22 The reliability and construct validity of student perceptions of teaching quality be found in the data collected with the questionnaire, the factor loadings of the items can be calculated. The studies by Marsh and Roche (1997) and Abrami et al. (1990) are classic examples of such a factor-analytic approach in the case of student perception questionnaires for higher education. However, as the factor analysis is always conducted on aggregated scores at the class level, it does not take into account differences in psychometric properties (e.g., reliability) of the questionnaire at both the student level and the class level. Because scores are aggregated, the analysis also does not deal with missing values (Marsh et al., 2012). As an improvement on this one-level approach, an item response theory (IRT) modelling approach can be used for statistically investigating construct validity. This approach is analogous to factor analysis, but also takes into account the multi-level structure of the data if scores are nested. Further, compared to classical test theory, it is easy to take missing data into account in IRT modelling, as the missing responses are not included in the estimates (they are not scored as zero). In IRT, it is assumed that all items in a test are indicators of one unidimensional construct. The parameter estimates of an IRT model on item scores, as well as goodness-of-f it indices, give insight into the contribution of individual items towards the reliability of the model, and thus, of the extent to which the items are indicators of the underlying construct. The use of IRT modelling has become the standard statistical approach for quantitatively determining construct validity (Hambleton et al., 1991; Lord, 1980). Based on statistical analyses alone, what the underlying construct is precisely remains unclear: do the items reflect the construct one is attempting to measure (Muijs, 2006)? Construct validity can be strengthened in other, more content-based ways. We will next explain how this was done. 2.2.2 T he content of the Impact! questionnaire and enhancing construct validity For the development of the Impact! questionnaire, the construct of “teaching quality” was conceptualized by first conducting a literature review of teacher effectiveness research to identify characteristics of teaching that positively affects student learning. This included a review of student perception questionnaires (Bijlsma, 2016) to identify constructs and items regarding teaching quality used in other student questionnaires. Several meta-analyses and other publications about effective teaching were found that showed a number of teaching practices that are known to be effective for student learning (Black & William, 2006; Creemers, 1994; Day et al., 2008; Drijvers, 2015;
RkJQdWJsaXNoZXIy MjY0ODMw