29 2 variance over time points 22 and the e interaction between teachers and time points do not appear in at some time points). As a follow-up, a generalization study in the reliability coefficient that occurs R ! 2 t s 2 jt s k a k a and the variance attributable to the interaction between teachers and time points 22 and the butable to the interaction between teachers and time points do not appear in at some time points). As a follow-up, a generalization study in the reliability coefficient that occurs R ! 2 t s 2 jt s k a k a do not appear in the denominator, because averaging the assessment over time points does not influence the reliability. All teachers are in principle assessed at the same fixed time points (no observations may be available for some teachers at some time points). As a follow-up, a generalization study (g-study) was conducted, in which the explained variance was decomposed into the following variance facets: teachers, students, and measurement moments. Furthermore, a decision study (d-study) was conducted that examined the decrease in the reliability coefficient that occurs when removing time points, students, or both. The local reliability of the instrument was also evaluated. Local reliability refers to the precision of the estimate of the latent variable of interest (in this study, the quality of a teacher’s teaching), which is determined by the discrimination parameters and location parameters of the items. The discrimination parameter ( 22 the average number of students who rated a teacher. The variance over time points and the variance attributable to the interaction between teachers and time points do not appear in the denominator, because averaging the assessment over time points does not influence the reliability. All teachers are in principle assessed at the same fixed time points (no observations m y be available for some teachers at some time points). As a follow-up, a generalization study (g-study) was c nducted, in which the explained variance was decomposed into the following variance facets: teachers, students, and measurement moments. Furthermore, a decision study (d-study) was conducted that examined the decrease in the reliability coefficient that occurs when rem ving time points, students, or both. The local reliability of the instrument was also evaluated. Local reliability refers to the precision of the estimate of the latent variable of interest (in this study, the quality of a teacher’s teaching), which is determined by the discrimination parameters and location para eters of the items. The discrimination parameter ( -values) indicates the contribution of the individual item to the scale and how well items can discriminate between students. The higher the - value, the more the item contributes to the estimation of overall teaching quality. The item 2 t s 2 jt s k a k a -values) indicates the contribution of the individual item to the scale and how well items can discriminate between students. The high r the 22 where stands for the weighted mean number of time points over students and stands for the average number of students who rated a teacher. T variance over time points nd the variance attributable to the interaction betw en teachers and t me p ints do not appear in the denominator, because averaging the assessment over time points does not influence the reliability. All teachers are in principle assessed at the same fixed time points (no observations may be available for some teachers at some ti e points). As a follow-up, a generalization study (g-study) was conducted, i which th explained variance was decomposed into the following variance facets: teachers, students, and measurement moments. Furthermore, a decision study (d-study) was conducted that examined the decrease in the reliability coefficient that occurs when removing time points, students, or both. The local reliability of the instrument was also evaluated. Local reliability refers to the precision of the estimate of the latent variable of interest (in this study, the quality of a teacher’s teaching), which is determined by the discrimination parameters and location parameters of the items. The discrimination parameter ( -values) indicates the contribution of the individual item to the scale and how well items can discriminate between students. The higher the - value, the more the item contributes to the estimation of overall teaching quality. The item T! R ! 2 t s 2 jt s k a k a -value, the mor th item contributes to t e estimation of overall teaching quality. The item location parameters, def ined as ters, defined as , were computed as an indication of the average 2 / 2 k k d d = k d , re co puted as an dication of the average diff iculty of an item. The higher the difficulty of an item. T e higher the -value, the more difficult it is for teachers to receive were well below 0.1 (ranging from 0.00 to 2 / 2 k k d d = k d-value, the more diff icult it is for teachers to receive high scores from students. Using these parameters, the information values of the items at f ive distinct points on the teaching quality scale were calculated. The higher the item information value, the higher the contribution of the item to the local reliability. The total test information coeff icient indicates the information provided by all items of the questionnaire in total about the quality of a certain teacher’s teaching. This coeff icient is calculated as one minus the inverse of the sum of the item information values, with a score between 0 (poor questionnaire quality) and 1 (high questionnaire quality). 2.5 RESULTS 2.5.1 Construct validity In the first investigation, the absolute differences between the observed average score and the posterior expected average score of the three subgroups of response patterns were calculated. The three score-level groups consisted of a group where total scores from students were low (below 11 out of the possible maximum score of 30 on the 15 items), an intermediate score level (from 11 to 15) and a high score level (above 15). The results are presented in Appendix C, Table
RkJQdWJsaXNoZXIy MjY0ODMw