22 Chapter 2 Interobserver agreement Table 3 shows the mean IOA for the categorical staging variables (as well as the diagnostic accuracy estimates based on the expert standard of reference). For most of the categorical staging items good (>80%) agreement was reached for the majority of patient cases. For three items – cTstage, cN-stage, and morphology/shape – IOA was <80% in the majority of cases with poorest results for cN-staging where good IOA was achieved in only 28% of the cases. For the six continuous variables, IOA (Krippendorff’s alpha) was α=0.72 for tumour length, α=0.84 for tumour height, α=0.33 for the tumour-MRF distance, α=0.73 for extramural invasion depth, α=0.55 for the total number of visible mesorectal nodes, and α=0.05 for the total number of suspicious mesorectal nodes. Good IOA was reached for the majority (71%) of cases to detect the presence of any high risk feature (i.e. ≥T3cd stage, N+, tumour deposits, or EMVI). Agreement with expert reference Good (>80%) agreement with the expert standard of reference (range 80-91%) was achieved for the dichotomized assessment of low versus high-risk cT-stage and cN0 vs N+ disease, for assessing the presence of lateral nodal metastases, tumour deposits, MRF involvement, and presence of sphincter invasion. Results for multi-categorical cT-staging and cN-staging were considerably lower (69% and 60% agreement with expert reference), as were results for EMVI (77%), tumour morphology (67-79%) and relation to the anterior peritoneal reflection (77%). In patients with suspected sphincter involvement, mean agreement with the expert reference to assess the level of involvement (internal sphincter, intersphincteric plane, external sphincter) was 51%. In patients staged as cT3, accuracy for cT3 subclassification into low risk (cT3ab) vs high risk (cT3cd) was 73%. Correlation with diagnostic confidence A significant positive correlation between diagnostic confidence and agreement with the expert reference was found for all staging items for which diagnostic confidence scores were available (see Table 1) with polychoric correlation coefficients ranging from 0.18-0.66 (p<0.001-p=0.003). Strongest effects (correlation coefficient >0.50) were found for presence of sphincter involvement, low versus high-risk cT-stage, cN0 vs. N+ stage , EMVI and MRF involvement. Confidence scores were lowest for assessing the level of sphincter involvement. Main problem areas Morphology (shape) As shown in Table 3, readers showed relatively poor IOA and agreement with an expert reference to differentiate between annular, semi-annular and polypoid tumours. Readers were mainly inconsistent in discerning semi-annular from polypoid tumours (accuracy 62-64%) (see Figure 1).
RkJQdWJsaXNoZXIy MjY0ODMw