A modified GRRAS checklist for the quality of reliability research in knee ultrasonography: An inter-rater agreement study

Introduction The original Guidelines for Reporting Reliability and Agreement Studies (GRRAS) checklist uses 15 items to assess the quality of reliability research. However, six of the 15 items have equal to or more than two sub-items which creates the potential for inconsistencies in reporting quality scoring within and between researchers. Therefore, as part of a wider systematic review examining the reliability of tibiofemoral joint articular cartilage ultrasonography (US), we modified the GRRAS checklist by separating all sub-items, yielding a new checklist with 24 items. Subsequently, the aim of this study was to determine the inter-rater agreement of the modified GRRAS (mGRRAS) checklist as a tool for assessing the quality of reliability research studies. Methods The mGRRAS checklist was used to assess the quality of studies selected specifically for inclusion in the systematic review (n=6). Each of the 24 items could be scored from zero to one, yielding a potential total score ranging from zero to 24. Higher scores represented overall higher quality of reliability research. Two researchers (R1, R2) scored the selected studies independently. If R1 or R2 were unable to decide on their score for any of the 24 items, a third researcher (R3) was available to facilitate the process further. Researchers’ scores were compared using the percent agreement statistic. Levels of agreement were classed: 71-80%, moderate; 81-90%, strong; 91-100%, almost perfect to perfect. Data were analysed using SPSS (v29). Results The total score for the six selected studies ranged from 15 to 21 out of 24. Neither R1 nor R2 required the assistance of R3 to decide on the score for any item. Inter-rater agreement was 83%, indicating that there was strong agreement between R1 and R2. Conclusion The mGRRAS checklist can improve quality scoring of reliability research studies due to its increased number of items. It can be used by raters independently, has strong inter-rater agreement, and may be useful for future assessment of the quality of knee US reliability research studies. Impact Measurement reliability is a prerequisite for valid interpretation of change in patients’ knee US data across time. The mGRRAS will assist clinicians to assess the quality of knee articular cartilage US reliability research and, in turn, choose which US method to employ in their clinical practice. References: Kottner J, Audige L, Brorson S, Donner A, Gajewski B, Hrobjartsson A, Roberts C, Shoukri M, Streiner D (2011) Guidelines for Reporting Reliability and Agreement Studies (GRRAS) Were Proposed. Journal of Clinical Epidemiology. 64, 96-106.