A process for establishing and maintaining inter-rater reliability for two observation instruments as a fidelity of implementation measure: A large-scale randomized controlled trial perspective
Academic Article
Overview
Research
Identity
Additional Document Info
Other
View All
Overview
abstract
2019 Elsevier Ltd In this study, we presented a practical framework to establish and monitor inter-rater reliability (IRR)of classroom observation instruments as a measure of fidelity of implementation (FOI)in the context of a large-scale, federally-funded, yearly-long randomized study targeting English learners and their teachers instruction. We utilized two different instruments and demonstrated the following steps: (a)establishing initial IRR, (b)continuously monitoring IRR, and (c)re-establishing IRR when it fell below the pre-determined cut-off value at each FOI checkpoint from 116 bilingual classrooms across seven school districts in Texas. The on-going training, coding procedures, and calculations of IRR across both treatment and control conditions were closely examined. While this project was equipped with extensive resources and a sophisticated study design, we have also addressed guidelines for establishing and maintaining IRR that can be applied across various designs in under-resourced programs to measure FOI through classroom observation.