A process for establishing and maintaining inter-rater reliability for two observation instruments as a fidelity of implementation measure: A large-scale randomized controlled trial perspective Academic Article uri icon

abstract

  • © 2019 Elsevier Ltd In this study, we presented a practical framework to establish and monitor inter-rater reliability (IRR)of classroom observation instruments as a measure of fidelity of implementation (FOI)in the context of a large-scale, federally-funded, yearly-long randomized study targeting English learners and their teachers’ instruction. We utilized two different instruments and demonstrated the following steps: (a)establishing initial IRR, (b)continuously monitoring IRR, and (c)re-establishing IRR when it fell below the pre-determined cut-off value at each FOI checkpoint from 116 bilingual classrooms across seven school districts in Texas. The on-going training, coding procedures, and calculations of IRR across both treatment and control conditions were closely examined. While this project was equipped with extensive resources and a sophisticated study design, we have also addressed guidelines for establishing and maintaining IRR that can be applied across various designs in under-resourced programs to measure FOI through classroom observation.

author list (cited authors)

  • Tong, F., Tang, S., Irby, B. J., Lara-Alecio, R., Guerrero, C., & Lopez, T.

citation count

  • 4

publication date

  • September 2019