Assessing Quality in Systematic Literature Reviews: A Study of Novice Rater Training Academic Article uri icon


  • This study investigated performance variability when graduate students critically appraised original studies from a systematic review. Fourteen doctoral students from different academic programs, with no systematic review experience, received training on the Methodological Quality Questionnaire (MQQ) rating scale. Participants were mostly male (71%) and non-native English speakers (79%). Each rater was randomly assigned one original study to independently assess using the MQQ. Their scores were compared to an expert rater. Statistical analysis comprised the following: percentage of agreement (POA), Kappa coefficient, and Kendalls tau-b correlation. On the completed MQQ rating scale, 43% of the novice raters had a POA of 78% or higher with the expert rater. From this case study, a guide for improving training on methodological quality assessment was developed. Benefits include the following: (a) developing and supporting critical reasoning as well as problem-solving skills and (b) increasing research skills and competencies in the systematic review process.

published proceedings


altmetric score

  • 3.85

author list (cited authors)

  • Acosta, S., Garza, T., Hsu, H., & Goodson, P.

citation count

  • 6

complete list of authors

  • Acosta, Sandra||Garza, Tiberio||Hsu, Hsien-Yuan||Goodson, Patricia

publication date

  • January 2020