What Is the State of the Art of Systematic Review in Engineering Education?
Academic Article
Overview
Research
Identity
Additional Document Info
Other
View All
Overview
abstract
2015 ASEE. Background: Systematic review refers to a growing set of research methodologies intended to critically appraise and synthesize research to inform policy and practice. Systematic review methods, widely used in fields such as medicine, psychology, and education, are being applied in engineering education. Analysis of current systematic review practices in engineering education can contribute to continuous improvement of an important set of synthesis methodologies. Purpose: To promote continuous improvement in systematic reviews, researchers are currently developing criteria to evaluate the quality of systematic review methodologies. The purpose of this article is to identify prior systematic reviews in areas of interest to the engineering education community, use these criteria to critique these systematic reviews, and make recommendations for improving use and quality of systematic review methods in engineering education research. Scope/Method: Using systematic review search methodologies, we identified 12 conference papers and 37 journal articles that are systematic reviews published since 1990. Two reviewers coded each by topic, education level, quantitative or qualitative sources and synthesis methods, and 18 quantitative measures of review quality. Interrater reliability was calculated using kappa as described by Landis and Koch, which ranged from 0.43 (moderate) to 0.80 (almost perfect) with overall agreement of 0.64 (substantial). Conclusions: We found that the 49 identified systematic reviews in engineering education are particularly strong in stating their purposes, rationale, and inclusion criteria. They can be improved by explaining reasons for excluding studies, appraising quality of included studies, engaging multiple coders in evaluation procedures, stating their limitations, and, in the case of meta-analyses, describing statistical methods in greater detail.
