External and Model Validity in a Systematic Review of Game-Based Interventions
Franz Coelho, Universidade Católica Portuguesa (Portugal)
Daniel Gonçalves, INESC-ID and Instituto Superior Técnico (Portugal)
Ana Maria Abreu, Universidade Católica Portuguesa and Forward College (Portugal)
Abstract
Game-based learning, training, exercises, serious games, and gamification represent distinct approaches, integrating games into diverse contexts. Lately, interventions based on these approaches have gained popularity due to their potential to enhance cognitive outcomes. The term game-based intervention (GBI) was adopted to describe the use of all these playful processes with the goals of cognition and behavior promotion. Here, we present complementary research originating from a comprehensive systematic review examining the influence of GBI on adult cognition. This additional research evaluates the external and model validity of the original studies included in a systematic review, adhering to a registered PROSPERO protocol and PRISMA guidelines. Our systematic methodology covered various databases, resulting in 1398 articles. Following examination, 42 studies (26 randomized control trials and 16 non-randomized control trials) were selected. External and model validity were assessed using the External Validity Assessment Tool (EVAT©). Half of the studies inadequately described recruitment (48%), and most poorly outlined participation (71%) in terms of external validity, obscuring the results' generalizability. However, most studies adequately described model validity (88%), clarifying the comprehension of staff, places, and facilities used. While the systematic review showcased encouraging results regarding the impact of GBI on adult cognition, the evaluation of external and model validity conducted here revealed challenges in generalizing these findings to real-world settings and other populations beyond the laboratory context. However, it underscored that the contextual environment and operational procedures are conducive to replicability.
Keywords |
Games, Technology, Digital, Education, Learning, Cognition |
References |
[1] S. de Freitas, “Are Games Effective Learning Tools? A Review of Educational Games,” Journal of Educational Technology & Society, vol. 21, no. 2, pp. 74–84, 2018, doi: https://www.jstor.org/stable/26388380. [2] R. N. Landers, E. M. Auer, A. B. Collmus, and M. B. Armstrong, “Gamification science, its history and future: Definitions and a research agenda,” Simulation & Gaming, vol. 49, no. 3, pp. 315–337, 2018, doi: https://doi.org/10.1177/10468781187743. [3] J. Krath, L. Schürmann, and H. F. O. von Korflesch, “Revealing the theoretical basis of gamification: A systematic review and analysis of theory in research on gamification, serious games and game-based learning,” Computers in Human Behavior, vol. 125, p. 106963, 2021, doi: https://doi.org/10.1016/j.chb.2021.106963. [4] F. Ferreira-Brito et al., “Game-based interventions for neuropsychological assessment, training and rehabilitation: Which game-elements to use? A systematic review,” Journal of biomedical informatics, vol. 98, p. 103287, 2019, doi: https://doi.org/10.1016/j.jbi.2019.103287. [5] M. J. Page et al., “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews,” BMJ, vol. 372, p. n71, 2021, doi: https://doi.org/10.1136/bmj.n71. [6] R. Khorsan and C. Crawford, “External Validity and Model Validity: A Conceptual Approach for Systematic Review Methodology,” Evidence-Based Complementary and Alternative Medicine, vol. 2014, p. 694804, Mar. 2014, doi: https://doi.org/10.1155/2014/694804. |