Longitudinal assessment of medical students: Is progress test appropriate?
DOI:
https://doi.org/10.18222/eae.v34.9220Keywords:
Education Evaluation, Medical Educational Measurement, External EvaluationAbstract
The purpose of this article is to assess whether the Progress Test is appropriate to evaluate programs and students during different stages of medical studies. The characteristics of the items and reliability of three previously applied progress tests were analyzed. For second-year students, 76.4% of the questions demonstrated poor quality (biserial < 0.2). This percentage decreased to 47.7% in the forth year and to 25.3% in the sixth year. The test’s reliability, measured by Cronbach’s alpha, was only 0.6 for second-year students and increased to 0.76 in the fourth year and to 0.87 for sixth-year students. The current form of the Progress Test showed low and unacceptable reliability for second-year students, reasonable for the fouth year, and excellent for the sixth year. An improvement of this longitudinal assessment is proposed.
Downloads
References
Albanese, M., & Case, S. M. (2016). Progress testing: Critical analysis and suggested practices. Advances in Health Sciences Education, 21(1), 221-234. https://doi.org/10.1007/s10459-015-9587-z
Albekairy, A. M., Obaidat, A. A., Alsharidah, M. S., Alqasomi, A. A., Alsayari, A. S., Albarraq, A. A., Aljabri, A. M., Alrasheedy, A. A., Alsuwayt, B. H., Aldhubiab, B. E., Almaliki, F. A., Alrobaian, M. M., Aref, M. A., Altwaijry, N. A., Alotaibi, N. H., Alkahtani, S. A., Bahashwan, S. A., & Alahmadi, Y. A. (2021). Evaluation of the potential of national sharing of a unified progress test among colleges of pharmacy in the Kingdom of Saudi Arabia. Advances in Medical Education and Practice, 12, 1465-1475. https://doi.org/10.2147/amep.s337266
Ali, K., Coombes, L., Kay, E., Tredwin, C., Jones, G., Ricketts, C., & Bennett, J. (2016). Progress testing in undergraduate dental education: The Peninsula experience and future opportunities. European Journal of Dental Education, 20(3), 129-134. https://doi.org/10.1111/eje.12149
Alkhalaf, Z. S. A., Yakar, D., Groot, J. C. de, Dierckx, R. A. J. O., & Kwee, T. C. (2021). Medical knowledge and clinical productivity: Independently correlated metrics during radiology residency. European Radiology, 31(7), 5344-5350. https://doi.org/10.1007/s00330-020-07646-3
Aubin, A.-S., Young, M., Eva, K., & St-Onge, C. (2020). Examinee cohort size and item analysis guidelines for health professions education programs: A Monte Carlo simulation study. Academic Medicine: Journal of the Association of American Medical Colleges, 95(1), 151-156.
https://doi.org/10.1097/acm.0000000000002888
Bica, R. B. da S., & Kornis, G. E. (2020). Exames de licenciamento em medicina: Uma boa ideia para a formação médica no Brasil? Interface – Comunicação, Saúde, Educação, 24, Artigo e180546. https://doi.org/10.1590/Interface.180546
Bicudo, A. M., Hamamoto, P., Filho, Abbade, J., Hafner, M. de L., & Maffei, C. (2019). Teste de Progresso em Consórcios para todas as escolas médicas do Brasil. Revista Brasileira de Educação Médica, 43(4), 151-156. https://doi.org/10.1590/1981-52712015v43n4RB20190018
Blake, J. M., Norman, G. R., Keane, D. R., Mueller, C. B., Cunnington, J., & Didyk, N. (1996). Introducing progress testing in McMaster University’s problem-based medical curriculum: Psychometric properties and effect on learning. Academic Medicine, 71(9), 1002-1007.
https://doi.org/10.1097/00001888-199609000-00016
Cecilio-Fernandes, D. (2019). Implementando o teste adaptativo computadorizado. Scientia Medica, 29(3), Artigo e34432. https://doi.org/10.15448/1980-6108.2019.3.34432
Cecilio-Fernandes, D., Bicudo, A. M., & Hamamoto, P. T., Filho. (2021). Progress testing as a pattern of excellence for the assessment of medical students’ knowledge – Concepts, history, and perspective. Medicina, 54(1), Article e-173770. https://doi.org/10.11606/issn.2176-7262.rmrp.2021.173770
Collares, C. F., & Cecilio-Fernandes, D. (2019). When I say… computerised adaptive testing. Medical Education, 53(2), 115-116. https://doi.org/10.1111/medu.13648
Damas, B. R., & Miranda, G. J. (2019). Preparação da Instituição para o Enade: Importa? In Anais do 3. Congresso UFU de Contabilidade. UFU. https://eventos.ufu.br/sites/eventos.ufu.br/files/documentos/030_artigo_completo.pdf
De Champlain, A. F. (2010). A primer on classical test theory and item response theory for assessments in medical education. Medical Education, 44(1), 109-117. https://doi.org/10.1111/j.1365-2923.2009.03425.x
Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38(9), 1006-1012. https://doi.org/10.1111/j.1365-2929.2004.01932.x
Epstein, R. M. (2007). Assessment in medical education. New England Journal of Medicine, 356(4), 387-396. https://doi.org/10.1056/nejmra054784
Hamamoto, P. T., Filho, & Bicudo, A. M. (2020a). Improvement of faculty’s skills on the creation of items for progress testing through feedback to item writers: a successful experience. Revista Brasileira de Educação Médica, 44(1), Article e018. https://doi.org/10.1590/1981-5271v44.1-20190130.ING
Hamamoto, P. T., Filho, & Bicudo, A. M. (2020b). Implementation of the Brazilian National Network for Practices and Research with Progress Testing – BRAZ-NPT. Revista Brasileira de Educação Médica, 44(3), Letter to the editor e074. https://doi.org/10.1590/1981-5271v44.3-20200089
Heeneman, S., Schut, S., Donkers, J., Vleuten, C. van der, & Muijtjens, A. (2017). Embedding of the progress test in an assessment program designed according to the principles of programmatic assessment. Medical Teacher, 39(1), 44-52. https://doi.org/10.1080/0142159x.2016.1230183
Henning, M., Pinnock, R., & Webster, C. (2017). Does Progress Testing violate the principles of constructive alignment? Medical Science Educator, 27(4), 825-829. https://doi.org/10.1007/s40670-017-0459-4
Herrmann, L., Beitz-Radzio, C., Bernigau, D., Birk, S., Ehlers, J. P., Pfeiffer-Morhenn, B., Preusche, I., Tipold, A., & Schaper, E. (2020). Status quo of progress testing in veterinary medical education and lessons learned. Frontiers in Veterinary Science, 7, Article 559.
https://doi.org/10.3389%2Ffvets.2020.00559
Kheyami, D., Jaradat, A., Al-Shibani, T., & Ali, F. A. (2018). Item analysis of multiple choice questions at the department of paediatrics, Arabian Gulf University, Manama, Bahrain. Sultan Qaboos University Medical Journal, 18(1), e68-e74. https://doi.org/10.18295/squmj.2018.18.01.011
Kibble, J. (2017). Best practices in summative assessment. Advances in Physiology Education, 41(1), 110-119. https://doi.org/10.1152/advan.00116.2016
Ministério da Educação. (2016). Avaliação Nacional Seriada dos Estudantes de Medicina – Documento Básico. Ministério da Educação.
Norcini, J., Anderson, M. B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., Hays, R., Palacios Mackay, M., Roberts, T., & Swanson, D. (2018). 2018 Consensus framework for good assessment. Medical Teacher, 40(11), 1102-1109. https://doi.org/10.1080/0142159x.2018.1500016
Oliveira, F. A. M., Martins, M. T., Ferraz, A. M. L., Júnior, Ribeiro, C. G., Oliveira, R. G. de, & Porto, F. R. (2020). Percepção dos acadêmicos de odontologia em relação ao Teste de Progresso. Revista da Abeno, 20(2), 26-37. https://doi.org/10.30979/rev.abeno.v20i2.821
Pasquali, L. (2004). Psicometria dos testes na psicologia e na educação (5a ed.). Vozes.
Primi, R. (2012). Psicometria: Fundamentos matemáticos da teoria clássica dos testes. Avaliação Psicológica, 11(2), 297-307.
Pugh, D., & Regehr, G. (2016). Taking the sting out of assessment: Is there a role for progress testing? Medical Education, 50(7), 721-729. https://doi.org/10.1111/medu.12985
Reberti, A. G., Monfredini, N. H., Ferreira, O., Filho, Andrade, D. F. de, Pinheiro, C. E., & Silva, J. C. (2020). Teste de Progresso na escola médica: Uma revisão sistemática acerca da literatura. Revista Brasileira de Educação Médica, 44(1), Artigo e015. https://doi.org/10.1590/1981-5271v44.1-20190194
Resolução n. 3, de 20 de junho de 2014. (2014). Institui Diretrizes Curriculares Nacionais do Curso de Graduação em Medicina e dá outras providências. Diário Oficial da União, Brasília, DF.
Ristoff, D. (2022). Mitos e meias-verdades: A educação superior sob ataque. Insular.
Rosa, M. I. da, Isoppoi, C., Cattaneoi, H., Madeirai, K., Adami, F., & Ferreira, O. F., Filho. (2017). O Teste de Progresso como indicador para melhorias em Curso de Graduação em Medicina. Revista Brasileira de Educação Médica, 41(1), 58-68. https://doi.org/10.1590/1981-52712015v41n1RB20160022
Rutgers, D., Raamt, F. van, Lankeren, W. van, Ravesloot, C., Gijp, A. van der, Ten Cate, T. J., & Schaik, J. van. (2018). Fourteen years of progress testing in radiology residency training: Experiences from The Netherlands. European Radiology, 28(5), 2208-2215. https://doi.org/10.1007%2Fs00330-017-5138-8
Sá, M. F. S. de, Romão, G. S., Fernandes, C. E., & Silva, A. L. da, Filho. (2021). The Individual Progress Test of Gynecology and Obstetrics Residents (TPI-GO): The Brazilian experience by Febrasgo. Revista Brasileira de Ginecologia e Obstetrícia, 43(6), 425-428. https://doi.org/10.1055/s-0041-1731803
Sahoo, D., & Singh, R. (2017). Item and distracter analysis of multiple choice questions (MCQs) from a preliminary examination of undergraduate medical students. International Journal of Research in Medical Sciences, 5(12), 5351-5355. http://dx.doi.org/10.18203/2320-6012.ijrms20175453
Sakai, M. H., Ferreira, O. F., Filho, Almeida, M., Mashima, D., & Marchese, M. (2008). Teste de Progresso e avaliação do curso: Dez anos de experiência da medicina da Universidade Estadual de Londrina. Revista Brasileira de Educação Médica, 32(2), 254-263. https://doi.org/10.1590/S0100-55022008000200014
Sartor, L. B., Rosa, L., Rosa, M. I. da, Madeirai, K., Uggioni, M. L., Ferreira, O., Filho. (2020). Undergraduate Medical Student’s Perception about the Progress Testing. Revista Brasileira de Educação Medica, 44(2), Artigo e062. https://doi.org/10.1590/1981-5271v44.2-20190286.ING
Scheffer, M., Cassenote, A., Guerra, A., Guilloux, A. G., Brandão, A. P., Miotto, B. A., Almeida, C. de J., Gomes, J. de O., & Miotto, R. A. (2020). Demografia médica no Brasil 2020. Conselho Federal de Medicina.
Scheffer, M., & Dal Poz, M. (2015). The privatization of medical education in Brazil: Trends and challenges. Human Resources for Health, 13(1), Article 96. https://doi.org/10.1186/s12960-015-0095-2
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53-55. https://doi.org/10.5116%2Fijme.4dfb.8dfd
Tavakol, M., & Dennick, R. (2013). Psychometric evaluation of a knowledge based examination using Rasch analysis: An illustrative guide: AMEE guide no. 72. Medical Teacher, 35(1), e838-e848. https://doi.org/10.3109/0142159X.2012.737488
Tavakol, M., & Dennick, R. (2017). The foundations of measurement and assessment in medical education. Medical Teacher, 39(10), 1010-1015. https://doi.org/10.1080/0142159x.2017.1359521
Troncon, L. E. (2019). Exames de licenciamento: Um componente necessário para avaliação externa dos estudantes e egressos dos cursos de graduação em Medicina. Interface – Comunicação, Saúde, Educação, 24, Artigo e190576. https://doi.org/10.1590/Interface.190576
Vleuten, C. van der. (2016). Revisiting ‘Assessing professional competence: From methods to programmes’. Medical Education, 50(9), 885-888. https://doi.org/10.1111/medu.12632
Vleuten, C. van der, Freeman, A., & Collares, C. F. (2018). Progress test utopia. Perspectives on Medical Education, 7(2), 136-138. https://doi.org/10.1007/s40037-018-0413-1
Vleuten, C. van der, & Schuwirth, L. W. T. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309-317. https://doi.org/10.1111/j.1365-2929.2005.02094.x
Vleuten, C. van der, & Schuwirth, L. W. T. (2019). Assessment in the context of problem-based learning. Advances in Health Sciences Education, 24(5), 903-914. https://doi.org/10.1007/s10459-019-09909-1
Vleuten, C. van der, Schuwirth, L. W. T., Muijtjens, A. M. M., Thoben, A. J. N. M., Cohen-Schotanus, J., & Boven, C. P. A. van. (2004). Cross institutional collaboration in assessment: A case on progress testing. Medical Teacher, 26(8), 719-725. https://doi.org/10.1080/01421590400016464
Walsh, J. L., Harris, B., Denny, P., & Smith, P. (2018). Formative student-authored question bank: Perceptions, question quality and association with summative performance. Postgraduate Medical Journal, 94(1108), 97-103. https://doi.org/10.1136/postgradmedj-2017-135018
Wrigley, W., Vleuten, C. van der, Freeman, A., & Muijtjens, A. (2012). A systemic framework for the progress test: Strengths, constraints and issues: AMEE guide no. 71. Medical Teacher, 34(9), 683-697. https://doi.org/10.3109/0142159x.2012.704437
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Estudos em Avaliação Educacional
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish in this journal agree to the following terms:
a. Authors retain the copyright and grant the journal the right to first publication, with the paper simultaneously licensed under the Creative Commons Attribution license that allows the sharing of the paper with acknowledgment of authorship and initial publication in this journal.
b. Authors are authorized to assume additional contracts separately, for non-exclusive distribution of the version of the paper published in this journal (for example publishing in institutional repository or as a book chapter), with acknowledgment of authorship and initial publication in this journal.
c. Authors are allowed and encouraged to publish and distribute their paper on-line (for example in institutional repositories or on their personal page) at any moment before or during the editorial process, as this can generate productive changes, as well as increase the impact and citation of the published paper (See The Effect of Open Access).
