Qualitative indicators in the evaluation of graduate programs in education
DOI:
https://doi.org/10.18222/eae.v35.10777Keywords:
Educational Evaluation, Postgraduates, Educational Policies, Large-scale EvaluationAbstract
This article analyzes the evaluation of graduate studies in the Education Area by Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Brazilian Federal Agency for Support and Evaluation of Graduate Education] between 2017-2020 highlighting the changes resulting from the expanded use of qualitative indicators. Following a discussion on the evaluation and theoretical aspects of the subject, the process adopted for the objectification of the evaluation and the results obtained is described. Regression analysis is used to analyze the behavior of both quantitative and qualitative indicators in defining the grades of the programs in the area. The article proposes a reduction in the total number of indicators, emphasizes the importance of qualitative indicators as valuable sources of the feedback needed to improve the program, and advocates for the adoption of objectification processes derived from the construction of evaluative criteria that parameterize the evaluation work.
Downloads
References
Barreto, J. A. E. (1993). Avaliação: Mitos e armadilhas. Ensaio: Avaliação e Políticas Públicas em Educação, 1, 45-54.
Biagioli, M. (2018). Quality to impact, text to metadata: Publication and evaluation in the age of metrics. KNOW: A Journal on the Formation of Knowledge, 2(2), 249-275. https://doi.org/10.1086/699152 DOI: https://doi.org/10.1086/699152
Borsboom, D., Mellenbergh, G. J., & Heerden, J. van. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071. https://doi.org/10.1037/0033-295x.111.4.1061 DOI: https://doi.org/10.1037/0033-295X.111.4.1061
Brasil, A. (2023). Advancing the evaluation of graduate education: Towards a multidimensional model in Brazil. Universiteit Leiden.
Castro, C. de M. (2006). A Capes na visão de um ex-diretor. Análise, 17(2), 360-376.
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes). (2021). Equivalência entre conceitos e notas. https://www.gov.br/capes/pt-br/centrais-de-conteudo/documentos/avaliacao/2021.04.05_RelatrioGTequivalenciaconceitosnotasversoformatadaV3.pdf
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes). (2022). Relatório de avaliação de Área de Educação. https://www.gov.br/capes/pt-br/centrais-de-conteudo/ documentos/avaliacao/19122022_RELATORIO_AVALIACAO_QUADRIENAL_ comnotaEducao.pdf
Fórum de Coordenadores de Programas de Pós-Graduação em Educação (Forpred). (2017). Carta de Fortaleza. (Documento não publicado).
Gatti, B., Andre, M., Favero, O., & Candau, V. M. F. (2003). O modelo de avaliação da Capes. Revista Brasileira de Educação, 22, 137-144. https://doi.org/10.1590/S1413-24782003000100012 DOI: https://doi.org/10.1590/S1413-24782003000100012
Gomez, M. N. G. de. (2014). Da questão da validade ao julgamento de valor: Mediação informacional da avaliação científica. In Anais do 15. Encontro Nacional de Pesquisadores em Ciência da Informação.
Khandker, S. R., Koolwal, G. B., & Samad, H. A. (2009). Handbook on impact evaluation: Quantitative methods and practices. World Bank Publications. DOI: https://doi.org/10.1596/978-0-8213-8028-4
Klecka, W. R. (1980). Discriminant analysis. Sage. DOI: https://doi.org/10.4135/9781412983938
Lamont, M. (2012). Toward a comparative sociology of valuation and evaluation. Annual Review of Sociology, 38(1), 201-221. https://doi.org/10.1146/annurev-soc-070308-120022 DOI: https://doi.org/10.1146/annurev-soc-070308-120022
Leite, D., Verhine, R. E., Dantas, L. M. V., & Bertolin, J. C. D. (2020). A autoavaliação na pós- -graduação (PG) como componente do processo avaliativo Capes. Avaliação: Revista da Avaliação da Educação Superior, 25(2), 339-353. https://doi.org/10.1590/S1414-4077/S1414-40772020000200006 DOI: https://doi.org/10.1590/s1414-4077/s1414-40772020000200006
Minayo, M. C. de S. (2009). Construção de indicadores qualitativos para avaliação de mudanças. Revista Brasileira de Educação Médica, 33(Supl. 1), 83-91. http://educa.fcc.org.br/pdf/rbem/v33s01/v33s01a09.pdf DOI: https://doi.org/10.1590/S0100-55022009000500009
Moreira, C. O. F., Hortale, V. A., & Hartz, Z. M. A. (2004). Avaliação da pós-graduação: Buscando consenso. Revista Brasileira de Pós-Graduação, 1(1), 26-40. https://www.arca.fiocruz.br/handle/icict/60206
Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34(5 Pt 2), 1189-1208.
Portaria n. 122, de 5 de agosto de 2021. (2021). Consolida os parâmetros e os procedimentos gerais da Avaliação Quadrienal de Permanência da pós-graduação stricto sensu no Brasil. Brasília, DF. http://cad.capes.gov.br/ato-administrativo-detalhar?idAtoAdmElastic=6742
Porter, T. M. (1995). Quantification and the accounting ideal in science. Social Studies in Science, 22(4), 633-652. https://www.jstor.org/stable/285458 DOI: https://doi.org/10.1177/030631292022004004
Reckase, M. D. (2023). The psychometrics of standard setting: Connecting policy and test scores. CRC Press. DOI: https://doi.org/10.1201/9780429156410
Richie, J., & Lewis, J. (Eds.) (2003). Qualitative research practice: A guide to social science students and researchers. Sage.
Secchi, L. (2014). Instrumentos de políticas públicas. In R. F. Boullosa (Org.), Dicionário para a formação em gestão social (pp. 103-105). CIAGS/UFBA.
Sguissardi, V. (2006). A avaliação defensiva no “modelo Capes de avaliação”: É possível conciliar avaliação educativa com processos de regulação e controle do Estado? Perspectiva, 24(1), 49-88. http://educa.fcc.org.br/pdf/rp/v24n01/v24n01a03.pdf
Spagnolo, F., & Calhau, M. G. (2002). Observadores internacionais avaliam a avaliação da Capes. Infocapes, 10(1), 7-34.
Stufflebeam, D., & Shinkfield, A. J. (2007). Evaluation theory, models and applications. HB Printing.
Tierney, R., & Simon, M. (2004). What’s still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research, and Evaluation, 9(1), Article 2. https://doi.org/10.7275/jtvt-wg68
Todeschini, R., & Baccini, A. (2016). Handbook of bibliometric indicators: Quantitative tools for studying and evaluating research. John Wiley & Sons. DOI: https://doi.org/10.1002/9783527681969
ToloData. (2021). Qualitative indicators and their relevance in monitoring and evaluation. https://www.toladata.com/blog/qualitative-indicators-and-their-relevance-in-monitoring-and-evaluation/
Tufte, E. R. (1974). Data analysis for politics and policy. Prentice-Hall.
United States Agency for International Development (Usaid). (2005). Handbook on qualitative indicators. https://pdf.usaid.gov/pdf_docs/PNAEB361.pdf
Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (2004). Avaliação de programas: Concepções e práticas. Gente.
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Estudos em Avaliação Educacional
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish in this journal agree to the following terms:
a. Authors retain the copyright and grant the journal the right to first publication, with the paper simultaneously licensed under the Creative Commons Attribution license that allows the sharing of the paper with acknowledgment of authorship and initial publication in this journal.
b. Authors are authorized to assume additional contracts separately, for non-exclusive distribution of the version of the paper published in this journal (for example publishing in institutional repository or as a book chapter), with acknowledgment of authorship and initial publication in this journal.
c. Authors are allowed and encouraged to publish and distribute their paper on-line (for example in institutional repositories or on their personal page) at any moment before or during the editorial process, as this can generate productive changes, as well as increase the impact and citation of the published paper (See The Effect of Open Access).
