Experimental designs in public policy evaluation: Uses and abuses
DOI:
https://doi.org/10.18222/eae.v34.9956Keywords:
Evaluation, Public Policy, Experiments, Quasi-ExperimentsAbstract
The present essay discusses the strengths and, above all, the limits of using experimental and quasi-experimental methods in evaluating public programs. It begins with a brief presentation of classical experimental design, its requirements and related concepts such as internal, external, and counterfactual validity. Next, it addresses the modalities of quasi-experimental evaluation design, wich relax the requirements of the classical experiment. Two critical sections point out the ethical, political, and operational limitations of experimental and quasi-experimental designs in program evaluation and, subsequently, the political-institutional motivations for the resilience of this approach in spite of well-known and recurrent robustness problems.
Downloads
References
Bamberguer, M., Segone, M., & Tateossian, F. (2016). Evaluating the Sustainable Development Goals with a “no one left behind” lens through equity-focused and gender-responsive evaluations. UN Women.
Batista, M., & Domingos, A. (2017). Mais que boas intenções: Técnicas quantitativas e qualitativas na avaliação de impacto de políticas públicas. Revista Brasileira de Ciências Sociais, 32(94), Artigo e329414. https://doi.org/10.17666/329414/2017 DOI: https://doi.org/10.17666/329414/2017
Cano, I. (2004). Introdução à avaliação de programas sociais. FGV.
Casa Civil da Presidência da República. (2018). Avaliação de políticas públicas: Guia prático de análise ex post (vol. 2). Casa Civil da Presidência da República.
Chianca, T. (2015). Um modelo alternativo ao estudo experimental para inferir causalidade em avaliações do impacto de projetos sociais. Revista Brasileira de Monitoramento e Avaliação, 9, 16-29. http://dx.doi.org/10.4322/rbma201509003 DOI: https://doi.org/10.4322/rbma201509003
Gertler, P., Martínez, S., Premand, P., Rawlings, L. B., & Vermeesch, C. M. J. (2015). Avaliação de impacto na prática. Banco Mundial.
Gussi, A. F., & Oliveira, B. R. (2017). Discutindo paradigmas contra-hegemônicos de avaliação de políticas públicas. Anais do 1. Encontro Nacional de Ensino e Pesquisa do Campo de Públicas. UFC.
Imas, L. G. M., & Rist, R. (2009). The road to results: Designing and conducting effective development evaluations. World Bank.
Jaime, P. C., Vaz, A. C. N., Nilson, E. A. F., Fonseca, J. C. G., Guadagnin, S. C., Silva, S. A., Sousa, M. F., & Santos, L. M. P. (2014). Desnutrição em crianças de até 5 anos beneficiárias do programa Bolsa Família: Análise transversal e painel longitudinal de 2008 a 2012. Cadernos de Estudos Desenvolvimento Social em Debate, (17), 49-61.
Jannuzzi, P. de M. (2016). Monitoramento e avaliação de programas sociais: Uma introdução aos conceitos e técnicas. Alínea.
Jannuzzi, P. de M. (2018). Mitos do desenho quase-experimental na avaliação de programas. Nau Social, 9(16), 76-90. https://doi.org/10.9771/ns.v9i16.31419 DOI: https://doi.org/10.9771/ns.v9i16.31419
Jannuzzi, P. de M., & Pinto, A. (2013). Bolsa Família e seus impactos nas condições de vida da população brasileira: Uma síntese dos principais achados da pesquisa de avaliação de impacto do Bolsa Família II. In M. Neri, & T. Campello (Orgs.), Programa Bolsa Família: Uma década de inclusão e cidadania (pp. 179-192). Ipea.
La Rovere, R. (2014). The consultative group on international agricultural research approach to impact evaluation on environment and natural resources management. In J. I. Uitto (Org.), Evaluating environment in international development (pp. 277-288). Routledge.
Leão, L. S., & Eyal, G. (August, 2016). Experiments in the wild: A historical perspective on the rise of randomized controlled trials in international development. Comparative Historical Sociology Mini-Conference. American Sociological Association, Seattle, USA.
Leeuw, F. L. (2010). On the contemporary history of experimental evaluations and its relevance for policy making. In O. Rieper, F. L. Leeuw, & T. Ling (Eds.), The evidence book: concepts, generation, and use of evidence (Comparative polyce evaluation, vol. 15, pp. 11-26). Routledge.
McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. F. (2019). Abandon statistical significance. The American Statistician, 73, 235-245. https://doi.org/10.1080/00031305.2018.1527253 DOI: https://doi.org/10.1080/00031305.2018.1527253
Moral-Arce, I. (2014). Elección del método de evaluación cuantitativa de una política pública: Buenas prácticas en América Latina y la Unión Europea. EuroSocial.
Patton, M. Q. (1990). Qualitative evaluation and research methods. Sage.
Patton, M. Q. (1997). Utilization-focused evaluation: The next century text. Sage.
Ravallion, M. (2009). Should the randomists rule? The Economists’ Voice, 6(2). https://doi.org/10.2202/1553-3832.1368 DOI: https://doi.org/10.2202/1553-3832.1368
Rossi, P. (1987). The iron law of evaluation and other metallic rules. In J. Miller, & M. Lewis, Research in social problems and public policy (vol. 4, pp. 3-20). Jai Press.
Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2004). Evaluation: A systematic approach. Sage.
Sawyer, A. G., & Peter, J. P. (1983). The significance of statistical significance tests in Marketing Research. Journal of Marketing Research, 20(2), 122-133. https://doi.org/10.2307/3151679 DOI: https://doi.org/10.1177/002224378302000203
Vaitsman, J., & Paes-Sousa, R. (2009). Avaliação de programas e transparência da gestão pública. In C. Franzese, C. Anjos, D. Ferraz, F. L. Abrucio, G. N. Cheli, G. M. B. P. Melo, J. Vaistman, J. L. D. Nehmé, L. Santoni, M. G. da Silva, M. Rubio, P. Yanes, S. Nahas, P. de M. Jannuzzi, & R. Paes-Sousa, Reflexões para Ibero-América: Avaliação de programas sociais (pp. 11-23). Enap.
Weiss, C. I. (1998). Evaluation research. Prentice Hall.
Worthern, B. R., Sanders, J. R., & Fitzpatrick, J. L. (2004). Avaliação de programas: Concepções e práticas. Gente.
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Estudos em Avaliação Educacional
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish in this journal agree to the following terms:
a. Authors retain the copyright and grant the journal the right to first publication, with the paper simultaneously licensed under the Creative Commons Attribution license that allows the sharing of the paper with acknowledgment of authorship and initial publication in this journal.
b. Authors are authorized to assume additional contracts separately, for non-exclusive distribution of the version of the paper published in this journal (for example publishing in institutional repository or as a book chapter), with acknowledgment of authorship and initial publication in this journal.
c. Authors are allowed and encouraged to publish and distribute their paper on-line (for example in institutional repositories or on their personal page) at any moment before or during the editorial process, as this can generate productive changes, as well as increase the impact and citation of the published paper (See The Effect of Open Access).
