[SE02-CCT] Opportunities and Challenges in Quality (Rigorous) Impact Evaluations: Lessons From Academia and the Field

DAY 1 – Wednesday 27 June – 13:30-15:00
Swiss Tech | Room 3C | Level Garden

 

Session Leaders
 

Marina Cracco

Ecole Polytechnique Fédérale de Lausanne, Switzerland

marina.cracco@epfl.ch

Marina Cracco is a sustainable development professional with experience working on project and program evaluations, biodiversity, climate change adaptation, environmental and social safeguards, gender and protected areas. She has over 18 years of experience in regional, non-governmental organizations and multilateral development funds. Marina task-managed the First Phase of the Adaptation Fund Evaluation, and the Cuba Country Portfolio Evaluation and the Peru Impact Study for the GEF Independent Evaluation Office. At EPFL-CODEV, she has been instrumental in developing courses and research projects on disaster risk reduction and leveraging the topic of development evaluation.

Neeta Goel

3iE

ngoel@3ieimpact.org

Neeta is responsible for the review and management of 3ie-funded research and impact evaluation grants. She has over seventeen years of experience in the international development sector. Her work includes the design, implementation and evaluation of NGO interventions focusing on disadvantaged children and communities. Prior to 3ie, Neeta worked in several national and international NGOs. In her most recent assignment, she served as the Program Director for Children International, managing programmes in ten countries across Africa, Asia and Latin America. 

 

Summary
 

Quality (Rigorous) impact evaluations are increasingly gaining prominence in development evaluation. These evaluations focus on clear causation or causal attribution by establishing the counterfactual. “Assessing the effects produced by an intervention, the net impact, requires a comparison of what has occurred with the intervention implemented – i.e. the factual – with the situation without the intervention – i.e. the counterfactual” – Silvestrini et al. 2015. However, establishing a counterfactual, overcoming selection bias, and contamination problems in the context of technology for development interventions come with its challenges. The aim of this session is to present examples of quality (rigorous) impact evaluations and to provide a valuable opportunity to discuss the challenges and opportunities when developing evaluations establishing counterfactuals for development interventions (Technology for Disaster Risk Reduction, ICT, Sustainable and Renewable Energy, Sustainable Habitats and Cities, etc.). The session also hopes to bring together participants interested in this topic.

Abstracts and Panelists

 

Transformative Accountability Tools for Securing and Sustaining Development Results in the 2030 Agenda

 

Fabio Losa1, Maurizio Bigotta1, Aw Boubacar2, Caesar Cheelo3, Theoneste Ntakirutimana4, Laila Smith5, Claudio Valsangiacomo1

1 Centre for Development and Cooperation and Department of Business Economics, Health and Social Work – University of Applied Sciences and Arts of Southern Switzerland (SUPSI), Lugano (Switzerland).

2 Centre for Learning on Evaluation and Results for Francophone Africa (CLEAR) – Centre Africain d’Études Supérieures en Gestion (CESAG), Dakar (Senegal).

3 Southern African Institute for Policy and Research (SAIPAR), Lusaka (Zambia)

4 College of Medicine and Health Sciences – University of Rwanda, Kigali (Rwanda)

5 Centre for Learning on Evaluation and Results for Anglophone Africa (CLEAR-AA) – University of Witwatersrand, Johannesburg (South Africa)

 

Email address: Fabio.losa@supsi.ch

Biography: Economist, applied researcher and monitoring and evaluation (M&E) specialist, with PhD and Habilitation in economics (2008). Currently researcher at University of Applied Sciences and Arts of Southern Switzerland, senior researcher with the Center for Research in Economics and Management (University of Rennes 1), and the Swiss National Centre of Competence in Research LIVES. Proficient development consultant with field experience in particularly in Africa.

Abstract

To exploit the unprecedented opportunity of the 2030 SDG Agenda and help achieving its goals, there is a need to retool development frameworks, policies and practices to secure and sustain development achievements.

The research is addressing this issue by developing and validating transformative accountability tools, in particular Sustainable Impact Evaluation (SIE) and Shared Accountability Frameworks (SAF). With regard to the 2030 Agenda, the research is rooted in SDG17 – Revitalize the global partnership for sustainable development – but reaches out to all the other SDGs. The empirical application and validation will be on the nexus between SDG17 and SDG6 on access to Water Supply and Sanitation (WSS), and in particular on WSS in rural Africa.

 

Opportunities and Challenges in Experimental Impact Evaluations: Lessons from an (Ongoing) RCT in Salvadorian Schools

 

Konstantin Büchel1

1 University of Bern, Bern, Switzerland

 

Email address: konstantin.buechel@vwi.unibe.ch

 

Biography: Konstantin Büchel is a lecturer and researcher at the University of Bern, where he obtained a PhD in Economics in 2017. His work focuses on topics in economic geography and development economics, with a particular interest in experimental evaluations.

 Abstract

Rigorous evaluations are an important instrument to inform decision-makers which policies work best, and how they can be implemented. I will illustrate these opportunities based on an ongoing randomized controlled trial in Salvadorian primary schools. Comparing the cost-effectiveness of policies across studies also has its caveats. I will focus on one particular challenge in the educational literature: Standardized test scores are both test and sample dependent, and thus cannot be meaningfully compared across different studies. Hence, conclusions from metastudies that compare the cost-effectiveness of various educational interventions should be interpreted with caution.

A community-based program promotes private latrine use in rural Mali

 

Maria Laura Alzua1

1CEDLAS-FCE- Universidad Nacional de la Plata –CONICET

 

Email address: malzua@depeco.econo.unlp.edu.ar

Biography: Maria Laura Alzua is Deputy Director at the Center for Distributional, Labor and Social Studies (CEDLAS) at Universidad Nacional de La Plata (Argentina) and the Research Director of the Policy Impact Analysis at PEP (Partnership for Economic Development. She co-founded the Impact Evaluation Network at Lacea to promote exchange between researchers and policymaking. She specializes in development economics and policy evaluation. Currently, she is involved in several multi-year projects evaluating several policies in developing countries.

Abstract

This paper adds to the existing evidence explaining the failure of many sanitation interventions to deliver strong impacts on health. Such a failure may in part be due to weak implementation of the behavioral components when programs include financial and supply incentives (Gertler et al. 2015, Briceño et al. 2015, Hammer and Spears 2016). Failure can also come from design issues.  In order to be able to disentangle design vs implementation, one needs to properly document programs as implemented. We provide evidence that CLTS in Mali was implemented as planned. Nor was there any major disruption from the political situation due to the conflict in the North of the country. The success of CLTS in Mali is thus consistent with proper implementation of a well-designed program.

 

Implementing and Evaluating Low-cost Interventions to Improve Latrine Use among Rural Households in Odisha, India

 

Fiona Majorin1, Gloria D. Sclar2, Bethany Caruso2, Parimita Routray1, Corey Nagel3, Steven Sola2, Belen Torondel1, Thomas Clasen2

1London School of Hygiene and Tropical Medicine, London, UK

2Rollins School of Public Health, Atlanta, Georgia, USA

3University of Arkansas, Little Rock, Arkansas, USA

 

Email address: Fiona.Majorin@lshtm.ac.uk

Biography: Fiona Majorin, PhD is a Research Fellow at the London School of Hygiene and Tropical Medicine with expertise in environmental health research, especially sanitation. Fiona has experience in quantitative and qualitative research methods and worked on projects in India and Rwanda. Her work has involved designing and managing impact evaluation studies.

Abstract

Despite large-scale efforts by the Government of India to improve access to sanitation facilities, open defecation remains a widespread and persistent challenge in India. We conducted formative research on the determinants of latrine use behavior in Puri district, Odisha and leveraged behavior change theory to develop an evidence-based, theoretically grounded intervention. We are conducting a cluster-randomized controlled trial in 33 intervention and 33 control villages to evaluate the impact of this innovative sanitation intervention on latrine use behavior among households with a latrine. While this evaluation includes a number of research opportunities, such as the development and use of a novel behavioral determinants metric, our team also faces challenges around separation of the evaluation and implementation teams, how to address the impact of past sanitation programming, length of follow-up, and more. We will expand upon our lessons learns from these opportunities and challenges during the presentation.