Impact Evaluations, Part 1: Do We Need Them in Rotary?

by Quentin Wodon

Monitoring and evaluation have become essential in development and service work. When providing funding, most foundations and donors now require some type of monitoring and evaluation either ex ante to allocate funding, or ex post to assess whether projects that have been funded have proved to be successful or not. The same is true for government agencies – when deciding whether to scale up a pilot intervention, most agencies now typically rely on the results of at least some type of evaluation.

Different types of evaluations can be conducted and not all are equal in terms of what we can learn from them. In this series of three posts, I will focus on impact evaluations (as opposed to process evaluations). Specifically, I will discuss 1) whether we need impact evaluations in Rotary; 2) how impact evaluations can be implemented in practice; and 3) what are some of the limits of impact evaluations that practitioners and policy makers should be aware of. But before doing so, it is probably useful to briefly explain what an impact evaluation entails.

What Is An Impact Evaluation?

Impact evaluations aim to measure the impact of specific projects, policies, or interventions on specific outcomes. The question they ask is not whether a given outcome has been achieved or not among a target population. They look instead at the specific contribution or impact of a well-defined intervention on a well-defined outcome. Said differently, they try to measure a counterfactual: what would have been the outcome without the intervention? By comparing the assessment of what the outcome would have been without the intervention and the outcome with the intervention, impact evaluations inform us about the success (or lack thereof) of specific interventions in improving specific outcomes. When data on the cost of alternative interventions and their benefits are available, impact evaluations are especially helpful in deciding whether some interventions should be maintained or scaled up, while others should be abandoned.

Estimating the counterfactual in an impact evaluation is no simple matter, in part because most interventions are not distributed randomly in a target population. Consider children going to private or charter schools. In most countries, those children perform better than children attending public schools. Does that in itself tell us anything about the comparative performance of public versus private or charter schools? It does not.

Children attending private or charter schools may for example come from wealthier families with well educated parents who are better able to help their children at home than parents from less privileged backgrounds. The higher performance of students in private or charter schools on tests aiming to measure learning may not be related to the school in which they are enrolled per se, but instead to their socio-economic and other characteristics. The proper counterfactual question would be: how well would students enrolled today in private or charter schools perform if they had enrolled instead in public schools? Data to answer such questions are often not readily available, so the counterfactual is often not known without some further data or analysis. Special techniques – ranging from randomized control trials (the gold standard) to various statistical and econometric approaches, are often needed in order to assess the impact of specific interventions on specific outcomes. I will discuss these techniques in the second post in this series.

Do We Need Impact Evaluations in Rotary?

If as Rotarians we “do good work” in the world, why would we need impact evaluations? Isn’t it enough to serve the less fortunate? I would argue that Rotary needs impact evaluations for at least three reasons.

  1. Doing good work is not enough – we need to do the best we can. We live in a world with scarce resources. What Rotarians can contribute – whether in terms of money or time and expertise – is finite. Our resources should be devoted to projects and initiatives that have the largest positive impact on those we aim to serve. And we will be able to assess what these projects or interventions are or should be only by evaluating our work (and relying as well on the evidence generated by other organizations that are implementing robust impact evaluations.)
  2. Who says we are doing good work? We may believe we are doing good work, but do we know whether some of our projects and interventions may have unintended consequences that could be detrimental to those we are trying to serve? To consider the example of education again, if we support one dynamic school in a poor community that selects its students on the basis of their academic potential, could this have a negative effect on another school in the area which might be left only with children who tend to perform less well for whatever reason. Given that peer effects are important drivers of learning, supporting one school may have negative consequences on another school. This is of course not necessarily the case, and this does not imply  that we should never support specific schools. But we need to be aware of the potential ripple effects that our projects may have, and impact evaluation can be useful for that.
  3. Rotary has an important role to play in piloting innovative interventions. Much of what Rotary clubs do on an on-going basis is not innovative, and there is nothing wrong with that. There are a lot of needs out there that we can contribute to meet without being innovative. Being innovative is not a requirement. But at the same time, we should promote some level of innovation in Rotary. Rotarians have technical expertise in many areas and they know (or should know) their community. They are in a way uniquely positioned to propose new ways of tackling old problems, and assessing whether such new ideas actually work. Furthermore, Rotary has a relatively large foundation, but in comparison to other actors, especially in the field of development, we are small. If we could pilot more innovative interventions, evaluate them properly, and let others with more resources scale up the most promising interventions, we could achieve more for those we aim to serve.

For these reasons I believe that we need impact evaluations in Rotary. Not all projects require an evaluation, especially since evaluations take some time and cost money. But we can probably do more in this area than we are doing today. In the next post in this series, I will discuss the “how to” of impact evaluations.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.