Impact Evaluations, Part 1: Do We Need Them in Rotary?

by Quentin Wodon

Monitoring and evaluation have become essential in development and service work. When providing funding, most foundations and donors now require some type of monitoring and evaluation either ex ante to allocate funding, or ex post to assess whether projects that have been funded have proved to be successful or not. The same is true for government agencies – when deciding whether to scale up a pilot intervention, most agencies now typically rely on the results of at least some type of evaluation.

Different types of evaluations can be conducted and not all are equal in terms of what we can learn from them. In this series of three posts, I will focus on impact evaluations (as opposed to process evaluations). Specifically, I will discuss 1) whether we need impact evaluations in Rotary; 2) how impact evaluations can be implemented in practice; and 3) what are some of the limits of impact evaluations that practitioners and policy makers should be aware of. But before doing so, it is probably useful to briefly explain what an impact evaluation entails.

What Is An Impact Evaluation?

Impact evaluations aim to measure the impact of specific projects, policies, or interventions on specific outcomes. The question they ask is not whether a given outcome has been achieved or not among a target population. They look instead at the specific contribution or impact of a well-defined intervention on a well-defined outcome. Said differently, they try to measure a counterfactual: what would have been the outcome without the intervention? By comparing the assessment of what the outcome would have been without the intervention and the outcome with the intervention, impact evaluations inform us about the success (or lack thereof) of specific interventions in improving specific outcomes. When data on the cost of alternative interventions and their benefits are available, impact evaluations are especially helpful in deciding whether some interventions should be maintained or scaled up, while others should be abandoned.

Estimating the counterfactual in an impact evaluation is no simple matter, in part because most interventions are not distributed randomly in a target population. Consider children going to private or charter schools. In most countries, those children perform better than children attending public schools. Does that in itself tell us anything about the comparative performance of public versus private or charter schools? It does not.

Children attending private or charter schools may for example come from wealthier families with well educated parents who are better able to help their children at home than parents from less privileged backgrounds. The higher performance of students in private or charter schools on tests aiming to measure learning may not be related to the school in which they are enrolled per se, but instead to their socio-economic and other characteristics. The proper counterfactual question would be: how well would students enrolled today in private or charter schools perform if they had enrolled instead in public schools? Data to answer such questions are often not readily available, so the counterfactual is often not known without some further data or analysis. Special techniques – ranging from randomized control trials (the gold standard) to various statistical and econometric approaches, are often needed in order to assess the impact of specific interventions on specific outcomes. I will discuss these techniques in the second post in this series.

Do We Need Impact Evaluations in Rotary?

If as Rotarians we “do good work” in the world, why would we need impact evaluations? Isn’t it enough to serve the less fortunate? I would argue that Rotary needs impact evaluations for at least three reasons.

  1. Doing good work is not enough – we need to do the best we can. We live in a world with scarce resources. What Rotarians can contribute – whether in terms of money or time and expertise – is finite. Our resources should be devoted to projects and initiatives that have the largest positive impact on those we aim to serve. And we will be able to assess what these projects or interventions are or should be only by evaluating our work (and relying as well on the evidence generated by other organizations that are implementing robust impact evaluations.)
  2. Who says we are doing good work? We may believe we are doing good work, but do we know whether some of our projects and interventions may have unintended consequences that could be detrimental to those we are trying to serve? To consider the example of education again, if we support one dynamic school in a poor community that selects its students on the basis of their academic potential, could this have a negative effect on another school in the area which might be left only with children who tend to perform less well for whatever reason. Given that peer effects are important drivers of learning, supporting one school may have negative consequences on another school. This is of course not necessarily the case, and this does not imply  that we should never support specific schools. But we need to be aware of the potential ripple effects that our projects may have, and impact evaluation can be useful for that.
  3. Rotary has an important role to play in piloting innovative interventions. Much of what Rotary clubs do on an on-going basis is not innovative, and there is nothing wrong with that. There are a lot of needs out there that we can contribute to meet without being innovative. Being innovative is not a requirement. But at the same time, we should promote some level of innovation in Rotary. Rotarians have technical expertise in many areas and they know (or should know) their community. They are in a way uniquely positioned to propose new ways of tackling old problems, and assessing whether such new ideas actually work. Furthermore, Rotary has a relatively large foundation, but in comparison to other actors, especially in the field of development, we are small. If we could pilot more innovative interventions, evaluate them properly, and let others with more resources scale up the most promising interventions, we could achieve more for those we aim to serve.

For these reasons I believe that we need impact evaluations in Rotary. Not all projects require an evaluation, especially since evaluations take some time and cost money. But we can probably do more in this area than we are doing today. In the next post in this series, I will discuss the “how to” of impact evaluations.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.

Public, Private Secular, and Faith-inspired Schools: How Do They Differ?

Private secular schools are also growing, albeit for different reasons related more directly to the perceived quality of the education they provide. Both types of private schools can usefully complement public schools and provide choice to families. And yet, relatively little systematic evidence is available on the market share, reach to the poor, cost for families, and quality of both types of schools.Two recent World Bank studies help bridge that gap in knowledge. The first study provides a comparative assessment of the role of public, private secular, and faith-inspired schools in sub-Saharan Africa. It is based on a systematic analysis of nationally representative household surveys for 16 countries that identify separately public, private secular, and faith-inspired schools, as well as in-depth qualitative data collection in two countries (Ghana and Burkina Faso).The second study focuses on the experience of the Jesuit-run Fe y Alegría network of schools in Latin America. That study combines articles that assess (quantitatively) the performance of the network’s schools with other articles looking at the schools’ pedagogical practices.

The first study on sub-Saharan Africa suggests that the market share of faith-inspired and private secular schools is smaller than often claimed, but still substantial. Across the 16 countries, the average market share for faith-inspired schools is at 14% at the primary level and 11% at the secondary level. The corresponding market shares for private secular schools are 12% and 16%. These estimates from household surveys are of an order of magnitude similar to the statistics on the market share of the private sector as a whole, reported by the UNESCO Institute of Statistics on the basis of administrative data provided by government.

The household surveys also suggest that public schools reach the poor more than faith-inspired schools, while private secular schools are the least ‘pro-poor’. This is not surprising given that the cost for households of private secular schools is higher than that of faith-inspired schools, with public schools being the most affordable. The fact that faith-inspired schools do not reach the poor in priority, statistically speaking, does not mean that they do not aim to do so. But they operate within tight budget constraints, especially when support from governments is limited, and this requires some level of cost recovery from households. Some private secular schools also aim to reach the poor, but this tends to not to be a widely shared feature of these schools.

Finally, and most importantly, the data suggest that both faith-inspired and private secular schools have higher satisfaction rates among parents than public schools, with private secular schools performing best according to that metrics.

Tentative evidence also suggests that student performance in terms of literacy and numeracy, as well as test scores, is also often (but not always) higher among faith-inspired and private secular schools than in public schools. In the case of faith-inspired schools, qualitative evidence suggests that the emphasis placed on religion for Islamic schools and values more than religion per se in Christian schools is appreciated by parents. In other words, these schools are highly valued and could probably be better supported by governments and donors provided that they also teach secular topics, which is often the case.

Why is the quality of education provided apparently higher in faith-inspired and private schools than in public schools?

Part of the explanation could come from teacher motivation. For example, teachers in faith-inspired schools might be “working for God” (to use the title of an article on faith-inspired health care by Reinikka and Svensson). If only motivation were at play, the ability to scale up those schools, or replicate their success in other schools might be limited, as reservoirs of highly motivated teachers are probably finite.

But while motivation plays a role, the management and teacher training systems that well performing faith-inspired and private secular schools have managed to put in place also matter greatly. This is clearly apparent in the second study on Fe y Alegría schools in Latin America. The management teams of the schools spend significant time and have succeeded in fostering a culture of accountability and dedication in the school that helps to place service to students at the very core of the teacher’s mission.

Overall, the message from both studies is that while it is not necessarily easy, fostering a culture of great service to students can be achieved, and the experience of faith-inspired and private secular schools is useful to better understand how it can be done.

The message is not necessarily to scale up provision of education by faith-inspired and private secular schools, but rather and more importantly to support existing schools that provide a great education to children. This is especially important when schools also aim to reach the poor. We should learn from their successes and see how they might be replicated in other types of schools.

Note: this post is reproduced with minor changes from a post on the Education for Global Development blog of the World Bank available at http://blogs.worldbank.org/education/