Impact Evaluations, Part 1: Do We Need Them in Rotary?

by Quentin Wodon

Monitoring and evaluation have become essential in development and service work. When providing funding, most foundations and donors now require some type of monitoring and evaluation either ex ante to allocate funding, or ex post to assess whether projects that have been funded have proved to be successful or not. The same is true for government agencies – when deciding whether to scale up a pilot intervention, most agencies now typically rely on the results of at least some type of evaluation.

Different types of evaluations can be conducted and not all are equal in terms of what we can learn from them. In this series of three posts, I will focus on impact evaluations (as opposed to process evaluations). Specifically, I will discuss 1) whether we need impact evaluations in Rotary; 2) how impact evaluations can be implemented in practice; and 3) what are some of the limits of impact evaluations that practitioners and policy makers should be aware of. But before doing so, it is probably useful to briefly explain what an impact evaluation entails.

What Is An Impact Evaluation?

Impact evaluations aim to measure the impact of specific projects, policies, or interventions on specific outcomes. The question they ask is not whether a given outcome has been achieved or not among a target population. They look instead at the specific contribution or impact of a well-defined intervention on a well-defined outcome. Said differently, they try to measure a counterfactual: what would have been the outcome without the intervention? By comparing the assessment of what the outcome would have been without the intervention and the outcome with the intervention, impact evaluations inform us about the success (or lack thereof) of specific interventions in improving specific outcomes. When data on the cost of alternative interventions and their benefits are available, impact evaluations are especially helpful in deciding whether some interventions should be maintained or scaled up, while others should be abandoned.

Estimating the counterfactual in an impact evaluation is no simple matter, in part because most interventions are not distributed randomly in a target population. Consider children going to private or charter schools. In most countries, those children perform better than children attending public schools. Does that in itself tell us anything about the comparative performance of public versus private or charter schools? It does not.

Children attending private or charter schools may for example come from wealthier families with well educated parents who are better able to help their children at home than parents from less privileged backgrounds. The higher performance of students in private or charter schools on tests aiming to measure learning may not be related to the school in which they are enrolled per se, but instead to their socio-economic and other characteristics. The proper counterfactual question would be: how well would students enrolled today in private or charter schools perform if they had enrolled instead in public schools? Data to answer such questions are often not readily available, so the counterfactual is often not known without some further data or analysis. Special techniques – ranging from randomized control trials (the gold standard) to various statistical and econometric approaches, are often needed in order to assess the impact of specific interventions on specific outcomes. I will discuss these techniques in the second post in this series.

Do We Need Impact Evaluations in Rotary?

If as Rotarians we “do good work” in the world, why would we need impact evaluations? Isn’t it enough to serve the less fortunate? I would argue that Rotary needs impact evaluations for at least three reasons.

  1. Doing good work is not enough – we need to do the best we can. We live in a world with scarce resources. What Rotarians can contribute – whether in terms of money or time and expertise – is finite. Our resources should be devoted to projects and initiatives that have the largest positive impact on those we aim to serve. And we will be able to assess what these projects or interventions are or should be only by evaluating our work (and relying as well on the evidence generated by other organizations that are implementing robust impact evaluations.)
  2. Who says we are doing good work? We may believe we are doing good work, but do we know whether some of our projects and interventions may have unintended consequences that could be detrimental to those we are trying to serve? To consider the example of education again, if we support one dynamic school in a poor community that selects its students on the basis of their academic potential, could this have a negative effect on another school in the area which might be left only with children who tend to perform less well for whatever reason. Given that peer effects are important drivers of learning, supporting one school may have negative consequences on another school. This is of course not necessarily the case, and this does not imply  that we should never support specific schools. But we need to be aware of the potential ripple effects that our projects may have, and impact evaluation can be useful for that.
  3. Rotary has an important role to play in piloting innovative interventions. Much of what Rotary clubs do on an on-going basis is not innovative, and there is nothing wrong with that. There are a lot of needs out there that we can contribute to meet without being innovative. Being innovative is not a requirement. But at the same time, we should promote some level of innovation in Rotary. Rotarians have technical expertise in many areas and they know (or should know) their community. They are in a way uniquely positioned to propose new ways of tackling old problems, and assessing whether such new ideas actually work. Furthermore, Rotary has a relatively large foundation, but in comparison to other actors, especially in the field of development, we are small. If we could pilot more innovative interventions, evaluate them properly, and let others with more resources scale up the most promising interventions, we could achieve more for those we aim to serve.

For these reasons I believe that we need impact evaluations in Rotary. Not all projects require an evaluation, especially since evaluations take some time and cost money. But we can probably do more in this area than we are doing today. In the next post in this series, I will discuss the “how to” of impact evaluations.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.

6 thoughts on “Impact Evaluations, Part 1: Do We Need Them in Rotary?

  1. Yes. It is necessary to have proper impact assessment before and after implementation of any project. The author is rightly pointed out to have the assessment done following appropriate methodology and stastical tools.

  2. I strongly support the need to measure impacts–even if they are proxies. For example, many WASH (water, Sanitation and Hygiene) projects have the goal of reducing sickness and disease. But, it is difficult to attribute better health directly to safer water and better hygiene. We can measure whether people are washing hands, whether they are getting safe water, whether girls have better attendance in school because of greater privacy in their toilets/latrines. These outcomes are good proxies for long-term impacts.
    “To measure is to know”. As Rotary clubs embark on more holistic sustainable projects we need to know whether our resources are producing the desired result–if not, why not. And, Rotary’s reputation is at stake. Too often well intentioned projects fail after a short time. How can we maintain our credibility (derived from Polio Plus) if wells, toilets, filters etc, with the Rotary name are not-functioning.

    (Comment posted by Ron Denham through discussion on LinkedIn)

  3. Impact evaluations are useful so long as the effort required to do the evaluation does not overshadow the effort required to do the work. I agree with observations that impact evaluations need not be difficult to do; and also agree that it is important to know what the impact of your work is. Unfortunately, TRF has made the process rather cumbersome with the effect of not only making the evaluation difficult, but also making the impact evaluation more important than actually doing the work.

  4. Six Questions To Ask & Answer Before Conducting An Impact Evaluation
    Rotary District 5960 Impact Evaluation Summary
    By Kathy Stutzman, M.A., M.A.

    Rotary District 5960 is developing a relevant impact evaluation; including why and when impact evaluation is important and what we hope to accomplish through an impact evaluation. In this particular initiative, Rotary District 5960 was conducting a pilot project, developing new methodologies and conducting new research about delivering services in the world in different ways. This executive summary reviews six questions that are critical to ask and answer before conducting an impact evaluation.

    Impact Evaluation Summary

    When conducting an “evaluation” the groups involved need to be clear about the following questions prior to beginning:
    1. What are we evaluating?
    2. Why are we evaluating?
    3. When are we evaluating (timeframe)?
    4. Who is conducting the evaluation, and who is being surveyed?
    5. What instruments are being used? Why? Standardized? Ease of collation, accessibility?
    6. What are we going to do with the data?

    These questions MUST be answered before you begin developing any type of evaluation. Clarity and understanding of what you hope to accomplish is critical to the fidelity of the evaluation.
    In any research or pilot project there are many levels of evaluation used, as knowledge and information evolves and as lessons learned require modification of direction or process. As part of this effort we studied a presentation by Jacqueline Novogratz of the Acumen Fund about new techniques that Acumen has been exploring in order to evaluate social impact. This is a brief summary:

    1. Pioneer Capital Evaluation is measuring lessons learned, inspiration and replicability. This evaluation is a longitudinal study. This is being done through reviewing original goals, objectives and hypothesis and charting them against lessons learned and the replicability of the overall Initiative. Funding, volunteer engagement and resources leveraged are part of the data being analyzed. The Simple Poverty Scorecard for Nicaragua developed by Microfinance Risk Management, L.L.C. is one of the measurement tools utilized in this evaluation.

    2. Growth Capital Evaluation is measuring increased capacity, leadership and project planning.
    a. The first evaluation instrument used was a “Community Assessment Survey” administered by the community itself and the local NGO, to the entire subject community. This survey was a valuable resource in determining demonstrated readiness and viability of project planning as well as indicators of community leadership and increased capacity. In the future it is recommended the survey questions become more standardized, the baseline questions from the Poverty Index be included and the data is analyzed using a statistical computing program. However, for purposes of this evaluation, the information that was extrapolated from the survey was useful in determining impact.
    b. A Process and Relationship Assessment” was conducted in person amongst a delegation from D-5960, the local NGO, and community members. The survey consists of a questionnaire which uses standardized questions to measure relationships, leadership and capacity. Also included were questions to gather antidotal information in order to inspire improvements and opportunities for growth.
    c. Relationships are an instrumental part of the Initiative, and there needs to be effort taken to include relationship questions during the pioneer capital stage of the development.
    d. Additional measurements of impact of growth capital include the increased social ties and leveraged resources and funding for identified plans – demonstrating the fidelity and increased capacity, leadership and project planning.

    3. Project or Impact Evaluation: Projects will have their own inherent evaluations and those evaluations may be funder driven, goal and objective driven or be determined by community demographics. To measure the effectiveness of the impact of the Initiative on the overall economic well-being of a community, the measurements will come from funder, or project driven evaluations, baseline data collected in the “Community Assessment” and a pre-and post-test of the Simple Poverty Scorecard for Nicaragua. An additional matrix has been developed to collate relevant pieces of data across the spectrum of project related evaluations.

    Before any group begins dedicating resources to “impact evaluation” please answer the first six questions listed above. Once you are clear about the how, why and whens, carefully consider who will be spearheading the evaluation process and how much of your resources you are going to dedicate to the evaluation process. Begin your evaluation conversation informed and by sharing a common language about what you hope to accomplish in your impact evaluation. It is through those conversations that each organization can answer the “impact evaluation” question in a manner that works best for your own group.

    Respectfully Submitted, Kathy Stutzman, M.A., M.A. kastutzm@smig.net (507) 219-0912 http://www.linkedin.com/in/kathystutzman or http://www.KathyStutzman.blogspot.com

    This initiative and resulting evaluations are the collective work of countless people. This executive summary is an excerpt from a larger body of work that an evaluation committee is developing to measure impact within the Rotary District 5960’s poverty eradication initiative in Nicaragua. We are currently in the early stages of disseminating information from evaluations and progress made over the past seven years. This summary is submitted solely by the author, who is a member of the Initiative’s Steering Committee and the evaluation subcommittee. To learn more about Rotary District 5960’s work on this impact evaluation, you may contact the author.

    Kathy Stutzman has been a Rotarian in the Austin Rotary Club for 24 years and has been a member of Rotary District 5960’s Fast For Hope Committee since 2007 when the group began exploring new ways to serve as Rotarians in the world. As a result of her experience within Rotary and the Fast For Hope Committee specifically, she was asked to speak at TEDxHoracePark in March of 2014 http://tedxtalks.ted.com/video/The-Power-Responsibility-of-One and her work with a team of women in Ghana was featured in The Rotarian Magazine in August 2014, http://therotarianmagazine.com/out-of-chaos As a consultant she has been conducting evaluations for businesses and organizations since 1999. An author and facilitator, Ms. Stutzman loves creating connections and then “watching the magic happen.”

  5. I believe the impact evaluations would have value for global grants with the downside of adding more requirements to a process that some already claim is cumbersome. At the club level, I think it is more important that the club is happy doing the projects than if they are doing the best they could.

  6. Evaluations will add a whole new level of respect for the work that Rotarians perform worldwide. In speaking with a local media contact yesterday he stated how impressed he is with the numerous projects that the Aurora Rotary club members manage and the number of individuals impacted by the volunteer work we do. Particularly demonstrative in light of NPO’s funding challenges in the state of IL.

Leave a Reply