Launch of the Rotarian Pro Bono Initiative in Capitol Hill

Rotarians could have a larger postive impact on their community if they used their professional skills to the benefit of local nonprofits.  I have mentioned the idea of the Pro Bono Rotarian on this blog in recent months. My club is launching a new pro bono pilot initiative on July 12 at the Hill Center in Washington, DC.

For readers of this blog living in the greater Washington, DC, area, I hope that you will be able to join us for the launch event. Our keynote speaker will be Eric Goldstein, the Founder and CEO of One World Education. Please spread the word about this event!

For those not living in the Washington, DC area who may be interested in the initiative, please don’t hesitate to post a comment on this blog or contact me if you would like to learn more about this initiative and how you could launch similar initiatives in your club.

The info on our launch event is provided here as well as below.

Launch of the Capitol Hill Pro Bono Initiative

Tuesday July 12, 2016 from 6:00 PM to 7:00 PM at the Hill Center

Old Naval Hospital, 921 Pennsylvania Ave SE, Washington, DC 20003

To help us plan, please register at

What? Help local nonprofits to achieve higher impact. As a lawyer, marketer, social media expert, evaluation specialist, or other professional, volunteer your skills to help nonprofits improve/expand their services.

Why? Because you can often make a larger impact in the community when you volunteer your skills to help nonprofits excel and grow.

How? Join an initiative from the Rotary Club of Capitol Hill in 2016-17 to provide pro bono advice to local nonprofits in Capitol Hill and beyond.

Who? This initiative is for Rotarians and others to engage in service work. Non-Rotarians are welcome to join teams advising participating nonprofits.

Keynote Speaker: Eric Goldstein, Founder of One World Education

One World Education is an innovative DC-based nonprofit running the largest writing program in DC public schools, reaching close to 6,000 students in 2015-16. A team from the Rotary Club of Capitol Hill and American University recently conducted an independent evaluation of One World Education, suggesting positive impacts and strong appreciation by teachers and students. Eric Goldstein will explain how the program works, why writing skills are essential for students to succeed in college and careers, and how nonprofits can benefit from professional pro bono advice.

Eric Goldstein is the founder of One World Education. Previously he was an educator in public, charter, and independent schools. He earned a US Department of the Interior Partners in Education Award while teaching in DC. Eric holds a Master’s in Education from the University of Vermont and a Master’s of International Policy from George Washington University. His career in education started after a solo 5,000-mile bicycle trip across the US in 1999.



Improving Immunization and Fighting Polio in Pakistan

Polio remains endemic in only two countries: Afghanistan and Pakistan. Apart from polio campaigns, broader support for immunization is essential to eradicate polio. Two weeks ago (on April 21, 2016), the World Bank approved an International Development Association (IDA) credit of $50 million to increase the availability of vaccines for infectious diseases, including polio, for children under two years of age in Pakistan.  Additional funding to the amount of $80 million is provided by a World Bank administered multi-donor trust fund, Gavi – the Vaccine Alliance, and the United States Agency for International Development. The Bill and Melinda Gates Foundation also participates through a buy-down mechanism (on what a buy-down amounts to, click here). Below is information on the project reproduced from the World Bank’s website (the original link for the information is here).


The National Immunization Support Project (NISP) is supporting the country’s Expanded Program on Immunization (EPI) that aims to immunize all children against eight vaccine preventable diseases:  tuberculosis, poliomyelitis, diphtheria, pertussis, tetanus, hepatitis B, haemophilus influenza type b (Hib), and measles. Strengthening EPI will also support Pakistan’s access to newer vaccines which are either in the process of roll out (pneumococcal vaccine) or under planning (rotavirus vaccine).

The Project is also receiving additional support of $80 million grant from a World Bank administered multi-donor trust fund, Gavi – the Vaccine Alliance, and the United States Agency for International Development. The Bill and Melinda Gates Foundation is also supporting the project through an innovative partial conversion of the IDA credit into a grant upon successful achievement of project objectives.

“Pakistan is grappling with the public health emergency of polio virus transmission. Ensuring strong routine immunization services is the first essential pillar in polio eradication”, says Illango Patchamuthu, World Bank Country Director for Pakistan. “The World Bank and other development partners are working with the Government of Pakistan to strengthen routine immunization services at the critical endgame stage of polio eradication, particularly as Pakistan introduces injectable polio vaccine into its routine schedule”.

The project will incentivize provincial government capacity for rigorous monitoring and effective implementation of its program, including strengthened vaccine logistics, and deploying and expanding qualified technical and managerial personnel.

“Pakistan’s performance in maternal and child health remains weak and inadequate immunization coverage is a major challenge. Childhood immunization against vaccine preventable diseases can help in significant reductions in disability and death”, says Robert Oelrichs, World Bank Task Team Leader of the Project. “The project will establish linkages of the federal and provincial EPI cells with private sector health providers and health-related civil society organizations (CSOs) working in low coverage catchment areas – especially urban slums.”

Children under two years of age in Pakistan are the main beneficiaries of NISP – particularly children belonging to the poorest households in which immunization coverage is lowest. In addition, all children will benefit from strengthened polio and measles interventions.

The credit is financed by IDA, the World Bank’s fund for the poor, with a maturity of 25 years, including a grace period of 5 years.

Pro Bono Rotarian Initiative

Rotary is about fellowship and service work. How do we increase the impact of our service work in order to achieve higher impact in our communities while also fostering fellowship among Rotarians and others committed to making a difference in the life of the less fortunate? One potential response is the concept of the pro bono Rotarian or Rotaractor.

In my (limited) experience, many clubs engage in service projects that do not really build on the professional expertise of their members. Beautifying a school before the start of the school year, serving food for the homeless, helping in the renovation of a house for a vulnerable family, distributing dictionaries to third graders, or even joining a polio vaccination drive for a short period of time are all worthwhile activities. Such activities should continue and they often enable many members in a club to be involved in the service projects of the club.

But these one-shot activities typically do not build on the expertise that Rotarians have developed over many years in their professional career. In addition to traditional (local) service projects, Rotarians should probably also engage in more extensive pro bono work, for example to provide advice to nonprofits as consultants would. While the term pro bono is often associated with free legal advise, pro bono work can be done in many other areas, building on a wide range of expertise that volunteers may have. The value of the volunteer time that Rotarians would allocate to pro bono consulting could be very high for local nonprofits, with potentially larger beneficial impacts for communities than is the case with traditional projects. Again, the idea is not to pitch one form of service work against another, but to expand on what clubs currently do in their service work.

Importantly, I believe that a pro bono consulting model may also be beneficial for fellowship among Rotarians. While for some issues faced by nonprofits pro bono consulting can be done effectively in a short period of time, for more complex issues analyzing the challenges faced by a nonprofit and suggesting a solution takes a few months. For these challenges, pro bono consulting is typically done by a small team of 3-5 volunteers who commit to dedicating a bit of their time for several months in order to provide in-depth professional and free advice to local nonprofits. As Rotarians work together on such pro bono projects, stronger fellowship and friendships will emerge, and the vitality of clubs will improve as well. The pro bono Rotarian concept can really be a win-win for local nonprofits, Rotary clubs, and the communities we serve.

This coming Rotary year, I will help my club explore in a systematic way pro bono consulting opportunities with local nonprofits in our area (Washington, DC). You will hear more about this in coming weeks and months through this blog. We will start small, and we will assess the value of our pro bono work along the way. But we hope that the idea will grow and strengthen our club, as well as other clubs that may adopt this model.

If you would like to move in this direction in your club as well or if you would like to discuss similar ideas you may have, don’t hesitate to comment on this blog or to send me if you prefer a private email through the Contact Me page. I will be happy to help if I can, and I look forward to learning from you if you have already adopted a pro bono consulting model in your own Rotary or Rotaract club.

Open Access World Bank Publications on Health, Nutrition, and Population (Resources Series No. 3)

Good health is fundamental for development. This post, the third in a series on open access World Bank publications, provides easy access to a selection of more than 50 books and reports published since 2010 by the World Bank on health, nutrition, and population. The publications were compiled as a resource for participants at the 2016 Rotary Presidential Conference on Disease Prevention and Treatment in Cannes, France. The list of publications is available here.


Rotary International has long recognized the importance of health. PolioPlus, the first global campaign aiming to eradicate polio through mass vaccination of children, was launched in 1985 by Rotary. In 1988 Rotary became a spearheading partner in the Global Polio Eradication Initiative (GPEI), a public-private partnership in which the Bill and Melinda Gates Foundation, UNICEF, the U.S. Centers for Disease Control and Prevention, the World Health Organization, and a number of national governments are also engaged and contributing.

In addition, disease prevention and treatment, and saving the lives of mothers and children, are two of the six areas of focus of the Rotary Foundation with large funding commitments, and many Rotary clubs and districts around the world implementing projects.

How can clubs and districts contribute to efforts to improve health, nutrition, and population outcomes worldwide? These are some of the questions that will be discussed at the Cannes conference, one of five flagship conferences organized by Rotary International in 2015-16. The other conferences are on peace and conflict resolution in California, economic development in Cape Town, literacy and WASH (water, sanitation, hygiene) in schools in Kolkata, and WASH in schools in Manila.

The compilation of recent World Bank publications on health, nutrition, and population made available here is provided as a service to Rotarians and others working on those areas without any endorsement of the World Bank as to which publications should be featured. Access is provided through the World Bank’s Open Knowledge Repository. In order to keep the list manageable, the focus is on books and reports published since 2010 as opposed to other publications.  Only publications from the World Bank are listed simply because covering (many) other organizations would be a rather complex task. At the same time, focusing on World Bank has the advantage of being able to go global with a single organization.

The hope is that the publications listed, and more generally the World Bank’s open access knowledge resources, will be useful to conference participants and others working on health, nutrition, and population.


Open Access Publications from the World Bank: Introduction (Resources Series No. 1)

This post is the first in a series on open access resources from the World Bank that could be useful to Rotarians as well as others involved in service work and development projects around the world. Probably more than any other development organization, the World Bank is making available a wealth of resources on topics related to development, including a large number of books and reports. The focus of most World Bank open access knowledge resources is on developing countries, but data and publications are also available for developed countries, and often lessons learned from the developing world have implications for service projects and social policy in developed countries as well.

In coming weeks, this blog will feature selections of recently published World Bank books and reports by topic, considering in priority the areas of focus of the Rotary Foundation (TRF), namely promoting peace, fighting disease, providing clean water, saving mothers and children, supporting education, and growing local economies apart from eradicating polio. The hope is that the featured publications will be beneficial not only to researchers, but also to practitioners and policy makers.

Why a Focus on Open Access Resources?

The inspiration for this series of posts on open access resources came in part from the fact that Rotary is organizing between January and March 2016 five conferences on the core areas of focus of the Rotary Foundation. The first will be the Rotary Presidential Conference on Peace and Conflict Prevention/Resolution or “World Peace Conference” to be held in January 2016 in Ontario, California. The other conferences are on disease prevention and treatment in Cannes, economic development in Cape Town, literacy and WASH (water, sanitation, hygiene) in schools in Kolkata, and WASH in schools in Manila. The dates of the five conferences are listed in the table below together with their websites.

Dates Topic Location Website
15-16 January Peace and conflict prevention/resolution Ontario, California, USA Click here
19-20 February Disease prevention & treatment Cannes, France Click here
27 February Economic development Cape Town, South Africa Click here
12-13 March Literacy & WASH in Schools Kolkata, India Click here
18-19 March WASH in Schools Pasay City, Philippines Click here

The conferences are sponsored jointly by Rotary International President K.R. Ravindran and TRF Trustee Chair Ray Klinginsmith. Each conference will be led by local Rotary districts and are open to all, whether Rotarians or not. The conferences will feature plenary sessions with world class speakers as well as parallel sessions on topics of interest and hands-on workshops.

The hope for this series of posts on open access resources is that selecting relevant publications on the topics to be discussed at the above five conferences could be useful not only to conference participants, but also to many others working or implementing service projects in those fields.

Why Focusing on World Bank Resources?

Only resources available from the World Bank will be included in this series even though many other organizations provide highly valuable open access resources. Restricting the focus on resources provided by the World Bank is driven by practicality. Including other organizations would yield a rather unwieldy list of relevant publications due to the scope of what would need to be included. At the same time, focusing on the World Bank has the advantage of being able to go global with a single organization, since the World Bank is engaged with the developing world as a whole. By contrast, many other organizations, including regional development banks, tend to have more of a regional focus.

In order to keep the list of publications and other resources highlighted through this series manageable, the focus in most cases will be on open access books and reports as opposed to other publications such as working papers, articles, and briefs. Even when restricting resources to books, a large number of World Bank publications directly relevant to the topics of the five Rotary conferences can be listed. In the case of the first conference on promoting peace for example, several dozen recent books and reports published since 2010 that relate closely to the topics of the conference can be listed.

Topics for Consideration

To keep things simple, the series of posts will consider in priority the six areas of focus of TRF, which also correspond to the topics selected for the five Rotary Presidential conferences (to a large extent, the conference on disease prevention and treatment also implicitly covers the area of focus of TRF devoted to saving mothers and children).

But the series will also feature a few cross-sectoral topics that are highly relevant to multiple areas of focus of TRF. One example is that of early childhood development, for which interventions are needed from virtually all six areas of focus of TRF. The series could also cover some topics in more depth than others, for example allocating more than one post to a single area of focus of TRF if this appears to be warranted.

So please, do not hesitate to share your views as to what should be covered by providing a comment on this post, so that your views can inform the final selection of topics and open access resources to be provided.

Increasing the Impact of Rotary (Partnerships Series No. 9)

This post is the last in a series of nine posts on partnerships, innovation, and evaluation in Rotary. The rationale for the series was my conviction that if Rotary is to have a larger impact globally, it must rely more than has been the case so far on partnerships, innovation, and evaluation (and in some areas advocacy, as has been the case with polio). Seven different projects or investments that have relied on partnerships, were innovative, and were evaluated at least in some way, were showcased. A compilation of the case studies together with a brief introduction is available here. Separate briefs are also available for each of the projects here.


As I mentioned it in the introduction to the series, partnerships help to implement larger projects and benefit from the expertise of organizations that are among the best in their field. Rotary’s Foundation was created almost 100 years ago (the Centennial is next year) and it has about $1 billion in assets. This is respectable, but in the world of development projects, which is in practice where Rotary is investing most of its funds, this remains small. Without innovation, the contribution of Rotary is an important drop, but still a drop in the development assistance bucket.

By contrast, if Rotary clubs and district innovate, successful pilots can then be scaled up by other organizations with deeper pockets, thereby potentially achieving much larger impact. However, for innovative projects to be recognized as such, proper evaluations are needed. We must be able to demonstrate the impact of pilot projects. Innovation and evaluation are like twins: they work best in pairs. Together, partnerships, innovation, and evaluation are the key to larger impact.

To encourage clubs and districts to think bigger and more strategically, stories of great projects were shared: an innovative financing mechanism for polio eradication; an award winning project fighting malaria and Ebola in Mali; a teacher training program that is transforming teaching and learning in Nepali classrooms; a project on obstetric fistula saving the lives of mothers and children in Nigeria; a program to invest in the writing skills of disadvantaged youth in the United States; a project to improve access to water and sanitation in Uganda; and a global network of Peace Centers and Peace Fellows to help promote peace.

Some of these programs and projects are large. Others are small. Most were implemented through global grants, but one was implemented through a district grant. All these projects have been in one way or another innovative. They have all leveraged partnerships not only to crowd in financial resources, but also – and even more importantly – to build on great expertise. And they have all relied on monitoring and evaluation mechanisms to assess their impact, at least partially.

Putting together great projects requires work. Fundraising is often time consuming in Rotary given the funding model of the Rotary Foundation that requires raising funds from many clubs and districts first before getting a match from the Foundation. Planning, implementing, and in addition evaluating projects also takes time, especially when one tries to do this in a professional way. Finally, in order to be innovative, Rotarians leading projects need to be aware of where the frontier is in their field, and what could be innovative. This also takes some time.

There is nothing wrong with clubs and districts funding and implementing traditional Rotary projects. Most projects will continue to be fairly simple, with funds provided to worthy charitable causes. These projects, as well as the volunteer time often contributed by Rotarians when implementing them, serve an important purpose. The beneficiaries of these projects are better off thanks to them. These projects help communities, and they also benefit Rotary through the goodwill that the projects create.

But if we want to raise the bar and achieve larger impact, we also need to do more innovative projects. Rotary needs to be bolder, more ambitious. It needs to better learn from its projects, both the great and not so great ones, and make sure that lessons learned are shared broadly, well beyond the Rotary family. The launch of the Future Vision model, despite some challenges, was a step in the right direction. As we celebrate the Centennial of the Rotary Foundation next year, let’s make sure that we have the right vision for what Rotary and its Foundation could accomplish in the next 100 years.

Buying Down Polio (Partnerships Series No. 2)

By partnering with the World Bank in an innovative way, Rotary has successfully leveraged  its funding for polio eradication, contributing to success towards one year without polio in Nigeria and in Africa. This post, the second in a series on partnerships, innovation, and evaluation, explains how the innovative polio buy-down mechanism has worked.

Nigeria’s President vaccinates his granddaughter – Photo courtesy of Dr. Etsano.

Last month, Africa achieved a key milestone towards polio eradication, with no case of polio observed for a full year. It will still take a few weeks for the World Health Organization to officially certify this milestone, and for the region to be declared polio-free, no polio cases should be observed for a period of three years. Still, tremendous progress towards polio eradication has been accomplished. Just a few years ago, hundreds of cases of polio were observed annually in Nigeria. The country achieved its first full year without polio on July 24, 2015. This will leave only Afghanistan and Pakistan on the list of polio-endemic countries.

As noted in a recent post on the World Bank health blog, achieving one year without polio in Nigeria required persistence and courage. In some areas, professionals and volunteers who led the polio campaigns risked their life: Boko Haram assassinated nine polio vaccinators two years ago in the north of the country. Vaccinators had to rely on “hit and run” tactics to reduce exposure to risk, vaccinating children quickly in the morning and leaving the area by the afternoon. (For an understanding of the role of a wide range of people at the heart of polio eradication (in the case of Afghanistan), see the great slide show provided by the Global Polio Eradication Initiative.)

The polio campaigns also required great effort and creativity from multiple agencies, including through an innovative buy-down mechanism implemented by the World Bank and funded by the Bill and Melinda Gates Foundation, as well as Rotary International and the U.S. Centers for Disease Control via the U.N. Foundation. (The Gates Foundation and Rotary International are the two largest donors worldwide towards polio eradication over the last 30 years.) Partnership with the government of Nigeria, the World Health Organization (WHO), and UNICEF, among others, was also crucial to the success of the campaigns.

How did the polio buy-down mechanism work? The basic idea was for the World Bank to fund polio eradication projects through concessional IDA (International Development Association) loans. In the case of Nigeria, two projects worth $285 million, including additional financing, were implemented over the last dozen years. The projects included clauses that allowed loans to Nigeria to become grants if the country achieved a high level of polio immunization coverage. In other words, if the immunization targets indicated in the loans were achieved and verified independently through in-depth audits, the government would receive grant funding for polio eradication without the need to repay the loans.

For the government of Nigeria, this was potentially a great deal. And for the Gates Foundation and the Rotary Foundation of Rotary International, this was also a pretty good investment. In general, investments towards polio eradication have been shown to be fairly cost-effective. But with the buy-down mechanism, these investments were especially cost-effective.

Due to the concessional nature of IDA loans (long-term zero or low-interest loans which grace repayment periods), for every dollar contributed to the buy-down, the actual amount of resources that could be transferred to the government for the polio campaigns was two times larger. The buy-down funds were transferred by the Gates Foundation and Rotary International (in the case of Rotary in partnership with the United Nations Foundation) to the World Bank at the start of the project, and used to repay the loan at the end of the project if the target immunization rates had been achieved.

Through this buy-down mechanism, the Gates Foundation and Rotary International were able to offset all future loan repayment obligations with a much smaller amount of funding to pay back IDA than the face value of the loans granted to Nigeria. Again, one dollar invested by these private donors generated about $2 for polio eradication in Nigeria, with a similar mechanism in place for Pakistan. The mechanism also had built-in incentives to encourage strong implementation performance by the government of Nigeria since the loans would be transformed into grants only if the specific immunization targets were to be achieved.

At the time of the first buy-down mechanism for polio, then-World Bank President James. D. Wolfensohn stated, “The partnership to buy-down loans to grants on the basis of good performance is an example of the innovative thinking occurring in the private sector and the World Bank about how to increase finances for the fight against global diseases. This financial innovation is bringing the goal of a polio-free world one large step closer to becoming reality.”

Could similar buy-down mechanisms be applied in other areas? That was probably the hope when this innovative mechanism was created for polio a dozen years ago. It seems however that with few exceptions the idea has not yet been replicated much in other development areas, even if it has been mentioned in a number of reports, including in a Results for Development report on education.

A number of conditions have to be met for this type of buy-down mechanism to be successful. But in the case of polio, it has been successful, enabling the Gates Foundations, individual Rotarian donors through the Rotary Foundation, the United Nations Foundation, and the World Bank to achieve higher impact towards polio eradication than would have been the case otherwise.

A brief on polio in Africa and the buy-down mechanism is available here.

This post is reproduced with minor changes from a post published by the author on September 2, 2015 on the World Bank’s Financing for Development blog at

Partnerships, Innovation, and Evaluation, 1: Introduction

This post is the first in a series on increasing the impact of Rotary. The series will feature case studies of great service projects that have achieved larger impact through partnerships, innovation, and evaluation. The hope is that the case studies will encourage clubs and districts to think bigger in their service work.  The series will cover each of the areas of focus of the Rotary Foundation, as well as polio.

Service work through volunteering or projects is at the heart of what Rotary is all about. Membership surveys suggest that the main reason why members join and remain in Rotary is the opportunity to serve (see my recent book on Rotary). Fellowship and networking are also very important, but service is first.

Rotary is a fairly decentralized organization with at its core the Rotary club. Rotarians come in many shapes and forms, beliefs and passions. There is amazing diversity in the types of service work that Rotarians engage in. This is a strength as members choose to contribute to the causes they are most passionate about.

Most of the service work that Rotarians engage in is done through volunteering, not through service projects that benefit from financial support from the Rotary Foundation (TRF). In adition, many projects implemented with TRF support are small and based on local opportunities identified by clubs. These projects may not rely on partnerships, they may not be especially innovative, and they may not be evaluated in depth. As long as it is clear to clubs and local communities that the projects are helpful, a lack of partnership, innovation or evaluation is not necessarily a major drawback. One straitjacket does not fit all in Rotary.

At the same time however, if Rotary is to have a larger impact globally, there is also a need to put together more and larger projects that do rely on partnerships, are innovative, and are monitored and evaluated properly.

Partnerships help to implement larger projects and benefit from the expertise of organizations that are among the best in their field. Partnerships may also generate visibility and media coverage for Rotary (polio is the best example). Partnerships have a cost since effort is required for collaborations to work. But if partnerships deliver scale, expertise, or visibility, gains outweigh the costs.

Innovation is even more important than partnerships to achieve larger impact and discover better ways to serve communities. Without innovation, the contribution of TRF is a drop in the development assistance bucket. TRF does have a respectable size, but in comparison to development funding, it is very small.

Total annual giving by the foundation represents less than half a percent of what the World Bank provides in development assistance every year, and this is just one of a number of development agencies. But if Rotary experiments and innovates, pilots that prove successful can be scaled up by other organizations with deeper pockets, thereby achieving larger impact.

Without serious monitoring and evaluation, innovation does not help much because impact on the ground must first be demonstrated at the pilot stage for a promising intervention to be scaled up. Innovation and evaluation are like twins: they work best as a pair. Evaluation is also needed for Rotary to learn internally from both successes and mistakes.

All three ingredients ̶ partnerships, innovation, and evaluation, can help increase the impact of Rotary’s service work. In order to encourage clubs and districts to move in that direction, this series will show how partnerships, innovation, and evaluation can be harnessed to serve Rotary’s mission of service above self.

The series will tell the story of projects in each of the areas of focus of TRF: promoting peace, fighting disease, providing clean water, saving mothers and children, supporting education, growing local economies, and eradicating polio.

You will learn about an innovative financing mechanism for polio eradication; an award winning project reducing under five mortality in Mali; a program that is transforming teaching and learning in Nepali classrooms; a project to save the life of mothers and children in Nigeria; a program to invest in the writing skills of disadvantaged youth in the United States; projects and initiatives to improve access to water and sanitation in Uganda; and the work done by Rotary with Peace Centers.

All these projects are in one way or another innovative. They all leverage partnerships. And virtually all build on solid monitoring and evaluation mechanisms. Hopefully, the series will give you additional insights into some of the great projects that clubs and districts are implementing around the world.

Please do not hesitate to send me an email through the Contact Me page of this blog if you believe other projects should be featured (perhaps in another series), and feel free to post comments on the projects that you find particularly inspiring.



Impact Evaluations, Part 3: What Are Their Limits?

by Quentin Wodon

In the first post of this series, I argued that impact evaluations could be highly valuable for organizations such as Rotary in order to assess the impact of innovative interventions that have the potential to be replicated and scaled up by others if successful. In the second post I suggested that a range of techniques are available to implement impact evaluations. In this third and last post in the series, I would like to mention some of the limits of impact evaluations. Specifically, I will discuss four limits: (1) limits as to what can be randomized or quasi-randomized; (2) limits in terms of external validity; (3) limits in terms of explanation as opposed to attribution; and finally (4) limits in terms of short-term versus long-term effects.

Can Everything Be Randomized?

The gold standard for impact evaluations is randomized controlled trials (RCTs), as discussed in the second post in this series. When it is not feasible to randomize the beneficiaries of an interventions, statistical and econometric techniques can sometimes be used to assess impact through “quasi-randomization”. But not all types of interventions can be randomized or quasi-randomized. If one wants to assess the impact on households of a major policy change in a country, this may be hard to randomize.

One example would be the privatization of a large public company with a monopoly in the delivery of a specific good. The company can be privatized, but typically it is difficult to privatize only part of it, so assessing the impact of privatization on households may be hard to do because of the absence of a good counterfactual. Another example would be a major change in the way public school teachers are evaluated or compensated nationally. At times, even with such reforms, it may be feasible to sequence the new policy, for example by covering first some geographic areas and not others, which can provide data and ways to assess impacts. But in many cases the choice is “all or nothing”. Under such circumstances techniques used for impact evaluations may not work. Some have argued that for many of the most important policies that affect development outcomes, the ability to randomize is the exception rather than the rule.

For the types of projects that most Rotary clubs are implementing, I would have doubts about an argument that randomization would not be feasible, at least at some level. This does not mean that all or even most of our projects should be evaluated. But we should recognize that most of our projects are small and local, which makes it easier to randomize (some of) them, when appropriate for evaluation. For larger programs or policy changes, one must however be aware that randomization or quasi-randomization are not always feasible.

Internal Versus External Validity

When RCTs or quasi-randomization are used to assess the impact of interventions, the evaluators often pay special attention to the internal validity of the evaluation. For example, are the control and treatment groups truly comparable, so that inferences about impact are legitimate? Careful evaluation design and research help in achieving internal validity.

But while good evaluations can be trusted in terms of their internal validity, do the results also have external validity? Do they apply beyond the design of the specific evaluation that has been carried out? Consider the case of a NGO doing great work in an area of health through an innovative pilot program. If the innovative model of that NGO is found to be successful and scaled up by a Ministry of Health, will the same results be observed nationally? Or is there a risk that with the scale-up, some of the benefits observed in the pilot will vanish, perhaps because the staff of the Ministry of Health are not as well trained or dedicated as the staff of the NGO? There have been cases of interventions when, as pilots were scaled up, their original promise did not materialize at scale.

Attribution Versus Explanation

Consider again the example of the dictionary project mentioned in the previous post. An impact evaluation could lead to the conclusion that the project improves some learning outcomes for children, or that it does not. Impact evaluations are great to attribute impacts and establish cause and effect. But they do not necessarily tell us why an impact is observed or not. For that, an understanding of the context of the intervention is needed. Such context is often provided by so-called process as opposed to impact evaluations. There is always a risk that an impact evaluation will be like a black box – impacts can be attributed, but the reasons for success or lack thereof may not be clear. This in turn can be problematic when scaling up programs that were successful as pilots because when doing so, it is often necessary to alter some of the parameters of the interventions that were evaluated, and without rich context, the potential consequences of altering some of the parameters of the original intervention may not be known.

Short Versus Long-term Effects

Another issue with impact evaluations is the time dimension to which they refer. Some interventions may have short-term positive impacts but no long-term gains. An evaluation carried out one or two years after an intervention may suggest positive impacts, but those could very well vanish after a few years. Conversely, other evaluations may have no clear impact in the short term, but positive impacts later on. Ideally, one would like to have information on both short-term and long-term impacts, but this may not be feasible. Most evaluations by design tend to look at short-term impacts rather than long-term impacts.

Implications of this Discussion

The above remarks should make it clear that impact evaluations are no panacea. They can be very useful – and I believe that Rotary should invest more in them for innovative projects that could be scaled up by others if successful – but they are not appropriate for all projects, and they should be designed with care.

I hope that this three-part series has helped some of you to understand better why impact evaluations have become so popular in development and service work, but also why they require hard work to set up well. Again, if you are considering impact evaluations in your service work, please let me know, and feel free to comment and share your own experience on this topic.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.


Impact Evaluations, Part 2: How Are They Done?

by Quentin Wodon

Having argued in the first post in this series of three that we need more impact evaluations in Rotary, the next question is: How are such evaluations to be done? One must first choose the evaluation question, and then use an appropriate technique to answer the question. The purpose of this post is to briefly describe these two steps. A  useful resource for those interested in knowing more is an open access book entitled Impact Evaluation in Practice published by the World Bank a few years ago. The book is thorough, yet not technical (or at least not mathematical), and thereby accessible to a large audience.

As mentioned in the first post in this series, impact evaluations seek to answer cause-and-effect questions such as: what is the impact of a specific program or intervention on a specific outcome? Not every project requires an impact evaluation – but it makes sense to evaluate the impact of selected projects that are especially innovative and relatively untested, replicable at larger scale, strategically relevant for the aims of the organization implementing them, and potentially influential if successful. It is also a good practice to combine impact evaluations with a cost-effectiveness analysis, but this will not be discussed here.

Evaluation Question

An impact evaluation starts with a specific project and a question to be asked about that project. Consider the dictionary project whereby hundreds if not thousands of Rotary clubs distribute free dictionaries to primary school students, mostly in the United States. This project has been going on for many years in many clubs. In Washington DC where I work, local Rotary clubs – and especially the Rotary Club of Washington DC – distribute close to 5,000 dictionaries every year to third graders. Some 50,000 dictionaries have been distributed in the last ten years. This is the investment made in just one city. My guess is that millions of dictionaries have been distributed by Rotarians in schools throughout the US.

The dictionary project is a fun and feel good activity for Rotarians, which also helps to federate members in a club because it is easy for many members to participate. I have distributed dictionaries in schools several times, the last time with my daughters and two other Interactors. Everybody was happy, especially the students who received the dictionary with big smiles. Who could argue against providing free dictionaries in public schools for children, many of whom are from underprivileged backgrounds?

I am not going to argue here against the dictionary project. But for this project as for many others, I would like to know whether it works to improve the prospects and life of beneficiaries – in this case the children who receive the dictionaries. It could perhaps be enough to justify the project that the children are happy to receive their own dictionary and that a few use it at home. But the project does have a cost, not only in terms of the direct cost of purchasing the dictionaries, but also in terms of the opportunity cost for Rotarians to go to the schools and distribute the dictionaries. Rotary clubs could decide to continue the project even if it were shown to have limited or no medium term impact on various measures of learning for the children. But having information on impact, as well as potential ways to increase impact, would be useful in making appropriate decisions to continue this type of service project or not. It would not matter much if dictionaries were distributed only by a few clubs in a few schools– but this is a rather large project for clubs in the US.

An impact evaluation question for the project would be of the form: “What is the impact of the distribution of free dictionaries on X?” X could be – among many other possibilities – the success rates at an English exam for the children, the propensity for children to read more at home, a measure of new vocabulary gained by children, or an assessment of the quality of the spelling in the children’s writing. One could come up with other potential outcomes that the project could  affect. In order to assess impact, one would need to compare students in schools where children did receive dictionaries to students in schools where children did not. This could be done some time after the dictionaries have been distributed.

About two years ago I tried to find whether any impact evaluation of the dictionary project had been done. I could not find any. May be I missed something (let me know if I did), but it seems that this project which requires quite a bit of funding from clubs as well as a lot of time spent by thousands of Rotarians every year has not been evaluated properly. It would be nice to know whether the project actually achieves results. This is precisely what impact evaluations are designed to do.

Evaluation Techniques

In order to estimate project impacts data collection is required. Typically for impact evaluations quantitative data are used. For the dictionary project, one could have children take a vocabulary test before receiving the dictionary and again one year after having received the dictionary. One would then compare a “treatment” group (those who received the dictionary) to a “control” group (those who did not). This could be done using data specifically collected for the evaluation, or using other information – such as standardized tests administered by schools, which would reduce the cost of an impact evaluation substantially, but would also limit the outcomes being considered for the impact evaluation to those on which students are being tested by schools.

The gold standard for establishing the treatment and control groups is randomized controlled trial (RCT). Under this design, a number of schools would be randomly selected to receive dictionaries, while other schools would not. Under most circumstances, comparisons of outcomes (say, reading proficiency) between students in schools with and without dictionaries would yield (unbiased) estimates of impacts. In many interventions, the randomization is applied to direct beneficiaries – here the students. But for the dictionary project that would probably not work – it would seem too unfair to give dictionaries to some students in a given school and not others, and the impact on some students could affect the other students, thereby making the impact evaluation not as clean as it should be (even if there may be ways to control for that). This issue of fairness in choosing beneficiaries in a RCT is very important, and typically the design of RCT evaluations has to be vetted ethically by institutional review boards (IRBs).

A number of other statistical and econometric techniques can be used to evaluate impacts when a RCT is not feasible or appropriate. These include (among others) regression discontinuity design, difference-in-difference estimation, and matching estimation. I will not discuss these techniques here because this would be too technical, but the open access Impact Evaluation in Practice book that I mentioned earlier does this very well.

Finally. apart from measuring the impact of programs through evaluations, it is also useful to better understand the factors that lead to impact or lack thereof – what is often referred to as the “theory of change” for how an intervention achieves impact. The question here is not whether a project is having the desired impact, but why it does or does not. This can be done in different ways, using both qualitative and quantitative data. For example, for the dictionary project, a few basic questions could be asked, such as: 1) did the child already have access to another dictionary at home when s/he received the dictionary provided by Rotary?; 2) how many times has the child looked at the dictionary over the last one month?; 3) did the dictionary provided by Rotary have unique features that led the child to learn new things?, etc… Having answers to this type of questions helps in interpreting the results of impact evaluations.


Only so much can be discussed in one post, and the question of how to implement impact evaluations is complex. Still, I hope that this post gave you a few ideas and some basic understanding of how impact evaluations are done, and why they can be useful. If you are considering an impact evaluation, please let me know, and if I can help I will be happy to. In the next and final post in this series, I will discuss some of the limits of impact evaluations.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.