Open Access World Bank Publications on Education (Resources Series No. 5)

Getting a good education is one of the best ways to escape poverty in the developing world. This post, the fifth in a series on open access World Bank publications, provides easy access to a selection of 50 books and reports published since 2010 by the World Bank on education and (to a lower extent) on WASH in schools. The publications were compiled as a resource for participants at the 2016 Rotary International Presidential Conferences on literacy and WASH in schools in Kolkata, India, and on WASH in schools in Pasay near Manila, Philippines. The list of publications is available here.

KolkataManilla

Rotary International has long recognized the importance of basic literacy and education, as well as WASH (water, sanitation, and hygiene). These areas have been recognized as two of the six areas of focus of the Rotary Foundation. Many clubs and districts around the world are implementing projects in those areas.

How can clubs and districts contribute to efforts towards literacy and education, including through WASH in schools? These are the questions that will be discussed at the Kolkata and Pasay Conferences. The conferences are part of five flagship conferences organized by Rotary International in 2015-16. The other conferences are on peace and conflict resolution in Ontario (California), disease prevention and treatment in Cannes, and economic development in Cape Town.

The compilation of recent World Bank publications on education made available here is provided as a service to Rotarians and others working on those areas without any endorsement of the World Bank as to which publications should be featured. Access is provided through the World Bank’s Open Knowledge Repository. In order to keep the list manageable, the focus is on books and reports published since 2010 as opposed to other publications. Only publications from the World Bank are listed simply because covering (many) other organizations would be a rather complex task. At the same time, focusing on World Bank has the advantage of being able to go global with a single organization.

The hope is that the publications listed, and more generally the World Bank’s open access knowledge resources, will be useful to conference participants and others working on education and WASH in schools.

 

Open Access Publications from the World Bank: Introduction (Resources Series No. 1)

This post is the first in a series on open access resources from the World Bank that could be useful to Rotarians as well as others involved in service work and development projects around the world. Probably more than any other development organization, the World Bank is making available a wealth of resources on topics related to development, including a large number of books and reports. The focus of most World Bank open access knowledge resources is on developing countries, but data and publications are also available for developed countries, and often lessons learned from the developing world have implications for service projects and social policy in developed countries as well.

In coming weeks, this blog will feature selections of recently published World Bank books and reports by topic, considering in priority the areas of focus of the Rotary Foundation (TRF), namely promoting peace, fighting disease, providing clean water, saving mothers and children, supporting education, and growing local economies apart from eradicating polio. The hope is that the featured publications will be beneficial not only to researchers, but also to practitioners and policy makers.

Why a Focus on Open Access Resources?

The inspiration for this series of posts on open access resources came in part from the fact that Rotary is organizing between January and March 2016 five conferences on the core areas of focus of the Rotary Foundation. The first will be the Rotary Presidential Conference on Peace and Conflict Prevention/Resolution or “World Peace Conference” to be held in January 2016 in Ontario, California. The other conferences are on disease prevention and treatment in Cannes, economic development in Cape Town, literacy and WASH (water, sanitation, hygiene) in schools in Kolkata, and WASH in schools in Manila. The dates of the five conferences are listed in the table below together with their websites.

Dates Topic Location Website
15-16 January Peace and conflict prevention/resolution Ontario, California, USA Click here
19-20 February Disease prevention & treatment Cannes, France Click here
27 February Economic development Cape Town, South Africa Click here
12-13 March Literacy & WASH in Schools Kolkata, India Click here
18-19 March WASH in Schools Pasay City, Philippines Click here

The conferences are sponsored jointly by Rotary International President K.R. Ravindran and TRF Trustee Chair Ray Klinginsmith. Each conference will be led by local Rotary districts and are open to all, whether Rotarians or not. The conferences will feature plenary sessions with world class speakers as well as parallel sessions on topics of interest and hands-on workshops.

The hope for this series of posts on open access resources is that selecting relevant publications on the topics to be discussed at the above five conferences could be useful not only to conference participants, but also to many others working or implementing service projects in those fields.

Why Focusing on World Bank Resources?

Only resources available from the World Bank will be included in this series even though many other organizations provide highly valuable open access resources. Restricting the focus on resources provided by the World Bank is driven by practicality. Including other organizations would yield a rather unwieldy list of relevant publications due to the scope of what would need to be included. At the same time, focusing on the World Bank has the advantage of being able to go global with a single organization, since the World Bank is engaged with the developing world as a whole. By contrast, many other organizations, including regional development banks, tend to have more of a regional focus.

In order to keep the list of publications and other resources highlighted through this series manageable, the focus in most cases will be on open access books and reports as opposed to other publications such as working papers, articles, and briefs. Even when restricting resources to books, a large number of World Bank publications directly relevant to the topics of the five Rotary conferences can be listed. In the case of the first conference on promoting peace for example, several dozen recent books and reports published since 2010 that relate closely to the topics of the conference can be listed.

Topics for Consideration

To keep things simple, the series of posts will consider in priority the six areas of focus of TRF, which also correspond to the topics selected for the five Rotary Presidential conferences (to a large extent, the conference on disease prevention and treatment also implicitly covers the area of focus of TRF devoted to saving mothers and children).

But the series will also feature a few cross-sectoral topics that are highly relevant to multiple areas of focus of TRF. One example is that of early childhood development, for which interventions are needed from virtually all six areas of focus of TRF. The series could also cover some topics in more depth than others, for example allocating more than one post to a single area of focus of TRF if this appears to be warranted.

So please, do not hesitate to share your views as to what should be covered by providing a comment on this post, so that your views can inform the final selection of topics and open access resources to be provided.

Increasing the Impact of Rotary (Partnerships Series No. 9)

This post is the last in a series of nine posts on partnerships, innovation, and evaluation in Rotary. The rationale for the series was my conviction that if Rotary is to have a larger impact globally, it must rely more than has been the case so far on partnerships, innovation, and evaluation (and in some areas advocacy, as has been the case with polio). Seven different projects or investments that have relied on partnerships, were innovative, and were evaluated at least in some way, were showcased. A compilation of the case studies together with a brief introduction is available here. Separate briefs are also available for each of the projects here.

TRF_Centennial_logo_lockup

As I mentioned it in the introduction to the series, partnerships help to implement larger projects and benefit from the expertise of organizations that are among the best in their field. Rotary’s Foundation was created almost 100 years ago (the Centennial is next year) and it has about $1 billion in assets. This is respectable, but in the world of development projects, which is in practice where Rotary is investing most of its funds, this remains small. Without innovation, the contribution of Rotary is an important drop, but still a drop in the development assistance bucket.

By contrast, if Rotary clubs and district innovate, successful pilots can then be scaled up by other organizations with deeper pockets, thereby potentially achieving much larger impact. However, for innovative projects to be recognized as such, proper evaluations are needed. We must be able to demonstrate the impact of pilot projects. Innovation and evaluation are like twins: they work best in pairs. Together, partnerships, innovation, and evaluation are the key to larger impact.

To encourage clubs and districts to think bigger and more strategically, stories of great projects were shared: an innovative financing mechanism for polio eradication; an award winning project fighting malaria and Ebola in Mali; a teacher training program that is transforming teaching and learning in Nepali classrooms; a project on obstetric fistula saving the lives of mothers and children in Nigeria; a program to invest in the writing skills of disadvantaged youth in the United States; a project to improve access to water and sanitation in Uganda; and a global network of Peace Centers and Peace Fellows to help promote peace.

Some of these programs and projects are large. Others are small. Most were implemented through global grants, but one was implemented through a district grant. All these projects have been in one way or another innovative. They have all leveraged partnerships not only to crowd in financial resources, but also – and even more importantly – to build on great expertise. And they have all relied on monitoring and evaluation mechanisms to assess their impact, at least partially.

Putting together great projects requires work. Fundraising is often time consuming in Rotary given the funding model of the Rotary Foundation that requires raising funds from many clubs and districts first before getting a match from the Foundation. Planning, implementing, and in addition evaluating projects also takes time, especially when one tries to do this in a professional way. Finally, in order to be innovative, Rotarians leading projects need to be aware of where the frontier is in their field, and what could be innovative. This also takes some time.

There is nothing wrong with clubs and districts funding and implementing traditional Rotary projects. Most projects will continue to be fairly simple, with funds provided to worthy charitable causes. These projects, as well as the volunteer time often contributed by Rotarians when implementing them, serve an important purpose. The beneficiaries of these projects are better off thanks to them. These projects help communities, and they also benefit Rotary through the goodwill that the projects create.

But if we want to raise the bar and achieve larger impact, we also need to do more innovative projects. Rotary needs to be bolder, more ambitious. It needs to better learn from its projects, both the great and not so great ones, and make sure that lessons learned are shared broadly, well beyond the Rotary family. The launch of the Future Vision model, despite some challenges, was a step in the right direction. As we celebrate the Centennial of the Rotary Foundation next year, let’s make sure that we have the right vision for what Rotary and its Foundation could accomplish in the next 100 years.

Providing Water and Sanitation in Uganda (Partnerships Series No. 3)

As in other low income African countries, access to water and sanitation remains limited in Uganda, especially for the poor. This third post in a series on partnerships, innovation, and evaluation tells the story of how Rotary is playing an important role in helping to meet some of the water and sanitation needs of Uganda’s population.

Apac

Water Projects

A first important initiative is the Uganda Rotary Water Plus (URWP) program. URWP coordinates work on water and sanitation done by 78 Rotary clubs (virtually all the clubs in Uganda). The program was launched by the Ugandan Minister for Water and Environment in October 2011. It promotes effective service delivery to rural and less privileged communities.

Clubs develop projects for the communities they wish to serve. For this purpose, they must first build strong relationships with the community and develop a needs assessment. Having identified needs, clubs then select partners to meet those needs, including other Rotary clubs for fund raising, non-profits and/or business partners for implementations, and local authorities. Co-funding is typically provided by the Rotary Foundation (TRF) and in some cases other funding agencies.

The design of projects must be based on adequate technologies for the community context, with attention paid to gender and environmental issues. Clubs are encouraged to link the projects to other areas of focus of TRF, for example by providing water and sanitation to schools or health clinics.

The idea is that water and sanitation alone can’t transform a community; the “Plus” in URWP refers to other areas of focus of TRF such as supporting education or fighting disease.

The model also encourages local management committees to oversee facilities cost recovery through tariffs so that funds are available for maintenance.

URWP aims to raise $7 million for more than 30 projects. Rotary International is also partnering in Uganda with USAID to invest $4 million over four years through additional projects, following previous successful similar collaborations in the Dominican Republic, Ghana, and the Philippines (this broader partnership is referred to as the International H20 Collaboration).

Beyond the mobilization of funds, the URWP initiative has also succeeded in uniting 4,000 Ugandan Rotarians, more than 3,000 Rotaractors and many members of Rotary Community Corps (RCCs) behind countrywide water and sanitation initiatives. Many have volunteered their time and financial resources to support the projects.

Community Needs Assessments

Another interesting initiative that is part of URWP has been the implementation of a detailed diagnostic of water and sanitation facilities in communities of Apac District located 250 kilometers north of Kampala.

The idea behind the water and sanitation community needs assessment was to prepare an inventory of resources as well as gaps to be used by the Ministry of Water and the Environment as well as Rotary and other funders for the prioritization of investments. Teams visited communities. After an initial meeting in each community, data collection involved implementing a survey, conducting interviews and focus groups, establishing an inventory of all water and sanitation assets in the community, and conducting community mapping exercise.

Data were collected using the FLOW (Field Level Operations Watch) system developed by Water for People. The application relies on Android cell phones together with GPS data and Google Earth software to document water and sanitation infrastructure as well as its functionality.

The community needs assessments was implemented with support from the Apac government and 16 organizations. Rotaractors served as field enumerators. Data were collected for communities as well as public institutions such as schools and health centers, with ratings provided on the quality of facilities and the satisfaction of users. Tests of water quality have also been conducted in some of the areas.

Conclusion

URWP represents a prime example of efforts by Rotary to invest in projects that have a larger impact through partnerships, innovation, and monitoring and evaluation.

The URWP team has established partnerships with multiple NGOs as well as USAID and Ministry of Water and the Environment. It has been innovative in project design to ensure a higher likelihood of sustainability. Evaluations of the projects are not yet available (many projects are still at the design or implementation stage), but monitoring systems are being put in place.

Finally, in the case of Apac district, extensive data collection has been conducted on water and sanitation assets and gaps at the level of communities in order to inform prioritization of future investments. This should also help in achieving higher impact through targeted interventions.

A brief on the URWP initiative as well as the water and sanitation context in Uganda is available here.

Partnerships, Innovation, and Evaluation, 1: Introduction

This post is the first in a series on increasing the impact of Rotary. The series will feature case studies of great service projects that have achieved larger impact through partnerships, innovation, and evaluation. The hope is that the case studies will encourage clubs and districts to think bigger in their service work.  The series will cover each of the areas of focus of the Rotary Foundation, as well as polio.

Service work through volunteering or projects is at the heart of what Rotary is all about. Membership surveys suggest that the main reason why members join and remain in Rotary is the opportunity to serve (see my recent book on Rotary). Fellowship and networking are also very important, but service is first.

Rotary is a fairly decentralized organization with at its core the Rotary club. Rotarians come in many shapes and forms, beliefs and passions. There is amazing diversity in the types of service work that Rotarians engage in. This is a strength as members choose to contribute to the causes they are most passionate about.

Most of the service work that Rotarians engage in is done through volunteering, not through service projects that benefit from financial support from the Rotary Foundation (TRF). In adition, many projects implemented with TRF support are small and based on local opportunities identified by clubs. These projects may not rely on partnerships, they may not be especially innovative, and they may not be evaluated in depth. As long as it is clear to clubs and local communities that the projects are helpful, a lack of partnership, innovation or evaluation is not necessarily a major drawback. One straitjacket does not fit all in Rotary.

At the same time however, if Rotary is to have a larger impact globally, there is also a need to put together more and larger projects that do rely on partnerships, are innovative, and are monitored and evaluated properly.

Partnerships help to implement larger projects and benefit from the expertise of organizations that are among the best in their field. Partnerships may also generate visibility and media coverage for Rotary (polio is the best example). Partnerships have a cost since effort is required for collaborations to work. But if partnerships deliver scale, expertise, or visibility, gains outweigh the costs.

Innovation is even more important than partnerships to achieve larger impact and discover better ways to serve communities. Without innovation, the contribution of TRF is a drop in the development assistance bucket. TRF does have a respectable size, but in comparison to development funding, it is very small.

Total annual giving by the foundation represents less than half a percent of what the World Bank provides in development assistance every year, and this is just one of a number of development agencies. But if Rotary experiments and innovates, pilots that prove successful can be scaled up by other organizations with deeper pockets, thereby achieving larger impact.

Without serious monitoring and evaluation, innovation does not help much because impact on the ground must first be demonstrated at the pilot stage for a promising intervention to be scaled up. Innovation and evaluation are like twins: they work best as a pair. Evaluation is also needed for Rotary to learn internally from both successes and mistakes.

All three ingredients ̶ partnerships, innovation, and evaluation, can help increase the impact of Rotary’s service work. In order to encourage clubs and districts to move in that direction, this series will show how partnerships, innovation, and evaluation can be harnessed to serve Rotary’s mission of service above self.

The series will tell the story of projects in each of the areas of focus of TRF: promoting peace, fighting disease, providing clean water, saving mothers and children, supporting education, growing local economies, and eradicating polio.

You will learn about an innovative financing mechanism for polio eradication; an award winning project reducing under five mortality in Mali; a program that is transforming teaching and learning in Nepali classrooms; a project to save the life of mothers and children in Nigeria; a program to invest in the writing skills of disadvantaged youth in the United States; projects and initiatives to improve access to water and sanitation in Uganda; and the work done by Rotary with Peace Centers.

All these projects are in one way or another innovative. They all leverage partnerships. And virtually all build on solid monitoring and evaluation mechanisms. Hopefully, the series will give you additional insights into some of the great projects that clubs and districts are implementing around the world.

Please do not hesitate to send me an email through the Contact Me page of this blog if you believe other projects should be featured (perhaps in another series), and feel free to post comments on the projects that you find particularly inspiring.

 

 

Impact Evaluations, Part 3: What Are Their Limits?

by Quentin Wodon

In the first post of this series, I argued that impact evaluations could be highly valuable for organizations such as Rotary in order to assess the impact of innovative interventions that have the potential to be replicated and scaled up by others if successful. In the second post I suggested that a range of techniques are available to implement impact evaluations. In this third and last post in the series, I would like to mention some of the limits of impact evaluations. Specifically, I will discuss four limits: (1) limits as to what can be randomized or quasi-randomized; (2) limits in terms of external validity; (3) limits in terms of explanation as opposed to attribution; and finally (4) limits in terms of short-term versus long-term effects.

Can Everything Be Randomized?

The gold standard for impact evaluations is randomized controlled trials (RCTs), as discussed in the second post in this series. When it is not feasible to randomize the beneficiaries of an interventions, statistical and econometric techniques can sometimes be used to assess impact through “quasi-randomization”. But not all types of interventions can be randomized or quasi-randomized. If one wants to assess the impact on households of a major policy change in a country, this may be hard to randomize.

One example would be the privatization of a large public company with a monopoly in the delivery of a specific good. The company can be privatized, but typically it is difficult to privatize only part of it, so assessing the impact of privatization on households may be hard to do because of the absence of a good counterfactual. Another example would be a major change in the way public school teachers are evaluated or compensated nationally. At times, even with such reforms, it may be feasible to sequence the new policy, for example by covering first some geographic areas and not others, which can provide data and ways to assess impacts. But in many cases the choice is “all or nothing”. Under such circumstances techniques used for impact evaluations may not work. Some have argued that for many of the most important policies that affect development outcomes, the ability to randomize is the exception rather than the rule.

For the types of projects that most Rotary clubs are implementing, I would have doubts about an argument that randomization would not be feasible, at least at some level. This does not mean that all or even most of our projects should be evaluated. But we should recognize that most of our projects are small and local, which makes it easier to randomize (some of) them, when appropriate for evaluation. For larger programs or policy changes, one must however be aware that randomization or quasi-randomization are not always feasible.

Internal Versus External Validity

When RCTs or quasi-randomization are used to assess the impact of interventions, the evaluators often pay special attention to the internal validity of the evaluation. For example, are the control and treatment groups truly comparable, so that inferences about impact are legitimate? Careful evaluation design and research help in achieving internal validity.

But while good evaluations can be trusted in terms of their internal validity, do the results also have external validity? Do they apply beyond the design of the specific evaluation that has been carried out? Consider the case of a NGO doing great work in an area of health through an innovative pilot program. If the innovative model of that NGO is found to be successful and scaled up by a Ministry of Health, will the same results be observed nationally? Or is there a risk that with the scale-up, some of the benefits observed in the pilot will vanish, perhaps because the staff of the Ministry of Health are not as well trained or dedicated as the staff of the NGO? There have been cases of interventions when, as pilots were scaled up, their original promise did not materialize at scale.

Attribution Versus Explanation

Consider again the example of the dictionary project mentioned in the previous post. An impact evaluation could lead to the conclusion that the project improves some learning outcomes for children, or that it does not. Impact evaluations are great to attribute impacts and establish cause and effect. But they do not necessarily tell us why an impact is observed or not. For that, an understanding of the context of the intervention is needed. Such context is often provided by so-called process as opposed to impact evaluations. There is always a risk that an impact evaluation will be like a black box – impacts can be attributed, but the reasons for success or lack thereof may not be clear. This in turn can be problematic when scaling up programs that were successful as pilots because when doing so, it is often necessary to alter some of the parameters of the interventions that were evaluated, and without rich context, the potential consequences of altering some of the parameters of the original intervention may not be known.

Short Versus Long-term Effects

Another issue with impact evaluations is the time dimension to which they refer. Some interventions may have short-term positive impacts but no long-term gains. An evaluation carried out one or two years after an intervention may suggest positive impacts, but those could very well vanish after a few years. Conversely, other evaluations may have no clear impact in the short term, but positive impacts later on. Ideally, one would like to have information on both short-term and long-term impacts, but this may not be feasible. Most evaluations by design tend to look at short-term impacts rather than long-term impacts.

Implications of this Discussion

The above remarks should make it clear that impact evaluations are no panacea. They can be very useful – and I believe that Rotary should invest more in them for innovative projects that could be scaled up by others if successful – but they are not appropriate for all projects, and they should be designed with care.

I hope that this three-part series has helped some of you to understand better why impact evaluations have become so popular in development and service work, but also why they require hard work to set up well. Again, if you are considering impact evaluations in your service work, please let me know, and feel free to comment and share your own experience on this topic.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.

 

Impact Evaluations, Part 2: How Are They Done?

by Quentin Wodon

Having argued in the first post in this series of three that we need more impact evaluations in Rotary, the next question is: How are such evaluations to be done? One must first choose the evaluation question, and then use an appropriate technique to answer the question. The purpose of this post is to briefly describe these two steps. A  useful resource for those interested in knowing more is an open access book entitled Impact Evaluation in Practice published by the World Bank a few years ago. The book is thorough, yet not technical (or at least not mathematical), and thereby accessible to a large audience.

As mentioned in the first post in this series, impact evaluations seek to answer cause-and-effect questions such as: what is the impact of a specific program or intervention on a specific outcome? Not every project requires an impact evaluation – but it makes sense to evaluate the impact of selected projects that are especially innovative and relatively untested, replicable at larger scale, strategically relevant for the aims of the organization implementing them, and potentially influential if successful. It is also a good practice to combine impact evaluations with a cost-effectiveness analysis, but this will not be discussed here.

Evaluation Question

An impact evaluation starts with a specific project and a question to be asked about that project. Consider the dictionary project whereby hundreds if not thousands of Rotary clubs distribute free dictionaries to primary school students, mostly in the United States. This project has been going on for many years in many clubs. In Washington DC where I work, local Rotary clubs – and especially the Rotary Club of Washington DC – distribute close to 5,000 dictionaries every year to third graders. Some 50,000 dictionaries have been distributed in the last ten years. This is the investment made in just one city. My guess is that millions of dictionaries have been distributed by Rotarians in schools throughout the US.

The dictionary project is a fun and feel good activity for Rotarians, which also helps to federate members in a club because it is easy for many members to participate. I have distributed dictionaries in schools several times, the last time with my daughters and two other Interactors. Everybody was happy, especially the students who received the dictionary with big smiles. Who could argue against providing free dictionaries in public schools for children, many of whom are from underprivileged backgrounds?

I am not going to argue here against the dictionary project. But for this project as for many others, I would like to know whether it works to improve the prospects and life of beneficiaries – in this case the children who receive the dictionaries. It could perhaps be enough to justify the project that the children are happy to receive their own dictionary and that a few use it at home. But the project does have a cost, not only in terms of the direct cost of purchasing the dictionaries, but also in terms of the opportunity cost for Rotarians to go to the schools and distribute the dictionaries. Rotary clubs could decide to continue the project even if it were shown to have limited or no medium term impact on various measures of learning for the children. But having information on impact, as well as potential ways to increase impact, would be useful in making appropriate decisions to continue this type of service project or not. It would not matter much if dictionaries were distributed only by a few clubs in a few schools– but this is a rather large project for clubs in the US.

An impact evaluation question for the project would be of the form: “What is the impact of the distribution of free dictionaries on X?” X could be – among many other possibilities – the success rates at an English exam for the children, the propensity for children to read more at home, a measure of new vocabulary gained by children, or an assessment of the quality of the spelling in the children’s writing. One could come up with other potential outcomes that the project could  affect. In order to assess impact, one would need to compare students in schools where children did receive dictionaries to students in schools where children did not. This could be done some time after the dictionaries have been distributed.

About two years ago I tried to find whether any impact evaluation of the dictionary project had been done. I could not find any. May be I missed something (let me know if I did), but it seems that this project which requires quite a bit of funding from clubs as well as a lot of time spent by thousands of Rotarians every year has not been evaluated properly. It would be nice to know whether the project actually achieves results. This is precisely what impact evaluations are designed to do.

Evaluation Techniques

In order to estimate project impacts data collection is required. Typically for impact evaluations quantitative data are used. For the dictionary project, one could have children take a vocabulary test before receiving the dictionary and again one year after having received the dictionary. One would then compare a “treatment” group (those who received the dictionary) to a “control” group (those who did not). This could be done using data specifically collected for the evaluation, or using other information – such as standardized tests administered by schools, which would reduce the cost of an impact evaluation substantially, but would also limit the outcomes being considered for the impact evaluation to those on which students are being tested by schools.

The gold standard for establishing the treatment and control groups is randomized controlled trial (RCT). Under this design, a number of schools would be randomly selected to receive dictionaries, while other schools would not. Under most circumstances, comparisons of outcomes (say, reading proficiency) between students in schools with and without dictionaries would yield (unbiased) estimates of impacts. In many interventions, the randomization is applied to direct beneficiaries – here the students. But for the dictionary project that would probably not work – it would seem too unfair to give dictionaries to some students in a given school and not others, and the impact on some students could affect the other students, thereby making the impact evaluation not as clean as it should be (even if there may be ways to control for that). This issue of fairness in choosing beneficiaries in a RCT is very important, and typically the design of RCT evaluations has to be vetted ethically by institutional review boards (IRBs).

A number of other statistical and econometric techniques can be used to evaluate impacts when a RCT is not feasible or appropriate. These include (among others) regression discontinuity design, difference-in-difference estimation, and matching estimation. I will not discuss these techniques here because this would be too technical, but the open access Impact Evaluation in Practice book that I mentioned earlier does this very well.

Finally. apart from measuring the impact of programs through evaluations, it is also useful to better understand the factors that lead to impact or lack thereof – what is often referred to as the “theory of change” for how an intervention achieves impact. The question here is not whether a project is having the desired impact, but why it does or does not. This can be done in different ways, using both qualitative and quantitative data. For example, for the dictionary project, a few basic questions could be asked, such as: 1) did the child already have access to another dictionary at home when s/he received the dictionary provided by Rotary?; 2) how many times has the child looked at the dictionary over the last one month?; 3) did the dictionary provided by Rotary have unique features that led the child to learn new things?, etc… Having answers to this type of questions helps in interpreting the results of impact evaluations.

Conclusion

Only so much can be discussed in one post, and the question of how to implement impact evaluations is complex. Still, I hope that this post gave you a few ideas and some basic understanding of how impact evaluations are done, and why they can be useful. If you are considering an impact evaluation, please let me know, and if I can help I will be happy to. In the next and final post in this series, I will discuss some of the limits of impact evaluations.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.

“We Love You Gringos!” Serving Remote Communities in Honduras

by Quentin Wodon

Every year, many Rotarians, Rotaractors, and Interactors travel internationally to participate in hands-on community service projects in developing countries. What does it take to implement such projects? Are the projects sustainable? Do travelling teams do useful work? Do the projects make sense or are they costly? What are their main benefits?

Honduras

A new Rotarian Economist Brief by Bill Phillips suggests answers to these questions. The brief is based on a decade-long commitment by Tennessee Rotarians to support families in remote communities near Choluteca in Honduras, among others through access to electricity and water. As for other posts showcasing briefs from this blog’s series, rather than summarizing the brief, I encourage you to read it in full here.

If you would like to submit a brief about your project, please send me an email through the Contact Me page of this blog.

Rotary Foundation Basics, Part 3: What’s Great, What Could Be Improved?

by Quentin Wodon

This last post in a series of three on The Rotary Foundation (TRF) looks at what is great about the foundation, and what could probably be improved. TRF support for Rotary projects is first discussed, based on my own perceptions and those of a few fellow Rotarians to whom I talked before writing this post. Ratings received by the foundation as a charity are then briefly reviewed.

TRF Support for Rotary Projects

On the plus side, TRF support for polio has been instrumental in the near eradication of the disease, as mentioned in the previous post in this series. The focus on polio has also helped Rotary in getting a seat at the table with major partners such as the World Health Organization and the Bill and Melinda Gates Foundation. Even more importantly for Rotarians involved in service projects, the matching system whereby TRF co-funds grants is well appreciated. Both district and global grants benefit from TRF support, but I will focus in this post on global grants.

TRF provides up to $200,000 in matching funds for global grants, with the minimum match being $15,000. This is for projects that reach a minimum size of $30,000 in overall cost/funding. The system for global grants has been fundamentally revised in recent years in order to have fewer but larger grants, which should help in ensuring that projects have a bigger impact on the ground and are well managed. Six areas of focus have been selected for the grants, which is also positive to narrow down a bit the scope of what is funded (even if this scope remains fairly broad). The rules of the game for putting together global grants are clear, which also helps.

In terms of potential areas for improvement, the Grants Online System may not be as friendly as it could be, given today’s technology. Several Rotarians mentioned to me that there may also be at times issues with the grant review process. Hopefully reviewers are as objective and qualified as they should be, but this is something that could be assessed. In addition, despite efforts to help Rotarians put together great global grants, more could be done in terms of e-learning resources and other tools to help the membership develop impactful projects beyond the management and processing aspects of grants.

Many global grants are complex and require substantial expertise. It is not always clear that project teams have enough expertise. The system relies largely on volunteer hours to prepare and implement grants. This helps not only for cost savings but also for getting Rotarians’ hands dirty. Personal experiences gained through hands-on work are invaluable, especially when working directly with project beneficiaries. But it may be useful in some cases to rely more on external paid expertise, especially for large grants. In principle Rotarians can get help from Rotarian Action Groups (RAGs) for the design and implementation of projects. These are great resources, but it is not fully clear how active and effective some of the RAGs are.

One area of concern is the ability of TRF to respond to crises, with the most recent case being Ebola in West Africa. There are two issues here. One issue is fundraising. TRF does not seem to have a good system to provide incentives (read matching funds) for individual Rotarians to donate in times of crisis. Many Rotarians donate when a major crisis hits, but they often do so through other organizations because TRF does not have a good system to attract these donations. If TRF could set aside funds to match individual donations by Rotarians for major crises, this could help the foundation raise more funds. It would also help TRF gain in visibility as a humanitarian organization. The other issue is about the allocation of the funds that could be raised. Part of the funds could be allocated to Rotary clubs in affected countries for their projects to respond to crises with some type of fast track approval. Part of the funds could also be transferred to well established national and international NGOs active on the ground in responding to crises. Overall, setting up a stronger crisis response mechanism within TRF could strengthen the Rotary brand while providing much needed rapid support to vulnerable groups in countries affected by major crises.

Finally, more expertise and commitment from TRF is needed for proper monitoring and evaluation of global grants, and for disseminating the results of such evaluations. My perception is that few projects are evaluated in-depth with baseline and endline data collection to assess impact. Impact evaluation can be expensive, so not all projects should be evaluated in that way. But more should be done in this area, including in partnership with some of the NGOs implementing TRF projects. If TRF could fund more innovative projects that would be evaluated seriously, it could have a larger impact because other organizations with more resources could then bring successful TRF pilots to scale.

Ratings for TRF as a Charity

The comments above point to some great features of TRF, but also some potential areas for improvement. One should not forget however that overall TRF is very well rated as a charity. Given that many of the followers of this blog are new, let me repeat here what I mentioned on TRF ratings a few months ago on this blog as well as in another post for Rotary Voices.

In the US, Charity Navigator provides ratings for charities. Three ratings are available for financial performance, accountability and transparency, and a combination of both. Charities can get one to four stars overall. TRF has the highest possible rating (four stars). The yellow dot in the Figure below shows exactly how the foundation is rated – it has a rating of 89.8 out of a maximum of 100 for financial performance, and 97.0 on accountability and transparency, which yields a four stars rating overall.

RI Foundation Graph

For financial performance, Charity Navigator considers seven main indicators: the share of the charity’s budget spent on programs and services, the share spent on administrative expenses, the share spent on fundraising expenses, the fundraising efficiency ratio, the primary revenue growth, the program expenses growth, and the working capital ratio. Details are available on the Charity Navigator website. For accountability and transparency, a total of 17 indicators are used. TRF could have scored even higher except for the fact that its donor privacy policy requires donors to opt out for their basic information not to be (potentially) shared with other charities.

Conclusion

Overall, TRF helps fund great projects on the ground, and it is also well rated as a charity. The reform of the global grants model of the last few years to define areas of focus and implement fewer but larger grants was smart. But as for any other organization, there are also areas where TRF could probably do better, especially in terms of the friendliness of the Grants Online System, the need to ensure that project teams have the expertise they need, the ability to respond to humanitarian crises, and the need to better evaluate the impact of projects that appear especially innovative. What do you think?

Note: This post is part of a series of three on TRF: Part 1, Part 2, Part 3.

Rotary Foundation Basics, Part 2: Where Does the Money Go?

by Quentin Wodon

This second post in a series of three looks at how funding provided by The Rotary Foundation (TRF) is allocated. TRF disbursed $232 million in program expenses last year. More than half ($131 million) was allocated PolioPlus, with the rest allocated to Rotary grants ($92 million) and other programs ($ 8 million). This post briefly describes and discusses those investments.

TRF Polio Vaccine

Polio

TRF gave $131 million in 2013-14 for polio. While the report does not state explicitly where the funding came from, simple calculations suggest that two thirds may have come from the Bill and Melinda Gates Foundation (BMGF), with the rest provided by Rotarians. This is because from 2013 to 2018, for every dollar raised by Rotary for PolioPlus, BMGF provides a 2 to 1 match up to a maximum of $70 million per year, as shown in the Figure below. In addition, TRF’s annual report mentions a previous $20 million match by BMGF for polio on the revenue side. If the $90 million in revenues provided by BMGF for polio were allocated the same year to TRF program expenses (this is not stated explicitly in the report), then Rotarians would have contributed in 2013-14 about a third of total TRF program expenses for polio. If my assumption is erroneous, please let me know!

TRF Polio

As mentioned in my first post for this blog, Polio used to be a devastating disease worldwide, affecting 30,000 children per year in the US alone in the mid-1950s. Thanks to vaccines and mass immunization, the number of polio cases has dropped to close to zero. This has been a great success built on public-private partnerships. While many governments have funded polio eradication campaigns, after the United States (with $2.2 billion in contributions and pledges) the two largest donors from 1985 to 2014 have been private foundations – BMGF ($1.9 billion) and Rotary International ($1.3 billion). Apart from financial donations, hundreds of thousands of volunteers – including many Rotarians – have participated in polio vaccination campaigns.

Today, it seems to me from informal conversations with fellow Rotarians that some wonder whether it still makes sense to spend that much money on a disease that now affects few children. Is this the best investment that TRF can make? This is a difficult question to answer, but there is evidence that at the very least, this is a good investment, simply because the cost of a spreading virus could be much higher than the cost of the polio eradication campaigns. A report prepared last year for BMGF suggests that previous investments of $9 billion since the creation in 1984 of the Global Polio Eradication Initiative (GPEI) may have generated $27 billion in net benefits out of $40-50 billion in potential benefits estimated by researchers in an economic analysis of GPEI. Investments in polio eradication campaigns do have higher initial costs than routine immunization, but they may also have greater long term payoffs.

At the same time, we need to be careful in what we promise. It is important to reach the last mile towards polio eradication, but this will not be easy. Vaccination remains difficult in conflict affected areas, and the risk of exportation of the virus from those areas to other countries is real. As the TRF report highlights, only three countries remain polio-endemic today (Pakistan, Afganisthan, and Nigeria). But reports documented polio outbreaks last year in Central Asia, the Middle East, and Central Africa, leading the World Health Organization to declare in May 2014 that the spread of the virus constituted an “extraordinary event”. In terms of costs and funding as well, there seem to be some challenges. In a February 2014 report, UNICEF and WHO estimated the price tag for polio eradication for the period 2013-18 at $5.5 billion. At the time, available and confirmed contributions amounted to $1.8 billion, so that there was a funding gap of $3.7 billion.

Rotary Grants and Other Programs

TRF’s annual report provides great stories of impact in other areas of interventions apart from polio, but relatively limited details on how funds are allocated by thematic area. The information provided focuses on the allocation of funds for global grants in each of six areas of focus of TRF apart from polio. A total of $47.3 million was disbursed for global grants in 2013-14. As shown in the Figure below, disease prevention and treatment received the largest allocation (265 grants for a total value of $14.2 million), followed by water and sanitation (198 grants and $11.2 million), economic and community development (148 grants and $7.8 million), basic education and literacy (121 grants and $6.5 million), maternal and child health (69 grants and $5.1 million), and finally peace and conflict prevention/resolution (67 grants and $2.7 million, excluding allocations to Rotary peace centers in a handful of universities).

TRF Global Grants

Information is also available in the TRF annual report on which regions benefit from the largest amount of funding all programs combined. Sub-Saharan Africa came first, with $104 million in funding provided, followed by South Asia ($56 million), East Asia and the Pacific ($24 million), North America ($19 million), the Middle East and North Africa as well as Europe (each $9 million), Central America and the Caribbean ($6 million), Latin America ($5 million), and finally Russia, Georgia, and the Commonwealth of Independent States (less than $1 million).

That’s it for the basics of how TRF program expenses are allocated. While a majority of funds allocated by TRF go to polio, quite a bit of this investment comes from matching funds provided by BMGF, so that a large share of the funds donated by Rotarians or earned by the foundation from its assets go to other priority areas. In the last post in this series, I will discuss the foundation’s performance.

Note: This post is part of a series of three on TRF: Part 1, Part 2, Part 3.