Evaluation is essential to assess what works and share stronger stories

Readers of this blog know that I have emphasized for some time the need to strengthen a culture of evaluation in Rotary. Evaluations should be undertaken not only for our service projects, but also to assess how our clubs meet, work, and grow – or wither away. This post is about a recent evaluation of an education project supported by my club, and how the evaluation is proving to be useful not only for the local nonprofit we worked with, but also for our club and more generally for practitioners and policy makers working in the field of education.

OWED celebration
Photo of OWEd scholarship winners with Brian Pick, Chief of Teaching and Learning for the District of Columbia Public Schools, and Dave Paris, member of the Board of OWEd.

For several years my club has supported One World Education (OWEd), a great nonprofit based in Washington, DC. OWEd runs the largest argumentative writing program in public and charter schools in the city. The nonprofit reached 5,800 middle and high school students this past school year. The aim of the program, which runs for 4-5 weeks in the schools, is to improve the research, writing, and presentation skills of the students, many of whom are from disadvantaged backgrounds and do not do very well in school.

In previous years, our support to OWEd consisted in providing a bit of funding and volunteering at some of their events. This year, we provided college scholarships for some of the high school students (seniors) who participated in the program and worked especially hard. But we also did more. Together with a team at American University, we designed an evaluation of the program to better measure its impact. For more than 550 students, teachers collected essays written in class before and after the program. The essays were graded by professors and instructors in the Department of Literature at American University. This enabled us to assess whether the program made a difference in the writing skills of middle and high school students.

The evaluation demonstrated that the program has a positive impact. The program generates statistically significant gains in writing quality, especially for students who performed worst on the initial pre-program assignment.  The positive impact of the program was confirmed through data on the perceptions of teachers and students about the program. Two summary briefs about those evaluation results have been written and are now available for public schools and for charter schools separately.

It is clear that this type of evaluation is beneficial for the nonprofits whose programs are evaluated, as the evaluations enable the nonprofits to measure their impact, and take corrective action when needed.  The evaluations are also beneficial for our club in reassuring members that we are investing in worthwhile initiatives.

But there is more. Many others are interested in such evaluations and may learn from them, possibly generating larger impacts beyond the specific programs being evaluated. And these evaluations provide for great stories to be featured in local newspapers or magazines as well as social media, giving more visibility not only to the nonprofits and programs being evaluated, but also to the Rotary clubs that supported those evaluations.

This is what we are focusing on now – making sure that the positive results obtained by OWEd through its program are better known in Washington, DC, and beyond. We are writing short articles that document those results, and some of the stories of the students who benefited from the program.  We have secured already two placements for stories in the local media and we hope to write additional articles for national publications about the results of the evaluation. In addition, we will also prepare technical papers for academic journals. It remains to be seen whether we will be successful, but we now have a stronger story to tell thanks to the evaluation.

Finally, as mentioned, the evaluation has been summarized in two easy-to-read briefs. The two briefs, together with briefs about the work of other nonprofits operating in the field of education and skills for youth in the city, will be included in a small brief series on innovations in education in Washington, DC to be published by the World Bank. We hope that this simple brief series will help attract attention to the nonprofits doing great work in the city, while also helping practitioners and policy makers learn from the experience of successful programs.

In summary, evaluation is essential not only to help improve service projects, whether implemented by Rotary clubs or nonprofits, but also to tell stronger stories about ways to improve the lives of the less fortunate. Investing more in evaluation seems to be a win-win for nonprofits as well as service clubs.  And for Rotary as a whole, as I mentioned it in a previous series of posts on this blog, focusing more on partnerships, innovation, and evaluation seems key to achieve larger impacts.

Providing Education for Girls and Employment Opportunities for Women: Deepa Willingham at the World Bank

If you want to provide more opportunities to girls, you shouldn’t only provide them with an education – you also need to change perceptions of gender roles so that, when they grow up, girls can (among other things) fully contribute to the household’s livelihood. To achieve this, combining education with interventions for entrepreneurship and employment is the right way to go.  This messages emerges not only from impact evaluations, but also from experiences on the ground and case studies of non-governmental organizations.

PACE home-banner
PACE Universal Website Photo – Program Participants

In celebration of International Women’s DayDeepa Willingham, a Rotarian and the founder of a program called Promise of Assurance to Children Everywhere or PACE, participated in a World Bank event on March 8 about inspiring women who made a difference in the world through innovative programs in the areas of education and health.

PACE is educating girls ages three to twenty-three in a village in West Bengal, India. The school started and remains small, with a total of about 350 girls enrolled since 2003. But retention rates are at 90 percent and almost 100 girls have now completed primary school. The schools currently admits 25 students each year, well below the demand as the school receives 100 applications each year.  Admission is need-based in order to give priority to the most disadvantaged families.

What I find especially interesting is the fact that, based on community feedback, PACE has started to help women in the village find decent work through various initiatives. To help expand employment opportunities in the village, PACE is providing literacy and vocational training courses for women, many of whom go on to craft jewelry products sold locally and in the US thanks to a micro-loan.

Additional income generating activities include planting 10,000 fruit bearing trees and providing cycle-vans. Recently, an organic garden was initiated on the school’s grounds as a training facility for local farmers.  PACE has also been actively upgrading water and sanitation facilities by installing 35 tube wells and 400 sanitation units. Without safe water at home, deworming children in the school did not work as well.

Deepa explained to me that when the project started, family incomes in the village were extremely low. There are signs that this has changed for the families that have benefited from the NGO’s programs, with many families making three to four times more than what they used to  bring in (according to the families’ applications for their children to enroll in the school).

The attitudes of fathers towards their daughters have changed, as measured for example by their presence during the school’s cultural activities. Also, in the past many newborn girls in the village did not get birth certificates. This is changing simply because an official birth certificate is required for admission in the school.

Is the project cost effective? The cost for the package of services provided to girls is $375 and paid mostly through grants and other resources raised by the NGO. This package of services includes not only schooling (following a board approved curriculum), two meals per day, school supplies and uniforms, access to health care as needed, and after school enrichment programs in music, art, theater, yoga and life-skills training.

How does this compare to public schools? This is not an easy question to answer, because of complex funding by federal, state, and municipal governments for basic services as well as complementary programs (such as school meals for example). Estimates from various studies can be found through a rapid search on the web. It seems that overall PACE’s programs may be more costly than a typical public school, and also more costly than the programs implemented by low cost private schools. But the range of services provided by PACE is clearly broader, and quality is likely to be (much) higher.

As for PACE’s services for women, their cost is estimated at $175 per year. This includes the cost of the adult literacy program for reading, writing in Bengali, and accounting, as well as the vocational training program for jewelry making and tailoring.

On the occasion of International Women’s Day, at least two important lessons emerge from PACE. The first lesson is that we can learn from the experience of NGOs like PACE on how to combine multiple interventions – in education, but also in vocational training and basic health, in order to make a larger impact in the life of girls by changing attitudes towards gender roles. The second, and most important lesson is that beyond the important role of the state that we often emphasize in development work, committed individuals such as Deepa can truly make a difference in the life of girls and women.

A recording of the event at the World Bank in celebration of International Women’s Day will soon be available here in case you could not watch it online on March 8.

This post is adapted with minor changes from a post published on March 8 on the World Bank’s Education for Global Development blog.

Rotary’s Family Health Days Featured at the World Bank

Today, in celebration of International Women’s Day, Marion Bunch, the founder of Family Health Days, participated in a well-attended event at the World Bank about inspiring women who made a difference in the world through innovative programs in the areas of education and health. The event was streamed online through World Bank Live, and a recording will soon be available if you missed it. What are family health days? Do these days make sense from a policy point of view? And who is Marion Bunch? This blog post answers these three questions.

Family Health days

What are family health days?

Imagine hundreds of thousands of families and individuals getting free health care for two or three days. Family Health Days makes this feasible. Individuals and families can be tested for HIV/AIDs as well as receive counseling. Testing and/or screening for tuberculosis, diabetes, high blood pressure, and other conditions such as cervical and breast cancer is provided. Children receive immunizations as well as Vitamin A supplements. Some sites also include dental clinic and hygiene education as well as eye examinations. Health counseling is also provided.

The program was launched in 2011 and is now active in a half dozen countries in sub-Saharan Africa and South Asia. The program is led by Rotary in partnership with the Coca-Cola Africa Foundation, the U.S. Mission – including the Centers for Disease Control, USAID and the health service delivery expertise of their implementing partners – and Ministries of Health that provide services and supplies at the sites. Media partners promote the days in each of the countries. Thousands of volunteers in each country, including many Rotarians, help in various ways.

Do family or child health days make sense from a policy point of view?

Intrigued by the concept of family health days, I conducted a rapid search of the literature and found that for purposes such as screening for illnesses and child immunization, holding such days appears to make a lot of sense. This type of programs appear to be beneficial and cost effective if implemented well.

For example, Fiedler and Chuko looked at the reach and cost of child health days in Ethiopia in 2008. They found that the program reached more than 10 million children at an average cost of half a dollar per child (one dollar when including measles). This made the program cost-effective, with potential additional benefits to be reaped through economies of scope in increasing coverage at delivery sites.

Palmer and others note in a 2010 article that child health days are becoming increasingly popular, implemented in more than 50 countries at the time of the study. They conclude that the days are helping countries achieve high and equitable coverage of essential health and nutrition services. Child health days were also recognized as beneficial in a 2013 World Health Organization report on essential nutrition actions for improving maternal, newborn, infant, and young child health and nutrition.

Improvements could be made, however. In a 2012 review of experiences integrating the delivery of maternal and child health services with childhood immunization programs, Wallace, Ryman, and Dietz emphasize the importance of proper planning and awareness when implementing these interventions, among other actions to reduce the risks of logistical difficulties, time-intensive interventions ill-suited for campaign delivery, concerns about harming existing services, and overlap of target age groups with other service delivery mechanisms. The review also revealed gaps in information about costs and impacts.

On costs, a 2014 paper by Fiedler and Semakula suggests that part of the reason why costs may appear so low in some countries (US$0.22 per child in their analysis for Uganda) is because of underpayment for Ministry of Health staff and volunteer allowances. Still, the authors find that child health days are successful in saving lives, making them highly cost-effective. Vijayaraghavan and others in a case study for Somalia also find child health days to be cost-effective in addressing leading causes of child mortality in a conflict settings. They rate child health days as one of health sector’s best buys in sub-Saharan Africa.

Who is Marion Bunch?

What is also interesting – especially as we celebrate International Women’s Day and the power of women to make a difference in the life of the less fortunate, is the personal story of Marion Bunch. Marion is the Chief Executive Officer of the Rotarian Action Group Rotarians for Family Health & AIDS Prevention. She has received numerous awards on behalf of her work for AIDS. But first and foremost, she simply considers herself a mother who helps in representing the human face of AIDS. She started her work after losing her son to the disease in 1994. As she explained it in an interview. “I never thought I’d do anything about it until one day, three years after his death, I felt a tap on the shoulder, and a voice in my ear said, ‘Mom, get up and get going; you haven’t done anything, and it’s been three years.’” Since then, the reach of Family Health days has expanded every year, in close partnership with Ministries of Health in the countries where the program is running. Marion’s story has been told in a documentary film that won two 2015 Telly Awards.

What can be learned from Marion’s project?

On the occasion of International Women’s Day, at least two important lessons emerge her project. The first lesson is that we can expand access to health care for children and families through Child and Family health Days, and we should invest in the expansion of these programs. But the second and most important lesson is that beyond the important role of the state that we often emphasize in development work, committed individuals can truly make a difference in the life of the less fortunate. The theme of the event at the World Bank was “Inspiring Women of Action”. There is no doubt that Marion is such a woman.

If you were not able to watch the event live, a recording will soon be made available here.

This post is adapted from a post published today on the World Bank’s Investing in Health blog.

Evaluation of Rotary’s 2016 World Peace Conference

Promoting peace is one of six areas of focus of the Rotary Foundation. In January 2016 Rotary International held a “World Peace Conference” in Ontario, California. This post summarizes the main results of an evaluation of the conference from the point of view of participants (a paper with more detailed results is available here). The conference appears to have been successful, in terms of both the satisfaction of participants and the promotion of work on peace and conflict prevention/resolution in Rotary.

Peace conference

The World Peace Conference was one of five flagship conferences organized by Rotary in 2015-16. The other conferences are on disease prevention and treatment in Cannes, economic development in Cape Town, literacy and WASH (water, sanitation, hygiene) in schools in Kolkata, and WASH in schools near Manila.

The evaluation is based on a survey administered shortly after the conference. A single email was sent to participants to invite them to provide feedback on the conference. The web link was kept open for a week. Some 211 participants provide feedback. Nine in ten participants at the conference were members of the Rotary family, and most were Rotarians as opposed to Rotaractors and Interactors. The quality of the conference tracks and plenary sessions was deemed high. The conference was considered better than previous Rotary conference attended by participants.

Most respondents rated the various aspects of the conference highly. Slightly lower marks were however reported for the quality of the food, the cost of the conference (often an issue for district conferences as well), publicity prior to the conference, and entertainment.

Open ended questions were asked about what participants liked best and least. The quality of speakers came up as the best feature of the conference, with especially high marks for Fr. Boyle, Dr. Wollschlaeger, and Claes Nobel. The possibility for participants to choose among many different tracks and sessions was also mentioned.

As to areas for improvement, a few plenary speakers were rated poorly, as is often the case with multiple plenaries. The House of Friendship did not get high marks. Some thought that the conference was too packed. A few respondents suggested that the Peace concert was too long, and that the quality of the food could have been higher. Technical difficulties, such as a late start for some sessions, were also mentioned.

Questions were also asked about the types of speakers and sessions that participants would like to see more of, or less of in future conferences. Participants would like to see more sessions on the specific topic of the conference, whether this relates to information and debates on peace/conflict in general, information and debates on Rotary’s role in peace/conflict, or sessions on successful Rotary projects. In terms of the types of speakers to invite, there is a desire in such conferences to have more academic/research speakers, motivational speakers, and government/public sector speakers especially at the international level.

Finally, questions were asked about whether participants are engaged in peace related work currently and whether attending the conference is likely to lead them to be more engaged in such work in the future. About half of participants stated being engaged in Rotary or other volunteer work related to peace, and for one in five peace or conflict prevention/resolution are topics on which they are engaged at work and in a volunteer capacity. For a third of participants, peace/conflict work is not something they are currently working on.

Almost one in two participants stated that due in part to the conference they would be likely to be working much more on those topics in the future, and for a third, they would be likely to work a little more on those issues. Many participants are also considering in part thanks to the conference implementing Rotary peace projects or incorporating peace in their Rotary work in the future. A third stated they would definitely do so. Finally about two thirds of participants did not donate to Rotary in the past for peace related work, but half would now consider doing so, some definitely.

Responses suggest however that some of the projects participants would like to work on may not be specifically focused on peace or conflict as traditionally defined (the approach at the conference in terms of what constitutes peace/conflict related work was also fairly broad). It may also be the case that after a conference enthusiasm is high to be active in the area, while the ability to actually do so in the future may be more limited.

Still, overall the conference seems to have had a positive impact on the desire of participants to be more engaged in peace and conflict related work in the future.

To access the paper with the more detailed results of the evaluation, please click here.

Open Access World Bank Publications on Peace, Conflict, and Violence (Resources Series No. 2)

Conflict and violence have dramatic negative consequences for development and the ability of populations to emerge from poverty. At least 1.5 billion people live in countries affected by repeated cycles of political and/or criminal violence. One fifth of the extreme poor worldwide live today in fragile and conflict-affected situations (FCS), but this proportion could double by 2030 if current trends continue.

Peace conference

Low-income FCS countries have not been able to achieve the targets set forth in the Millennium Development Goals in part because of conflict and violence. In recognition of the impact of conflict and violence on development, the Sustainable Development Goals recently approved by the international community include a goal on promoting peaceful and inclusive societies for sustainable development, providing access to justice for all, and building effective, accountable and inclusive institutions.

Rotary’s Peace Conference

How do conflict and violence affect development, and what can be done to reduce the risks of conflict and violence and instead promote peace? These are some of the questions that will be discussed at the Rotary Presidential Conference on Peace and Conflict Prevention/Resolution or World Peace Conference to be held on January 15-16, 2016 in Ontario, California. The conference is one of five flagship conferences organized by Rotary International in 2015-16. The other conferences will be on disease prevention and treatment in Cannes, economic development in Cape Town, literacy and WASH (water, sanitation, hygiene) in schools in Kolkata, and WASH in schools in Manila.

The World Peace Conference will include more than 80 panel and facilitated sessions as well as plenary sessions. It is expected to attract a couple of thousand participants. You are encouraged to attend, as it promises to be a great experience!

Apart from Rotary International President K. R. Ravindran and Rotary Foundation Chair Ray Klinginsmith, keynote speakers will include Sal Khan (founder and CEO of Khan Academy), Sharon Stone (Actress), Father Greg Boyle (Executive Director of Homeboy Industries), Carrie Hessler-Radelet (Director of the Peace Corps), Dr. Bernd Wollschläger (author of A German Life: Against All Odds Change is Possible), Barbara Winton (the daughter of Sir Nicholas Winton who organized the rescue of Jewish children from Czechoslovakia in 1939), Steve Killelea (Founder of the Institute for Economics and Peace), and Mary Peters (United States Ambassador).

Open Access Resources

Rotary is of course not the only organization emphasizing peace in its service and development work. Issues related to peace, fragility, conflict, and violence have been at the core of a substantial part of the work of development organizations for many years. This means that the World Bank as well as other organizations have substantial knowledge to share with researchers, practitioners, and policy makers in these areas.

As a contribution to Rotary’s World Peace Conference, this blog is providing a guide to selected open access publications from the World Bank that could help conference participants think about conflict, violence, and development. The publications listed are made available through the World Bank’s Open Knowledge Repository. The focus on resources provided by the World Bank is driven by practicality as including other organizations would yield a rather unwieldy list of available resources. At the same time, focusing on the World Bank has the advantage of being able to go global with a single organization.  In order to keep the guide manageable, the focus is on open access books as opposed to other publications such as working papers, articles, and briefs.

Selected Recent Books and Reports

You can access 45 selected World Bank books and reports published since 2010 on conflict, violence, and adversity either by downloading the guide prepared for conference participants, or by going to the Promoting peace page of this blog. The selection of the books and reports was based on the topics to be considered at Rotary’s Peace Conference. The scope of the conference is broad, with 13 parallel tracks apart from plenary sessions. The 13 tracks of the conference have been “aggregated” into 9 topics for listing World Bank publications: (1) Conflict, Development, and Trade; (2) Fighting Crime, Violence, and Terrorism; (3) Proving Services in Contexts of Adversity; (4) Middle East Region; (5) Equity and Discrimination; (6) Social Norms and Violence Against Women; (7) Jobs and Employment; (8) Education and Health, Including Role of Faith-based Providers; and finally (9) Governance and Institutions.

The hope is that the publications selected, and more generally the World Bank’s open access knowledge resources, will be useful to conference participants and others dealing directly or indirectly with issues of conflict, violence, and adversity when implementing projects in developing and developed countries alike. Please don’t hesitate to let me know if these resources are useful, or not so much so!

Open Access Publications from the World Bank: Introduction (Resources Series No. 1)

This post is the first in a series on open access resources from the World Bank that could be useful to Rotarians as well as others involved in service work and development projects around the world. Probably more than any other development organization, the World Bank is making available a wealth of resources on topics related to development, including a large number of books and reports. The focus of most World Bank open access knowledge resources is on developing countries, but data and publications are also available for developed countries, and often lessons learned from the developing world have implications for service projects and social policy in developed countries as well.

In coming weeks, this blog will feature selections of recently published World Bank books and reports by topic, considering in priority the areas of focus of the Rotary Foundation (TRF), namely promoting peace, fighting disease, providing clean water, saving mothers and children, supporting education, and growing local economies apart from eradicating polio. The hope is that the featured publications will be beneficial not only to researchers, but also to practitioners and policy makers.

Why a Focus on Open Access Resources?

The inspiration for this series of posts on open access resources came in part from the fact that Rotary is organizing between January and March 2016 five conferences on the core areas of focus of the Rotary Foundation. The first will be the Rotary Presidential Conference on Peace and Conflict Prevention/Resolution or “World Peace Conference” to be held in January 2016 in Ontario, California. The other conferences are on disease prevention and treatment in Cannes, economic development in Cape Town, literacy and WASH (water, sanitation, hygiene) in schools in Kolkata, and WASH in schools in Manila. The dates of the five conferences are listed in the table below together with their websites.

Dates Topic Location Website
15-16 January Peace and conflict prevention/resolution Ontario, California, USA Click here
19-20 February Disease prevention & treatment Cannes, France Click here
27 February Economic development Cape Town, South Africa Click here
12-13 March Literacy & WASH in Schools Kolkata, India Click here
18-19 March WASH in Schools Pasay City, Philippines Click here

The conferences are sponsored jointly by Rotary International President K.R. Ravindran and TRF Trustee Chair Ray Klinginsmith. Each conference will be led by local Rotary districts and are open to all, whether Rotarians or not. The conferences will feature plenary sessions with world class speakers as well as parallel sessions on topics of interest and hands-on workshops.

The hope for this series of posts on open access resources is that selecting relevant publications on the topics to be discussed at the above five conferences could be useful not only to conference participants, but also to many others working or implementing service projects in those fields.

Why Focusing on World Bank Resources?

Only resources available from the World Bank will be included in this series even though many other organizations provide highly valuable open access resources. Restricting the focus on resources provided by the World Bank is driven by practicality. Including other organizations would yield a rather unwieldy list of relevant publications due to the scope of what would need to be included. At the same time, focusing on the World Bank has the advantage of being able to go global with a single organization, since the World Bank is engaged with the developing world as a whole. By contrast, many other organizations, including regional development banks, tend to have more of a regional focus.

In order to keep the list of publications and other resources highlighted through this series manageable, the focus in most cases will be on open access books and reports as opposed to other publications such as working papers, articles, and briefs. Even when restricting resources to books, a large number of World Bank publications directly relevant to the topics of the five Rotary conferences can be listed. In the case of the first conference on promoting peace for example, several dozen recent books and reports published since 2010 that relate closely to the topics of the conference can be listed.

Topics for Consideration

To keep things simple, the series of posts will consider in priority the six areas of focus of TRF, which also correspond to the topics selected for the five Rotary Presidential conferences (to a large extent, the conference on disease prevention and treatment also implicitly covers the area of focus of TRF devoted to saving mothers and children).

But the series will also feature a few cross-sectoral topics that are highly relevant to multiple areas of focus of TRF. One example is that of early childhood development, for which interventions are needed from virtually all six areas of focus of TRF. The series could also cover some topics in more depth than others, for example allocating more than one post to a single area of focus of TRF if this appears to be warranted.

So please, do not hesitate to share your views as to what should be covered by providing a comment on this post, so that your views can inform the final selection of topics and open access resources to be provided.

Increasing the Impact of Rotary (Partnerships Series No. 9)

This post is the last in a series of nine posts on partnerships, innovation, and evaluation in Rotary. The rationale for the series was my conviction that if Rotary is to have a larger impact globally, it must rely more than has been the case so far on partnerships, innovation, and evaluation (and in some areas advocacy, as has been the case with polio). Seven different projects or investments that have relied on partnerships, were innovative, and were evaluated at least in some way, were showcased. A compilation of the case studies together with a brief introduction is available here. Separate briefs are also available for each of the projects here.

TRF_Centennial_logo_lockup

As I mentioned it in the introduction to the series, partnerships help to implement larger projects and benefit from the expertise of organizations that are among the best in their field. Rotary’s Foundation was created almost 100 years ago (the Centennial is next year) and it has about $1 billion in assets. This is respectable, but in the world of development projects, which is in practice where Rotary is investing most of its funds, this remains small. Without innovation, the contribution of Rotary is an important drop, but still a drop in the development assistance bucket.

By contrast, if Rotary clubs and district innovate, successful pilots can then be scaled up by other organizations with deeper pockets, thereby potentially achieving much larger impact. However, for innovative projects to be recognized as such, proper evaluations are needed. We must be able to demonstrate the impact of pilot projects. Innovation and evaluation are like twins: they work best in pairs. Together, partnerships, innovation, and evaluation are the key to larger impact.

To encourage clubs and districts to think bigger and more strategically, stories of great projects were shared: an innovative financing mechanism for polio eradication; an award winning project fighting malaria and Ebola in Mali; a teacher training program that is transforming teaching and learning in Nepali classrooms; a project on obstetric fistula saving the lives of mothers and children in Nigeria; a program to invest in the writing skills of disadvantaged youth in the United States; a project to improve access to water and sanitation in Uganda; and a global network of Peace Centers and Peace Fellows to help promote peace.

Some of these programs and projects are large. Others are small. Most were implemented through global grants, but one was implemented through a district grant. All these projects have been in one way or another innovative. They have all leveraged partnerships not only to crowd in financial resources, but also – and even more importantly – to build on great expertise. And they have all relied on monitoring and evaluation mechanisms to assess their impact, at least partially.

Putting together great projects requires work. Fundraising is often time consuming in Rotary given the funding model of the Rotary Foundation that requires raising funds from many clubs and districts first before getting a match from the Foundation. Planning, implementing, and in addition evaluating projects also takes time, especially when one tries to do this in a professional way. Finally, in order to be innovative, Rotarians leading projects need to be aware of where the frontier is in their field, and what could be innovative. This also takes some time.

There is nothing wrong with clubs and districts funding and implementing traditional Rotary projects. Most projects will continue to be fairly simple, with funds provided to worthy charitable causes. These projects, as well as the volunteer time often contributed by Rotarians when implementing them, serve an important purpose. The beneficiaries of these projects are better off thanks to them. These projects help communities, and they also benefit Rotary through the goodwill that the projects create.

But if we want to raise the bar and achieve larger impact, we also need to do more innovative projects. Rotary needs to be bolder, more ambitious. It needs to better learn from its projects, both the great and not so great ones, and make sure that lessons learned are shared broadly, well beyond the Rotary family. The launch of the Future Vision model, despite some challenges, was a step in the right direction. As we celebrate the Centennial of the Rotary Foundation next year, let’s make sure that we have the right vision for what Rotary and its Foundation could accomplish in the next 100 years.

Improving Teaching and Learning in Nepal (Partnerships Series No. 8)

Many developing countries have made substantial progress towards improving education attainment (the level of education attained by students) over the last two decades. At the same time, the instruction provided by teachers to students often remains of limited quality. This results in less than stellar education achievement (how much students actually learn). While students may do well enough on portions of examinations that rely for the most part on memorization, they tend to do less well when asked to think creatively or solve complex problems. This post, which is part of a series on partnerships, innovation, and evaluation in Rotary, tells the story of an innovative teacher training program in Nepal that has the potential of improving student learning substantially.

NTTI

Importance of Teacher Training

Outstanding in-service teacher training programs can make a major difference in how teachers teach, how much students learn, and how much they learn, especially among disadvantaged groups. Many factors influence student achievement, including factors that are beyond the control of schools such as a student’s socio-economic context. But teachers are the most important factor under the control of education systems to improve learning. Teachers also account for the bulk of public spending for education in developed and developing countries alike. For these reasons, there is increasing interest in finding ways to attract, retain, develop, and motivate great teachers.

The tasks of attracting, retaining, and motivating teachers fall squarely within the scope of the mission of Ministries of Education. Developing teachers is also a key responsibility and priority for the Ministries, but in this area there is also scope for nonprofits and organizations such as Rotary to play a role by helping to create great in-service training programs for teachers. The importance of in-service training and professional development to improve instruction is recognized by practitioners and policy makers. Three lessons emerge from the literature.

First, opportunities for teacher training and professional development should be made available. But not all programs achieve the same results. When in-service programs focus on changing pedagogy, the evidence suggests that they can improve teaching and as a result student achievement. By contrast, programs that merely provide additional teaching materials for teachers do not generate substantial gains.

Second, the contents of training programs aiming to change pedagogy matter as well. In-service training programs that expose teachers to best practices in instruction and actually show teachers how to implement these practices are more likely to generate positive change. Promoting collaboration between teachers, among others through teacher networks where teachers can exchange ideas is useful. Mentoring programs whereby junior teachers benefit from the guidance of experience teachers also tend to be effective. Other approaches tend to be less successful.

Third, it is important that in-service training and professional development programs target in priority the teachers who need help the most. Teachers who are struggling may benefit more than already great teachers from various programs. Similarly, students from disadvantaged backgrounds or living in poor areas also tend to benefit more from a higher quality instruction than better off students who have more help from their family at home. Identifying priority pockets of needs is most beneficial when implementing and teacher training programs.

Innovative Program in Nepal

Traditional instruction in Nepal relies on lecturing by teachers and memorization by students. Together with the Nepali NGO PHASE, NTTI (Nepal Teacher Training Innovations) has implemented innovative teacher training programs in Nepal for several years.  NTTI aims to train public schools teachers to make the classroom more interactive by coaching them on how to lead classroom discussions, facilitate group work, and ask questions to students to encourage individual thought. Instead of relying on punishment and at times shaming in the classroom to control student behavior, teachers are trained to use dynamic inquiry-based instruction methods and provide positive encouragement to motivate the students to learn. As the classroom becomes more participatory, students engage in their own learning.

The PHASE-NTTI model does not rely on one-off training. Instead it involves a cumulative cycle of trainings and intensive follow-up support to individual teachers. The aim is to help teachers move from an awareness of effective teaching practices to actual implementation of the practices in their own classrooms. The training model includes a series of teacher development courses: Introduction to Best Teaching Practices; Girls’ Sensitivity Training; and a Training of Trainers for those selected as Mentor Teachers.

The model includes pre- and post-training classroom observations, individual feedback received by teachers from Master Trainers, and follow up individual teacher support by Mentor Overall, the program is implemented over a two-year period in each school.

While no impact evaluation is yet available to measure the impact of the program, quantitative data obtained through pre- and post-training classroom observation are encouraging. In contrast to teacher-driven and student-silent classrooms, classrooms with trained teachers seem to be closer to functioning as hubs of learning.

Instead of only lecturing trained teachers lead classroom discussions, facilitate group work, and ask questions to encourage individual thought. Students learn how to make their own novel connections and think critically about what they hear and read. Qualitative data suggest that the program is appreciated by teachers and students.

Remaining Challenges and Conclusion

There have been challenges to which the program has had to adapt. The program did not work as well in secondary schools, so it now focuses on primary schools. Support from principals for teachers changing their pedagogical approach is needed, but not guaranteed. Distances to schools in rural areas make it hard to maintain regular contact after initial trainings. Lack of time for teachers to prepare lessons as advocated by the program is also a constraint. The structure of classroom time may limit creativity and inquiry-based teaching. The persistence of traditions harmful to girls in parts of the country is a major challenge to keep girls in school.

The PHASE-NTTI program does not have all the answers to these challenges, but it does have the key features that tend to be associated with successful in-service training programs. The program is also a great example of partnership (with the Ministry of Education and public schools), innovation (in teacher training), and evaluation (at least through monitoring of teacher pedagogy). A Rotary global grant proposal has been submitted to help develop the PHASE-NTTI program further and implement it in additional areas.

Fighting Malaria and Ebola in Mali (Partnerships Series No. 7)

As part of this series of posts on increasing Rotary’s impact through partnerships, innovation, and evaluation, I had to include Muso, a nonprofit that has successfully fought malaria and Ebola in Mali, in part with support from Rotary. As some readers may remember it, I talked about Muso in this blog previously, and this post is based in large part on previous posts. But at the risk of repetition, it is worth emphasizing again that Muso is a great example of an innovative approach that has been evaluated and has the potential of being replicated at scale thanks to partnerships.

Muso2

A Successful Pilot

Muso started in Mali as a project to provide basic care to communities, focusing initially in part on malaria. Two years ago a Harvard/University of California San Francisco study documented a tenfold difference in child mortality rates after the rollout of the Muso model in the program’s catchment area. Muso relies on professionalized community health workers to diagnose illnesses quickly in its catchment area, refer patients to clinics and hospitals as needed, and reduce financial barriers to care for families.

One needs to be careful in inferring causality between the intervention and the reduction in child mortality since the evaluation was based on repeated cross-sectional data as opposed to a randomized control trial. Still, the impact appears to have been major and obtained at relatively low cost. The Muso model was recently recognized as a finalist for two major prizes – the GSK Global Healthcare Innovation Award and the Caplow Children’s Prize.

Support to the Ministry of Health

Some successful pilot interventions in health are implemented without ever being scaled up, so that their benefits for a country’s population as a whole may be limited. This is not the case for Muso. In addition to implementing and evaluating an innovative model, Muso worked closely from the start with the Malian Ministry of Health as well as other partners to explore opportunities to expand the model nationally.

In November last year, based in part on the Muso model, Mali’s Ministry of Health Division of Community Health Systems announced a strategic plan to scale up professionalized community health workers throughout the country. How did this transformation happen? As just mentioned, Muso and other partners have been actively working with the Ministry of Health for seven years. The partnership was launched in 2008. The collaboration has been not only operational, but also scientific, with support from university researchers. Though this partnership, Muso has been able to provide technical assistance to help the Ministry develop a national plan for scaling-up the community health worker model.

Factors contributed to the success of this partnership and to the scaling-up announcement according to Dr. Ari Johnson, the co-founder of Muso, who was interviewed last year. “First, we started the partnership early on. By setting an operational research partnership at the design phase of the project, we were able to ensure that the pilot would focus on the priorities of the Ministry of Health, which meant a focus on child survival and disease-specific targets, including early effective treatment for malaria.”

A second factor for success was the ability to work with other NGOs to test the robustness of the community health worker model in different parts of the country. “The Malian Ministry of Health worked simultaneously with several NGOs on operational research to test community health workers models. This included, but was not limited to Muso with the operational research study in Yirimadjo and Doctors without Borders (Médecins sans Frontières) with another study in Kanbaga. These experiences with multiple partners in several locations provided the Ministry of Health with converging evidence for scaling up paid, professionalized community health workers,” explains Dr. Johnson.

The third factor for success was the support of international organizations. Multilateral and bilateral global health institutions are all trying to strengthen health systems in Africa. In Mali, Dr. Johnson explains that UNICEF and the Global Fund provided important support for the adoption and expansion of the community health workers model by the Ministry of Health.

Finally Dr. Johnson insists that operational research partnerships must be long-term to succeed. “Longitudinal operational research partnerships take time to implement, but over time, they help build relationships between public sector policy makers and hubs of research. These relationships become avenues for translating research into evidence-based policy change at scale.” Dr. Johnson adds that “the long view is critical, and often neglected in global health work. Short-term funding cycles push organizations to move on after a few years and abandon the foundations of a strong partnership. A long term partnership for iterative and ongoing research is crucial to support Ministries in their strategic plans and thereby achieve scale.”

Many questions remain. What should be the health care financing system for community health workers? How can those workers be deployed across both urban and rural areas? How is the supervision of the workers to be worked out? These and other questions will need to be answered. But progress is being made.

Relevance for Ebola

How does all this relate to the Ebola crisis that recently hit West African countries? Apart from their role in preventing and treating malaria as well as other common illnesses, community health workers can be essential in the fight against Ebola. Ebola arrived in Mali through a two-year-old girl who had traveled with her grandmother from Guinea died.  Mali became the sixth West African country with a confirmed Ebola case.

Why are community health workers so important for the fight against Ebola? They are crucial in part because they tend to be trusted members of their communities. They can not only help in providing information about Ebola and promoting appropriate behaviors, but they can also help to trace and monitor those who have been in contact with the virus. This must be done for at least 21 days – the period during which symptoms do not yet emerge, and it must be continued after that period if individuals become sick. In the other countries affected by the virus – including Guinea, Liberia, and Sierra Leone, community health workers already play that vital role.

As Dr, Johnson explained it, “there is huge potential for community health workers to accelerate the effort to stop Ebola across West Africa, by supporting epidemiologic surveillance, contact monitoring, returning traveler monitoring, community engagement, and prevention counseling.”

Conclusion

Muso has piloted an innovative new model of health care delivery that appears to have contributed to reducing child mortality in its area of intervention. The model has also proved valuable in fighting the Ebola epidemics. Rotary provided crucial support to Muso when the NGO was still small and not as well-known as it is today, with fewer resources. Rotarians and Rotaractors have volunteered with Muso in Mali, and others have contributed to making the project better known internationally.

Today, Muso is scaling up, aiming to raise substantial funds to expand its program. It is also launching a rigorous impact evaluation through a randomized controlled trial to measure its effectiveness. While many organizations and individuals have contributed to Muso’s success, especially the Muso team working on the ground, at times taking substantial risks to help the population as was the case during the Ebola epidemics, Rotary and Rotarians have played a small supporting role as well.

Investing in Disadvantaged Youth in the United States (Partnerships Series No. 6)

Growing local economies requires many different ingredients, but one of the most important ones is a skilled workforce, especially among youth. Skills tend to be acquired through the education system. As part of a series on increasing Rotary’s impact through partnerships, innovation, and evaluation, this brief tells the story of an innovative program in Washington, DC that is improving writing skills for high school seniors in public schools and preparing them for college in part with support from Rotary.

A student presents his papers at One World Education's Fair
A student presents his papers at One World Education’s Fair

The United States benefitted for decades from one of the most skilled workforce in the world, but there are concerns that this is not the case anymore. Within the US, the District of Columbia has been struggling and often ranks at the bottom of the National Assessment of Educational Progress league tables. There are many reasons for the poor performance of the District. In spite of major improvements in economic development in the last decade, a substantial share of its population remains poor, and poverty is one of the main drivers of poor performance in school. But some programs are helping.

One World Education

One World Education (OWEd) trains teachers and helps students improve their writing skills, and think about their college options at the same time. OWEd was created in 2006 by two teachers, Eric Goldstein and Emily Chiariello, who taught at one of the charter schools in Washington, DC. Their idea was to use students’ reflective writing as the foundation for what was discussed in the classroom. The model proved successful as students became more engaged and, in turn, started to develop better research, writing, and analytical thinking skills. The non-profit was launched in 2007 and has been growing.  OWEd recently signed an agreement with DCPS (District of Columbia Public Schools) to expand its programs in all public high schools in the city. As a result, OWEd has become the largest nonprofit program operating in the District’s public schools.

The program focuses on writing skills and is adapted to various grades. For example, the Grade 12 program helps students analyze, research, write argumentative essays, and lead presentations about the college and career issues that await them after graduation. It includes a comprehensive seven week coaching period. Essays written by students can serve as their Senior Project. Selected student essays are published on OWEd’s website, providing recognition for students and creating a cycle of peer-to-peer learning.

More generally, for all grades where the program is implemented (Grades 8, 10, and 12), students and teachers can access a number of resources provided by OWED, including the following:

  1. Common Core Aligned Lesson Plans: All lesson plans are created by teachers, for teachers, and are aligned to multiple research, writing, and presentation Common Core State Standards. Lessons are accompanied by rubrics for teacher evaluations and peer-to-peer reviews;
  2. Student Writer’s Notebook: the notebook leads students to analyze exemplary, peer-authored essays before guiding them through researching, outlining, drafting, and revising their own argumentative essays.
  3. Student and Educator Portals: Students and teachers will have access to easy-to-access lesson plans, rubrics, research sources, and related resources for teachers and students are available online.

Evaluating Program Impacts

Randomized controlled trials have not yet been implemented to assess the impact of the programs run by One World Education, but other data suggests that the program is having an impact. Specifically, evaluations by students and faculty at American University and George Washington University suggest gains in writing quality and self-confidence for students that have participated in OWEd’s programs.

In order to assess gains in the quality of the writing of participants, a sample of students participating in the program take a writing test before the start of the program and at the end of the program. The test is graded by university professors. Results suggest important gains after program participation.

Feedback from teachers – and more importantly students who have participated in the program is positive. For example, in the 2014 DCPS Grade 10 evaluation by students, participants reported improvements in terms of their ability to make a claim (87 percent); Provide research to support a claim (87 percent); Write (85 percent); Research information (84 percent); Analyze research (84 percent); Create an outline (79 percent); Create a draft (78 percent); Establish a research plan (75 percent); and Revise their essay (75 percent).

These and other positive evaluations of the program in partnership with two local universities have been a key factor in the agreement reached by OWEd with DCPS to substantially expand the program in grades 9, 10, and 12. All public high school students in the District in those grades will now have the opportunity to participate in the program.

How Has Rotary Helped?

Rotarians from the Rotary Club of Capitol Hill in Washington, DC, have supported the project in various ways. The club has donated funds to, and volunteered with, OWEd for several years. In 2015-16 the club’s donation will be matched with a district grant using so-called district designated funds from the Rotary Foundation.

Each year student essays are assessed by a panel of judges at a College and Career Writer’s Challenge each year. This enables students to learn how to make an argumentative pitch to a panel.  One student from each school is eligible to earn a college or vocational training scholarship, and every participating school can nominate a number of seniors to participate in the event. Rotary club and district grants will allow OWEd to provide small scholarships for college to 10 students who have written especially good essays thanks to the program.

In addition, Rotarians have participated in OWEd’s programs in a number of volunteering capacities, including as judges for the essay competitions taking place at the College and Career Writer’s Challenge.

Conclusion

In supporting OWEd, Rotary builds on the benefits from partnerships, innovation, and evaluation. OWEd itself has partnered with District of Columbia Public Schools to substantially expand the reach of its program. The program is innovative in the way writing skills for students are being developed using a range of different resources and mechanisms. Evaluations of OWEd’s programs have shown that the programs generate measurable gains in middle and high school students’ writing skills, and in their self-confidence. The program not only improved the student’s writing, but it also helps in preparing them for college and career-level writing.

For Rotarians, OWEd’s programs have also offered unique opportunities to personally support students from disadvantaged backgrounds by contributing in the programs in various ways. This had been done through donations, but also through volunteering.