Evaluation is essential to assess what works and share stronger stories

Readers of this blog know that I have emphasized for some time the need to strengthen a culture of evaluation in Rotary. Evaluations should be undertaken not only for our service projects, but also to assess how our clubs meet, work, and grow – or wither away. This post is about a recent evaluation of an education project supported by my club, and how the evaluation is proving to be useful not only for the local nonprofit we worked with, but also for our club and more generally for practitioners and policy makers working in the field of education.

OWED celebration
Photo of OWEd scholarship winners with Brian Pick, Chief of Teaching and Learning for the District of Columbia Public Schools, and Dave Paris, member of the Board of OWEd.

For several years my club has supported One World Education (OWEd), a great nonprofit based in Washington, DC. OWEd runs the largest argumentative writing program in public and charter schools in the city. The nonprofit reached 5,800 middle and high school students this past school year. The aim of the program, which runs for 4-5 weeks in the schools, is to improve the research, writing, and presentation skills of the students, many of whom are from disadvantaged backgrounds and do not do very well in school.

In previous years, our support to OWEd consisted in providing a bit of funding and volunteering at some of their events. This year, we provided college scholarships for some of the high school students (seniors) who participated in the program and worked especially hard. But we also did more. Together with a team at American University, we designed an evaluation of the program to better measure its impact. For more than 550 students, teachers collected essays written in class before and after the program. The essays were graded by professors and instructors in the Department of Literature at American University. This enabled us to assess whether the program made a difference in the writing skills of middle and high school students.

The evaluation demonstrated that the program has a positive impact. The program generates statistically significant gains in writing quality, especially for students who performed worst on the initial pre-program assignment.  The positive impact of the program was confirmed through data on the perceptions of teachers and students about the program. Two summary briefs about those evaluation results have been written and are now available for public schools and for charter schools separately.

It is clear that this type of evaluation is beneficial for the nonprofits whose programs are evaluated, as the evaluations enable the nonprofits to measure their impact, and take corrective action when needed.  The evaluations are also beneficial for our club in reassuring members that we are investing in worthwhile initiatives.

But there is more. Many others are interested in such evaluations and may learn from them, possibly generating larger impacts beyond the specific programs being evaluated. And these evaluations provide for great stories to be featured in local newspapers or magazines as well as social media, giving more visibility not only to the nonprofits and programs being evaluated, but also to the Rotary clubs that supported those evaluations.

This is what we are focusing on now – making sure that the positive results obtained by OWEd through its program are better known in Washington, DC, and beyond. We are writing short articles that document those results, and some of the stories of the students who benefited from the program.  We have secured already two placements for stories in the local media and we hope to write additional articles for national publications about the results of the evaluation. In addition, we will also prepare technical papers for academic journals. It remains to be seen whether we will be successful, but we now have a stronger story to tell thanks to the evaluation.

Finally, as mentioned, the evaluation has been summarized in two easy-to-read briefs. The two briefs, together with briefs about the work of other nonprofits operating in the field of education and skills for youth in the city, will be included in a small brief series on innovations in education in Washington, DC to be published by the World Bank. We hope that this simple brief series will help attract attention to the nonprofits doing great work in the city, while also helping practitioners and policy makers learn from the experience of successful programs.

In summary, evaluation is essential not only to help improve service projects, whether implemented by Rotary clubs or nonprofits, but also to tell stronger stories about ways to improve the lives of the less fortunate. Investing more in evaluation seems to be a win-win for nonprofits as well as service clubs.  And for Rotary as a whole, as I mentioned it in a previous series of posts on this blog, focusing more on partnerships, innovation, and evaluation seems key to achieve larger impacts.

Improving Teaching and Learning in Nepal (Partnerships Series No. 8)

Many developing countries have made substantial progress towards improving education attainment (the level of education attained by students) over the last two decades. At the same time, the instruction provided by teachers to students often remains of limited quality. This results in less than stellar education achievement (how much students actually learn). While students may do well enough on portions of examinations that rely for the most part on memorization, they tend to do less well when asked to think creatively or solve complex problems. This post, which is part of a series on partnerships, innovation, and evaluation in Rotary, tells the story of an innovative teacher training program in Nepal that has the potential of improving student learning substantially.

NTTI

Importance of Teacher Training

Outstanding in-service teacher training programs can make a major difference in how teachers teach, how much students learn, and how much they learn, especially among disadvantaged groups. Many factors influence student achievement, including factors that are beyond the control of schools such as a student’s socio-economic context. But teachers are the most important factor under the control of education systems to improve learning. Teachers also account for the bulk of public spending for education in developed and developing countries alike. For these reasons, there is increasing interest in finding ways to attract, retain, develop, and motivate great teachers.

The tasks of attracting, retaining, and motivating teachers fall squarely within the scope of the mission of Ministries of Education. Developing teachers is also a key responsibility and priority for the Ministries, but in this area there is also scope for nonprofits and organizations such as Rotary to play a role by helping to create great in-service training programs for teachers. The importance of in-service training and professional development to improve instruction is recognized by practitioners and policy makers. Three lessons emerge from the literature.

First, opportunities for teacher training and professional development should be made available. But not all programs achieve the same results. When in-service programs focus on changing pedagogy, the evidence suggests that they can improve teaching and as a result student achievement. By contrast, programs that merely provide additional teaching materials for teachers do not generate substantial gains.

Second, the contents of training programs aiming to change pedagogy matter as well. In-service training programs that expose teachers to best practices in instruction and actually show teachers how to implement these practices are more likely to generate positive change. Promoting collaboration between teachers, among others through teacher networks where teachers can exchange ideas is useful. Mentoring programs whereby junior teachers benefit from the guidance of experience teachers also tend to be effective. Other approaches tend to be less successful.

Third, it is important that in-service training and professional development programs target in priority the teachers who need help the most. Teachers who are struggling may benefit more than already great teachers from various programs. Similarly, students from disadvantaged backgrounds or living in poor areas also tend to benefit more from a higher quality instruction than better off students who have more help from their family at home. Identifying priority pockets of needs is most beneficial when implementing and teacher training programs.

Innovative Program in Nepal

Traditional instruction in Nepal relies on lecturing by teachers and memorization by students. Together with the Nepali NGO PHASE, NTTI (Nepal Teacher Training Innovations) has implemented innovative teacher training programs in Nepal for several years.  NTTI aims to train public schools teachers to make the classroom more interactive by coaching them on how to lead classroom discussions, facilitate group work, and ask questions to students to encourage individual thought. Instead of relying on punishment and at times shaming in the classroom to control student behavior, teachers are trained to use dynamic inquiry-based instruction methods and provide positive encouragement to motivate the students to learn. As the classroom becomes more participatory, students engage in their own learning.

The PHASE-NTTI model does not rely on one-off training. Instead it involves a cumulative cycle of trainings and intensive follow-up support to individual teachers. The aim is to help teachers move from an awareness of effective teaching practices to actual implementation of the practices in their own classrooms. The training model includes a series of teacher development courses: Introduction to Best Teaching Practices; Girls’ Sensitivity Training; and a Training of Trainers for those selected as Mentor Teachers.

The model includes pre- and post-training classroom observations, individual feedback received by teachers from Master Trainers, and follow up individual teacher support by Mentor Overall, the program is implemented over a two-year period in each school.

While no impact evaluation is yet available to measure the impact of the program, quantitative data obtained through pre- and post-training classroom observation are encouraging. In contrast to teacher-driven and student-silent classrooms, classrooms with trained teachers seem to be closer to functioning as hubs of learning.

Instead of only lecturing trained teachers lead classroom discussions, facilitate group work, and ask questions to encourage individual thought. Students learn how to make their own novel connections and think critically about what they hear and read. Qualitative data suggest that the program is appreciated by teachers and students.

Remaining Challenges and Conclusion

There have been challenges to which the program has had to adapt. The program did not work as well in secondary schools, so it now focuses on primary schools. Support from principals for teachers changing their pedagogical approach is needed, but not guaranteed. Distances to schools in rural areas make it hard to maintain regular contact after initial trainings. Lack of time for teachers to prepare lessons as advocated by the program is also a constraint. The structure of classroom time may limit creativity and inquiry-based teaching. The persistence of traditions harmful to girls in parts of the country is a major challenge to keep girls in school.

The PHASE-NTTI program does not have all the answers to these challenges, but it does have the key features that tend to be associated with successful in-service training programs. The program is also a great example of partnership (with the Ministry of Education and public schools), innovation (in teacher training), and evaluation (at least through monitoring of teacher pedagogy). A Rotary global grant proposal has been submitted to help develop the PHASE-NTTI program further and implement it in additional areas.

Impact Evaluations, Part 1: Do We Need Them in Rotary?

by Quentin Wodon

Monitoring and evaluation have become essential in development and service work. When providing funding, most foundations and donors now require some type of monitoring and evaluation either ex ante to allocate funding, or ex post to assess whether projects that have been funded have proved to be successful or not. The same is true for government agencies – when deciding whether to scale up a pilot intervention, most agencies now typically rely on the results of at least some type of evaluation.

Different types of evaluations can be conducted and not all are equal in terms of what we can learn from them. In this series of three posts, I will focus on impact evaluations (as opposed to process evaluations). Specifically, I will discuss 1) whether we need impact evaluations in Rotary; 2) how impact evaluations can be implemented in practice; and 3) what are some of the limits of impact evaluations that practitioners and policy makers should be aware of. But before doing so, it is probably useful to briefly explain what an impact evaluation entails.

What Is An Impact Evaluation?

Impact evaluations aim to measure the impact of specific projects, policies, or interventions on specific outcomes. The question they ask is not whether a given outcome has been achieved or not among a target population. They look instead at the specific contribution or impact of a well-defined intervention on a well-defined outcome. Said differently, they try to measure a counterfactual: what would have been the outcome without the intervention? By comparing the assessment of what the outcome would have been without the intervention and the outcome with the intervention, impact evaluations inform us about the success (or lack thereof) of specific interventions in improving specific outcomes. When data on the cost of alternative interventions and their benefits are available, impact evaluations are especially helpful in deciding whether some interventions should be maintained or scaled up, while others should be abandoned.

Estimating the counterfactual in an impact evaluation is no simple matter, in part because most interventions are not distributed randomly in a target population. Consider children going to private or charter schools. In most countries, those children perform better than children attending public schools. Does that in itself tell us anything about the comparative performance of public versus private or charter schools? It does not.

Children attending private or charter schools may for example come from wealthier families with well educated parents who are better able to help their children at home than parents from less privileged backgrounds. The higher performance of students in private or charter schools on tests aiming to measure learning may not be related to the school in which they are enrolled per se, but instead to their socio-economic and other characteristics. The proper counterfactual question would be: how well would students enrolled today in private or charter schools perform if they had enrolled instead in public schools? Data to answer such questions are often not readily available, so the counterfactual is often not known without some further data or analysis. Special techniques – ranging from randomized control trials (the gold standard) to various statistical and econometric approaches, are often needed in order to assess the impact of specific interventions on specific outcomes. I will discuss these techniques in the second post in this series.

Do We Need Impact Evaluations in Rotary?

If as Rotarians we “do good work” in the world, why would we need impact evaluations? Isn’t it enough to serve the less fortunate? I would argue that Rotary needs impact evaluations for at least three reasons.

  1. Doing good work is not enough – we need to do the best we can. We live in a world with scarce resources. What Rotarians can contribute – whether in terms of money or time and expertise – is finite. Our resources should be devoted to projects and initiatives that have the largest positive impact on those we aim to serve. And we will be able to assess what these projects or interventions are or should be only by evaluating our work (and relying as well on the evidence generated by other organizations that are implementing robust impact evaluations.)
  2. Who says we are doing good work? We may believe we are doing good work, but do we know whether some of our projects and interventions may have unintended consequences that could be detrimental to those we are trying to serve? To consider the example of education again, if we support one dynamic school in a poor community that selects its students on the basis of their academic potential, could this have a negative effect on another school in the area which might be left only with children who tend to perform less well for whatever reason. Given that peer effects are important drivers of learning, supporting one school may have negative consequences on another school. This is of course not necessarily the case, and this does not imply  that we should never support specific schools. But we need to be aware of the potential ripple effects that our projects may have, and impact evaluation can be useful for that.
  3. Rotary has an important role to play in piloting innovative interventions. Much of what Rotary clubs do on an on-going basis is not innovative, and there is nothing wrong with that. There are a lot of needs out there that we can contribute to meet without being innovative. Being innovative is not a requirement. But at the same time, we should promote some level of innovation in Rotary. Rotarians have technical expertise in many areas and they know (or should know) their community. They are in a way uniquely positioned to propose new ways of tackling old problems, and assessing whether such new ideas actually work. Furthermore, Rotary has a relatively large foundation, but in comparison to other actors, especially in the field of development, we are small. If we could pilot more innovative interventions, evaluate them properly, and let others with more resources scale up the most promising interventions, we could achieve more for those we aim to serve.

For these reasons I believe that we need impact evaluations in Rotary. Not all projects require an evaluation, especially since evaluations take some time and cost money. But we can probably do more in this area than we are doing today. In the next post in this series, I will discuss the “how to” of impact evaluations.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.

Public, Private Secular, and Faith-inspired Schools: How Do They Differ?

Private secular schools are also growing, albeit for different reasons related more directly to the perceived quality of the education they provide. Both types of private schools can usefully complement public schools and provide choice to families. And yet, relatively little systematic evidence is available on the market share, reach to the poor, cost for families, and quality of both types of schools.Two recent World Bank studies help bridge that gap in knowledge. The first study provides a comparative assessment of the role of public, private secular, and faith-inspired schools in sub-Saharan Africa. It is based on a systematic analysis of nationally representative household surveys for 16 countries that identify separately public, private secular, and faith-inspired schools, as well as in-depth qualitative data collection in two countries (Ghana and Burkina Faso).The second study focuses on the experience of the Jesuit-run Fe y Alegría network of schools in Latin America. That study combines articles that assess (quantitatively) the performance of the network’s schools with other articles looking at the schools’ pedagogical practices.

The first study on sub-Saharan Africa suggests that the market share of faith-inspired and private secular schools is smaller than often claimed, but still substantial. Across the 16 countries, the average market share for faith-inspired schools is at 14% at the primary level and 11% at the secondary level. The corresponding market shares for private secular schools are 12% and 16%. These estimates from household surveys are of an order of magnitude similar to the statistics on the market share of the private sector as a whole, reported by the UNESCO Institute of Statistics on the basis of administrative data provided by government.

The household surveys also suggest that public schools reach the poor more than faith-inspired schools, while private secular schools are the least ‘pro-poor’. This is not surprising given that the cost for households of private secular schools is higher than that of faith-inspired schools, with public schools being the most affordable. The fact that faith-inspired schools do not reach the poor in priority, statistically speaking, does not mean that they do not aim to do so. But they operate within tight budget constraints, especially when support from governments is limited, and this requires some level of cost recovery from households. Some private secular schools also aim to reach the poor, but this tends to not to be a widely shared feature of these schools.

Finally, and most importantly, the data suggest that both faith-inspired and private secular schools have higher satisfaction rates among parents than public schools, with private secular schools performing best according to that metrics.

Tentative evidence also suggests that student performance in terms of literacy and numeracy, as well as test scores, is also often (but not always) higher among faith-inspired and private secular schools than in public schools. In the case of faith-inspired schools, qualitative evidence suggests that the emphasis placed on religion for Islamic schools and values more than religion per se in Christian schools is appreciated by parents. In other words, these schools are highly valued and could probably be better supported by governments and donors provided that they also teach secular topics, which is often the case.

Why is the quality of education provided apparently higher in faith-inspired and private schools than in public schools?

Part of the explanation could come from teacher motivation. For example, teachers in faith-inspired schools might be “working for God” (to use the title of an article on faith-inspired health care by Reinikka and Svensson). If only motivation were at play, the ability to scale up those schools, or replicate their success in other schools might be limited, as reservoirs of highly motivated teachers are probably finite.

But while motivation plays a role, the management and teacher training systems that well performing faith-inspired and private secular schools have managed to put in place also matter greatly. This is clearly apparent in the second study on Fe y Alegría schools in Latin America. The management teams of the schools spend significant time and have succeeded in fostering a culture of accountability and dedication in the school that helps to place service to students at the very core of the teacher’s mission.

Overall, the message from both studies is that while it is not necessarily easy, fostering a culture of great service to students can be achieved, and the experience of faith-inspired and private secular schools is useful to better understand how it can be done.

The message is not necessarily to scale up provision of education by faith-inspired and private secular schools, but rather and more importantly to support existing schools that provide a great education to children. This is especially important when schools also aim to reach the poor. We should learn from their successes and see how they might be replicated in other types of schools.

Note: this post is reproduced with minor changes from a post on the Education for Global Development blog of the World Bank available at http://blogs.worldbank.org/education/