Evaluation is essential to assess what works and share stronger stories

Readers of this blog know that I have emphasized for some time the need to strengthen a culture of evaluation in Rotary. Evaluations should be undertaken not only for our service projects, but also to assess how our clubs meet, work, and grow – or wither away. This post is about a recent evaluation of an education project supported by my club, and how the evaluation is proving to be useful not only for the local nonprofit we worked with, but also for our club and more generally for practitioners and policy makers working in the field of education.

OWED celebration
Photo of OWEd scholarship winners with Brian Pick, Chief of Teaching and Learning for the District of Columbia Public Schools, and Dave Paris, member of the Board of OWEd.

For several years my club has supported One World Education (OWEd), a great nonprofit based in Washington, DC. OWEd runs the largest argumentative writing program in public and charter schools in the city. The nonprofit reached 5,800 middle and high school students this past school year. The aim of the program, which runs for 4-5 weeks in the schools, is to improve the research, writing, and presentation skills of the students, many of whom are from disadvantaged backgrounds and do not do very well in school.

In previous years, our support to OWEd consisted in providing a bit of funding and volunteering at some of their events. This year, we provided college scholarships for some of the high school students (seniors) who participated in the program and worked especially hard. But we also did more. Together with a team at American University, we designed an evaluation of the program to better measure its impact. For more than 550 students, teachers collected essays written in class before and after the program. The essays were graded by professors and instructors in the Department of Literature at American University. This enabled us to assess whether the program made a difference in the writing skills of middle and high school students.

The evaluation demonstrated that the program has a positive impact. The program generates statistically significant gains in writing quality, especially for students who performed worst on the initial pre-program assignment.  The positive impact of the program was confirmed through data on the perceptions of teachers and students about the program. Two summary briefs about those evaluation results have been written and are now available for public schools and for charter schools separately.

It is clear that this type of evaluation is beneficial for the nonprofits whose programs are evaluated, as the evaluations enable the nonprofits to measure their impact, and take corrective action when needed.  The evaluations are also beneficial for our club in reassuring members that we are investing in worthwhile initiatives.

But there is more. Many others are interested in such evaluations and may learn from them, possibly generating larger impacts beyond the specific programs being evaluated. And these evaluations provide for great stories to be featured in local newspapers or magazines as well as social media, giving more visibility not only to the nonprofits and programs being evaluated, but also to the Rotary clubs that supported those evaluations.

This is what we are focusing on now – making sure that the positive results obtained by OWEd through its program are better known in Washington, DC, and beyond. We are writing short articles that document those results, and some of the stories of the students who benefited from the program.  We have secured already two placements for stories in the local media and we hope to write additional articles for national publications about the results of the evaluation. In addition, we will also prepare technical papers for academic journals. It remains to be seen whether we will be successful, but we now have a stronger story to tell thanks to the evaluation.

Finally, as mentioned, the evaluation has been summarized in two easy-to-read briefs. The two briefs, together with briefs about the work of other nonprofits operating in the field of education and skills for youth in the city, will be included in a small brief series on innovations in education in Washington, DC to be published by the World Bank. We hope that this simple brief series will help attract attention to the nonprofits doing great work in the city, while also helping practitioners and policy makers learn from the experience of successful programs.

In summary, evaluation is essential not only to help improve service projects, whether implemented by Rotary clubs or nonprofits, but also to tell stronger stories about ways to improve the lives of the less fortunate. Investing more in evaluation seems to be a win-win for nonprofits as well as service clubs.  And for Rotary as a whole, as I mentioned it in a previous series of posts on this blog, focusing more on partnerships, innovation, and evaluation seems key to achieve larger impacts.

Investing in Disadvantaged Youth in the United States (Partnerships Series No. 6)

Growing local economies requires many different ingredients, but one of the most important ones is a skilled workforce, especially among youth. Skills tend to be acquired through the education system. As part of a series on increasing Rotary’s impact through partnerships, innovation, and evaluation, this brief tells the story of an innovative program in Washington, DC that is improving writing skills for high school seniors in public schools and preparing them for college in part with support from Rotary.

A student presents his papers at One World Education's Fair
A student presents his papers at One World Education’s Fair

The United States benefitted for decades from one of the most skilled workforce in the world, but there are concerns that this is not the case anymore. Within the US, the District of Columbia has been struggling and often ranks at the bottom of the National Assessment of Educational Progress league tables. There are many reasons for the poor performance of the District. In spite of major improvements in economic development in the last decade, a substantial share of its population remains poor, and poverty is one of the main drivers of poor performance in school. But some programs are helping.

One World Education

One World Education (OWEd) trains teachers and helps students improve their writing skills, and think about their college options at the same time. OWEd was created in 2006 by two teachers, Eric Goldstein and Emily Chiariello, who taught at one of the charter schools in Washington, DC. Their idea was to use students’ reflective writing as the foundation for what was discussed in the classroom. The model proved successful as students became more engaged and, in turn, started to develop better research, writing, and analytical thinking skills. The non-profit was launched in 2007 and has been growing.  OWEd recently signed an agreement with DCPS (District of Columbia Public Schools) to expand its programs in all public high schools in the city. As a result, OWEd has become the largest nonprofit program operating in the District’s public schools.

The program focuses on writing skills and is adapted to various grades. For example, the Grade 12 program helps students analyze, research, write argumentative essays, and lead presentations about the college and career issues that await them after graduation. It includes a comprehensive seven week coaching period. Essays written by students can serve as their Senior Project. Selected student essays are published on OWEd’s website, providing recognition for students and creating a cycle of peer-to-peer learning.

More generally, for all grades where the program is implemented (Grades 8, 10, and 12), students and teachers can access a number of resources provided by OWED, including the following:

  1. Common Core Aligned Lesson Plans: All lesson plans are created by teachers, for teachers, and are aligned to multiple research, writing, and presentation Common Core State Standards. Lessons are accompanied by rubrics for teacher evaluations and peer-to-peer reviews;
  2. Student Writer’s Notebook: the notebook leads students to analyze exemplary, peer-authored essays before guiding them through researching, outlining, drafting, and revising their own argumentative essays.
  3. Student and Educator Portals: Students and teachers will have access to easy-to-access lesson plans, rubrics, research sources, and related resources for teachers and students are available online.

Evaluating Program Impacts

Randomized controlled trials have not yet been implemented to assess the impact of the programs run by One World Education, but other data suggests that the program is having an impact. Specifically, evaluations by students and faculty at American University and George Washington University suggest gains in writing quality and self-confidence for students that have participated in OWEd’s programs.

In order to assess gains in the quality of the writing of participants, a sample of students participating in the program take a writing test before the start of the program and at the end of the program. The test is graded by university professors. Results suggest important gains after program participation.

Feedback from teachers – and more importantly students who have participated in the program is positive. For example, in the 2014 DCPS Grade 10 evaluation by students, participants reported improvements in terms of their ability to make a claim (87 percent); Provide research to support a claim (87 percent); Write (85 percent); Research information (84 percent); Analyze research (84 percent); Create an outline (79 percent); Create a draft (78 percent); Establish a research plan (75 percent); and Revise their essay (75 percent).

These and other positive evaluations of the program in partnership with two local universities have been a key factor in the agreement reached by OWEd with DCPS to substantially expand the program in grades 9, 10, and 12. All public high school students in the District in those grades will now have the opportunity to participate in the program.

How Has Rotary Helped?

Rotarians from the Rotary Club of Capitol Hill in Washington, DC, have supported the project in various ways. The club has donated funds to, and volunteered with, OWEd for several years. In 2015-16 the club’s donation will be matched with a district grant using so-called district designated funds from the Rotary Foundation.

Each year student essays are assessed by a panel of judges at a College and Career Writer’s Challenge each year. This enables students to learn how to make an argumentative pitch to a panel.  One student from each school is eligible to earn a college or vocational training scholarship, and every participating school can nominate a number of seniors to participate in the event. Rotary club and district grants will allow OWEd to provide small scholarships for college to 10 students who have written especially good essays thanks to the program.

In addition, Rotarians have participated in OWEd’s programs in a number of volunteering capacities, including as judges for the essay competitions taking place at the College and Career Writer’s Challenge.

Conclusion

In supporting OWEd, Rotary builds on the benefits from partnerships, innovation, and evaluation. OWEd itself has partnered with District of Columbia Public Schools to substantially expand the reach of its program. The program is innovative in the way writing skills for students are being developed using a range of different resources and mechanisms. Evaluations of OWEd’s programs have shown that the programs generate measurable gains in middle and high school students’ writing skills, and in their self-confidence. The program not only improved the student’s writing, but it also helps in preparing them for college and career-level writing.

For Rotarians, OWEd’s programs have also offered unique opportunities to personally support students from disadvantaged backgrounds by contributing in the programs in various ways. This had been done through donations, but also through volunteering.

Promoting Peace with Peace Centers and Fellows (Partnerships Series No. 5)

In terms of global grants, promoting peace is one of the smallest portfolios among the areas of focus of the Rotary Foundation. But this does not mean that examples of partnerships, innovation, and evaluation cannot be found in the peace portfolio of the Foundation. The largest program for promoting peace that Rotary invests in is actually managed outside of the global grant model. Rotary provides funding for six Peace Centers established in universities around the world as well as Peace Fellow scholarships for individuals to obtain Master’s degree or Certificate program at the Peace Centers.

Chulalongkorn University in Thailand
Chulalongkorn University in Thailand

The Peace Fellows program is good an example of partnership (with universities), with components that are innovative (especially the Certificate program for professionals working in the area of peace), and for which at least some monitoring and evaluation data have been collected by Rotary through tracer studies of graduates of the program as well as assessments of the perceived quality of events organized for Rotary’s Peace Community of Practice.

Peace Centers and Peace Fellows

Up to 100 Peace Fellows are provided with a Rotary scholarship each year among a pool of applicants recommended by Rotary clubs and districts. Rotary provides funding for the scholarships given to the Peace Fellows as well as part of the operating costs of six Peace Centers at which the Peace Fellows undertake their training.

Five of the six Peace Centers and associated universities offer Master’s degrees, with up to 50 Peace Fellows selected each year.  These Peace Centers are affiliated with Duke University and the University of North Carolina at Chapel Hill in the United States (joint Center), International Christian University in Japan, the University of Bradford in England, the University of Queensland in Australia, and Uppsala University in Sweden. The fellowships are for Master’s programs that take 15 to 24 months to complete and include a practical internship of two to three months during the summer break.  The sixth Peace Center is affiliated with Chulalongkorn University in Thailand. It offers a three months Certificate program for up to 50 Fellows per year.

A Different Model for Scholarships

Rotary has a long tradition of providing scholarships for graduate students, but the Certificate program at Chulalongkorn University is different. It is cheaper per person than the master’s degree program, and probably better targeted to individuals committed to work on peace and conflict resolution since it serves practicing professionals. The certificate takes eleven weeks to complete including two to three weeks of field study. The program aims to provide Fellows with a comprehensive overview of peace and conflict studies with four modules of study: (1) Concepts and Values of Peace and Conflict Studies (introduction to the field); (2) Diagnosis and Analysis of Conflict (assessment of conflict and peace interventions); (3) Conflict Resolution Skills, Approaches, and Strategies (including negotiation, mediation, facilitation, and communication); and (4) Conflict Transformation and Building a Sustainable Peace (ways to move from conflict to peace with proper stakeholder participation in society). Two practical field studies experiences are included in the program, one after the third module in Thailand, and an international field study at the end of the fourth module. The program relies in part on guest lecturers with governmental, NGO, corporate, and security backgrounds.

Tracer Studies

In-depth evaluation of the Peace Fellows program have not yet been conducted, but results from tracer studies among graduates suggest a high level of satisfaction with the program among graduates. In addition, the tracer studies suggest that most graduates appear to be indeed working on peace and conflict resolution broadly defined.

Since the first class of peace fellows graduated in 2004, a total of 930 living alumni have graduated from the program, 603 with a Master’s degree and 333 with a Certificate (six have completed both). Virtually all Fellows (94 percent) have reported their post-graduation area of employment to Rotary through tracer studies at least once, and nearly two thirds (62 percent) have done so over the last 24 months.

Interesting findings emerge from the tracer studies. At least two thirds of graduates work as practitioners in peace, conflict resolution and development. This includes working for NGOs or other peace-related organizations (36 percent), a government agency or the military (15 percent), a United Nations agency (six percent), police or law enforcement agencies (three percent), and the World Bank (one percent). One fourth of the Fellows engage in research, teaching, or further study (eight percent each as teachers/professors, students, and researchers/academic support staff). The rest are working as lawyers (three percent), journalists (two percent), and as other professionals (seven percent). Four percent are looking for work.

Program alumni work and live all around the world, including in North America (30 percent), Asia (22 percent), Europe (15 percent), Africa (11 percent), Australia and Oceania (nine percent), South America (seven percent), the Middle East (four percent) and Central America and the Caribbean (two percent). This provides a potentially impactful worldwide network or community of practice of individuals committed to peace and conflict resolution. The question, then, is how to mobilize this network, including in collaboration with Rotary and Rotarians.

Building a Community of Practice

Rotary is investing in building a community of practice among Peace Fellows and Rotarians interested in promoting peace. One tool is the Rotarian Action Group (RAG) for Peace. Another is the Rotary Peace Symposia organized every three years. The last and fourth triennial Symposium was held just before the Rotary International convention in São Paulo in June 2015. This was an occasion for Rotarians and Peace Fellows to discuss collaborations and potential service projects together. Oscar Arias at the 2015 Peace Symposium in São Paulo. Photo: Rotary International.

The event was held for two days. It was attended by 354 participants, including 72 Peace Fellows, Rotarians (some of whom are members of the RAG for Peace), representatives of the six Peace Centers, and leaders of NGOs working on peace and conflict resolution. Oscar Arias, the former President of Costa Rica and a Nobel Peace Laureate, was a keynote speaker. Nine in ten attendees surveyed after the Symposium were satisfied or very satisfied with the event, suggesting potential for the community of practice.

The Sierra Leone Education Fund: Small but Impactful

by Quentin Wodon

While some of the projects implemented in developing countries by service clubs are large, most are not.  Smaller projects may be small, but they can nevertheless be impactful, making a real difference in the life of their beneficiaries.  A good example is the Sierra Leone Education Fund.

Sierra Leone

A new Rotarian Economist Brief by Jennifer Carr Pilholski and Eric Wolvovsky tells the story of the fund and how it works.  The fund was launched as a project of the Howard County Rotaract Club with additional support from the Columbia Rotary Club.  It is now a 501c(3) organization in the US, and all the funds donated go to a primary school in Sierra Leone for materials and scholarships. As for other posts showcasing projects through briefs from this blog’s series, rather than summarizing the brief, I encourage you to read it in full here.

If you would like to submit a brief about your project, please send me an email through the Contact Me page of this blog.