Evaluation is essential to assess what works and share stronger stories

Readers of this blog know that I have emphasized for some time the need to strengthen a culture of evaluation in Rotary. Evaluations should be undertaken not only for our service projects, but also to assess how our clubs meet, work, and grow – or wither away. This post is about a recent evaluation of an education project supported by my club, and how the evaluation is proving to be useful not only for the local nonprofit we worked with, but also for our club and more generally for practitioners and policy makers working in the field of education.

OWED celebration
Photo of OWEd scholarship winners with Brian Pick, Chief of Teaching and Learning for the District of Columbia Public Schools, and Dave Paris, member of the Board of OWEd.

For several years my club has supported One World Education (OWEd), a great nonprofit based in Washington, DC. OWEd runs the largest argumentative writing program in public and charter schools in the city. The nonprofit reached 5,800 middle and high school students this past school year. The aim of the program, which runs for 4-5 weeks in the schools, is to improve the research, writing, and presentation skills of the students, many of whom are from disadvantaged backgrounds and do not do very well in school.

In previous years, our support to OWEd consisted in providing a bit of funding and volunteering at some of their events. This year, we provided college scholarships for some of the high school students (seniors) who participated in the program and worked especially hard. But we also did more. Together with a team at American University, we designed an evaluation of the program to better measure its impact. For more than 550 students, teachers collected essays written in class before and after the program. The essays were graded by professors and instructors in the Department of Literature at American University. This enabled us to assess whether the program made a difference in the writing skills of middle and high school students.

The evaluation demonstrated that the program has a positive impact. The program generates statistically significant gains in writing quality, especially for students who performed worst on the initial pre-program assignment.  The positive impact of the program was confirmed through data on the perceptions of teachers and students about the program. Two summary briefs about those evaluation results have been written and are now available for public schools and for charter schools separately.

It is clear that this type of evaluation is beneficial for the nonprofits whose programs are evaluated, as the evaluations enable the nonprofits to measure their impact, and take corrective action when needed.  The evaluations are also beneficial for our club in reassuring members that we are investing in worthwhile initiatives.

But there is more. Many others are interested in such evaluations and may learn from them, possibly generating larger impacts beyond the specific programs being evaluated. And these evaluations provide for great stories to be featured in local newspapers or magazines as well as social media, giving more visibility not only to the nonprofits and programs being evaluated, but also to the Rotary clubs that supported those evaluations.

This is what we are focusing on now – making sure that the positive results obtained by OWEd through its program are better known in Washington, DC, and beyond. We are writing short articles that document those results, and some of the stories of the students who benefited from the program.  We have secured already two placements for stories in the local media and we hope to write additional articles for national publications about the results of the evaluation. In addition, we will also prepare technical papers for academic journals. It remains to be seen whether we will be successful, but we now have a stronger story to tell thanks to the evaluation.

Finally, as mentioned, the evaluation has been summarized in two easy-to-read briefs. The two briefs, together with briefs about the work of other nonprofits operating in the field of education and skills for youth in the city, will be included in a small brief series on innovations in education in Washington, DC to be published by the World Bank. We hope that this simple brief series will help attract attention to the nonprofits doing great work in the city, while also helping practitioners and policy makers learn from the experience of successful programs.

In summary, evaluation is essential not only to help improve service projects, whether implemented by Rotary clubs or nonprofits, but also to tell stronger stories about ways to improve the lives of the less fortunate. Investing more in evaluation seems to be a win-win for nonprofits as well as service clubs.  And for Rotary as a whole, as I mentioned it in a previous series of posts on this blog, focusing more on partnerships, innovation, and evaluation seems key to achieve larger impacts.

Pro Bono Rotarian Initiative

Rotary is about fellowship and service work. How do we increase the impact of our service work in order to achieve higher impact in our communities while also fostering fellowship among Rotarians and others committed to making a difference in the life of the less fortunate? One potential response is the concept of the pro bono Rotarian or Rotaractor.

In my (limited) experience, many clubs engage in service projects that do not really build on the professional expertise of their members. Beautifying a school before the start of the school year, serving food for the homeless, helping in the renovation of a house for a vulnerable family, distributing dictionaries to third graders, or even joining a polio vaccination drive for a short period of time are all worthwhile activities. Such activities should continue and they often enable many members in a club to be involved in the service projects of the club.

But these one-shot activities typically do not build on the expertise that Rotarians have developed over many years in their professional career. In addition to traditional (local) service projects, Rotarians should probably also engage in more extensive pro bono work, for example to provide advice to nonprofits as consultants would. While the term pro bono is often associated with free legal advise, pro bono work can be done in many other areas, building on a wide range of expertise that volunteers may have. The value of the volunteer time that Rotarians would allocate to pro bono consulting could be very high for local nonprofits, with potentially larger beneficial impacts for communities than is the case with traditional projects. Again, the idea is not to pitch one form of service work against another, but to expand on what clubs currently do in their service work.

Importantly, I believe that a pro bono consulting model may also be beneficial for fellowship among Rotarians. While for some issues faced by nonprofits pro bono consulting can be done effectively in a short period of time, for more complex issues analyzing the challenges faced by a nonprofit and suggesting a solution takes a few months. For these challenges, pro bono consulting is typically done by a small team of 3-5 volunteers who commit to dedicating a bit of their time for several months in order to provide in-depth professional and free advice to local nonprofits. As Rotarians work together on such pro bono projects, stronger fellowship and friendships will emerge, and the vitality of clubs will improve as well. The pro bono Rotarian concept can really be a win-win for local nonprofits, Rotary clubs, and the communities we serve.

This coming Rotary year, I will help my club explore in a systematic way pro bono consulting opportunities with local nonprofits in our area (Washington, DC). You will hear more about this in coming weeks and months through this blog. We will start small, and we will assess the value of our pro bono work along the way. But we hope that the idea will grow and strengthen our club, as well as other clubs that may adopt this model.

If you would like to move in this direction in your club as well or if you would like to discuss similar ideas you may have, don’t hesitate to comment on this blog or to send me if you prefer a private email through the Contact Me page. I will be happy to help if I can, and I look forward to learning from you if you have already adopted a pro bono consulting model in your own Rotary or Rotaract club.

Investing in Disadvantaged Youth in the United States (Partnerships Series No. 6)

Growing local economies requires many different ingredients, but one of the most important ones is a skilled workforce, especially among youth. Skills tend to be acquired through the education system. As part of a series on increasing Rotary’s impact through partnerships, innovation, and evaluation, this brief tells the story of an innovative program in Washington, DC that is improving writing skills for high school seniors in public schools and preparing them for college in part with support from Rotary.

A student presents his papers at One World Education's Fair
A student presents his papers at One World Education’s Fair

The United States benefitted for decades from one of the most skilled workforce in the world, but there are concerns that this is not the case anymore. Within the US, the District of Columbia has been struggling and often ranks at the bottom of the National Assessment of Educational Progress league tables. There are many reasons for the poor performance of the District. In spite of major improvements in economic development in the last decade, a substantial share of its population remains poor, and poverty is one of the main drivers of poor performance in school. But some programs are helping.

One World Education

One World Education (OWEd) trains teachers and helps students improve their writing skills, and think about their college options at the same time. OWEd was created in 2006 by two teachers, Eric Goldstein and Emily Chiariello, who taught at one of the charter schools in Washington, DC. Their idea was to use students’ reflective writing as the foundation for what was discussed in the classroom. The model proved successful as students became more engaged and, in turn, started to develop better research, writing, and analytical thinking skills. The non-profit was launched in 2007 and has been growing.  OWEd recently signed an agreement with DCPS (District of Columbia Public Schools) to expand its programs in all public high schools in the city. As a result, OWEd has become the largest nonprofit program operating in the District’s public schools.

The program focuses on writing skills and is adapted to various grades. For example, the Grade 12 program helps students analyze, research, write argumentative essays, and lead presentations about the college and career issues that await them after graduation. It includes a comprehensive seven week coaching period. Essays written by students can serve as their Senior Project. Selected student essays are published on OWEd’s website, providing recognition for students and creating a cycle of peer-to-peer learning.

More generally, for all grades where the program is implemented (Grades 8, 10, and 12), students and teachers can access a number of resources provided by OWED, including the following:

  1. Common Core Aligned Lesson Plans: All lesson plans are created by teachers, for teachers, and are aligned to multiple research, writing, and presentation Common Core State Standards. Lessons are accompanied by rubrics for teacher evaluations and peer-to-peer reviews;
  2. Student Writer’s Notebook: the notebook leads students to analyze exemplary, peer-authored essays before guiding them through researching, outlining, drafting, and revising their own argumentative essays.
  3. Student and Educator Portals: Students and teachers will have access to easy-to-access lesson plans, rubrics, research sources, and related resources for teachers and students are available online.

Evaluating Program Impacts

Randomized controlled trials have not yet been implemented to assess the impact of the programs run by One World Education, but other data suggests that the program is having an impact. Specifically, evaluations by students and faculty at American University and George Washington University suggest gains in writing quality and self-confidence for students that have participated in OWEd’s programs.

In order to assess gains in the quality of the writing of participants, a sample of students participating in the program take a writing test before the start of the program and at the end of the program. The test is graded by university professors. Results suggest important gains after program participation.

Feedback from teachers – and more importantly students who have participated in the program is positive. For example, in the 2014 DCPS Grade 10 evaluation by students, participants reported improvements in terms of their ability to make a claim (87 percent); Provide research to support a claim (87 percent); Write (85 percent); Research information (84 percent); Analyze research (84 percent); Create an outline (79 percent); Create a draft (78 percent); Establish a research plan (75 percent); and Revise their essay (75 percent).

These and other positive evaluations of the program in partnership with two local universities have been a key factor in the agreement reached by OWEd with DCPS to substantially expand the program in grades 9, 10, and 12. All public high school students in the District in those grades will now have the opportunity to participate in the program.

How Has Rotary Helped?

Rotarians from the Rotary Club of Capitol Hill in Washington, DC, have supported the project in various ways. The club has donated funds to, and volunteered with, OWEd for several years. In 2015-16 the club’s donation will be matched with a district grant using so-called district designated funds from the Rotary Foundation.

Each year student essays are assessed by a panel of judges at a College and Career Writer’s Challenge each year. This enables students to learn how to make an argumentative pitch to a panel.  One student from each school is eligible to earn a college or vocational training scholarship, and every participating school can nominate a number of seniors to participate in the event. Rotary club and district grants will allow OWEd to provide small scholarships for college to 10 students who have written especially good essays thanks to the program.

In addition, Rotarians have participated in OWEd’s programs in a number of volunteering capacities, including as judges for the essay competitions taking place at the College and Career Writer’s Challenge.

Conclusion

In supporting OWEd, Rotary builds on the benefits from partnerships, innovation, and evaluation. OWEd itself has partnered with District of Columbia Public Schools to substantially expand the reach of its program. The program is innovative in the way writing skills for students are being developed using a range of different resources and mechanisms. Evaluations of OWEd’s programs have shown that the programs generate measurable gains in middle and high school students’ writing skills, and in their self-confidence. The program not only improved the student’s writing, but it also helps in preparing them for college and career-level writing.

For Rotarians, OWEd’s programs have also offered unique opportunities to personally support students from disadvantaged backgrounds by contributing in the programs in various ways. This had been done through donations, but also through volunteering.

Technology in Nepal’s classrooms: Using impact evaluation as a learning device

Students use laptops and digital resources provided by OLE Nepal
Students use laptops and digital resources provided by OLE Nepal

Impact evaluations are becoming essential in the way we think about development and service projects. Pilot programs suggesting statistically significant impacts are hailed as breakthroughs and as candidates for scaling up. Programs without such clear impact tend to be looked down upon and may be terminated. This may not be warranted. A primary function of impact evaluations should be to improve existing programs, especially in fields where evidence of positive impacts remains scarce. The experience of OLE Nepal, which is part of the OLE network and aims to improve learning and teaching through technology, is instructive in this regard.

Last week, Rabi Karmacharya shared his experience at OLE Nepal at a seminar co-sponsored by the World Bank, the Global Partnership for Education and the Rotary Club of Capitol Hill, a supporter of Rotary Club of Kathmandu Mid Town in expanding OLE Nepal’s programs. After a successful career in engineering and technology in California, he launched OLE Nepal in 2007. Now a social entrepreneur, Rabi wants to use technology to transform the way children learn through engagement, exploration and experimentation.

OLE Nepal has achieved quite a bit over the last eight years: deploying 5,000+ laptops in 100+ schools, training 600+ teachers on integrating ICT in the classroom teaching-learning process, developing 600+ learning modules for use by teachers, and creating a digital library with 6000+ books and other items used in schools and community libraries. Its ultimate objective is to help transform and improve Nepal’s education system with technology, working closely with the Ministry of Education and other partners.

What I found most interesting about Rabi’s presentation is how OLE Nepal – as well as donors that support the NGO such as the World Food Program, responded to an evaluation of its programs in 2009-11. The evaluation used a quasi-experimental design: English and mathematics tests administered to students, with collection of additional data through student, teacher and household surveys.

The results were disappointing, with no effect on student test scores even though both teachers and students reported liking the digital contents and finding them useful. The results contradicted field observations and other evidence from teachers and students on the fact that the program was making a difference.

Several factors may have led to an apparent lack of impact on student learning, according to Rabi. First, the program had been implemented only for a relatively short period of time, and improving student learning takes time. Second, it turned out that not all teachers reported to the one-week intensive training session that was held before launching the program, which may have decreased overall impacts in the schools that benefitted from the program. Third, some teachers may not have used as the available digital resources as much as expected due to the increase in workload that this entailed. Fourth, after the initial training, the support provided to teachers was limited. Fifth, it could be that the digital content, while following the curriculum taught in school, was too difficult for students to grasp.

These and a number of factors (including questions about variations in exam difficulty between the baseline and endline tests in the evaluation, and the possibility that some students spent more time on games available through the digital libraries rather than on study for examinations) may have led to the results.

What is important is that the evaluation did not lead to the demise of the program. Witnessing firsthand the changes in classroom dynamic and student engagement brought about by the program, donors continued to fund the NGO, which has been able to grow with support from the World Food Program, the Embassies of Denmark and Finland, and most recently the Air Asia Foundation.

In response to the evaluation, OLE Nepal introduced a number improvements in its programs:

  • It now promotes a shared model between grades for a more effective use of laptops and digital resources by schools and teachers;
  • The use of resources by students is better supervised and teachers receive more extensive support after the initial one-week training. In addition, a volunteers program is available for additional support;
  • The educational contents have been revised to follow the curriculum even more closely; and
  • More emphasis will be put on enhancing the ability of students to read in early grades, since this is a prerequisite for them to be able to learn subsequently and use digital resources effectively.

OLE Nepal’s experience is a great lesson in social entrepreneurship, humility, and resilience from an innovative NGO that strives to help teachers and students harness the power of technology in the classroom.

Note: this post is reproduced with minor changes from a post by the author for the World Bank’s Education for Global Development blog, available at http://blogs.worldbank.org/education/.

Webinar with Rotarian Founder of OLE Nepal: Learning from Pilots and Scaling Up

Established in September 2007, OLE (Open Learning Exchange) Nepal has pioneered the integration of technology in classroom teaching-learning process. OLE Nepal partnered with Nepal’s Department of Education to launch laptop-based integrated learning initiative in 2008, and has gradually expanded the program to more schools over the years. It has also designed and developed digital learning materials, trained teachers to use them to enhance student learning, and set up appropriate infrastructure in schools to enable learning using technology.

On August 4, 2015 at 12:30 PM (EST), Rabi Karmacharya will present at the World Bank a seminar/webinar on Learning from Pilots and Scaling Up: Integrating Technology in Classrooms in Nepal. Rabi will discuss successes and challenges in implementing OLE’s programs from their pilot phases to scaling up.

Rabi is a social entrepreneur who helped launch OLE Nepal with the vision to use technology to improve the quality of primary education in Nepal’s public schools, and to transform the way children learn through engagement, exploration and experimentation. He has extensive experience in technological innovation and management, and a conviction that young educated Nepalis have a critical role to play in nation building.

Prior to launching OLE Nepal in 2007, he headed HimalayanTechies, one of the first successful software outsourcing companies that he co-founded in Nepal in 2001. He currently serves as the Chair of the company’s Board of Directors. Rabi holds M.Eng and B.Sc. Degrees in Electrical Engineering & Computer Science from the Massachusetts Institute of Technology (MIT). He worked at 3Com Corporation in Santa Clara, California for three years as design engineer prior to his return to Nepal. Rabi is Asia Society’s Asia 21 Young Leader (2010) and Asia 21 Fellow (2011). He is also an active Rotarian from the Rotary club of Kathmandu Mid Town.

The seminar/webinar will be held at the World Bank “G” building (701 18th Street, NW, Washington D.C.) in room J7-044, It will be accessible remotely through WebEx (please send me an email through the Contact Me page if you would like to get the web link). If you live in the Washington DC area and would like to attend this seminar in person, please let me know as well through the Contact Me page. Light Lunch will be served.

This seminar and webinar is sponsored by the Education Global Practice at the World Bank, the Global Partnership for Education, and the Rotary Club of Capitol Hill that is helping to put together a global Rotary grant to expand OLE’s digital library program.

Experiencing the Wilderness

by Divya Wodon, Naina Wodon, and Quentin Wodon

Peter Kyle, the immediate past  Governor for District 7620 and a member of the Capitol Hill Rotary Club, has been a Rotarian for almost 40 years. He has been a member of Rotary clubs in the Philippines, New Zealand, and the US. As District Governor, he spent most of his time running the District and supporting clubs. But as his DG bio suggests, his third love (after Margaret and Rotary) has been Outward Bound, a nonprofit founded in 1941 that serves 70,000 students and teachers annually.

April Issue - Outward Bound

The organization sends students on tough expeditions into the wilderness, immersing them in unfamiliar and sometimes uncomfortable situations so that they can push their own limits. Students form teams and learn about individuality, strength, and character. They also learn about how to use these traits to make a difference and serve their community as well as the larger world. Apart from its expeditions, Outward Bound also offers courses for teenagers and other groups who have health complications or social, financial or other specific educational needs.

Peter first became involved with Outward Bound in New Zealand in 1967 when he participated on one of its programs. He subsequently became very involved on the administrative side rising to the position of Senior Vice President.  Following his move to the US in 1992 he helped found Outward Bound International and still serves as its Chairman Emeritus.

As Peter put it “Outward Bound is a cross between Boy Scouts and boot camp”. He described its mission as “to expose young people to a variety of physical and mental challenges to prove to them that if they try a little harder then they can do a lot better—they can run faster, swim for longer and climb higher thereby proving that there is more in you than you think.”

Peter would love for Rotarians, Rotaractors and Interactors to experience Outward Bound by participating on one of its expeditions. And if Rotarians are lost in the wilderness of the implementation of service projects, his advice, in typical Outward Bound fashion, is simply “to commit and do what you can (however hard it may be) to help alleviate poverty and promote peace, international understanding and goodwill.”

Note: This story is reproduced with minor changes from a book published by the authors entitled Membership in Service Clubs: Rotary’s Experience (Palgrave Macmillan, 2014).