The sixth free ebook in the Rotarian Economist Short Books series has been released. The book tells the story of an initiative by a Rotary club to improve its public image by writing articles in the local media about volunteering opportunities for residents to make a difference in their community. The articles feature great local nonprofits, some of which the club is partnering with in order to implement service projects. The initiative appears to have been a success. To download your free copy, please go here.
As readers of this blog may be aware, my Rotary club launched last month a number of partnerships with key nonprofits in our community as part of a “pro bono initiative”. These partnerships bring several benefits: 1) better service opportunities for our members and larger impact in the community; 2) more visibility for our partners and our club; and 3) new members. Let me briefly explain these three benefits in case they may inspire other clubs to adopt a similar model.
Better service opportunities and larger impact: Most Rotarians are professionals and/or business leaders. We are building on these skills in our club by providing pro bono strategic advise with small teams of 4-5 individuals (both Rotarians and non-Rotarians) that support local nonprofits. This makes our club more interesting for our members in terms of the service opportunities we provide, and it also increases the impact that we have on the community through local nonprofits. I mentioned this pro bono initiative in previous blog posts, so let me focus here on the other two benefits.
More visibility for our partners and our club: This higher visibility is achieved is several ways. First, we are sharing our work on social media using some of the better known blogs in our community. The main blog for our community is “The Hill Is Home”. So we started writing posts for that blog, not directly about our club, but about the great work of our nonprofit partners … and the fact that we are working with them. We also started writing short articles about our partner nonprofits in the main monthly magazine for the community. Again, the stories are about our partner nonprofits but they mention in passing that our club works with them. These efforts should give us more visibility, and they also help our nonprofit partners who truly appreciate the visibility they get with this initiative. Finally, we have started placing small posters in local cafes, libraries, and other locations to advertise the fact that our nonprofit partners are invited as speakers to our club meetings. We indicate when they are speaking, which can bring us more visitors.
More members: Our club has been losing members for quite a few years. As mentioned in a separate post on this blog in which I shared our club’s strategic plan, our top priority this year is to attract new members and revitalize the club. It is too early to assess whether we will be successful, but the last few weeks have been promising. On July 1, we had 18 members, down from 31 a few years ago. Right now, we are back to 26 members thanks to 8 new members who joined in the last three weeks. Our pro bono initiative and our partnerships with local nonprofits have helped us in recruiting some of these new members and we have a number of other potential members we are in contact with thanks to the initiative. We will loose a few members in coming weeks/months due to relocations (Washington DC is for some a temporary location), but we are hopeful that we will achieve a substantial net gain in membership this year thanks in large part to the pro bono initiative and the benefits it brings not only to the club, but more importantly to local nonprofits and the community.
There are multiple ways for Rotary clubs to partner with local nonprofits in a strategic way, and some clubs have a long history in doing so. Our new model emphasizing pro bono consulting teams working closely with local nonprofits may not be the right model for all clubs, but it appears to be working for us, and it ties in nicely with our efforts at improving our public image and recruiting new members. If you would like to know more about our new model, please do not hesitate to post a comment on this post, or to email me through the Contact Me page of the blog.
Readers of this blog know that I have emphasized for some time the need to strengthen a culture of evaluation in Rotary. Evaluations should be undertaken not only for our service projects, but also to assess how our clubs meet, work, and grow – or wither away. This post is about a recent evaluation of an education project supported by my club, and how the evaluation is proving to be useful not only for the local nonprofit we worked with, but also for our club and more generally for practitioners and policy makers working in the field of education.
For several years my club has supported One World Education (OWEd), a great nonprofit based in Washington, DC. OWEd runs the largest argumentative writing program in public and charter schools in the city. The nonprofit reached 5,800 middle and high school students this past school year. The aim of the program, which runs for 4-5 weeks in the schools, is to improve the research, writing, and presentation skills of the students, many of whom are from disadvantaged backgrounds and do not do very well in school.
In previous years, our support to OWEd consisted in providing a bit of funding and volunteering at some of their events. This year, we provided college scholarships for some of the high school students (seniors) who participated in the program and worked especially hard. But we also did more. Together with a team at American University, we designed an evaluation of the program to better measure its impact. For more than 550 students, teachers collected essays written in class before and after the program. The essays were graded by professors and instructors in the Department of Literature at American University. This enabled us to assess whether the program made a difference in the writing skills of middle and high school students.
The evaluation demonstrated that the program has a positive impact. The program generates statistically significant gains in writing quality, especially for students who performed worst on the initial pre-program assignment. The positive impact of the program was confirmed through data on the perceptions of teachers and students about the program. Two summary briefs about those evaluation results have been written and are now available for public schools and for charter schools separately.
It is clear that this type of evaluation is beneficial for the nonprofits whose programs are evaluated, as the evaluations enable the nonprofits to measure their impact, and take corrective action when needed. The evaluations are also beneficial for our club in reassuring members that we are investing in worthwhile initiatives.
But there is more. Many others are interested in such evaluations and may learn from them, possibly generating larger impacts beyond the specific programs being evaluated. And these evaluations provide for great stories to be featured in local newspapers or magazines as well as social media, giving more visibility not only to the nonprofits and programs being evaluated, but also to the Rotary clubs that supported those evaluations.
This is what we are focusing on now – making sure that the positive results obtained by OWEd through its program are better known in Washington, DC, and beyond. We are writing short articles that document those results, and some of the stories of the students who benefited from the program. We have secured already two placements for stories in the local media and we hope to write additional articles for national publications about the results of the evaluation. In addition, we will also prepare technical papers for academic journals. It remains to be seen whether we will be successful, but we now have a stronger story to tell thanks to the evaluation.
Finally, as mentioned, the evaluation has been summarized in two easy-to-read briefs. The two briefs, together with briefs about the work of other nonprofits operating in the field of education and skills for youth in the city, will be included in a small brief series on innovations in education in Washington, DC to be published by the World Bank. We hope that this simple brief series will help attract attention to the nonprofits doing great work in the city, while also helping practitioners and policy makers learn from the experience of successful programs.
In summary, evaluation is essential not only to help improve service projects, whether implemented by Rotary clubs or nonprofits, but also to tell stronger stories about ways to improve the lives of the less fortunate. Investing more in evaluation seems to be a win-win for nonprofits as well as service clubs. And for Rotary as a whole, as I mentioned it in a previous series of posts on this blog, focusing more on partnerships, innovation, and evaluation seems key to achieve larger impacts.
Growing local economies requires many different ingredients, but one of the most important ones is a skilled workforce, especially among youth. Skills tend to be acquired through the education system. As part of a series on increasing Rotary’s impact through partnerships, innovation, and evaluation, this brief tells the story of an innovative program in Washington, DC that is improving writing skills for high school seniors in public schools and preparing them for college in part with support from Rotary.
The United States benefitted for decades from one of the most skilled workforce in the world, but there are concerns that this is not the case anymore. Within the US, the District of Columbia has been struggling and often ranks at the bottom of the National Assessment of Educational Progress league tables. There are many reasons for the poor performance of the District. In spite of major improvements in economic development in the last decade, a substantial share of its population remains poor, and poverty is one of the main drivers of poor performance in school. But some programs are helping.
One World Education
One World Education (OWEd) trains teachers and helps students improve their writing skills, and think about their college options at the same time. OWEd was created in 2006 by two teachers, Eric Goldstein and Emily Chiariello, who taught at one of the charter schools in Washington, DC. Their idea was to use students’ reflective writing as the foundation for what was discussed in the classroom. The model proved successful as students became more engaged and, in turn, started to develop better research, writing, and analytical thinking skills. The non-profit was launched in 2007 and has been growing. OWEd recently signed an agreement with DCPS (District of Columbia Public Schools) to expand its programs in all public high schools in the city. As a result, OWEd has become the largest nonprofit program operating in the District’s public schools.
The program focuses on writing skills and is adapted to various grades. For example, the Grade 12 program helps students analyze, research, write argumentative essays, and lead presentations about the college and career issues that await them after graduation. It includes a comprehensive seven week coaching period. Essays written by students can serve as their Senior Project. Selected student essays are published on OWEd’s website, providing recognition for students and creating a cycle of peer-to-peer learning.
More generally, for all grades where the program is implemented (Grades 8, 10, and 12), students and teachers can access a number of resources provided by OWED, including the following:
- Common Core Aligned Lesson Plans: All lesson plans are created by teachers, for teachers, and are aligned to multiple research, writing, and presentation Common Core State Standards. Lessons are accompanied by rubrics for teacher evaluations and peer-to-peer reviews;
- Student Writer’s Notebook: the notebook leads students to analyze exemplary, peer-authored essays before guiding them through researching, outlining, drafting, and revising their own argumentative essays.
- Student and Educator Portals: Students and teachers will have access to easy-to-access lesson plans, rubrics, research sources, and related resources for teachers and students are available online.
Evaluating Program Impacts
Randomized controlled trials have not yet been implemented to assess the impact of the programs run by One World Education, but other data suggests that the program is having an impact. Specifically, evaluations by students and faculty at American University and George Washington University suggest gains in writing quality and self-confidence for students that have participated in OWEd’s programs.
In order to assess gains in the quality of the writing of participants, a sample of students participating in the program take a writing test before the start of the program and at the end of the program. The test is graded by university professors. Results suggest important gains after program participation.
Feedback from teachers – and more importantly students who have participated in the program is positive. For example, in the 2014 DCPS Grade 10 evaluation by students, participants reported improvements in terms of their ability to make a claim (87 percent); Provide research to support a claim (87 percent); Write (85 percent); Research information (84 percent); Analyze research (84 percent); Create an outline (79 percent); Create a draft (78 percent); Establish a research plan (75 percent); and Revise their essay (75 percent).
These and other positive evaluations of the program in partnership with two local universities have been a key factor in the agreement reached by OWEd with DCPS to substantially expand the program in grades 9, 10, and 12. All public high school students in the District in those grades will now have the opportunity to participate in the program.
How Has Rotary Helped?
Rotarians from the Rotary Club of Capitol Hill in Washington, DC, have supported the project in various ways. The club has donated funds to, and volunteered with, OWEd for several years. In 2015-16 the club’s donation will be matched with a district grant using so-called district designated funds from the Rotary Foundation.
Each year student essays are assessed by a panel of judges at a College and Career Writer’s Challenge each year. This enables students to learn how to make an argumentative pitch to a panel. One student from each school is eligible to earn a college or vocational training scholarship, and every participating school can nominate a number of seniors to participate in the event. Rotary club and district grants will allow OWEd to provide small scholarships for college to 10 students who have written especially good essays thanks to the program.
In addition, Rotarians have participated in OWEd’s programs in a number of volunteering capacities, including as judges for the essay competitions taking place at the College and Career Writer’s Challenge.
In supporting OWEd, Rotary builds on the benefits from partnerships, innovation, and evaluation. OWEd itself has partnered with District of Columbia Public Schools to substantially expand the reach of its program. The program is innovative in the way writing skills for students are being developed using a range of different resources and mechanisms. Evaluations of OWEd’s programs have shown that the programs generate measurable gains in middle and high school students’ writing skills, and in their self-confidence. The program not only improved the student’s writing, but it also helps in preparing them for college and career-level writing.
For Rotarians, OWEd’s programs have also offered unique opportunities to personally support students from disadvantaged backgrounds by contributing in the programs in various ways. This had been done through donations, but also through volunteering.
by Quentin Wodon
The United States and especially the District of Columbia are lagging behind in STEM education, as discussed in the first blog post of this series. When Don and his team designed the small Rotary-led tutoring program described in the second post of the series, they did not start with a review of the evidence from the literature on what works. But through the experience of the teachers and principal at the school, as well as their own experience, they had a pretty good idea of what could be useful. As a result, the design of the program actually corresponds to what the literature recommends.
Lessons from the Literature
The literature on tutoring and out-of-school-time programs (see for example the review by Heinrich and Burch) suggests that in order to achieve impact, it is often useful to: (1) provide consistent and sustained instructional time, for a total of at least 40-45 hours; (2) provide tutoring to small groups of students, preferably less than ten at a time; (3) follow a curriculum that is rich in content and takes into account the specific needs of students while being also closely related to what students learn during the regular school day; (4) ensure that tutoring sessions are active and varied (for example by combining structured and unstructured instruction, as well as individual and collective work time) and focused on targeting the development of specific skills; (5) foster positive relationships between tutors and students; and finally (6) foster collaboration between teachers and tutors with support of administrators, including for constructive evaluation. All of these features are at work in Don’s program.
There is substantial interest in tutoring today in the US. As mentioned in the first post in this series, under the ‘No Child Left Behind’ Act adopted a dozen years ago, public schools not making enough progress in learning assessments for two consecutive years must provide tutoring services to children. Tutoring initiatives are being implemented throughout the country. Earlier this year Mayor Emanuel announced the expansion (with private funding) of a mathematics tutoring program in Chicago that University of Chicago researchers found helpful for at-risk students in public schools (see the review of the study in the New York Times).
Examples of Great Programs
Another example of intensive tutoring program having impact is Higher Achievement. The NGO operates in Washington, DC, Baltimore, Richmond, and Pittsburgh. Students in the program meet three days a week during the school year. They first complete homework with support from teachers and volunteers. They then have dinner and work on a specific subject in small groups of two or three with a trained volunteer mentor. This is a rigorous program – overall, students spend a total of 650 hours a year in the program between 5th and 8th grade.
Data from Higher Achievement suggest that three fourth of the enrolled students improve their grade point average (GPA) by at least one letter grade, and 96% graduate from high school – two times the rate of their peers. Three fourths of the students also go on to graduate from college – four times the rate of their peers. The program has been evaluated rigorously by MDRC, a nonprofit, nonpartisan research organization. Researchers from the University of Texas at Austin compared Higher Achievement students (“scholars”) with a control group of students who applied to the program, met the admissions criteria, but were not selected to participate through a randomized lottery.
According to the evaluation of Higher Achievement published last year, the program had a statistically significant positive impact after one year in the program on mathematics proficiency and reading comprehension, as measured by standardized tests. The mathematics impacts lasted four years after enrollment in the program. The program also increased the probability that the students would enroll in high performing private high schools. These findings suggest that intensive OST (out-of-school-time) programs like Higher Achievement can be beneficial.
Another program that also operates in Washington, DC, and that has been rigorously evaluated by MDRC is Reading Partners. The program serves more than 7,000 students in over 130 schools in California, Colorado, New York, Oklahoma, Maryland, South Carolina, Texas, and Washington, DC. As was the case with the small Rotary-funded program in Washington, DC, and the larger program operated by Higher Achievement, Reading Partners works in (large) part with volunteers, which helps in keeping costs down. The evaluation of Reading Partners was conducted in 2012-13 in a subset of the schools where the program operates. Results suggest gains in reading proficiency. While this evaluation was not about STEM, it suggests again that tutoring programs can make a difference.
Policy and What You Can Do
From a policy point of view, there are legitimate questions about the cost effectiveness of some tutoring programs. This cost effectiveness issue must be looked at carefully on a case by case basis. But when the programs are staffed in part or fully by volunteers, they are more likely to be cost effective. Tutoring may also in some cases – especially when it is profit-motivated, act as a substitute for good quality teaching. This may be a serious problem in some developing countries (as an example, see this paper on Nepal), but probably much less so in developed countries. In most situations, tutoring is likely to lead to positive changes.
For those who care about helping disadvantaged students better succeed in schools, the good news is that there are many ways to contribute. If you have or can take the necessary time to do so, you can get personally involved like Don and his fellow Rotarians are doing, going every week to a school and working with a few students. But if you do not have the time, you can still help by contributing funding to organizations that are doing a great job on the ground.
by Quentin Wodon
Imagine a group of elementary school students gathering in a school gymnasium as part of a tutoring session. The students are trying to estimate the gravitational acceleration force on an object at sea level, where Washington, DC, is located. The students throw a golf ball in the air in the gymnasium. They record the time it takes for the ball to fall from apogee to the floor using a simple stop watch. They repeat the exercise 25 times. They also estimate the distance from apogee to the top of the ceiling, which is done by first measuring the distance from floor to ceiling and next by guessing by how much the ball misses the ceiling. The students’ estimate of ‘g’, the gravitational acceleration due to the force exerted by the earth on the golf ball, turns out to be within three percent of the accepted value for Washington, DC, even though each of the 25 individual computations per throw varied widely. This showed to the students how approximate values, when averaged, may converge on true values with reasonable accuracy.
Another experiment used a hygrometer, an instrument for measuring humidity or moisture content as well as temperatures. This was coupled with water and iced water in cans. Students had to figure out the temperature at which beads of water formed on the outside of the tin cans, which was followed by a discussion of what fog is, how temperature affects relative humidity, why clouds form and sometimes rain or snow is produced.
These scenes are not from a movie, but from a volunteer-based tutoring program run in a public school located in Anacostia, the poorest area of the city. Until recently, few children at the school passed standardized mathematics and reading tests, but things have improved. The tutoring program has now been in existence for six years. It is run by Dr. Don Messer a member of the Rotary Club of Washington, DC in District 7620. Together with teachers, school administrators, and a half dozen other tutors from his Rotary club Don designed the program in an innovative way.
The program focuses on mathematics and reading, and on the types of questions asked in standardized tests. This is not to “teach to the test”, but to ensure that children understand potential test questions well. Tutors work with students in small groups of three or four to generate interactions and more learning. The groups meet once or twice a week for the entire school year. The goal is not only to help the students learn, but also to help them understand that there is a future for them that often they didn’t know existed.
Tutoring can work to improve learning – this is why so many parents who have the means to do so invest in tutoring (there is a rather large literature on private tutoring – as just one recent example see this paper on Vietnam). But children from disadvantaged backgrounds do not have such opportunities, which is why volunteer-run programs are so important for those children.
To work well, tutoring sessions should be active, varied, and even fun. Sessions should combine structured and unstructured instruction, as well as individual and collective work, and they should focus on specific skills. In Don’s small but effective volunteer program the first part of each tutoring session focuses on prior test problems from DC standardized tests. These tests are augmented by problems that tutors or teachers prepare to emphasize special themes. In mathematics for example, a package would contain around 80 problems, ranging from routine arithmetic operations to data analysis (histograms, bar charts, tables), basic geometry, and problems that require reading to make sense of what is to be done. The problem set is paced by student progress, not by a time schedule. Tutors make sure that if a problem is difficult to understand for one or more of the students, all students understand what the problem is driving at before they start to work on the problem. Students work on the problem until all have finished, but if the tutor sees that at least one student remains confused, a group discussion is launched to help the students get the correct solution. The tutors also try to interject simple science illustrations within the problems to be solved, as illustrated earlier with the gravity constant and hygrometer experiments.
Impact and Recognition
How successful has Don’s program been? No impact evaluation is available to say for sure, but success rates at standardized tests have been systematically higher for tutored than non-tutored students year after year. The results, albeit not based on a randomized study, are encouraging. In part thanks to this program, the Rotary club of Washington, DC, was recognized two years ago as Volunteer Group of the Year by Chancellor Henderson of the District of Columbia Public Schools. For the Rotarian tutors, the experience has been highly rewarding. And in Don’s case, there was no better reward than having a fifth-grader tell him: “You know Dr. Messer, you’re my grandpa.”
In the third and last post in this series, I will discuss results from several programs that operate in Washington, DC, and have been rigorously evaluated, including Higher Achievement and Reading Partners.
Note: Part of this blog post is adapted from a section in a book published by the author entitled Membership in Service Clubs: Rotary’s Experience (Palgrave Macmillan, 2014).
by Quentin Wodon
Brandon was a quiet student enrolled in a primary school located in one of the poorest areas of Washington, DC, the capital city of the United States. Students in that area tend to have very low scores on standardized tests. Upon the recommendation of his teachers Brandon started to participate in the school’s tutoring program. He said little, but it was clear that he was absorbing the material being taught like a sponge. When the results from the District of Columbia’s comprehensive assessment system (DC-CAS) tests were announced, Brandon achieved proficiency in both mathematics and English. For his efforts and success, Brandon received a well-deserved award during the fifth grade graduation ceremony!
Tutoring and other supplemental education programs have received renewed attention in the United States. Under the much debated ‘No Child Left Behind’ Act adopted a dozen years ago, public schools that have not made enough progress in learning assessments for two consecutive years are in principle required to provide tutoring services to children. This makes sense given that there is scientific evidence that tutoring programs can make a difference in learning achievement if they are well implemented.
Series of Three Posts
This series of three posts on STEM (Science, Technology, Engineering, and Mathematics) education and tutoring in the capital city is written in recognition of World Science Day for Peace and Development celebrated each year on November 10. The day raises awareness of the importance of science and aims to bridge the gap between science and societies. The focus of World Science Day celebrations this year is about quality science education.
Improving science education is needed not only in developing countries, but also in developed countries, and especially so in the capital city of Washington, DC. This first post in the series documents the state of science education in the United States and in the District of Columbia. The second post will show how as individuals we can make a difference. That post will tell the story of Rotarians who have been actively involved in mathematics and science tutoring in one of the city’s schools for several years. The third post will argue that tutoring can be brought to scale and be part of the solution. That post will report on the impact of a tutoring program implemented in Washington, DC, and a few other cities by Higher Achievement.
Performance of the US
When Brandon received his award, he was enrolled in one of the worst performing public schools in Washington, DC (the schools has since made substantial progress under new management). The District of Columbia itself is one of the worst performing areas in the United States according to national assessment data. And the performance of the United States is one of the lowest among OECD and other developed countries according to international assessment data. Before talking about the potential promise of tutoring programs, providing a few statistics and basic facts about the performance of the United States, the District of Columbia, and schools within the District may be useful to underscore the magnitude of the problem we face.
Consider first the performance of the US as a nation. International comparable data on the performance of school systems in science, mathematics, and reading are available from PISA (Program for International Student Assessment). PISA measures skills for reading, mathematics and science literacy among 15 year olds. The test has been conducted every three years among a sample of students in each participating country since 2000. The latest round of data collection took place in 2012 with 65 countries participating. Results were released in December 2013.
Among 34 OECD countries, the US ranked 27th in mathematics, 17th in reading, and 20th in science, with no statistically significant improvement over time. This is despite the fact that the U.S. spends more per student than most other countries (only Austria, Luxembourg, Norway, and Switzerland spend more, but these countries do much better). More than one in four US students did not show basic mathematics proficiency on the test. The US also had a below-average share of top performers, and (not surprisingly) students from disadvantaged backgrounds performed worse on average.
Performance of the District of Columbia
Consider next the performance of the District of Columbia within the US. Comparable data on state-level performance are available from the 2013 National Assessment of Educational Progress (NAEP). Data on performance in mathematics are available in 4th and 8th grades.
Nationally, the average score for fourth-graders in mathematics was 242 in 2013. For the District of Columbia, the average was 229, the lowest score in the nation. Nationally, 83 percent of students performed at or above basic level. In the District, that share was 66 percent, again the lowest in the nation. Some 42 percent of students showed proficiency nationally, but in the District the proportion was only 28 percent. Only two states (Louisiana and Mississippi) performed worse. Gaps between the District and the nation are also large in eighth grade.
Whether those gaps are due to poor teaching or the fact that many children come from disadvantaged background is beyond the scope of this blog post (for an analysis of teacher value added in the district, see this recent paper). But whatever the reasons, the fact remains that many students in the District do not perform well. Furthermore, within the District of Columbia, inequalities in student performance also tend to be high between the well-to-do and the less fortunate.
Mentioning this inequality in performance between groups is just another way to emphasize how beyond broad averages, for the poor the likelihood to perform well on standardized tests in the District is really low. One way to show this inequality at work is to share a little known fact about the NMSQT/PSAT test administered each year in 11th grade by the College Board. For the high school class of 2015, the District (together with New Jersey) had the highest required qualifying scores for students to become National Merit Semifinalists. Students in the District had to obtain a score of 224 out of a maximum of 240 to qualify, a much higher threshold than in many other states. This is because while many students do poorly in the Districts, a few do very well, and the threshold to become a National Merit Semifinalist is state-specific and percentage based.
To sum up, the District of Columbia tends to be at the bottom in terms of average performance in mathematics (as well as science and reading) within the United States, with the United States also faring poorly internationally. That’s the problem. In the next two posts, I will discuss part of the solution – whether tutoring could help make a difference.