Free ebook 2 – Partner, Innovate, Evaluate: Increasing Rotary’s Impact

The second ebook in the Rotarian Economist Short Books Series has been published. Partnerships, innovation, and evaluation can increase the quality, scope, and reach of Rotary’s service work in communities. The book suggests with case studies how this can be done. All books in the series are free and available here in multiple formats.  Please share this link widely with others for them to be able to benefit from this resource. And if you like the books in the series, please consider writing a quick review at Smashwords!

2-partner-innovate-evaluate

Increasing the Impact of Rotary (Partnerships Series No. 9)

This post is the last in a series of nine posts on partnerships, innovation, and evaluation in Rotary. The rationale for the series was my conviction that if Rotary is to have a larger impact globally, it must rely more than has been the case so far on partnerships, innovation, and evaluation (and in some areas advocacy, as has been the case with polio). Seven different projects or investments that have relied on partnerships, were innovative, and were evaluated at least in some way, were showcased. A compilation of the case studies together with a brief introduction is available here. Separate briefs are also available for each of the projects here.

TRF_Centennial_logo_lockup

As I mentioned it in the introduction to the series, partnerships help to implement larger projects and benefit from the expertise of organizations that are among the best in their field. Rotary’s Foundation was created almost 100 years ago (the Centennial is next year) and it has about $1 billion in assets. This is respectable, but in the world of development projects, which is in practice where Rotary is investing most of its funds, this remains small. Without innovation, the contribution of Rotary is an important drop, but still a drop in the development assistance bucket.

By contrast, if Rotary clubs and district innovate, successful pilots can then be scaled up by other organizations with deeper pockets, thereby potentially achieving much larger impact. However, for innovative projects to be recognized as such, proper evaluations are needed. We must be able to demonstrate the impact of pilot projects. Innovation and evaluation are like twins: they work best in pairs. Together, partnerships, innovation, and evaluation are the key to larger impact.

To encourage clubs and districts to think bigger and more strategically, stories of great projects were shared: an innovative financing mechanism for polio eradication; an award winning project fighting malaria and Ebola in Mali; a teacher training program that is transforming teaching and learning in Nepali classrooms; a project on obstetric fistula saving the lives of mothers and children in Nigeria; a program to invest in the writing skills of disadvantaged youth in the United States; a project to improve access to water and sanitation in Uganda; and a global network of Peace Centers and Peace Fellows to help promote peace.

Some of these programs and projects are large. Others are small. Most were implemented through global grants, but one was implemented through a district grant. All these projects have been in one way or another innovative. They have all leveraged partnerships not only to crowd in financial resources, but also – and even more importantly – to build on great expertise. And they have all relied on monitoring and evaluation mechanisms to assess their impact, at least partially.

Putting together great projects requires work. Fundraising is often time consuming in Rotary given the funding model of the Rotary Foundation that requires raising funds from many clubs and districts first before getting a match from the Foundation. Planning, implementing, and in addition evaluating projects also takes time, especially when one tries to do this in a professional way. Finally, in order to be innovative, Rotarians leading projects need to be aware of where the frontier is in their field, and what could be innovative. This also takes some time.

There is nothing wrong with clubs and districts funding and implementing traditional Rotary projects. Most projects will continue to be fairly simple, with funds provided to worthy charitable causes. These projects, as well as the volunteer time often contributed by Rotarians when implementing them, serve an important purpose. The beneficiaries of these projects are better off thanks to them. These projects help communities, and they also benefit Rotary through the goodwill that the projects create.

But if we want to raise the bar and achieve larger impact, we also need to do more innovative projects. Rotary needs to be bolder, more ambitious. It needs to better learn from its projects, both the great and not so great ones, and make sure that lessons learned are shared broadly, well beyond the Rotary family. The launch of the Future Vision model, despite some challenges, was a step in the right direction. As we celebrate the Centennial of the Rotary Foundation next year, let’s make sure that we have the right vision for what Rotary and its Foundation could accomplish in the next 100 years.

Rotary Foundation Basics, Part 1: How Large is the Foundation?

by Quentin Wodon

The Rotary Foundation of Rotary International is a major player in the work of Rotarians worldwide. True, most activities organized by clubs and indeed most service projects are implemented independently of the Rotary Foundation, which I will refer to as “TRF” in this post. Many clubs have their own foundations and many projects do not require a foundation to be implemented. But for most investments at scale, TRF does play a key role, so Rotarians should have at least a basic knowledge of the foundation. My guess is that this is not the case today, so I thought it might be useful to run a three post series on some of the basics of the foundation. This first post discusses assets and expenses. The next will look at categories of expenses by thematic areas. The third will discuss management and suggest a few ideas to make TRF more impactful.

RTF Report

How large is TRF In Terms of Assets?

A widely used measure of the size of a foundation is its assets. According to their latest annual report (consolidated statements of activities), TRF and Rotary International had $1.2 billion in total assets as of the end the 2013-14 fiscal year, and $1.09 billion in assets net of liabilities. This represented an increase in net assets of about $130 million versus the previous year (2012-13) when net assets were at $961 million and total assets were at $1.08 billion.

One billion dollars is no small change, but some other foundations are even larger. Within the US, TRF would rank about 71st in terms of assets according to data from the Foundation Center (for some reason, TRF is not listed in the top 100 foundations put together by the Foundation Center, so the exact ranking is not available). The largest US foundation is the Bill and Melinda Gates Foundation, which had more than $40 billion in assets at the end of 2013. A dozen other US foundations have assets between $5 billion and $15 billion, and many more had assets between $1 billion and $5 billion.

In other words, in comparison to some of the largest US foundations, TRF could be considered as mid-size, even if it remains large in comparison to most foundations that tend to be much smaller. In comparison to foundations from other service club organizations, TRF is also the largest by far. Lions Clubs International now has more members than Rotary worldwide, but the Lions Clubs International Foundation had total assets in 2014 of $318 million. This is still large, but quite a bit smaller than TRF. The Kiwanis International Foundation is much smaller in terms of assets ($28 million in 2013).

How large is TRF In Terms of Grants?

Grants matter more than assets, since this is where the foundations make a difference. In 2013-14, TRF and RI had consolidated total expenses of $350 million. About two thirds of that amount ($232 million) were allocated to program expenses. The rest went to TRF development expenses ($16 million), TRF general administration ($5 million), RI operating expenses ($73 million) and RI service and other activities ($25 million).

With $232 million in program giving last year, TRF would have ranked about 25th among US foundation according to the list from the Foundation Center. This is a pretty good ranking. By comparison, the Lions Clubs International Foundation provided $44 million in grants in 2014, and the Kiwanis International Foundation made $18 million in grants in 2013. Only one US foundation – the Bill and Melinda Gates Foundations – gives more than one billion dollar in grants per year (it gives over $3 billion per year).

Why is there such a jump in terms of ranking for TRF among US foundations when considering grants or contributions instead of assets? In large part because many US foundations rely mostly or quasi exclusively on their endowments to make grants, without necessarily a lot of extra funding coming in annually (apart from returns on equity). By contrast TRF is able to rely also on donations from Rotarians, among others through its annual fund. In 2013-14, revenues from the annual fund reached $117 million. A separate endowment fund grew by $24 million. Annual giving by Rotarians, which is invested by the Foundation, is what makes it feasible for TRF to be able to make more grants in a sustainable and long-term basis.

Annual Fund

Still, we should all realize that while substantial, $232 million in grants/contributions per year remains small in comparison to some of the investments made by other players in the field of development. Official Development Assistance flows estimated by the Organization for Economic Cooperation and Development stood at $134.5 billion in 2013 (the largest recipient was Afghanistan, with more than $5 billion in aid). And some groups such as World Vision or Catholic Charities tend to have larger footprints than TRF. For example, World Vision provides funding to the tune of $2.3 billion per year for international programming as well as relief and rehabilitation, or about 10 times the level of contributions of TRF.

Who Gives the Most to TRF?

Since annual contributions are essential for the future of TRF, it is useful to look at who gives. The country that gives the most to TRF is (not surprisingly) the US. This is not surprising because the US has also the largest membership in Rotary. For the Rotary year 2013-14, TRF received $174 million from donors in the US. This includes $90 million in matches from the Bill and Melinda Gates Foundation for polio, so that individual and other forms of giving reached $84 million. Japan came in second, with $15 million in giving, followed by India with $13 million, Korea with $12 million, Taiwan with $9 million, Italy with $8 million, Canada and Germany with about $7 million each, Australia with $6 million, and Brazil with $5 million.

Giving TRF

How about giving per member? On a per Rotarian basis Taiwan comes on top with $216 in giving per Rotarian, followed by Korea ($182), Canada ($168), Japan ($134), the US ($124, not including the Gates foundation matching funds), and Australia ($123). In other countries, average giving per Rotarian to TRF is below $100 per year. Considering that membership in Rotary costs much more than that including for meals in many clubs, these levels of individual giving are frankly too low, especially in the United States where donors benefit from a tax exemption when giving since TRF is a registered 501c(3) charity. But this is another matter that will not be discussed here.

Giving per person

So that’s is for the basic financial information on the size of TRF that I wanted to summarize in this post. The conclusion is that TRF is a relatively large foundation, especially in terms of contributions/grants disbursed annually. At the same time, in comparison to overall flows for projects in developing countries, TRF is not at the same level as a number of other players, which makes the issue of strategic positioning essential for the foundation. That will be the topic of the next post in this series, by looking at the categories of expenses of TRF by thematic area.

Note: This post is part of a series of three on TRF: Part 1, Part 2, Part 3.

 

 

STEM Education and Tutoring in the Capital City: Part 3 – Achieving Impact

Note: This post is part of a series of three on tutoring for science and mathematics among disadvantaged students. Part 1 looks at needs. Part 2 and part 3 give examples of successful programs.

by Quentin Wodon

The United States and especially the District of Columbia are lagging behind in STEM education, as discussed in the first blog post of this series. When Don and his team designed the small Rotary-led tutoring program described in the second post of the series, they did not start with a review of the evidence from the literature on what works. But through the experience of the teachers and principal at the school, as well as their own experience, they had a pretty good idea of what could be useful. As a result, the design of the program actually corresponds to what the literature recommends.

Don, a teacher, and a few of the tutored students
Don, a teacher, and a few of the tutored students

Lessons from the Literature

The literature on tutoring and out-of-school-time programs (see for example the review by Heinrich and Burch) suggests that in order to achieve impact, it is often useful to: (1) provide consistent and sustained instructional time, for a total of at least 40-45 hours; (2) provide tutoring to small groups of students, preferably less than ten at a time; (3) follow a curriculum that is rich in content and takes into account the specific needs of students while being also closely related to what students learn during the regular school day; (4) ensure that tutoring sessions are active and varied (for example by combining structured and unstructured instruction, as well as individual and collective work time) and focused on targeting the development of specific skills; (5) foster positive relationships between tutors and students; and finally (6) foster collaboration between teachers and tutors with support of administrators, including for constructive evaluation. All of these features are at work in Don’s program.

There is substantial interest in tutoring today in the US. As mentioned in the first post in this series, under the ‘No Child Left Behind’ Act adopted a dozen years ago, public schools not making enough progress in learning assessments for two consecutive years must provide tutoring services to children. Tutoring initiatives are being implemented throughout the country. Earlier this year Mayor Emanuel announced the expansion (with private funding) of a mathematics tutoring program in Chicago that University of Chicago researchers found helpful for at-risk students in public schools (see the review of the study in the New York Times).

Examples of Great Programs

Another example of intensive tutoring program having impact is Higher Achievement. The NGO operates in Washington, DC, Baltimore, Richmond, and Pittsburgh. Students in the program meet three days a week during the school year. They first complete homework with support from teachers and volunteers. They then have dinner and work on a specific subject in small groups of two or three with a trained volunteer mentor. This is a rigorous program – overall, students spend a total of 650 hours a year in the program between 5th and 8th grade.

Data from Higher Achievement suggest that three fourth of the enrolled students improve their grade point average (GPA) by at least one letter grade, and 96% graduate from high school – two times the rate of their peers. Three fourths of the students also go on to graduate from college – four times the rate of their peers. The program has been evaluated rigorously by MDRC, a nonprofit, nonpartisan research organization. Researchers from the University of Texas at Austin compared Higher Achievement students (“scholars”) with a control group of students who applied to the program, met the admissions criteria, but were not selected to participate through a randomized lottery.

According to the evaluation of Higher Achievement published last year, the program had a statistically significant positive impact after one year in the program on mathematics proficiency and reading comprehension, as measured by standardized tests. The mathematics impacts lasted four years after enrollment in the program. The program also increased the probability that the students would enroll in high performing private high schools. These findings suggest that intensive OST (out-of-school-time) programs like Higher Achievement can be beneficial.

Another program that also operates in Washington, DC, and that has been rigorously evaluated by MDRC is Reading Partners. The program serves more than 7,000 students in over 130 schools in California, Colorado, New York, Oklahoma, Maryland, South Carolina, Texas, and Washington, DC. As was the case with the small Rotary-funded program in Washington, DC, and the larger program operated by Higher Achievement, Reading Partners works in (large) part with volunteers, which helps in keeping costs down. The evaluation of Reading Partners was conducted in 2012-13 in a subset of the schools where the program operates. Results suggest gains in reading proficiency. While this evaluation was not about STEM, it suggests again that tutoring programs can make a difference.

Policy and What You Can Do

From a policy point of view, there are legitimate questions about the cost effectiveness of some tutoring programs. This cost effectiveness issue must be looked at carefully on a case by case basis. But when the programs are staffed in part or fully by volunteers, they are more likely to be cost effective. Tutoring may also in some cases – especially when it is profit-motivated, act as a substitute for good quality teaching. This may be a serious problem in some developing countries (as an example, see this paper on Nepal), but probably much less so in developed countries. In most situations, tutoring is likely to lead to positive changes.

For those who care about helping disadvantaged students better succeed in schools, the good news is that there are many ways to contribute. If you have or can take the necessary time to do so, you can get personally involved like Don and his fellow Rotarians are doing, going every week to a school and working with a few students. But if you do not have the time, you can still help by contributing funding to organizations that are doing a great job on the ground.

STEM Education and Tutoring in the Capital City: Part 2 – Measuring Gravity

Note: This post is part of a series of three on tutoring for science and mathematics among disadvantaged students. Part 1 looks at needs. Part 2 and part 3 give examples of successful programs.

by Quentin Wodon

Innovative Tutoring

Imagine a group of elementary school students gathering in a school gymnasium as part of a tutoring session. The students are trying to estimate the gravitational acceleration force on an object at sea level, where Washington, DC, is located. The students throw a golf ball in the air in the gymnasium. They record the time it takes for the ball to fall from apogee to the floor using a simple stop watch. They repeat the exercise 25 times. They also estimate the distance from apogee to the top of the ceiling, which is done by first measuring the distance from floor to ceiling and next by guessing by how much the ball misses the ceiling. The students’ estimate of ‘g’, the gravitational acceleration due to the force exerted by the earth on the golf ball, turns out to be within three percent of the accepted value for Washington, DC, even though each of the 25 individual computations per throw varied widely. This showed to the students how approximate values, when averaged, may converge on true values with reasonable accuracy.

Two students perform the gravity experiment
Two students perform the gravity experiment

Another experiment used a hygrometer, an instrument for measuring humidity or moisture content as well as temperatures. This was coupled with water and iced water in cans. Students had to figure out the temperature at which beads of water formed on the outside of the tin cans, which was followed by a discussion of what fog is, how temperature affects relative humidity, why clouds form and sometimes rain or snow is produced.

Two students work with a hygrometer
Two students work with a hygrometer

Program Characteristics

These scenes are not from a movie, but from a volunteer-based tutoring program run in a public school located in Anacostia, the poorest area of the city. Until recently, few children at the school passed standardized mathematics and reading tests, but things have improved. The tutoring program has now been in existence for six years. It is run by Dr. Don Messer a member of the Rotary Club of Washington, DC in District 7620. Together with teachers, school administrators, and a half dozen other tutors from his Rotary club Don designed the program in an innovative way.

The program focuses on mathematics and reading, and on the types of questions asked in standardized tests. This is not to “teach to the test”, but to ensure that children understand potential test questions well. Tutors work with students in small groups of three or four to generate interactions and more learning. The groups meet once or twice a week for the entire school year. The goal is not only to help the students learn, but also to help them understand that there is a future for them that often they didn’t know existed.

Tutoring can work to improve learning – this is why so many parents who have the means to do so invest in tutoring (there is a rather large literature on private tutoring – as just one recent example see this paper on Vietnam). But children from disadvantaged backgrounds do not have such opportunities, which is why volunteer-run programs are so important for those children.

To work well, tutoring sessions should be active, varied, and even fun. Sessions should combine structured and unstructured instruction, as well as individual and collective work, and they should focus on specific skills. In Don’s small but effective volunteer program the first part of each tutoring session focuses on prior test problems from DC standardized tests. These tests are augmented by problems that tutors or teachers prepare to emphasize special themes. In mathematics for example, a package would contain around 80 problems, ranging from routine arithmetic operations to data analysis (histograms, bar charts, tables), basic geometry, and problems that require reading to make sense of what is to be done. The problem set is paced by student progress, not by a time schedule. Tutors make sure that if a problem is difficult to understand for one or more of the students, all students understand what the problem is driving at before they start to work on the problem. Students work on the problem until all have finished, but if the tutor sees that at least one student remains confused, a group discussion is launched to help the students get the correct solution. The tutors also try to interject simple science illustrations within the problems to be solved, as illustrated earlier with the gravity constant and hygrometer experiments.

Impact and Recognition

How successful has Don’s program been? No impact evaluation is available to say for sure, but success rates at standardized tests have been systematically higher for tutored than non-tutored students year after year. The results, albeit not based on a randomized study, are encouraging. In part thanks to this program, the Rotary club of Washington, DC, was recognized two years ago as Volunteer Group of the Year by Chancellor Henderson of the District of Columbia Public Schools. For the Rotarian tutors, the experience has been highly rewarding. And in Don’s case, there was no better reward than having a fifth-grader tell him: “You know Dr. Messer, you’re my grandpa.”

In the third and last post in this series, I will discuss results from several programs that operate in Washington, DC, and have been rigorously evaluated, including Higher Achievement and Reading Partners.

Note: Part of this blog post is adapted from a section in a book published by the author entitled Membership in Service Clubs: Rotary’s Experience (Palgrave Macmillan, 2014).

 

STEM Education and Tutoring in the Capital City: Part 1 – The Problem

Note: This post is part of a series of three on tutoring for science and mathematics among disadvantaged students. Part 1 looks at needs. Part 2 and part 3 give examples of successful programs.

by Quentin Wodon

Brandon was a quiet student enrolled in a primary school located in one of the poorest areas of Washington, DC, the capital city of the United States. Students in that area tend to have very low scores on standardized tests. Upon the recommendation of his teachers Brandon started to participate in the school’s tutoring program. He said little, but it was clear that he was absorbing the material being taught like a sponge. When the results from the District of Columbia’s comprehensive assessment system (DC-CAS) tests were announced, Brandon achieved proficiency in both mathematics and English. For his efforts and success, Brandon received a well-deserved award during the fifth grade graduation ceremony!

Brandon receives an award for his hard work
Brandon receives an award for his hard work

Tutoring and other supplemental education programs have received renewed attention in the United States. Under the much debated ‘No Child Left Behind’ Act adopted a dozen years ago, public schools that have not made enough progress in learning assessments for two consecutive years are in principle required to provide tutoring services to children. This makes sense given that there is scientific evidence that tutoring programs can make a difference in learning achievement if they are well implemented.

Series of Three Posts

This series of three posts on STEM (Science, Technology, Engineering, and Mathematics) education and tutoring in the capital city is written in recognition of World Science Day for Peace and Development celebrated each year on November 10. The day raises awareness of the importance of science and aims to bridge the gap between science and societies. The focus of World Science Day celebrations this year is about quality science education.

Improving science education is needed not only in developing countries, but also in developed countries, and especially so in the capital city of Washington, DC. This first post in the series documents the state of science education in the United States and in the District of Columbia. The second post will show how as individuals we can make a difference. That post will tell the story of Rotarians who have been actively involved in mathematics and science tutoring in one of the city’s schools for several years. The third post will argue that tutoring can be brought to scale and be part of the solution. That post will report on the impact of a tutoring program implemented in Washington, DC, and a few other cities by Higher Achievement.

Performance of the US

When Brandon received his award, he was enrolled in one of the worst performing public schools in Washington, DC (the schools has since made substantial progress under new management). The District of Columbia itself is one of the worst performing areas in the United States according to national assessment data. And the performance of the United States is one of the lowest among OECD and other developed countries according to international assessment data. Before talking about the potential promise of tutoring programs, providing a few statistics and basic facts about the performance of the United States, the District of Columbia, and schools within the District may be useful to underscore the magnitude of the problem we face.

Consider first the performance of the US as a nation. International comparable data on the performance of school systems in science, mathematics, and reading are available from PISA (Program for International Student Assessment). PISA measures skills for reading, mathematics and science literacy among 15 year olds. The test has been conducted every three years among a sample of students in each participating country since 2000. The latest round of data collection took place in 2012 with 65 countries participating. Results were released in December 2013.

Among 34 OECD countries, the US ranked 27th in mathematics, 17th in reading, and 20th in science, with no statistically significant improvement over time. This is despite the fact that the U.S. spends more per student than most other countries (only Austria, Luxembourg, Norway, and Switzerland spend more, but these countries do much better). More than one in four US students did not show basic mathematics proficiency on the test. The US also had a below-average share of top performers, and (not surprisingly) students from disadvantaged backgrounds performed worse on average.

Performance of the District of Columbia

Consider next the performance of the District of Columbia within the US. Comparable data on state-level performance are available from the 2013 National Assessment of Educational Progress (NAEP). Data on performance in mathematics are available in 4th and 8th grades.

Nationally, the average score for fourth-graders in mathematics was 242 in 2013. For the District of Columbia, the average was 229, the lowest score in the nation. Nationally, 83 percent of students performed at or above basic level. In the District, that share was 66 percent, again the lowest in the nation. Some 42 percent of students showed proficiency nationally, but in the District the proportion was only 28 percent. Only two states (Louisiana and Mississippi) performed worse. Gaps between the District and the nation are also large in eighth grade.

Whether those gaps are due to poor teaching or the fact that many children come from disadvantaged background is beyond the scope of this blog post (for an analysis of teacher value added in the district, see this recent paper). But whatever the reasons, the fact remains that many students in the District do not perform well.  Furthermore, within the District of Columbia, inequalities in student performance also tend to be high between the well-to-do and the less fortunate.

Mentioning this inequality in performance between groups is just another way to emphasize how beyond broad averages, for the poor the likelihood to perform well on standardized tests in the District is really low. One way to show this inequality at work is to share a little known fact about the NMSQT/PSAT test administered each year in 11th grade by the College Board. For the high school class of 2015, the District (together with New Jersey) had the highest required qualifying scores for students to become National Merit Semifinalists. Students in the District had to obtain a score of 224 out of a maximum of 240 to qualify, a much higher threshold than in many other states. This is because while many students do poorly in the Districts, a few do very well, and the threshold to become a National Merit Semifinalist is state-specific and percentage based.

To sum up, the District of Columbia tends to be at the bottom in terms of average performance in mathematics (as well as science and reading) within the United States, with the United States also faring poorly internationally. That’s the problem. In the next two posts, I will discuss part of the solution – whether tutoring could help make a difference.