Education innovations for disadvantaged students in Washington DC

Can internship and mentorship programs help students graduate from high school and prepare them for colleges and careers? What type of support is needed for the most disadvantaged youth, including those who suffer from homelessness? Do tutoring programs help students learn? How can we improve the research, writing, and presentation skills of middle and high school students? 

These are some of the questions considered in a new series of briefs on innovations in education in the greater Washington DC, area of the United States.  The series is launched jointly by the World Bank Group’s (WBG) education team, the WBG Community Connections Program (the WBG’s outreach program to the local community), and the Rotary Club of Capitol Hill. We hope to contribute to better education outcomes not only in the DC area but also elsewhere by showcasing innovative programs that make a positive difference in the life of students and how well they learn.

Why focus on DC? The WBG’s mission is to end extreme poverty and boost shared prosperity. While living standards in DC are better than in the developing work, poverty rates remain high and only about two thirds of students complete high school according to the Office of the State Superintendent of Education. While improvements have been achieved in recent years, innovations are needed to improve education and employment outcomes for disadvantaged children and youth.

The  briefs, in a small but meaningful way, will hopefully contribute to debates on how to improve education by featuring successful programs and policies. The programs and policies featured in the briefs will likely be relevant for other regions in the United States and developing countries with similar challenges.

The first set of briefs in the series feature non-profits that benefited from monitoring and evaluation grants from the WBG’s Community Connections Program or pro bono evaluation support provided by World Bank staff. The Latin America Youth Center (LAYC), One World Education, and the Urban Alliance are among the nonprofits featured in the briefs released on the occasion of the WBG’s Volunteer Awareness Day on May 10.

Below is a brief description of the work of the three organizations together with links to the briefs on these organizations.

LAYC: Some 17,400 young adults aged 18 – 24 who are from the Washington Metropolitan Area are disconnected from work and school. These youth are often from low-income families, are not in school, and are out of work. They typically face multiple challenges, including homelessness, issues with the courts, or substance abuse. These challenges prevent them from successfully transitioning into adulthood. LAYC uses an innovative approach called the ‘Promotor Pathway’ to target high-risk youth.  It’s a long-term, intensive, holistic case management and mentorship intervention which has led to positive changes in school enrollment, birth rates, and homelessness.

One World Education: During the 2015-16 school year, One World Education worked with District of Columbia Public Schools as a partner for the ‘Cornerstone’ initiative. Cornerstones are high-quality, in-depth core curricular experiences such as argumentative writing programs. Cornerstones aim to provide rigorous content to students, improved professional development for teachers, and continuity and consistency across grades and subjects. Some 5,200 students in 15 DC public schools in grades 10 and 12 have already participated in the One World Education programs.

Urban Alliance:  This non-profit organization, which operates in DC, Baltimore, Chicago, and Northern Virginia, facilitates the transition of disadvantage youth from high school to college to employment. It does so by running a comprehensive early employment program that provides access and exposure to professional networks for youth enrolled in the program. Urban Alliance staff train and mentor the students through their first professional employment opportunities, which help propel them to future success. Over 90 percent of Urban Alliance alumni go on to college. The WBG has participated in the Urban Alliance program since 1997 and supported over 300 students through internships. Results from an external evaluation suggest that the Urban Alliance program improves high school graduation rates and the likelihood that students will go to college after graduation.

The objective of the series of briefs is to document these and other successful programs, so that they can inform education policy and practice not only in DC, but also elsewhere. If you are living in the Greater Washington, DC, area and if you have an idea for a potential brief in the series, please let me know or post your idea in the comment section below.

This post is reproduced with minor changes from a post published by the author on May 9, 2016 on the World Bank Education for Global Development blog.

 

 

Impact Evaluations, Part 1: Do We Need Them in Rotary?

by Quentin Wodon

Monitoring and evaluation have become essential in development and service work. When providing funding, most foundations and donors now require some type of monitoring and evaluation either ex ante to allocate funding, or ex post to assess whether projects that have been funded have proved to be successful or not. The same is true for government agencies – when deciding whether to scale up a pilot intervention, most agencies now typically rely on the results of at least some type of evaluation.

Different types of evaluations can be conducted and not all are equal in terms of what we can learn from them. In this series of three posts, I will focus on impact evaluations (as opposed to process evaluations). Specifically, I will discuss 1) whether we need impact evaluations in Rotary; 2) how impact evaluations can be implemented in practice; and 3) what are some of the limits of impact evaluations that practitioners and policy makers should be aware of. But before doing so, it is probably useful to briefly explain what an impact evaluation entails.

What Is An Impact Evaluation?

Impact evaluations aim to measure the impact of specific projects, policies, or interventions on specific outcomes. The question they ask is not whether a given outcome has been achieved or not among a target population. They look instead at the specific contribution or impact of a well-defined intervention on a well-defined outcome. Said differently, they try to measure a counterfactual: what would have been the outcome without the intervention? By comparing the assessment of what the outcome would have been without the intervention and the outcome with the intervention, impact evaluations inform us about the success (or lack thereof) of specific interventions in improving specific outcomes. When data on the cost of alternative interventions and their benefits are available, impact evaluations are especially helpful in deciding whether some interventions should be maintained or scaled up, while others should be abandoned.

Estimating the counterfactual in an impact evaluation is no simple matter, in part because most interventions are not distributed randomly in a target population. Consider children going to private or charter schools. In most countries, those children perform better than children attending public schools. Does that in itself tell us anything about the comparative performance of public versus private or charter schools? It does not.

Children attending private or charter schools may for example come from wealthier families with well educated parents who are better able to help their children at home than parents from less privileged backgrounds. The higher performance of students in private or charter schools on tests aiming to measure learning may not be related to the school in which they are enrolled per se, but instead to their socio-economic and other characteristics. The proper counterfactual question would be: how well would students enrolled today in private or charter schools perform if they had enrolled instead in public schools? Data to answer such questions are often not readily available, so the counterfactual is often not known without some further data or analysis. Special techniques – ranging from randomized control trials (the gold standard) to various statistical and econometric approaches, are often needed in order to assess the impact of specific interventions on specific outcomes. I will discuss these techniques in the second post in this series.

Do We Need Impact Evaluations in Rotary?

If as Rotarians we “do good work” in the world, why would we need impact evaluations? Isn’t it enough to serve the less fortunate? I would argue that Rotary needs impact evaluations for at least three reasons.

  1. Doing good work is not enough – we need to do the best we can. We live in a world with scarce resources. What Rotarians can contribute – whether in terms of money or time and expertise – is finite. Our resources should be devoted to projects and initiatives that have the largest positive impact on those we aim to serve. And we will be able to assess what these projects or interventions are or should be only by evaluating our work (and relying as well on the evidence generated by other organizations that are implementing robust impact evaluations.)
  2. Who says we are doing good work? We may believe we are doing good work, but do we know whether some of our projects and interventions may have unintended consequences that could be detrimental to those we are trying to serve? To consider the example of education again, if we support one dynamic school in a poor community that selects its students on the basis of their academic potential, could this have a negative effect on another school in the area which might be left only with children who tend to perform less well for whatever reason. Given that peer effects are important drivers of learning, supporting one school may have negative consequences on another school. This is of course not necessarily the case, and this does not imply  that we should never support specific schools. But we need to be aware of the potential ripple effects that our projects may have, and impact evaluation can be useful for that.
  3. Rotary has an important role to play in piloting innovative interventions. Much of what Rotary clubs do on an on-going basis is not innovative, and there is nothing wrong with that. There are a lot of needs out there that we can contribute to meet without being innovative. Being innovative is not a requirement. But at the same time, we should promote some level of innovation in Rotary. Rotarians have technical expertise in many areas and they know (or should know) their community. They are in a way uniquely positioned to propose new ways of tackling old problems, and assessing whether such new ideas actually work. Furthermore, Rotary has a relatively large foundation, but in comparison to other actors, especially in the field of development, we are small. If we could pilot more innovative interventions, evaluate them properly, and let others with more resources scale up the most promising interventions, we could achieve more for those we aim to serve.

For these reasons I believe that we need impact evaluations in Rotary. Not all projects require an evaluation, especially since evaluations take some time and cost money. But we can probably do more in this area than we are doing today. In the next post in this series, I will discuss the “how to” of impact evaluations.

Note: This post is part of a series of three on impact evaluations. The three posts are available here: Part 1, Part 2, and Part 3.