STEM Education and Tutoring in the Capital City: Part 3 – Achieving Impact

Note: This post is part of a series of three on tutoring for science and mathematics among disadvantaged students. Part 1 looks at needs. Part 2 and part 3 give examples of successful programs.

by Quentin Wodon

The United States and especially the District of Columbia are lagging behind in STEM education, as discussed in the first blog post of this series. When Don and his team designed the small Rotary-led tutoring program described in the second post of the series, they did not start with a review of the evidence from the literature on what works. But through the experience of the teachers and principal at the school, as well as their own experience, they had a pretty good idea of what could be useful. As a result, the design of the program actually corresponds to what the literature recommends.

Don, a teacher, and a few of the tutored students
Don, a teacher, and a few of the tutored students

Lessons from the Literature

The literature on tutoring and out-of-school-time programs (see for example the review by Heinrich and Burch) suggests that in order to achieve impact, it is often useful to: (1) provide consistent and sustained instructional time, for a total of at least 40-45 hours; (2) provide tutoring to small groups of students, preferably less than ten at a time; (3) follow a curriculum that is rich in content and takes into account the specific needs of students while being also closely related to what students learn during the regular school day; (4) ensure that tutoring sessions are active and varied (for example by combining structured and unstructured instruction, as well as individual and collective work time) and focused on targeting the development of specific skills; (5) foster positive relationships between tutors and students; and finally (6) foster collaboration between teachers and tutors with support of administrators, including for constructive evaluation. All of these features are at work in Don’s program.

There is substantial interest in tutoring today in the US. As mentioned in the first post in this series, under the ‘No Child Left Behind’ Act adopted a dozen years ago, public schools not making enough progress in learning assessments for two consecutive years must provide tutoring services to children. Tutoring initiatives are being implemented throughout the country. Earlier this year Mayor Emanuel announced the expansion (with private funding) of a mathematics tutoring program in Chicago that University of Chicago researchers found helpful for at-risk students in public schools (see the review of the study in the New York Times).

Examples of Great Programs

Another example of intensive tutoring program having impact is Higher Achievement. The NGO operates in Washington, DC, Baltimore, Richmond, and Pittsburgh. Students in the program meet three days a week during the school year. They first complete homework with support from teachers and volunteers. They then have dinner and work on a specific subject in small groups of two or three with a trained volunteer mentor. This is a rigorous program – overall, students spend a total of 650 hours a year in the program between 5th and 8th grade.

Data from Higher Achievement suggest that three fourth of the enrolled students improve their grade point average (GPA) by at least one letter grade, and 96% graduate from high school – two times the rate of their peers. Three fourths of the students also go on to graduate from college – four times the rate of their peers. The program has been evaluated rigorously by MDRC, a nonprofit, nonpartisan research organization. Researchers from the University of Texas at Austin compared Higher Achievement students (“scholars”) with a control group of students who applied to the program, met the admissions criteria, but were not selected to participate through a randomized lottery.

According to the evaluation of Higher Achievement published last year, the program had a statistically significant positive impact after one year in the program on mathematics proficiency and reading comprehension, as measured by standardized tests. The mathematics impacts lasted four years after enrollment in the program. The program also increased the probability that the students would enroll in high performing private high schools. These findings suggest that intensive OST (out-of-school-time) programs like Higher Achievement can be beneficial.

Another program that also operates in Washington, DC, and that has been rigorously evaluated by MDRC is Reading Partners. The program serves more than 7,000 students in over 130 schools in California, Colorado, New York, Oklahoma, Maryland, South Carolina, Texas, and Washington, DC. As was the case with the small Rotary-funded program in Washington, DC, and the larger program operated by Higher Achievement, Reading Partners works in (large) part with volunteers, which helps in keeping costs down. The evaluation of Reading Partners was conducted in 2012-13 in a subset of the schools where the program operates. Results suggest gains in reading proficiency. While this evaluation was not about STEM, it suggests again that tutoring programs can make a difference.

Policy and What You Can Do

From a policy point of view, there are legitimate questions about the cost effectiveness of some tutoring programs. This cost effectiveness issue must be looked at carefully on a case by case basis. But when the programs are staffed in part or fully by volunteers, they are more likely to be cost effective. Tutoring may also in some cases – especially when it is profit-motivated, act as a substitute for good quality teaching. This may be a serious problem in some developing countries (as an example, see this paper on Nepal), but probably much less so in developed countries. In most situations, tutoring is likely to lead to positive changes.

For those who care about helping disadvantaged students better succeed in schools, the good news is that there are many ways to contribute. If you have or can take the necessary time to do so, you can get personally involved like Don and his fellow Rotarians are doing, going every week to a school and working with a few students. But if you do not have the time, you can still help by contributing funding to organizations that are doing a great job on the ground.

STEM Education and Tutoring in the Capital City: Part 2 – Measuring Gravity

Note: This post is part of a series of three on tutoring for science and mathematics among disadvantaged students. Part 1 looks at needs. Part 2 and part 3 give examples of successful programs.

by Quentin Wodon

Innovative Tutoring

Imagine a group of elementary school students gathering in a school gymnasium as part of a tutoring session. The students are trying to estimate the gravitational acceleration force on an object at sea level, where Washington, DC, is located. The students throw a golf ball in the air in the gymnasium. They record the time it takes for the ball to fall from apogee to the floor using a simple stop watch. They repeat the exercise 25 times. They also estimate the distance from apogee to the top of the ceiling, which is done by first measuring the distance from floor to ceiling and next by guessing by how much the ball misses the ceiling. The students’ estimate of ‘g’, the gravitational acceleration due to the force exerted by the earth on the golf ball, turns out to be within three percent of the accepted value for Washington, DC, even though each of the 25 individual computations per throw varied widely. This showed to the students how approximate values, when averaged, may converge on true values with reasonable accuracy.

Two students perform the gravity experiment
Two students perform the gravity experiment

Another experiment used a hygrometer, an instrument for measuring humidity or moisture content as well as temperatures. This was coupled with water and iced water in cans. Students had to figure out the temperature at which beads of water formed on the outside of the tin cans, which was followed by a discussion of what fog is, how temperature affects relative humidity, why clouds form and sometimes rain or snow is produced.

Two students work with a hygrometer
Two students work with a hygrometer

Program Characteristics

These scenes are not from a movie, but from a volunteer-based tutoring program run in a public school located in Anacostia, the poorest area of the city. Until recently, few children at the school passed standardized mathematics and reading tests, but things have improved. The tutoring program has now been in existence for six years. It is run by Dr. Don Messer a member of the Rotary Club of Washington, DC in District 7620. Together with teachers, school administrators, and a half dozen other tutors from his Rotary club Don designed the program in an innovative way.

The program focuses on mathematics and reading, and on the types of questions asked in standardized tests. This is not to “teach to the test”, but to ensure that children understand potential test questions well. Tutors work with students in small groups of three or four to generate interactions and more learning. The groups meet once or twice a week for the entire school year. The goal is not only to help the students learn, but also to help them understand that there is a future for them that often they didn’t know existed.

Tutoring can work to improve learning – this is why so many parents who have the means to do so invest in tutoring (there is a rather large literature on private tutoring – as just one recent example see this paper on Vietnam). But children from disadvantaged backgrounds do not have such opportunities, which is why volunteer-run programs are so important for those children.

To work well, tutoring sessions should be active, varied, and even fun. Sessions should combine structured and unstructured instruction, as well as individual and collective work, and they should focus on specific skills. In Don’s small but effective volunteer program the first part of each tutoring session focuses on prior test problems from DC standardized tests. These tests are augmented by problems that tutors or teachers prepare to emphasize special themes. In mathematics for example, a package would contain around 80 problems, ranging from routine arithmetic operations to data analysis (histograms, bar charts, tables), basic geometry, and problems that require reading to make sense of what is to be done. The problem set is paced by student progress, not by a time schedule. Tutors make sure that if a problem is difficult to understand for one or more of the students, all students understand what the problem is driving at before they start to work on the problem. Students work on the problem until all have finished, but if the tutor sees that at least one student remains confused, a group discussion is launched to help the students get the correct solution. The tutors also try to interject simple science illustrations within the problems to be solved, as illustrated earlier with the gravity constant and hygrometer experiments.

Impact and Recognition

How successful has Don’s program been? No impact evaluation is available to say for sure, but success rates at standardized tests have been systematically higher for tutored than non-tutored students year after year. The results, albeit not based on a randomized study, are encouraging. In part thanks to this program, the Rotary club of Washington, DC, was recognized two years ago as Volunteer Group of the Year by Chancellor Henderson of the District of Columbia Public Schools. For the Rotarian tutors, the experience has been highly rewarding. And in Don’s case, there was no better reward than having a fifth-grader tell him: “You know Dr. Messer, you’re my grandpa.”

In the third and last post in this series, I will discuss results from several programs that operate in Washington, DC, and have been rigorously evaluated, including Higher Achievement and Reading Partners.

Note: Part of this blog post is adapted from a section in a book published by the author entitled Membership in Service Clubs: Rotary’s Experience (Palgrave Macmillan, 2014).

 

STEM Education and Tutoring in the Capital City: Part 1 – The Problem

Note: This post is part of a series of three on tutoring for science and mathematics among disadvantaged students. Part 1 looks at needs. Part 2 and part 3 give examples of successful programs.

by Quentin Wodon

Brandon was a quiet student enrolled in a primary school located in one of the poorest areas of Washington, DC, the capital city of the United States. Students in that area tend to have very low scores on standardized tests. Upon the recommendation of his teachers Brandon started to participate in the school’s tutoring program. He said little, but it was clear that he was absorbing the material being taught like a sponge. When the results from the District of Columbia’s comprehensive assessment system (DC-CAS) tests were announced, Brandon achieved proficiency in both mathematics and English. For his efforts and success, Brandon received a well-deserved award during the fifth grade graduation ceremony!

Brandon receives an award for his hard work
Brandon receives an award for his hard work

Tutoring and other supplemental education programs have received renewed attention in the United States. Under the much debated ‘No Child Left Behind’ Act adopted a dozen years ago, public schools that have not made enough progress in learning assessments for two consecutive years are in principle required to provide tutoring services to children. This makes sense given that there is scientific evidence that tutoring programs can make a difference in learning achievement if they are well implemented.

Series of Three Posts

This series of three posts on STEM (Science, Technology, Engineering, and Mathematics) education and tutoring in the capital city is written in recognition of World Science Day for Peace and Development celebrated each year on November 10. The day raises awareness of the importance of science and aims to bridge the gap between science and societies. The focus of World Science Day celebrations this year is about quality science education.

Improving science education is needed not only in developing countries, but also in developed countries, and especially so in the capital city of Washington, DC. This first post in the series documents the state of science education in the United States and in the District of Columbia. The second post will show how as individuals we can make a difference. That post will tell the story of Rotarians who have been actively involved in mathematics and science tutoring in one of the city’s schools for several years. The third post will argue that tutoring can be brought to scale and be part of the solution. That post will report on the impact of a tutoring program implemented in Washington, DC, and a few other cities by Higher Achievement.

Performance of the US

When Brandon received his award, he was enrolled in one of the worst performing public schools in Washington, DC (the schools has since made substantial progress under new management). The District of Columbia itself is one of the worst performing areas in the United States according to national assessment data. And the performance of the United States is one of the lowest among OECD and other developed countries according to international assessment data. Before talking about the potential promise of tutoring programs, providing a few statistics and basic facts about the performance of the United States, the District of Columbia, and schools within the District may be useful to underscore the magnitude of the problem we face.

Consider first the performance of the US as a nation. International comparable data on the performance of school systems in science, mathematics, and reading are available from PISA (Program for International Student Assessment). PISA measures skills for reading, mathematics and science literacy among 15 year olds. The test has been conducted every three years among a sample of students in each participating country since 2000. The latest round of data collection took place in 2012 with 65 countries participating. Results were released in December 2013.

Among 34 OECD countries, the US ranked 27th in mathematics, 17th in reading, and 20th in science, with no statistically significant improvement over time. This is despite the fact that the U.S. spends more per student than most other countries (only Austria, Luxembourg, Norway, and Switzerland spend more, but these countries do much better). More than one in four US students did not show basic mathematics proficiency on the test. The US also had a below-average share of top performers, and (not surprisingly) students from disadvantaged backgrounds performed worse on average.

Performance of the District of Columbia

Consider next the performance of the District of Columbia within the US. Comparable data on state-level performance are available from the 2013 National Assessment of Educational Progress (NAEP). Data on performance in mathematics are available in 4th and 8th grades.

Nationally, the average score for fourth-graders in mathematics was 242 in 2013. For the District of Columbia, the average was 229, the lowest score in the nation. Nationally, 83 percent of students performed at or above basic level. In the District, that share was 66 percent, again the lowest in the nation. Some 42 percent of students showed proficiency nationally, but in the District the proportion was only 28 percent. Only two states (Louisiana and Mississippi) performed worse. Gaps between the District and the nation are also large in eighth grade.

Whether those gaps are due to poor teaching or the fact that many children come from disadvantaged background is beyond the scope of this blog post (for an analysis of teacher value added in the district, see this recent paper). But whatever the reasons, the fact remains that many students in the District do not perform well.  Furthermore, within the District of Columbia, inequalities in student performance also tend to be high between the well-to-do and the less fortunate.

Mentioning this inequality in performance between groups is just another way to emphasize how beyond broad averages, for the poor the likelihood to perform well on standardized tests in the District is really low. One way to show this inequality at work is to share a little known fact about the NMSQT/PSAT test administered each year in 11th grade by the College Board. For the high school class of 2015, the District (together with New Jersey) had the highest required qualifying scores for students to become National Merit Semifinalists. Students in the District had to obtain a score of 224 out of a maximum of 240 to qualify, a much higher threshold than in many other states. This is because while many students do poorly in the Districts, a few do very well, and the threshold to become a National Merit Semifinalist is state-specific and percentage based.

To sum up, the District of Columbia tends to be at the bottom in terms of average performance in mathematics (as well as science and reading) within the United States, with the United States also faring poorly internationally. That’s the problem. In the next two posts, I will discuss part of the solution – whether tutoring could help make a difference.