Proposals for education reform generally focus on teachers and curricula. But the most important factor in education may be the student himself or herself. A growing number of states, including Georgia, Michigan, New York, and Massachusetts, have established programs that provide financial rewards in the form of merit scholarships for college for students who perform well academically. However, such programs are controversial with some educators, and the structure of many existing programs in the United States makes it difficult to evaluate rigorously the impact of such incentive programs because it is hard to identify for comparison a credible group of students who were not eligible for the program.
Examining the experiences of programs outside the United States may well be informative in helping to understand the impact of incentives for students. We collected evidence from a program in Kenya, in which girls in public schools who performed well were offered merit scholarships that covered the cost of the fees charged by public schools at the time. Their families were offered grants to help cover the cost of school supplies. The program was implemented by a nonprofit organization, which phased it into a number of schools in random order, allowing us to compare schools that were eligible for the program with other schools where the program had not yet been introduced. The results of our evaluation, conducted in 2001 and 2002, indicate that the program significantly improved the test scores of girls. Moreover, the program had salutary spillover effects: test scores of students who were not eligible for-or had no hope of earning-the award also improved, as did school attendance for both students and teachers.
A Deficit of Data on Merit Scholarships
Prior to our study, we had little information on the value of using scholarships as incentives to learn. One of the few existing sources of evidence in the United States is Georgia’s HOPE (Helping Outstanding Pupils Educationally) scholarship program, which awards college scholarships to high school students who graduate with at least a B average and attend college in-state. After the program was introduced in 1993, the average SAT score for the Peach State’s high school seniors rose almost 40 points. But since all students in the state were eligible, there was no way of determining, with a reliable degree of certainty, whether factors other than the scholarship had also contributed to the outcomes.
Outside the United States, a randomized study among high school students in Israel conducted by economists Joshua Angrist of MIT and Victor Lavy of Hebrew University found that students eligible for cash awards for good performance were 6 to 8 percentage points more likely to pass their matriculation exams than other students in the same school.
The opportunity to evaluate the Kenyan program has enabled us to address many of the problems of earlier research on merit scholarships. Because the schools participating in the Kenyan scholarship program were randomly selected, we could credibly identify its effects. We were also able to collect data on student and teacher attendance, purchases of school supplies, students’ use of time, and a range of student attitudes about school that allowed us to explore the mechanisms by which merit scholarships affect schooling. Finally, as this same region in Kenya has been the site of randomized evaluations of several other education interventions, we can compare the cost-effectiveness of the scholarship program with alternative programs such as those that provide textbooks to students or performance-based incentives for teachers.
Remarkably, we found little evidence to support the common criticisms of merit scholarships. The Kenyan program, for instance, did not appear to have led students to focus on test performance at the expense of other dimensions of learning. We also found no evidence that incentives led to better performance only during the time frame of the program, as might be expected if the improvements were due to cramming for-or cheating on-the exam; in fact, test score gains remained large in the year following the competition, and there was no increase in the frequency of extra “prep” sessions before tests. And in our surveys of students, we found no evidence that external rewards interfere with a student’s motivation to learn.
 |
| Even though only girls in public schools were offered merit scholarships (covering the cost of a year’s worth of school fees), both boys’ and girls’ test scores improved.
|
History, Math, Science, and Swahili
Like the United States, the Kenyan educational system includes eight years of primary and four years of secondary school. Though nominally free, various fees have, historically, created financial barriers to education in Kenya, as in many other developing countries. (Kenya recently abolished these fees for primary school, but at the time our evaluation was conducted, they were still in place.) Thus, while approximately 85 percent of children of primary school age in western Kenya, where our study was conducted, enrolled in school, only about one-third of all Kenyan children finished 8th grade; and only a fraction of these students enter secondary school. Because of the few slots available, getting a place at a good high school (and even, to some extent, getting a place at any high school) depends on performance on the government’s Kenya Certificate of Primary Education (KCPE) exam taken in grade 8. In preparation for the KCPE, students in grades 4-8 typically take standardized exams at the end of each school year. These tests-in English, geography and history, mathematics, science, and Swahili-are often the difference between moving on or not. More troublesome for many families: students have to pay to take the exams and to cover the cost of school supplies such as textbooks, writing materials, and the uniforms required in Kenyan schools.
Thus, when a Dutch-funded Kenyan nonprofit, International Child Support (ICS) Africa, launched a program to pay school fees for girls who scored in the top 15 percent of year-end grade 6 exams, we realized it was a wonderful opportunity to study the effects of such payments on academic achievement. The payment to the family to help cover the cost of uniforms, textbooks, writing supplies, and other school expenses, was US$12.80, a substantial amount in a region with a per capita income of only about US$350 per year. The amount set aside to pay for school fees was about US$6.50.
In March 2001, ICS Africa randomly invited half of an initial sample of 127 primary schools to participate in the Girls Scholarship Program. Each of the schools was located in one of two rural districts, Busia or Teso, in Kenya’s populated and fertile western region just north of Lake Victoria. Both districts, which cover an area about half the size of Rhode Island (more than 700 square miles), have similar asset and income profiles. In fact, they were part of the same school district until 1995. But they are very different. Busia’s 370,600 people are mainly from a Bantu-speaking ethnic group, the Luhya, who have agricultural traditions. Teso’s 192,700 people are primarily a Nilotic-speaking group, the Teso, with nomadic cattle- and goat-herding traditions. Indigenous religious beliefs and traditional taboos remain stronger in Teso than in Busia. Historically, Tesos have been educationally disadvantaged relative to the Luhya, and our survey data confirms this disparity: parents of students in Teso district have on average 0.4 years less schooling than Busia district parents.
These differences in language, history, and customs created some challenges for the program. The first came in the form of a lightning strike on the Korisai public school in Teso just as the scholarship program was being introduced. The lightning killed 7 students, injured 27, and severely damaged the school. Secondly, though Korisai was not part of the ICS Africa scholarship program, local suspicions of ICS, which was headquartered in Busia and employed mostly members of the Luhya ethnic group, did contribute to the withdrawal of 5 of the original 58 sample schools in the Teso district, as well as 1 nearby Busia school. Three of the six were “program schools” (where scholarships were offered), and three were in the comparison group.
Five additional schools-three in Teso and two in Busia-had incomplete exam scores for 2000, 2001, or 2002 and were also dropped from the evaluation, leaving 116 schools and 7,258 students in the final study sample. Overall, 51 percent of the students in our final sample were from program schools.
The scholarship program provided high-scoring grade 6 girls in participating schools with an award for the following two academic years-that is, for grades 7 and 8, through the end of primary school. In each of those years, the award comprised a grant of US$6.40, paid to the girl’s school to cover fees; a grant of US$12.80 for school supplies, paid directly to the girl’s family; and public recognition at a school awards assembly. Although there was no way to guarantee that parents spent the award money on school supplies, the fact that the money was presented to parents in a public ceremony is likely to have created community pressure on them to use the money in ways that benefited their daughter’s education.
As mentioned, ICS Africa awarded scholarships to the highest-scoring 15 percent of grade 6 girls in the program schools in each district, based on a student’s total score across the five subjects tested. The competition for scholarships therefore took place across a large number of schools and among a large number of students, not within individual schools with a fixed number of winners in each-making it less likely that the program would undermine cooperation between students within schools and classrooms. Some 57 percent of program schools (36 of 63 schools) had at least one winner in 2001 (the first year of the program), with an average of 5.6 winners in each of those schools. ICS Africa then held school assemblies-for students, parents, teachers, and local government officials-in January 2002 to announce and publicly recognize the 2001 winners. In the second round of the scholarship competition in 2002, 70 percent of the program schools (44 of 63 schools) had at least one winner. All told, 78 percent of program schools had at least one winner in either 2001 or 2002.
As expected, scholarship winners came from somewhat more advantaged families than the other students in the sample. Parents of scholarship winners, for example, had nearly three years more schooling than nonwinners (7.7 years versus 4.8 years). However, there was no notable difference between winners and nonwinners in household ownership of iron roofs or latrines, important household assets in this area.
During unannounced attendance checks in 2002, ICS Africa personnel administered questionnaires to students in grades 5-7, collecting information on school attendance, study effort, habits, and attitudes toward school. There were four such attendance checks in each school each year. We did not use attendance data from official school registers, which are often unreliable in less-developed countries like Kenya. The surveys confirmed that nearly 90 percent of eligible girls had heard of the scholarship and that the vast majority of boys correctly knew that they were ineligible for scholarships.
Test Score Impacts
Because the schools participating in the Girls Scholarship Program were chosen at random, we can reliably gauge the program’s impact by comparing their students’ test scores in 2001 and 2002 to those of students in the comparison schools. We make these comparisons using two different samples of students: those in 6th grade in 2001 and those in 6th grade in 2002. For the former sample, we can use data from end-of-year grade 5 exams in 2000-the year before the program was launched-to control for differences in performance at baseline. For the latter, we rely instead on the average test scores in their school in 2000, since individual data were not available for all these students.
We looked at the combined sample in both districts for children in grade 6 in 2001, finding that the scholarship program raised test scores significantly, by 0.12 standard deviations on average, in 2001 and 2002. Surprisingly, there was no statistically significant difference in the effect of the program on girls and its effect on boys, indicating that the program also indirectly benefited students who were not eligible for the scholarships. The pattern of effects differed dramatically, however, between Busia and Teso. The average program impact for students in Busia was a remarkable 0.19 standard deviations, while the program had no impact whatsoever on students in Teso. A gain of 0.1 standard deviation corresponds to moving from the 50th to 54th percentile of the distribution of test scores.
The effects on the test scores of students from both grades are quite similar, with positive effects of the program in Busia and none in Teso. Within this larger sample of students, however, the improvements registered by girls were roughly twice as large as those of the boys (see Figure 1).
Why the differences in results across the two districts? Through subsequent interviews with a representative sample of 64 teachers in 18 program schools, we learned of stark differences in the reception of the program across Busia and Teso, perhaps in part due to the lightning strike. For instance, when teachers were asked to rate local parental support for the program, 90 percent of the Busia teachers claimed that parents were either “very positive” or “somewhat positive” toward the scholarship program, but the analogous rate in Teso was only 58 percent. And one girl in Teso who won the ICS scholarship in 2001 actually refused to accept the reward.
Patterns of attrition from the study make determining the impact of the program difficult in Teso. Not only did fewer students in program schools than in comparison schools take the 2001 exam in Teso (53 percent versus 65 percent, respectively), but the students who were tested in program schools had lower initial (baseline) test scores on average than did the tested students in comparison schools. These differences likely led us to underestimate the effect of the program in Teso. In Busia, by contrast, attrition from the sample was much lower overall and was evenly balanced between treatment and control groups. In the following analysis, we therefore focus entirely on students from Busia district, where we are more confident that our results capture the program’s actual effects.
The effects were large and persistent. When the Busia students were in grade 7, those participating in the scholarship program scored roughly 0.2 grades higher than those not eligible to participate. As expected, girls just below the winning threshold showed the largest test-score gains. But there are also marked gains among girls with low initial scores, again indicating the spillover benefits of the program, since girls with low scores had little chance of winning. Busia boys showed similar patterns, although gains at the top of the baseline test distribution are somewhat more pronounced for boys than for girls.
The performance differences between students in program and comparison schools continued into 2002, despite the fact that the original cohort of girls was now in 7th grade and no longer competing for a scholarship. This persistence suggests that the test score gains were not due to cramming or cheating, but the result of lasting learning effects of the program.
Collateral Advantages
It is also useful to explore potential channels for test-score gains due to the cash incentives, since some behavioral changes, such as increased coaching or cramming, might raise test scores without improving actual learning.
Perhaps the most interesting finding in this regard was that the scholarship program increased student attendance by as much as 5 percentage points for both girls and boys in Busia, as measured by the unannounced visits in 2001 and 2002-a reduction in absenteeism of almost one-third (see Figure 2). The improvements in attendance were evenly distributed across the school year, so there is no evidence that gains were due to cramming in the run-up to exams.
The large attendance gains for boys in Busia immediately suggest that increases in effort were not simply investments made to increase the chance of winning an award. So too the increased attendance among program-school girls with low initial test scores-indicating little chance of winning an award.
Indirect impacts seem almost as important as direct ones. For example, absentee rates among teachers in Busia also fell-by more than 6 percentage points, or roughly one-third, an effect as large as the attendance gain among students. This finding provides a plausible explanation for the positive ancillary benefits experienced by boys in Busia program schools, namely greater teacher effort directed to the class as a whole.
We found no evidence of the adverse changes in students’ attitudes that some psychologists warn will result from using material rewards to encourage learning. We asked students a series of eight questions centered on the subject of how much they liked a school activity-doing homework, for instance-relative to a nonschool activity, such as fetching water or playing sports. Overall, students preferred the school activity 72 percent of the time, with no statistically significant differences between program and comparison schools or between girls and boys. This suggests that external incentives did not dampen students’ intrinsic motivation to learn.
Finally, when we compared the cost-effectiveness of six primary school interventions that had recently been evaluated in the Kenyan study area-the merit scholarship program that is the focus of this paper, a teacher incentive program, a textbook provision program, a flip-chart program, a deworming program, and a child sponsorship program that provided a range of education resources, including free uniforms-we found that cash incentives were a highly cost-effective way to improve both test scores and attendance.
The average test-score gain in merit scholarship program schools, for both female and male students in Busia and Teso districts, in both years of the program, was roughly 0.12 standard deviations, while the comparable gains for schools participating in the teacher incentive program over two years was just 0.07 standard deviations, and for textbook program schools the average gain was only 0.04 standard deviations. (The deworming, flip-chart, and child sponsorship programs did not produce statistically significant effects on test scores.)
As to the per pupil cost of increasing test scores by 0.1 standard deviation, the Girls Scholarship Program cost wasUS$1.41, while the teacher incentive program cost US$1.36. Costs of raising scores using the textbook program, however, were much higher, at US$5.61. (See Figure 3.) In Busia, where the girls’ scholarship program was well received, the per pupil cost of the program fell to US$0.75, making student merit awards a much more cost-effective way to boost student scores than all the other programs. Though the provision of inexpensive deworming medicine proved by far the most cost-effective way of increasing student attendance-US$3.50 per additional year of school participation compared with US$90 for the Girls Scholarship Program-merit scholarships were the second most cost-effective way of improving attendance.
Winning Margins
While merit-based scholarships have historically played an important part of the education finance system in many countries, we never knew if-or how-they worked. Evidence from Kenya now suggests that such programs can induce extra effort to raise test scores. Moreover, the benefits seem to spill over to ineligible students and to encourage more effort by teachers as well. All this suggests that boosting study effort among students improves the learning environment in the classroom.
A common reservation about merit-based scholarships is the possibility of adverse equity effects, and it is likely that advantaged students, specifically those whose parents had more schooling, benefited most from the program we studied. Still, groups with little chance at winning an award, including girls with low initial test scores, gained enough academically from the merit scholarship program to make it cost-effective for them, too.
One way to spread the benefits of merit scholarships even more widely would be to restrict the scholarship competition to needy schools, regions, or populations, or, alternatively, to conduct multiple competitions, each restricted to a small geographic area. For instance, if each Kenyan location (a small administrative unit) awarded merit scholarships to its residents independently of other locations, children would compete only against others who live in the same area, where many households live in broadly comparable socioeconomic conditions. This would allow children even in disadvantaged areas to have a reasonable shot at an award. To the extent that such a policy would put more students near the margin of winning a scholarship, it would presumably generate greater incentive effects and spillover benefits like those we found.
The important lesson from Kenya, though, is that while most education research focuses on the effect of material inputs such as class size or of school organization on student outcomes, this work suggests that the most important input in the education production function may very well be the study effort on the part of the child-effort that may be significantly enhanced by the promise of material rewards.
Michael Kremer is professor, department of economics, Harvard University. Edward Miguel is assistant professor, department of economics, University of California, Berkeley. Rebecca Thornton is a Ph.D. candidate in the department of economics, Harvard University.
The unabridged version of this article may be found at www.educationnext.org.
