It was only ten years ago that the country adopted a standard for measuring its high school graduation rate. The US Department of Education under Secretary Margaret Spellings added a regulation to the No Child Left Behind accountability structure that states needed to report the ‘adjusted cohort graduation rate’ of their high schools.
The reported rate has been going up. This seems like a positive trend and good news, but, maybe not. Recent reports suggest some states and districts have counted students as graduates who should not have been counted. A nagging question is how much of this behavior is happening. Graduating from high school is an important education milestone, but we would like graduates to meet standards for graduation and not simply leave the system with a piece of paper and deficient skills.
The high school graduation rate is a simple concept
The ‘adjusted cohort graduation rate’ is a mouthful but it is easy to measure and consistent between states and districts. Every year, a high school has some number of students enter its ninth grade—the cohort. Four years later, that cohort has some number of its members graduating. The on-time graduation rate is the number graduating divided by the number entering ninth grade four years earlier.
The cohort of students entering ninth grade needs to be ‘adjusted’ because students transfer in or out. [1] Crucially, the regulations require that even if a student no longer appears to be attending a high school, the school cannot remove a student from the cohort unless the school receives a request for records from another school. That request is the paper trail that the student transferred to another school. The paper-trail clause prevents schools from counting students as transfers when they actually had dropped out.
In the first school year after the regulation was in force, 2010-2011, the country learned that 79 percent of its high school students graduated on time. [2] Since then, the rate has been rising, and is now 83 percent. That 17 percent of students do not graduate on time still is worrisome, but the trend seems encouraging, and some of those students graduate in five or six years.
Still, social scientists know that when it comes to numbers and accountability, Campbell’s Law needs to be kept in mind: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” [3] Campbell’s Law is cited regularly in discussions of achievement-test scores, but high schools are accountable for their on-time graduation rate, so the law would apply to it.
And, indeed, increases in the graduation rate have been followed by reports about states, districts, and schools playing games with numbers that made their graduation rates higher. Which raises a question—is the increase ‘real?’ Did the K-12 system do a better job preparing students for high school and getting them through it? Or did the system find ways to ‘help’ students graduate?
Campbell’s Law at work
Four examples consistent with Campbell’s Law emerged recently. A fifth example is a curious program known as ‘credit recovery.’ In their own way, the five examples suggest that the country has an implicit two-zone approach for high school graduation. Students in the first zone attend school, pass classes, get credits, and are given diplomas. Students in the second zone don’t attend school, fail classes, don’t have enough credits, and are given diplomas.
Alabama’s on-time graduation rate increased sharply from 72 percent in 2010-2011 to 86 percent in 2013-2014. The timing of the increase itself should have raised eyebrows. Students who graduated from high school in 2014 would have entered high school in 2010, the first year in which the requirement for the new cohort graduation rate was in place. How could high schools—about the most reform-resistant institution in K-12 education—have improved so much during the few years these students were attending them?
The Inspector General for the US Department of Education conducted an audit of how the state calculated its graduation rate for the 2013-2014 school year. [4] The IG found that the state inflated the rate in two ways. The first is that the state department of education shared its preliminary estimates of the graduation rate with local districts, who then ‘adjusted’ them. Mostly, districts removed students from cohorts for reasons the IG could not determine—there was no paper trail for many adjustments. Removing students from cohorts inflates graduation rates.
The second way the state inflated its rate is that the state superintendent at the time decided (evidently unilaterally) that the state would count an alternative diploma it awarded to special-education students as a regular diploma. The alternative diploma did not meet the state’s course requirements for a diploma, but the state superintendent said in a 2012 e-mail to district superintendents that not counting alternative diplomas ‘is just wrong.’ But counting them was contrary to the state’s own plan for calculating graduation rates it had submitted to ED and which had been approved. Alternative diplomas were not included in that plan. Including them also inflated graduation rates.
The state has not yet accounted for these alternative diplomas and submitted revised numbers. Regardless, the reported increase of its graduation rate is at least partly bogus.
In Florida, regular high schools in the Orange County school district, eighth-largest in the nation and the 2014 winner of the Broad Prize for Urban Education, created relationships with for-profit charter schools that inflated graduation rates of regular high schools, according to an investigation by ProPublica. [5] Students who were unlikely to graduate on time from a regular high school were recruited by an alternative charter school. Based on how graduation rates are calculated, described above, students who transferred to the charter school no longer counted when calculating the regular school’s graduation rate. Those transfers benefited both sides—regular high schools had higher graduation rates and test scores, and alternative charter schools received ‘management fees’ that, in the schools investigated by ProPublica, were larger than what was spent on instruction. It is common for instructional expenses to be about 70 percent of a school’s budget—having instructional expenses be less than 50 percent of a budget is atypical.
The ProPublica article noted that the Orange County district had reported 211 dropouts in 2015, a dropout rate of less than one percent for a district enrolling about 58,000 students in its regular high schools. For the country as a whole, about five percent of students dropped out of school in 2014—they stopped attending high school and had not graduated. [6] If the district had the same dropout rate as the country, it would have reported about 3,000 dropouts. Where did they go? ProPublica reported that in the five alternative charter schools it investigated, more than 1,000 students had withdrawn to enter adult education. Florida regulations did not require that alternative schools confirm that these students actually enrolled in adult education—a paper trail was not needed—so whether these students were dropping out rather than enrolling in adult education is an open question. A comparison of withdrawals to adult education and adult-education enrollments might find the missing dropouts. They will be students coded as withdrawing from the charter schools to attend adult education but not found in adult education. Florida’s Board of Education has begun taking a closer look. [7]
In June 2017, in the District of Columbia, Ballou High School announced remarkable good news. One hundred percent of its graduates had enrolled in college. For a struggling high school in one of the poorest sections of DC with a reported cohort graduation rate of 64 percent [8] to have 100 percent of its graduates enrolled in college was likely to draw attention one way or the other—either the school had done the nearly impossible and others would want to know the ingredients of its success, or something was amiss in the numbers. The local National Public Radio affiliate, WAMU, decided to take a closer look.
What it found focused on the reported graduation rate: many students who graduated had been absent more than three months of their senior year. Under district policies, even 30 days of absences should have triggered automatic failure of courses and students should have been short of credits to graduate. Two months before the date of graduation, an internal e-mail between staff in the school noted that 57 students were on track to graduate—but 164 students graduated. Speaking mostly off the record, teachers told NPR that they were pressured to pass students, such as by giving students grades of 50 percent on assignments they did not hand in, instead of the more accurate grade of zero. And some students were placed in credit-recovery programs before they had even failed the course for which credit needed to be recovered (more on credit recovery below), in violation of district policies.
In the aftermath of the NPR report, the Office of the State Superintendent commissioned an audit to examine graduation and attendance practices at all (non-charter) high schools in the district. The audit reported that 34 percent of graduates from DCPS in 2017 had irregularities that should have precluded their graduation, such as violating the attendance policy or getting credits through ‘credit recovery’ programs before having failed the course in question, which was a requirement to enter a credit recovery program (more on credit recovery programs below). [ 9]
The district responded by suspending or firing supervisory staff and pledging to do better. But the audit report makes it clear that it was the district’s culture that created the context for the fraudulent behavior. “District of Columbia Public Schools teachers and principals are subject to a variety of institutional and administrative pressures that have contributed to a culture in which passing and graduating students is expected, sometimes in contradiction to standards of academic rigor and integrity” (page 3). Which is a business-like way of saying DCPS did not care whether its students learned, but it did care whether it looked like they did.
That incident may have inspired whistleblowers in the Prince George’s County, Maryland district, which is adjacent to the District of Columbia and enrolls more than twice as many students. Members of the Maryland state legislature learned of improprieties in that district’s graduation rate, which prompted the state to engage an auditor to take a closer look. The auditor found evidence of rampant grade-changing for thousands of students in the weeks before students were to graduate. [10]
Unlike these local incidents, ‘credit recovery programs’ are a more national phenomenon. Credit recovery works like summer school except students can take the courses during the regular school year. For example, a high school student might have failed English and needs that credit to graduate. The school or district signs the student up for credit recovery. Often the courses are delivered online, and students can take them in the same school they attend. The student earns the credit and graduates. Problem solved.
Credit recovery programs are common. Nationally, the US Department of Education reports that 89 percent of high schools offered at least one credit-recovery course, and 15 percent of their students participated in some credit recovery. [11] These numbers imply that 2 million or more high school students are participating in credit recovery programs each year.
Creating second chances is sensible as long as students learn what they were supposed to have learned to pass the course they failed. But the programs have a long history of questions about whether students actually learn material or are just going through motions so that the school can say the student has passed and now can graduate. A series of pieces prepared by students at the Columbia School of Journalism described frauds and abuses in credit-recovery programs and documented a history of concerns raised about the programs in states and districts around the country. [12] In Los Angeles, which reported that 16,000 students took at least one credit recovery course in the 2016-2017 school year, a student described raising his biology grade from an F to a C in one week. [13]
What next?
Before the on-time graduation rate was introduced, the Federal government and states used a wide range of approaches to calculate graduation rates. Each had strengths and weaknesses, but their inconsistency made it difficult to see the whole picture. [14] The consistency of the on-time graduation rate is central to seeing the picture. Consistency also was the theme of a remarkable paper published in 2010 by James Heckman and Peter LaFontaine. They wove together large amounts of data from different sources and showed that America’s post-war high school graduation rate was stable for four decades. [15] If anything, their research showed that the graduation rate is hard to move. It had remained stable through wars, social trends, and a wide range of educational reforms. It began moving upward after it became part of an accountability structure, which should trigger skepticism.
Could the reported increase in the graduation rate be entirely spurious? It’s plausible. The national graduation rate of 79 percent in 2010-2011 implies about 3,275,000 students graduated from the cohort that started ninth grade in 2007-2008. The graduation rate of 83 percent in 2013-2014 implies about 3,302,000 students graduated from the cohort that started ninth grade in 2010-2011. [16] The difference in the number of graduates between these two years is about 45,000 students. When a whole state and large urban districts such as DC and Prince Georges misreport their graduates, and if the many credit-recovery programs around the country serving 2 million students are included as a possible source of misreporting, it is easy to create scenarios that get to 45,000 students. Indeed, if even a handful of the largest urban districts miscounted graduates in ways discussed here, the ‘true’ graduation rate may be falling.
Under the ‘Every Student Succeeds Act,’ states and districts are required to intervene in high schools with on-time graduation rates lower than 67 percent. That number is written in the law, but what it means to graduate is not written in that law. It is left to states to determine. And there Campbell’s Law applies.
Remarking on issues arising from his law, Donald Campbell urged that we seek out and institutionalize ‘objectivity-preserving features.’ [17] What preserves objectivity when numbers are generated by actors in the system who are accountable to them? More audits may help. Giving parents and school staff a means to report on improprieties without fear of retaliation may help. Another path is to administer a national test to high school seniors to assess whether they had attained the skills and knowledge the country deems appropriate to graduate from high school. That seems unlikely; indeed, only a few states require their high school students to take a proficiency test to graduate. And a national test hints at a national curriculum, which is touching the third rail in K-12 education.
We are at an impasse. Today’s elementary and middle school students will move on to high school, and one day they may graduate. Did they accomplish a real milestone, or were they just moved along? They are owed an answer.
— Mark Dynarski
Mark Dynarski is a Nonresident Senior Fellow at Economic Studies, Center on Children and Families, at Brookings.
This post originally appeared as part of Evidence Speaks, a weekly series of reports and notes by a standing panel of researchers under the editorship of Russ Whitehurst.
The author(s) were not paid by any entity outside of Brookings to write this particular article and did not receive financial support from or serve in a leadership position with any entity whose political or financial interests could be affected by this article.
Notes:
1. Students also can be removed from the cohort if they emigrate to another country, enter a prison or juvenile facility, or die. All of these must be confirmed in writing. Students also can be considered tp have transferred our of high school if they begin being home-schooled, as long as the homeschool program meets state requirements. https://www2.ed.gov/policy/elsec/leg/essa/essagradrateguidance.pdf
2. https://nces.ed.gov/programs/coe/pdf/Indicator_COI/COE_COI_2013_05.pdf
3. Donald T. Campbell, ‘Assessing the impact of planned social change.’ Evaluation and Program Planning, Vol. 2, Issue 1, 1979, Pages 67-90.
4. https://www2.ed.gov/about/offices/list/oig/auditreports/fy2017/a02p0010.pdf.
5. https://www.propublica.org/article/alternative-education-using-charter-schools-hide-dropouts-and-game-system
6. https://nces.ed.gov/pubs2018/2018117.pdf
7. http://www.orlandosentinel.com/features/education/school-zone/os-florida-board-alternative-graduation-rates-20170322-story.html
8. https://osse.dc.gov/sites/default/files/dc/sites/osse/publication/attachments/2016-17%20School%20Year%20Public%20ACGR%20Rates.xlsx
9. https://osse.dc.gov/sites/default/files/dc/sites/osse/release_content/attachments/Report%20on%20DCPS%20Graduation%20and%20Attendance%20Outcomes%20-%20Alvarez%26Marsal.pdf
10. https://www.washingtonpost.com/local/education/probe-finds-late-grade-changes-for-5500-in-prince-georges/2017/11/03/5e54e10c-be62-11e7-959c-fe2b598d8c00_story.html?utm_term=.b6eb57806e53
11. https://www2.ed.gov/rschstat/eval/high-school/credit-recovery.pdf
12. http://www.slate.com/articles/news_and_politics/schooled.html
13. http://www.latimes.com/local/education/la-me-edu-pass-credit-recovery-20170701-htmlstory.html
14. The National Research Council reviewed these strengths and weaknesses in its 2011 monograph: High School Dropout, Graduation, and Completion Rates: Better Data, Better Measures, Better Decisions. Washington, DC: The National Academies Press. https://doi.org/10.17226/13035.
15. James J. Heckman & Paul A. LaFontaine, 2010. “The American High School Graduation Rate: Trends and Levels,” The Review of Economics and Statistics, vol. 92(2), pages 244-262.
16. https://nces.ed.gov/programs/digest/d13/tables/dt13_203.10.asp
