Do the NCTQ Rankings Identify Schools of Education that Produce Graduates Who Are Effective in the Classroom?
The National Council on Teacher Quality, in conjunction with U. S. News and World Report, has issued an ambitious report evaluating the quality of teacher preparation programs in schools of education across the United States. But its critics argue that the report fails to show that its measure of program quality is correlated with the classroom effectiveness of a school’s graduates. If the information available to us for a few teacher preparation schools in Florida is at all representative, the critics may have a point.
NCTQ’s composite measure of the quality of teacher preparation focuses on the characteristics of the training program itself. Its measure includes, among other factors, the selectiveness of those admitted into the program, content of courses as revealed by course syllabi, extent of instruction in lesson planning and classroom management, and the availability of practical training in the instruction of disadvantaged students. Although it also says that it measures whether or not “the program’s graduates have a positive impact on student learning,” the specifics as to how that measurement was made are not provided.
Using these critieria, experts rank over 608 of the 1,130 institutions that “prepare 99 percent of the nation’s traditionally trained new teachers” on a five-point scale that awarded education schools anywhere from zero to four stars. Two major findings are quite disturbing. First, many of the nation’s schools of education were unwilling to “share data with NCTQ,” a shocking display of willful non-transparency. Even worse, the study team concluded that the education professoriate “rejected any notion that its role is to train the next generation of teachers.” In a separate Education Next essay written by NCTQ director Kate Walsh, she says ed schools view their mission as being to help “form” teacher thinking rather than to “train” them in techniques useful for classroom management and lesson preparation. By calling public attention to the problems in teacher training across the country, NCTQ has provided a valuable public service.
Given our enthusiasm for NCTQ’s efforts, we were curious to discover whether its ranking of teacher preparation programs was consistent with the estimates of preparation effectiveness of selected education schools in Florida that Matthew Chingos and I reported in a 2011 article published in the Economics of Education Review. Our study gathered no information about the teacher education programs themselves, but did examine carefully the classroom effectiveness of graduates who taught within Florida. For our investigation, we used individual test-score information on the Florida state assessments in math and reading that are available for as many as 500,000 Florida public-school student observations in grades four through eight for the eight years 2002 to 2009. The database includes information about each student’s classroom teacher in a given year, which allows us to estimate how much the student learned in that year and to connect that information to such professional characteristics as teacher certification, acquisition of a master’s degree, teacher experience, teacher test performance, and the specific school of education the teacher had attended within Florida, if the teacher had attended one of the eleven schools for which adequate numbers of teacher observations were available. In estimating teacher effectiveness we controlled for many other factors, including student demographic characteristics, student test performance the previous year, the quality of their classroom peers, and the fixed effects of the school the students attended, whether the teacher had an M. A. degree, years of experience in the classroom, and many other factors.
NCTQ released evaluations for six of the eleven elementary teacher training programs for which we have estimates of effectiveness. None of the six earned either earned three or four stars (which were given to only 9 percent of all teacher preparation programs across the country). NCTQ nonetheless detected a noticeable difference in teacher preparation quality among the six schools. St. Petersburg College was not given any star at all and was placed on NCTQ’s endangered list, created to warn potential teachers away from the school. Florida Atlantic was given only one star; Florida International and the University of West Florida were given one and a half stars; Florida Gulf Coast University was given 2 stars; and Florida State was given 2 and a half stars, the highest rating among the six institutions..
When we measured the actual effectiveness of graduates from the six programs, we could not discern any statistically significant differences among the six. However, the graduates of “endangered” St. Petersburg lifted average student math performance by 1.6 to 1.8 percent of a standard deviation higher during the course of a year than did the graduates of top-ranked Florida State. In fact, graduates of St. Petersburg did better than any other of the six teacher training programs. The lowest-performing institutions at training graduates for math instruction were the University of West Florida and Florida Gulf Coast University. Still, none of these differences were statistically significant.
At reading instruction, graduates of St. Petersburg also outperformed graduates of Florida State–by nearly 2 percent of a standard deviation over the course of a year, on average. Florida Gulf Coast University graduates did very slightly better than St. Petersburg graduates, while the remaining schools very closely resembled Florida State. Once again, none of these differences were statistically significant, so we cannot be sure that they did not occur simply by chance.
At the master’s degree level, the University of Central Florida was found by NCTQ to be the weak sister among the four institutions evaluated. But in our data set it was the graduates of Florida Atlantic who were significantly less effective at teaching reading to students in sixth through eighth grades. Student performance in the classes of its graduates trailed by 6 percent of a standard deviation those of the graduates of the University of Central Florida. In math the graduates of the University of Florida, the state’s premier university, outperformed the other institutions at teaching students in fourth to eighth grade by as much as 10 percent of a standard deviation, even though NCTQ gave it no better rating than Florida State or Florida Atlantic.
Florida is only one state in the union, and our data set overlaps that of NCTQ for only a few institutions. Generalizing from these few cases is risky. But unfortunately, we could not demonstrate that the ranking scheme NCTQ employs correlates with the effectiveness of the graduates of specific schools of education. At least in Florida, NCTQ’s experts do not seem to have nailed it.
—Paul E. Peterson