The countryside 30 miles west of Milwaukee was a great place to escape to in the middle of the 20th century, and that’s just what Alfred Lunt and Lynn Fontanne wanted. The premier theater couple of that era made Ten Chimneys their home in Genesee Depot, Wisconsin. It was a secluded spot where many of the biggest stars of American and British entertainment joined them for peaceful getaways.
But in the early years of the 21st century, there are no getaways from the demands of a fast-paced world. On the same turf that Lunt and Fontanne envisioned as a retreat, school leaders see their students as upcoming members of a global economy.
“Our results are generally high, but compared to whom?” asked Patricia Deklotz, superintendent of the Kettle Moraine School District, which covers a large piece of once heavily rural land, where more subdivisions appear every year.
Kettle Moraine High School, about a half dozen miles from Ten Chimneys, aims to answer Deklotz’s question. Compared to surrounding schools? Compared to the rest of Wisconsin? How about compared to students around the globe, especially those in high-performing nations?
Kettle Moraine High is trying to make the global comparison and thereby improve student outcomes.
The vehicle is the OECD Test for Schools, a small but growing program in which samples of 15-year-old students at selected schools take a version of the Program for International Student Assessment (PISA) exam created by the Paris-based Organization for Economic Cooperation and Development (OECD). The original PISA tests were launched in 2000 and have become the most recognized basis for comparing the achievement of students across nations and, in some cases, regions or parts of nations around the globe. The tests are given to a random sample of high-school students in each “economy,” as the OECD labels them. PISA tests generally require two hours to complete the reading, math, and science portions. Questions aim to assess the critical-thinking and problem-solving abilities of students rather than specific skills. Students also answer questions about their attitudes toward learning and the learning environments at their schools, yielding insights into the strengths and weaknesses in different systems of schooling.
Starting with a trial run in 2012 that involved more than 100 U.S. schools, the OECD Test for Schools has been offered to individual American schools in an effort to provide local school administrators with an international benchmark. School-level results can be compared to those obtained by economies that administer the PISA. The campaign to enlist schools to administer the new OECD tests—and, more importantly, to make good use of the results—has been led by America Achieves, a New York‒based nonprofit that wants to “fire up” the education system to be more ambitious and effective in improving student achievement. “We need to get better faster,” Jon Schnur, the executive chairman, explained. The OECD work of America Achieves has been supported by several large foundations, including Bloomberg Philanthropies, the Kern Family Foundation (headquartered only a few miles from Kettle Moraine High School), and the William and Flora Hewlett Foundation.
Schnur said he is pleased by both the number of schools that have taken the test and by how they’re using the results. “We’ve seen hundreds of schools really actively involved in a learning community around this, eager to learn, modify what they’re doing,” he said.
Raising Expectations
Superintendent Deklotz said her district was eager to join the OECD effort from the start. “This is a very good school district,” she said. Good teachers, good kids. The challenges of urban education seem far away: only about 10 percent of the 1,300 students at Kettle Moraine High qualify for free or reduced-price lunch, and about 90 percent are white.
The percentage of Kettle Moraine students rated as proficient or advanced in reading and math on Wisconsin’s standardized tests has been consistently above the state average in recent years (see Figure 1). The average score on the ACT college admission test was above the state average in 2013‒14, when more than 96 percent of Kettle Moraine students graduated from high school in four years (in Milwaukee, the figure was 61 percent). By the second fall after graduation, 75 percent of Kettle Moraine students had enrolled in postsecondary education (compared to 44 percent in Milwaukee).
But there was a bit of a “Lake Wobegon” problem in the district, said Steve Plum, principal of Kettle Moraine High School of Health Sciences, a charter school within the larger high school. As the Prairie Home Companion radio program jokes, all the children were above average, and it was a challenge to convince some staff members, parents, and others of the need to aim higher.
The local school board was not among those needing convincing. It set a goal of having graduates meet “international expectations.” Deklotz and her team were intent on pursuing that. “I wanted to have a lever to help my staff understand the need for continuous improvement with some urgency,” Deklotz said. “I wanted to say, ‘Guys, we’re good, but we can be better.’” She didn’t want the district to go crazy over the drive to improve, but she also didn’t want people to be unduly content.
Deklotz tried to have the high school included in the 2012‒13 trial run of the OECD test. It wasn’t selected, with no reason given.
But for 2013‒14, the Kern Family Foundation, a leading funder of efforts nationwide to increase the international competitiveness of students, stepped in. Kern offered to cover the costs ($10,000 to $12,000 per school) for first-year participation by Wisconsin schools.
Jack Linehan, a retired suburban Milwaukee school superintendent, was hired to recruit schools. Linehan said it was not an easy sell. Many schools said they were having problems with “test fatigue” and did not want to take on another test, even if might have useful results, was fairly brief (about three and a half hours on one day), and involved only a sample of the school’s students (50 to 85 15-year-olds). In the end, 13 Wisconsin schools took part.
and human anatomy
As for the testing process itself, there were “bumps on the road typical of first-time testing,” Deklotz said. It was not easy to recruit Kettle Moraine High students to take the test; a sample of 85 were chosen, but only 59 actually took part. Tue Halgreen, the OECD’s project manager for the Test for Schools, said, “We would normally expect around 75 students to show up on the day of testing. If less than 75 students take the test, then the confidence intervals become larger,” increasing the amount of caution necessary in interpreting results.
Kettle Moraine administrators said some students didn’t take part because they were focused at the time on upcoming Advanced Placement tests and the OECD test carried no individual consequences for them; results are not reported for each student. Administrators also said there were indications that some students who took the test did not give it their best effort.
Nonetheless, Deklotz said the results were useful, even if they could not be stated with the same confidence a larger sample would have brought. She said the school learned a lot, both from the results on reading, math, and science and from answers to an additional set of questions asking students about such things as their engagement in learning and the learning climate in their classes.
Overall, the results supported the view that Kettle Moraine was good, but not that good, compared to global high performers. PISA describes results on a scale with six levels. In reading and science, none of the Kettle Moraine High students were on levels five or six, the top ratings. Only 10 percent placed in the top levels on math. On the other hand, few placed on level one or below; the large majorities—83 percent in reading, 81 percent in math, and 100 percent in science—ranked in the middle. Among schools in the United States participating in the OECD Test for Schools in the first year, Kettle Moraine’s scores were above average in math and science, and a bit below the middle of the pack in reading.
If the 2012 PISA tests were the comparison point, Kettle Moraine High School’s 2014 scale score in math would place it among the top five countries, while its 2014 score in reading would place it in the bottom ten (see Figure 2).
Jeff Walters, principal of Kettle Moraine High, said the results framed things clearly: “Where are we at and are we satisfied with that? And we’re not.”
Improving the Learning Culture
The results from the students’ responses to the learning culture questions on the OECD Test for Schools did provide some valuable help, Deklotz said.
For example, students were asked to what degree they agreed with the statement, “I get along with most of my teachers.” Deklotz said the expectation of teachers was that nearly 100 percent of students would agree. In higher-performing schools that administered the test, the percentage is indeed very high. But at Kettle Moraine, the 2014 result was closer to 80 percent—like the overall scores, good, but not great. Deklotz said this was a signal that the staff needed to work on building relationships with students.
A second example: students were asked how often their math classes begin long after the bell rings. About 70 percent said that was not an issue in their classes; in higher-performing schools the percentage is closer to 85 percent. “We would like to be closer to what the highest-performing schools reported,” Deklotz said.
a biomedical sciences lab
Kettle Moraine educators met before the beginning of the 2014‒15 school year to review how they were doing, including the OECD test data, and to consider how to do better. One change made for 2014‒15 was a revamped approach to using “advisory” periods, when 12 to 15 students meet with a teacher (sort of like homeroom periods of days gone by). The goal was to have more dialogue between a teacher and students over perceptions of what is going on in school and how to turn those impressions in more positive directions.
Michael Comiskey, principal of Kettle Moraine Middle School and director of math learning for the district, said that when he was a student, teachers never talked about what students were learning and why. Now, “we need to have more conversations with students about our approach,” he said. Learning needs to be more personalized to fit each student, Comiskey said, and the OECD test results help shape that.
The Kettle Moraine district has placed a big bet on improving outcomes through charter schools that serve elementary, middle school, and high school students. The charter schools generally offer more individualized and unconventionally structured programs aimed at increasing student engagement. In addition to what is call the “legacy school,” the high school building houses three charter schools: the High School of Health Sciences; KM Perform (emphasizing arts and performance); and KM Global (emphasizing individually guided, project-oriented work).
School leaders hope that the range of options will lead students to deeper, more engaged learning and higher achievement.
Weighing Costs and Benefits
The experience of Kettle Moraine and the Wisconsin schools as a whole illuminates hurdles facing advocates of wider use of the new OECD test.
Unhappiness on the part of educators, as well as students, about standardized testing, including the total load of such tests, is one big hurdle. The context includes growing opposition nationwide to standardized tests and the amount of time they take away from instruction as well as uncertainty about future federal and state accountability and testing policies. Wisconsin had an experience in spring 2015 that was rocky, at best, with its first (and, as it turns out, last) round of Common-Core testing in conjunction with its chosen test provider, the Smarter Balanced Assessment Consortium.
Leaders of many schools where the OECD Test for Schools was given in 2014 chose not to take part again in 2015. Of the 13 Wisconsin districts involved in the first year, only two (including Kettle Moraine) took part in the second year, and one other district took the test for the first time. Some may choose to do it every other year; for others, once seems like enough to get the input they want.
High-performing district Whitefish Bay, in Milwaukee’s suburbs, was one that did not participate in the second year. Maria Kucharski, director of teaching and learning for the district, said there were positive aspects to the experience. “Having an idea of where we are globally can be a lighthouse to ensure we are doing our best academically in preparing our students,” she said. “This should be done with caution, however, because we don’t view our goal of educating children to be based on achievement as much as many other skills, like critical thinking, collaboration, and a spirit and drive for entrepreneurship.” She added that school leaders liked the collaborative work with other districts that America Achieves facilitated.
But Whitefish Bay prefers tests that give information on performance of individual students and to use school time for instruction, Kucharski said. “At this point in time, Whitefish Bay does not feel that the benefits of the assessment that we identified outweigh the cost in instructional learning time,” she said.
Financial factors also are a hurdle to taking part in the OECD test. Is $11,000 a lot of money when it comes to the total budget of a high school? Many schools in Wisconsin are saying yes. State aid to schools has been reduced, and the “revenue cap” that the state imposes on the combination of aid and property tax income has gone down or stayed close to flat in recent years. The battle in 2011 between Governor Scott Walker and public employee unions, which attracted extensive national attention, produced some savings as some health insurance and retirement contributions were shifted to public employees. But the impact of the savings is wearing off, and school budgets are stressed. As much as she likes the OECD test, even Deklotz said it will be a year-to-year decision whether to take part. “While it is an expense, the data we receive back is well worth the investment,” Deklotz said. “The possibility that OECD will provide an online version at a reduced cost is very encouraging.”
America Achieves leaders say districts around the country are considering every-other-year use, although the organization recommends annual participation so that change can be monitored better.
The Kern Foundation remains supportive of the OECD initiative, but its funding was intended to pay only for first-year costs for Wisconsin schools. Ryan S. Olson, director of the K12 program for the foundation said, “There doesn’t seem to be any question that there is value in a tool such as this…. A lot of the people involved think this needs to become a movement.” As far as continuing to pay for schools to take part, he said, “Our intention was and is to have a tool that schools can use and afford that can become part of their regular system.”
America Achieves leaders say they are working with OECD in hopes of finding ways to reduce the cost of participation.
Year Two
Kettle Moraine made a few changes in the second year of the test. Administrators enlisted two groups of students: 68 students from the legacy high school and 43 students from the charter school KM Perform. (KM Perform has fewer than 150 students, and the small number of test-takers brought a warning in the OECD report that results “need to be interpreted with caution.”) School leaders put considerable energy into selling students on giving the test their best effort. Pep talks to students described the goal of the testing program and delivered the message that the students were representing Kettle Moraine in a global event. The students also received water bottles and T-shirts celebrating their participation.
Six of the students interviewed afterward said that they did their best and thought the test was better than some other standardized tests they’ve taken—more thought-provoking, with questions that more closely resembled real-world situations.
“It worked something in my brain,” said Maddi Racine, a sophomore at KM Perform. She said the questions were more in line with the way students are taught and called for critical thinking more than specific knowledge. Her comments reflect the goals of those who designed the tests.
Ethan Suhr, also a KM Perform sophomore, took the idea of being on the school “team” seriously. “We are representing something we are very passionate about being part of,” he said, referring to the charter school program.
When the results from the second year’s tests arrived in May, there were encouraging signs (see Figure 3). The scale score for Kettle Moraine High School students was particularly improved in reading, with 9 percent scoring in level five, the second-highest level. The science score also rose, but the math scores were unchanged; 9 percent of students scored in level five in science, and 15 percent scored in levels five or six in math. Average scores for students at KM Perform, the charter school, were higher than for the legacy school students.
Deklotz said the legacy school scores were above average for the United States. “This may be the result of building a culture of engagement around the assessment’s purpose and the use of results, and engaging staff in analyzing assessment results for the purpose of goal setting,” she said. She said the KM Perform test-takers gave high ratings for student engagement and student-teacher relationships, and the school’s results on these measures were commensurate with the top 10 percent of 2012 PISA test-takers in the United States.
There is no direct connection or alignment between the OECD Test for Schools and the Common Core State Standards effort that is shaping education and, in many states, roiling politics. Deklotz said her perspective was that “the shared value [of these efforts] would be increased expectation for student performance, especially over previous state assessments.” She said, “The OECD tests higher-order thinking and requires applied critical thinking and problem solving. Common Core standards raise the bar of academic expectation and application.”
About 446 schools in the U.S. have taken part in the OECD test so far. Add in Canada, Spain, and the United Kingdom, and the total worldwide is some 800 schools.
Schnur said he is encouraged by the growth of the OECD Test for Schools effort across the United States. “This tool has really taken off with a surprising degree of energy, both around the U.S. and the world,” Schnur said.
A recent report by America Achieves shows that while there is some modest progress in improving the performance of U.S. students in the lower quarter of the economic spectrum, results in the middle-income range have been flat.
“We have more than one kind of hill to climb in improving education and opportunity in the U.S.,” Schnur said. The needs of low-income, underserved, and minority children are urgent. But the proficiency rates of middle- and upper-income kids lag behind those for children in the same groupings in high-scoring nations around the world, he said. Schnur said he expects participation in the OECD test to grow and results to increasingly help schools improve.
What does Deklotz like about the OECD test overall? The perspective it provides on how Kettle Moraine students are doing, measured against the world, plus the insight into what school leaders otherwise wouldn’t know about their students, including how the students see their school experience.
The OECD results, Deklotz said, “raise lots of great questions.” But, she said, they don’t provide the answers. You have to find those yourself.
Alan J. Borsuk is senior fellow at Marquette University Law School. A longtime education reporter for the Milwaukee Journal Sentinel, he continues to write a Sunday column for the paper.
This article appeared in the Fall 2015 issue of Education Next. Suggested citation format:
Borsuk, A.J. (2015). Wisconsin High Schools Learn From New PISA Test: International comparison drives efforts to improve. Education Next, 15(4), 43-49.
