If this sounds alien to you, clearly you haven’t signed up for Twitter. This five-year-old phenomenon allows individuals to dash off short comments to their friends, families, professional colleagues, and whoever else might be interested in their stream of consciousness. The technology has already been credited with bringing down oppressive regimes and creating whole new ways of reporting breaking news. It’s a truly open marketplace of ideas, with no editors, gatekeepers, or quality control. So what does it mean for the education debate?

The first thing to understand about Twitter is that most of its messages amount to, “Hey, check this out,” followed by a link to a newspaper article or blog post. It’s a handy device for telling the world (or at least the people in your own world) about news or columns that you find compelling. It’s also a form of self-promotion; quite a few tweets announce posts the tweeter herself has written.

But in the hands of a gifted provocateur, Twitter can be so much more. Take scholar-turned-reform-apostate Diave Ravitch, who according to Klout.com is the most influential tweeter in the education policy space (see sidebar). As Alexander Russo, a freelance writer and blogger, remarked sardonically, “a 72-year-old grandmother has won the Internet.” She’s done it not only by linking to columns and articles she agrees with, but by offering bumper sticker–style statements that tend to set the web aflame. For instance, “Accountability is only for teachers and principals, not for students, families, elected officials, district leadership.” Or: “Last places to go to find out how to ‘reform’ schools: Congress/State Legislature/US Dept of Education.”

Want to follow the top tweeters in education?
Twitter lists made up of the Top 25 Education Policy/Media Tweeters and the
Top 25 Education Tweeters may be found at the Education Next Twitter page.

This might not exactly be H. L. Mencken, but it surely provides raw emotional relief for educators and others who feel besieged by the modern-day reform movement. They “retweet” Ravitch’s rants and, thanks to the multiplication effects of networks, soon tens of thousands of people receive them. In fact, Ravitch’s tweets are so influential that an anonymous someone has created the Twitter handle “@NOTDianeRavitch” to argue the positions held by the education historian before she changed her mind on most education policy issues.
Not that reformers don’t have their own Twitter heroes. Former District of Columbia schools chancellor Michelle Rhee is within striking distance of Ravitch’s influence and serves up a steady diet of can-do reform truisms. Tom Vander Ark, an entrepreneur formerly of the Bill and Melinda Gates Foundation, offers an optimistic take on the burgeoning field of online learning. Secretary of Education Arne Duncan promotes his administration’s policies via @arneduncan. And @EdTrust offers its patented progressive take on education and social justice.

It’s hard to know whether all this tweeting adds up to anything significant. Of course, much the same was once said of blogs; now it’s well-accepted that a well-written blog post can be just as influential as a newspaper op-ed. Twitter offers a nonstop stream of views, ideas, opinions, and emotions; get yourself in the flow or be left behind.

It’s an unseasonably warm winter morning in San Jose, California, and the two dozen students at Rocketship Mateo Sheedy Elementary School get situated quickly in the computer lab, donning headphones and peering into monitors displaying their names. The kindergartners follow a moment later, until 43 seats are filled. The effect is of a miniature, and improbably enthusiastic, call center.

This lab—and the larger plan for the school surrounding it—has probably done more than any other single place to create enthusiasm for “hybrid schools.” Such schools combine “face-to-face” education in a specific place (what used to be called “school”) with online instruction. (Rocketship uses the term “hybrid,” rather than the increasingly prevalent term “blended learning,” because the computers are not actually “blended” with face-to-face instruction in the same classroom.) It’s a sign of how young the hybrid and blended field is that this school at the epicenter hails all the way back to 2007. Rocketship Education, a small but burgeoning network of charter schools that serves an overwhelmingly low-income immigrant community in San Jose, has made a name through its, forgive the phrase, high-flying student performance. Two of its three schools are old enough to have test scores. They rank among the 15 top-performing high-poverty schools statewide, and the site that opened in 2009 was the number-one first-year school in the state in the high-poverty category. But what positions Rocketship on the cutting edge of school reform is its vision for how technology will integrate with, and change, the structure of the school. (Disclosure: Our firm, NewSchools Venture Fund, is a significant investor in the work of Rocketship and of several other organizations mentioned in this article.)

The scene in the computer lab represents the first steps toward realizing the Rocketship vision. In the lab, the 1st graders log in by selecting from a group of images that acts as a personal password, and then race through a short assessment that covers math and reading problems. Faced with the prompt “Put all the striped balls in one basket and all the polka-dotted balls in the other basket,” a student named Jazmine uses her mouse to move the objects to their places. Then it’s on to the core activity of her 90 minutes in the lab: a lesson on counting and grouping using software from DreamBox. The scenarios are slightly surreal—more objects to move, in this case mostly fruit, and the reward for getting it right involves an animated monkey bringing yet more fruit to a stash on her island—but she and most other students take on the task assiduously. It may be a lesson, but that’s not how Jazmine sees it. “This game is really easy,” she says. A bit later, she’ll read a book from a box targeted at her exact reading level, and make a return visit to the computer to take a short quiz about what she read.

Despite the kids’ engagement in the online lesson, no one is claiming that time in front of the computer is directly responsible for the extraordinary performance of Rocketship students. Rather, the online work is essential to the long-term vision for the school’s instructional model—and for Rocketship’s growth trajectory. Crucially, the lab requires an adult who has experience with children, but no teaching credential (nor, indeed, bachelor’s degree) is required. For this class, it’s a young mother named Coral De Dios, who dispenses help and order as the moment requires. Her ability to monitor the 43 kids here means that the school requires less staff, ultimately saving hundreds of thousands of dollars each year that can be plowed back into resources for the school, including staff salaries. In cash-strapped California, that’s no small matter.

But the larger impact of the technology is still ahead, in the ways it will integrate with, and alter, classroom practice. Rocketship is building a model in which kids learn much of their basic skills via adaptive technology like the DreamBox software, leaving classroom teachers free to focus on critical-thinking instruction and extra help where kids are struggling. Likewise, teachers will be able to “prescribe” online attention to specific skills. Part of the model involves providing teachers with a steady stream of data that will help them adjust instruction to kids’ specific needs, and to guide afterschool tutors. Today, those linkages between the computer lab and the classroom remain incomplete, in part because the data from various online systems aren’t sufficiently standardized; the many data points from different systems could be overwhelming to teachers.

Rocketship has probably done more than any other single place to create the market for “hybrid schools.”

Rocketship’s data guru, Charlie Bufalino, says that to date, vendors haven’t invested sufficiently in the R&D and technical fixes that would make a standardized stream of data possible and take menial tasks like attendance out of teachers’ hands. As more schools like Rocketship build hybrid and blended systems, however, and as more entrepreneurs develop the missing-piece systems, the tipping point may be reached, fueling rapid growth of this new approach to schooling.

Rocketship and the other school models we describe here offer a vision for what deeply integrated technology can mean for children’s education, for the way schools are structured, and for the promise of greater efficiency amid a lengthy economic downturn. This is much more than simply taking a class online. Already, millions of children take one or more online courses, ranging from credit recovery to Advanced Placement. And there’s a wide range of ways that the school facility and online learning—“bricks and clicks”—mix. (Michael B. Horn and Heather Staker offer an excellent guide to the landscape in their recent paper, “The Rise of K–12 Blended Learning.”) Our interest is specifically in schools and platforms that use technology intensively and thoughtfully to tailor instruction to individual students’ needs, and provide robust, frequent data on their performance. Most of our examples are high-performing charter schools, which have become a particular hotbed for the type of hybrid and blended models we are describing. Their designs call for bringing new productivity to the way schools deploy staff and dollars. They all share an ambition to prepare their students for success not just on tests, but in college.

Visitors need only to walk into School of One’s classroom space to see what customized education looks like.

School of One

Much of the enthusiasm for the potential of blended learning comes from what is currently a math program. School of One, operating inside three New York City public middle schools, is an exciting experiment interweaving a wide range of online learning possibilities with classroom instruction. Indeed, a visitor needs only to walk into School of One’s classroom space at Intermediate School 228 in Brooklyn to see what customized education looks like. The classroom is an open space that runs the length of the building wing, but is subdivided by bookshelves into workspaces where small groups of students work with the teacher or individually with laptops. The first sight that greets the eye is an airport-style video display, listing not cities and flights, but students’ names and how they will receive their instruction during that period. For those who are starting on the computer, a press of a button will take them to a lesson provided by 1 of more than 50 content providers. Each lesson runs about half an hour, and students may switch from one content provider to another on the same skill. Others work in small groups with a teacher, who will typically oversee two or three groups of students, the content and groupings informed by data from the student’s work online.

“You understand way better,” says Edwin, a 12-year-old 7th grader clad in basketball-ready dark blue T-shirt, shorts, and athletic shoes. Thanks to the unusual structure of math classes at School of One, he says, teachers work with only 9 or 10 students at one time, while at other schools, “the teacher doesn’t have time to go over things with every student.” He adds, “It’s a really good program for kids who have trouble with math.”

Behind the flashy images on the laptop screens, the real power behind School of One is in its brawny “back end” systems, which enable the creation of real-time, hourly reports of students’ progress and shortfalls. Teachers review these reports daily, both individually and in a collaborative planning period when they discuss the progress of individual students as well as student groups. Teachers can review the information before school, after school, during their prep period, or even while they are overseeing instruction (so they can identify the students in a group who, according to previous assessment data, may be struggling to learn a skill). “We get data every single day to help us understand what’s working and what’s not,” says founder Joel Rose. When a student struggles on Tuesday, she can be assigned to a small group for help from a teacher on Wednesday, and with enough data and enough flexibility, it will even be possible to assign her to a teacher who is particularly good at teaching that lesson. It’s a model that seems certain to make us question assumptions about how we organize classrooms and schools.

Like the teachers, students can see a map of their accomplishments. That map is tied to state standards and will later align with the Common Core standards. As at Rocketship, aligning lessons to these standards is no small matter; School of One had veteran math teachers codify the precursors and dependencies for each skill. They sourced more than 25,000 lessons for middle-school math, from which they chose the top 5,000. Many lessons were not included because they did not closely align to their map. School of One has enough faith in the power of its standards and assessments that it will soon offer students the option to press a “prove it” button that allows them to demonstrate mastery at an upcoming task and, if successful, skip it. The button stands as a testament to a core notion of the blended idea: learning that proceeds at a pace the student is ready for, rather than one set by the needs of an entire class.

Technology is everywhere as one strolls through DSST’s Stapleton campus in northeast Denver, just barely within sight of peaks of the Rocky Mountains.

DSST Public Schools

The constant, real-time stream of student assessment data is a crucial element of the most promising tech-enabled schools, including some high flyers that don’t fit neatly under the blended label. One of the most interesting is charter school network DSST Public Schools, named for its flagship, the Denver School of Science and Technology. DSST enrolls a mostly-minority, 47 percent low-income student population and has achieved national renown for its extraordinary results, including the second-highest longitudinal growth rate in student test scores statewide. Among graduates, 100 percent have been accepted to four-year colleges, where an astonishing 1 percent require remedial courses, in comparison to 56 percent for the Denver district. Technology is everywhere as one strolls through DSST’s Stapleton campus in northeast Denver, just barely within sight of peaks of the Front Range of the Rocky Mountains. In a 6th-grade social studies class recently, students used collaborative user-made web sites called wikis to access and respond to in-class and homework assignments. The teacher projected a map of Asia and posted prompts on the wiki for students to respond to as they learned about the geography of the region.

DSST has achieved national renown for its extraordinary results, including the second-highest longitudinal growth rate in student test scores statewide.

DSST’s assessment system provides real-time, instant feedback to teachers and students on students’ progress, measured through quick assessments that students take on netbooks. Teachers at DSST have been developing these informal assessments and in the 2010–11 school year are working with a consultant to review the validity of the assessment items and gather feedback that will in turn make teachers better item writers. The data enable teachers to differentiate instruction and connect instructional strategies with student results. As at School of One, both teachers and students at DSST can track mastery on a particular standard. Teachers can quickly adjust groups and/or identify topics for re-teaching. Through these assessments and classroom observations, teachers identify students in need of extra support, who are then assigned to afterschool tutoring the same day. Teachers use the information to plan lessons, deciding whether to spend more class time on a certain area or focus on individual tutoring based on class scores. DSST is also using data to analyze teacher performance. “The technology enables us to collect good data on our school performance, which is used to drive and motivate student achievement,” says founder and CEO Bill Kurtz. “We believe that education innovation will be driven by common data.”

Carpe Diem Collegiate High School

Elsewhere in the charter universe, schools are incorporating hybrid and blended structures into already successful school organizations, which increasingly seek efficiency, even as they expand and work to maintain excellent student achievement. The impact has been dramatic, for example, at Carpe Diem Collegiate High School of Yuma, Arizona. Carpe Diem represents what will likely be a crucial chapter in the story of blended schools: a turn to a blended model because of financial or facilities challenges. The charter school, which serves 250 mostly low-income students in 6th through 12th grades, faced a crisis after losing its lease on a church building. Its founders radically transformed it from a traditional structure to one heavily dependent on online instruction, and in 2006 completed a facility tailored to the new model. In the reinvented school, small groups take classes directly from teachers, while most students take online classes in a learning center that features 300 low-sided cubicles in one brightly painted room. Student cubicles have a desktop computer and monitor; many have been personalized and decorated with artwork. The learning center is staffed by the principal, two instructional assistants, and a course manager, who also talks with students about their progress.

Students begin their day by logging onto a software system called e2020 and accessing the calendar, selecting a subject area, and looking at their lists of assignments for the week. On any given day, based on the data, teachers may gather an entire grade or a subset of students, sometimes in groups as small as one or two. Some students work through all subjects each day, while others focus on math for the week on one day, science for the week on another day. Carpe Diem has been a state leader in student growth for the past two years.

Hallways at High Tech High are lined by prizewinning robotics projects.

High Tech High

Yet, even in schools that have been aggressive in incorporating technology, there is such a thing as too much in adopting blended approaches. Such is the case at High Tech High, whose campus near the San Diego airport is perhaps the most eye-poppingly technology-rich in the country. Rooms within the warehouse-sized buildings are delineated with glass walls 15 feet high, leaving the remaining space under the 25-foot ceilings for a chaotic crisscross of air ducts, structural supports, and wires. Mixed-media art hangs from every wall, door, and metal roof beam, and gee-whiz technology is everywhere. Students use the same computer-aided design systems that they would find in a professional design firm as they model real-life, design-forward chairs. The hallways are lined by prize-winning robotics projects. And outside, students further their studies of air pressure by racing hovercraft they have designed using large circles of plywood with plastic-bag cushion edges and leaf-blower engines.

High Tech High has taken gentle steps into blended territory through its use of ALEKS, which bills itself as “a Web-based, artificially intelligent assessment and learning system.” ALEKS, which runs on computers on the periphery of a 9th-grade classroom, provides teachers with detailed diagnostics, helping them to focus on the areas where students are struggling, and lets students take lessons at their own pace. A student logs on to ALEKS and begins by taking an adaptive assessment, each question chosen on the basis of previous answers. With this information, ALEKS develops a snapshot of a student’s knowledge in a given content area, recognizing which topics he has mastered and which he has not. This information is represented for both the student and teacher by a multicolored pie chart, which is constantly being updated as the student masters new topics. Once a student has mastered a specific topic, new ones become available for the student to choose from. “It doesn’t slow you down,” says Danie, a 15-year-old boy with a dark mop of hair that he regularly brushes off his forehead. Danie, wearing untied high-tops and faded black jeans, confesses matter-of-factly that he is repeating the 9th grade. “Students learn at different speeds,” he says with marked confidence. He hastens to add that the technology augments, rather than replaces, the teacher. “Nothing,” he says, “can replace human interaction.” Danie’s teacher, Jane Armstrong, agrees, saying ALEKS gives her more flexibility in grouping students. Today, Armstrong has divided the class in two. Half of the students are using ALEKS while Armstrong is working with the other half in small groups. “This setting allows me to get to know all of my students,” she says. “If I’m just lecturing them, I don’t get to know what they’ve mastered.”

California’s budget situation today is nothing short of disastrous. Yet High Tech High recently rejected a much more aggressive move into the blended field, a “flex” plan that would have brought students to campus only once a week, with the other four days spent online, typically from home. The plan would have created enormous cost savings by allowing five different cohorts of students to use one building each week. Yet teachers, students, and parents rejected the idea of giving up the daily campus experience, and teachers were not enthusiastic about doing a large proportion of their teaching online. “We’re not drinking the Kool-Aid,” said founder and CEO Larry Rosenstock.

High Tech High is perhaps the most eye-poppingly technology-rich charter school in the country. Mixed-media art hangs from every wall, door, and metal roof beam, and gee-whiz technology is everywhere.

On the Verge

Indeed, it seems likely that, just as happened with charter management organizations, rapid growth will take place only when the pioneers can demonstrate proof points of excellence in student performance. “In order for there to be larger market traction, the overall industry has to see more results,” says Anthony Kim of Education Elements, a nascent firm that designs the technical back end for blended schools. “We’re at the very early adopter stage right now.”

Blended schooling is dawning at a time when, as recent public opinion polls show, people are open to online learning. According to the 2010 EdNext-PEPG Survey (“Meeting of the Minds,” features, Winter 2011), support for online coursework jumped 8 to 10 percent in a single year. Yet as much as anything, the blended effort is being driven by a new fiscal reality. In a widely regarded speech at the American Enterprise Institute called “The New Normal: Doing More with Less,” education secretary Arne Duncan noted that a loss of housing valuation meant that education funds are down sharply and aren’t coming back anytime soon. In the spirit of never wasting a crisis, he said he hoped the difficult financial straits would help bring an end to “the factory model of education” and an increase in productivity in schools. He said, “Our schools must prepare all students for college and careers—and do far more to personalize instruction and employ the smart use of technology.”

Is the blended school the model he’s looking for? Tom Vander Ark, a former head of education for the Bill & Melinda Gates Foundation and now a partner in a private equity fund focused on education innovation, thinks so. In the past, technology actually made schooling more expensive, as computers were layered onto an existing model without adding any efficiency. Technology-driven productivity, he says, stands to change that. “We can make learning far more productive,” says Vander Ark. “It’s the first chance in history to change the curve.”

Jonathan Schorr and Deborah McGriff are partners at NewSchools Venture Fund, a nonprofit venture philanthropy firm that supports entrepreneurial innovation to improve public education for low-income children.

Additional photographs of the hybrid schools are available here.

The ubiquity of “satisfactory” ratings stands in contrast to a rapidly growing body of research that examines differences in teachers’ effectiveness at raising student achievement. In recent years, school districts and states have compiled datasets that make it possible to track the achievement of individual students from one year to the next, and to compare the progress made by similar students assigned to different teachers. Careful statistical analysis of these new datasets confirms the long-held intuition of most teachers, students, and parents: teachers vary substantially in their ability to promote student achievement growth.

The quantification of differences has generated a flurry of policy proposals to promote teacher quality over the past decade, and the Obama administration’s recent Race to the Top program only accelerated interest. Yet, so far, little has changed in the way that teachers are evaluated, in the content of pre-service training, or in the types of professional development offered. A primary stumbling block has been a lack of agreement on how best to identify and measure effective teaching.

A handful of school districts and states—including Dallas, Houston, Denver, New York, and Washington, D.C.—have begun using student achievement gains as indicated by annual test scores (adjusted for prior achievement and other student characteristics) as a direct measure of individual teacher performance. These student-test-based measures are often referred to as “value-added” measures. Yet even supporters of policies that make use of value-added measures recognize the limitations of those measures. Among the limitations are, first, that these performance measures can only be generated in the handful of grades and subjects in which there is mandated annual testing. Roughly one-quarter of K–12 teachers typically teach in grades and subjects where obtaining such measures is currently possible. Second, test-based measures by themselves offer little guidance for redesigning teacher training or targeting professional development; they allow one to identify particularly effective teachers, but not to determine the specific practices responsible for their success. Third, there is the danger that a reliance on test-based measures will lead teachers to focus narrowly on test-taking skills at the cost of more valuable academic content, especially if administrators do not provide them with clear and proven ways to improve their practice.

Student-test-based measures of teacher performance are receiving increasing attention in part because there are, as yet, few complementary or alternative measures that can provide reliable and valid information on the effectiveness of a teacher’s classroom practice. The approach most commonly in use is to evaluate effectiveness through direct observation of teachers in the act of teaching. But as “The Widget Effect” reports, such evaluations are a largely perfunctory exercise.

In this article, we report a few results from an ongoing study of teacher classroom observation in the Cincinnati Public Schools. The motivating research question was whether classroom observations—when performed by trained professionals external to the school, using an extensive set of standards—could identify teaching practices likely to raise achievement.

We find that evaluations based on well-executed classroom observations do identify effective teachers and teaching practices. Teachers’ scores on the classroom observation components of Cincinnati’s evaluation system reliably predict the achievement gains made by their students in both math and reading. These findings support the idea that teacher evaluation systems need not be based on test scores alone in order to provide useful information about which teachers are most effective in raising student achievement.

The Cincinnati Evaluation System

Jointly developed by the local teachers union and district more than a decade ago, the Cincinnati Public Schools’ Teacher Evaluation System (TES) is often cited as a rare example of a high-quality evaluation program based on classroom observations. At a minimum, it is a system to which the district has devoted considerable resources. During the yearlong TES process, teachers are typically observed and scored four times: three times by a peer evaluator external to the school and once by a local school administrator. The peer evaluators are experienced classroom teachers chosen partly based on their own TES performance. They serve as full-time evaluators for three years before they return to the classroom. Both peer evaluators and administrators must complete an intensive training course and accurately score videotaped teaching examples.

The system requires that all new teachers participate in TES during their first year in the district, again to receive tenure (usually in their fourth year), and every fifth year thereafter. Teachers tenured before 2000–01 were gradually phased into the five-year rotation. Additionally, teachers may volunteer to be evaluated; most volunteers do so to post the high scores necessary to apply for selective positions in the district (for example, lead teacher or TES evaluator).

The TES scoring rubric used by the evaluators, which is based on the work of educator Charlotte Danielson, describes the practices, skills, and characteristics that effective teachers should possess and employ. We focus our analysis on the two (out of four total) domains of TES evaluations that directly address classroom practices: “Creating an Environment for Student Learning” and “Teaching for Student Learning.” (The other two TES domains assess teachers’ planning and professional contributions outside of the classroom; scores in these areas are based on lesson plans and other documents included in a portfolio reviewed by evaluators.) These two domains, with scores based on classroom observations, contain more than two dozen specific elements of practice that are grouped into eight “standards” of teaching. Table 1 provides an example of two elements that comprise one standard. For each element, the rubric provides language describing what performance looks like at each scoring level: Distinguished (a score of 4), Proficient (3), Basic (2), or Unsatisfactory (1).

Data and Methodology

Cincinnati provided us with records of each classroom observation conducted between the 2000–01 and 2008–09 school years, including the scores that evaluators assigned for each specific practice element as a result of that observation. Using these data, we calculated a score for each teacher on the eight TES “standards” by averaging the ratings assigned during the different observations of that teacher in a given year on each element included under the standard. We then collapsed these eight standard-level scores into three summary indexes that measure different aspects of a teacher’s practice:

• The first, which we call Overall Classroom Practices, is simply the teacher’s average score across all eight standards. This index captures the general importance of the full set of teaching practices measured by the evaluation.

• The second, Classroom Management vs. Instructional Practices, measures the difference in a teacher’s rating on standards that evaluate classroom management and that same teacher’s rating on standards that assess instructional practices. A teacher who is more skilled at managing the classroom environment, as compared to her ability to engage in desired instructional activities, will receive a higher score on this index than a teacher who engages in these instructional practices but who is less skilled at managing the classroom.

• The third, Questions/Discussion vs. Standards/Content, measures the difference between a teacher’s rating on a single standard that evaluates the use of questions and classroom discussion as an instructional strategy, and that same teacher’s average rating on three standards that assess teaching practices that focus on classroom management routines, on conveying standards-based instructional objectives to students, and on demonstrating content-specific knowledge in teaching these objectives.

Our main analysis below examines the degree to which these summary indices predict a teacher’s effectiveness in raising student achievement. Note, however, that we did not construct the indices based on any hypotheses of our own about which aspects of teaching practice measured by TES were most likely to influence student achievement. Rather, we used a statistical technique known as principal components analysis, which identifies the smaller number of underlying constructs that the eight different dimensions of practice are trying to capture. As it turns out, scores on these three indices explain 87 percent of the total variation in teacher performance across all eight standards.

For all teachers in our sample, the average score on the Overall Classroom Practices index was 3.21, or between the “Proficient” and “Distinguished” categories. Yet one-quarter of teachers received an overall score higher than 3.53 and one-quarter received a score lower than 2.94. In other words, despite the fact that TES evaluators tended to assign relatively high scores on average, there is a fair amount of variation from teacher to teacher that we can use to examine the relationship between TES ratings and classroom effectiveness.

In addition to TES observation results, Cincinnati provided student data for the 2003–04 through 2008–09 school years, including information on each student’s gender, race/ethnicity, English proficiency status, participation in special education or gifted and talented programs, class and teacher assignments by subject, and state test scores in math and reading. This rich dataset allows us to study students’ math and reading test-score growth from year to year in grades four through eight (where end of year and prior year tests are available), while also taking account of differences in student backgrounds.

Our primary goal was to examine the relationship between teachers’ TES ratings and their assigned students’ test-score growth. This task is complicated, however, by the possibility that factors not measured in our data, such as the level of social cohesion among the students or unmeasured differences in parental engagement, could independently affect both a TES observer’s rating and student achievement. To address this concern, we use observations of student achievement from teachers’ classes in the one or two school years prior to and following TES measurement, but we do not use student achievement gains from the year in which the observations were conducted. (If some teachers are assigned particularly engaged or cohesive classrooms year after year, the results could still be biased; this approach, however, does eliminate bias due to year-to-year differences in unmeasured classroom traits being related to classroom observation scores.)

We restrict our comparisons to teachers and students within the same schools in order to eliminate any potential influence of differences between schools on both TES ratings and student achievement. In other words, we ask whether teachers who receive higher TES ratings than other teachers in their school produce larger gains in student achievement than their same-school colleagues.

Results

We find that teachers’ classroom practices, as measured by TES scores, do predict differences in student achievement growth. Our main results, which are based on a sample of 365 teachers in reading and 200 teachers in math, indicate that improving a teacher’s Overall Classroom Practices score by one point (e.g., moving from an overall rating of “Proficient” [3] to “Distinguished” [4]) is associated with one-seventh of a standard deviation increase in reading achievement, and one-tenth of a standard deviation increase in math (see Figure 1).

The specific point system that TES uses to rate teachers as Proficient and Distinguished is somewhat arbitrary. For a better sense of the magnitude of these estimates, consider a student who begins the year at the 50th percentile and is assigned to a top-quartile teacher as measured by the Overall Classroom Practices score; by the end of the school year, that student, on average, will score about three percentile points higher in reading and about two points higher in math than a peer who began the year at the same achievement level but was assigned to a bottom-quartile teacher.

This difference might not seem large but, of course, a teacher is just one influence on student achievement scores (and classroom observations are only one way to assess the quality of a teacher’s instruction). By way of comparison, we can estimate the total effect a given teacher has on her students’ achievement growth; that total effect includes the practices measured by the TES process along with everything else a teacher does. The difference between being taught by a top-quartile total-effect teacher versus a bottom-quartile total-effect teacher would be about seven percentile points in reading and about six points in math (see Figure 2). This total-effect measure is one example of the kind of “value-added” approach taken in current policy proposals.

From these data, we can also discern relationships between more specific teaching practices and student outcomes across academic subjects (see Figure 1). Among students assigned to different teachers with the same Overall Classroom Practices score, math achievement will grow more for students whose teacher is better than his peers at classroom management (i.e., has a higher score on our Classroom Management vs. Instructional Practices measure). We also find that reading scores increase more among students whose teacher is relatively better than his peers at engaging students in questioning and discussion (i.e., has a high score on Questions/Discussion vs. Standards/Content). This does not mean, however, that students’ math achievement would rise if their teachers were to become worse at a few carefully selected instructional practices. Although this might raise their Classroom Environment vs. Instructional Practices score it would also lower the Overall Classroom Practices score, and any real teacher is the combination of these three scores.

Do these statistics provide any insight that teachers can use to focus their efforts? First, our finding that Overall Classroom Practices is the strongest predictor of student achievement in both subjects indicates that improved practice in any of the areas considered in the TES process should be encouraged. In other words, the practices captured by the TES rubric do predict better outcomes for students. If, however, teachers must choose a smaller number of practices on which to focus their improvement efforts (for example, because of limited time or professional development opportunities), our results suggest that math achievement would likely benefit most from improvements in classroom management skills before turning to instructional issues. Meanwhile, reading achievement would benefit most from time spent improving the practice of asking thought-provoking questions and engaging students in discussion.

Can we be confident that the various elements of practice measured by TES are the reasons that students assigned to highly rated teachers make larger achievement gains? Skeptical readers may worry that better teachers engage in more of the practices encouraged by TES, but that these practices are not what make the teacher more effective. To address this concern, we take advantage of the fact that some teachers were evaluated by TES multiple times. For these teachers, we can test whether improvement over time in the practices measured by TES is related to improvement in the achievement gains made by the teachers’ students. This is exactly what we find. Since this exercise compares each teacher only to his own prior performance, we can be more confident that it is differences in the use of the TES practices themselves that promote student achievement growth, not just the teachers who employ these strategies.

Conclusion

Is TES worth the considerable effort and cost? Does the intensive TES process (with its multiple observations and trained peer evaluators) produce more accurate information on teachers’ effectiveness in raising student achievement gains than do more-subjective evaluations? In fact, studies of informal surveys of principals (see “When Principals Rate Teachers,” research, Spring 2006) and teacher ratings by mentor teachers find that these more-subjective evaluation methods have similar power to detect differences in teacher effectiveness as the TES ratings. These studies may lead some to question the need for the more detailed TES process. We contend, however, that evaluations based on observations of classroom practice are valuable, even if they do not predict student achievement gains considerably better than more subjective methods like principal ratings of teachers.

The additional information the TES system provides can be used in several important ways. First, the data gleaned from the observations allow researchers to connect specific teaching practices with student achievement outcomes, providing evidence of effective teaching practices that can be widely shared.

The TES program also has the advantage of furnishing teachers and administrators with details about the specific practices that contributed to each teacher’s score. The descriptions of practices, and different performance levels for each practice, that comprise the TES rubric can help teachers and administrators map out professional development plans. A school administrator who desires to differentiate the support she provides to individual teachers would benefit from knowing the components of each teacher’s overall scores. A teacher who would like to improve his classroom management skills may find that he has scored relatively low in a particular standard, and then take steps to improve his practice in response to that information.

Finally, scoring individual practices allows for understanding of more fine-grained variations in skill among teachers with similar overall ratings. It is notable, especially given “The Widget Effect” study, that nearly 90 percent of teachers in our sample received an overall “Satisfactory” rating (i.e., “Distinguished” or “Proficient” in Cincinnati’s terms). Still, there are readily discernible differences in mastery of specific skills within that 90 percent, and those differences in skills predict differences in student achievement.

There are other aspects of the Cincinnati system that may or may not account for the results we observed. First, the observers were external to the school and, in most cases, had no personal relationship with the person they were observing. Second, the observers were trained beforehand and were required to demonstrate their ability to score some sample videos in a manner consistent with expert scores. Simply handing principals a checklist with the same set of standards may not lead to a similar outcome.

The results presented here constitute the strongest evidence to date on the relationship between teachers’ observed classroom practices and the achievement gains made by their students. The nature of the relationship between practices and achievement supports teacher evaluation and development systems that make use of multiple measures. Even if one is solely interested in raising student achievement, effectiveness measures based on classroom practice provide critical information to teachers and administrators on what actions they can take to achieve this goal.

Thomas J. Kane is professor of education and economics at the Harvard Graduate School of Education. Eric S. Taylor is a doctoral student at the Stanford University School of Education. John H. Tyler is associate professor of education, economics, and public policy at Brown University. Amy L. Wooten is a doctoral student at the Harvard Graduate School of Education. Reflecting equal contributions to this work, authors are listed alphabetically. This article is based in part on a larger study which is forthcoming in the Journal of Human Resources.