Two major public opinion polls have just been released. First, Education Next (EdNext) released its ninth annual survey of over 4,083 respondents, which is administered by Knowledge Networks. (Along with Michael Henderson, we are responsible for the design and analysis of this survey.) Shortly thereafter, Phi Delta Kappan (PDK) released its own survey of 3,499 respondents, which is administered by Gallup.
The two surveys are complementary, because they ask about different topics. But when they ask about the same topic, they sometimes get different results even when asking questions that appear to be quite similar. The curious reader is bound to wonder: Who is right? Our answer: Both are correct, but to understand the findings, you must pay close attention to how the questions were asked.
The issue is not the sampling techniques that are used. If questions are worded the same way, PDK and EdNext get almost identical results. For example, PDK finds that 21% of Americans think the nation’s schools deserve either an A or a B, while EdNext finds that 23% do. PDK reports that 51% of Americans give an A or a B to schools in their community, while EdNext finds that 52% do. Statistically speaking, the results are identical.
Yet PDK’s results differ from EdNext’s on several topics. These differences almost certainly arise from differences in the way the question and response options are worded. Let’s take a look at a few specifics.
Opting Out
EdNext finds a clear majority opposed to parental opt out, while PDK finds the public more or less evenly split. Here’s what each survey asked:
EdNext: Some people say that ALL students should take state tests in math and reading. Others say that parents should decide whether or not their children take these tests. Do you support or oppose letting parents decide whether to have their children take state math and reading tests?
Results: Support: 25%; Oppose: 59%; Neither support nor oppose (Neither below): 16%.
PDK: Do you think that all parents with children in the public schools should be allowed to excuse their child from taking one or more standardized tests?
Results: Yes, parents should be allowed to excuse their child: 41%; No, parents should not be allowed to excuse their child: 44%; Don’t know: 16%.
Notice that EdNext begins by presenting each side of the debate: some say all students should take state tests, while others say parents should decide. PDK simply asks whether parents should “be allowed to excuse their child” from “standardized” tests and does not indicate that the tests are required by state law.
PDK finds higher levels of support for parental opt out, because its question is worded in a way that is friendly to the opt-out idea. EdNext offers arguments both for and against before asking for the respondent’s views, making it (in our view) a more neutral question.
Charter Schools
At first glance it looks like PDK finds broader support for charter schools. They report that nearly two-thirds favor the idea, while EdNext reports that only about half the public is so inclined. Can both be right? Let’s look at the questions and response options.
EdNext (long question administered to a random half of the sample): As you may know, many states permit the formation of charter schools, which are publicly funded but are not managed by the local school board. These schools are expected to meet promised objectives, but are exempt from many state regulations. Do you support or oppose the formation of charter schools?
Results: Support: 51%; Oppose: 27%; Neither: 22%.
EdNext (short question administered to the other half): Do you support or oppose the formation of charter schools?
Results: Support: 47%; Oppose: 19%; Neither: 35%.
PDK: Do you favor or oppose the idea of charter schools?
Results: Favor: 64%; Oppose: 25%; Don’t know: 11%.
Notice that a smaller fraction of the public selects PDK’s “don’t know” response than selects the EdNext “neither” option, probably because it is easier for people to say that they neither support nor oppose a policy than to admit they don’t know. If we ignore those without a clear opinion, 71% of EdNext and 72% of PDK respondents say they support charters. However, opposition to charters increases moderately—from 19% to 27% when the question includes a description of charter schools. Apparently, some people become more opposed to charters when they are given specific information about them.
Common Core
EdNext reports a clear plurality in favor of the Common Core when it notes that the standards will be used hold schools accountable and an evenly divided public when the accountability connection is not made. PDK, on the other hand, reports a majority opposed to Common Core. Can these results be reconciled? Let’s again take a look at the questions:
EdNext (long question administered to a random half of the sample): As you may know, in the last few years states have been deciding whether or not to use the Common Core, which are standards for reading and math that are the same across the states. In the states that have these standards, they will be used to hold public schools accountable for their performance. Do you support or oppose the use of the Common Core standards in your state?
Results: Support: 49%; Oppose: 35%; Neither: 16%.
EdNext (short question administered to the other half): Same as above with sentence about accountability deleted.
Results: Support: 39%; Oppose: 37%; Neither: 23%.
PDK: Do you favor or oppose having the teachers in your community use the Common Core State Standards to guide what they teach?
Results: Favor: 24%; Oppose: 54%; Don’t know: 22%.
Taken together, these questions reveal what the public likes and dislikes about Common Core. Apparently, some people favor using common standards to hold schools accountable, but they don’t want those standards to guide classroom instruction. This may explain why debate over the Common Core often focuses on whether the standards will only set a common framework or will also dictate what teachers do in their classrooms. One should expect that debate to continue.
School Vouchers
Public opinion about school vouchers is especially sensitive to decisions about question wording. Two issues are critical: whether the voucher program is universal or targeted to low-income families, and whether it is described in a way that highlights the expanded choice vouchers offer parents or the fact that vouchers would be paid for with government funds. To shed light on the role of question wording, EdNext asked four versions of its questions to randomly selected groups within its sample. PDK asked all respondents a single question.
EdNext (universal vouchers, choice emphasis): A proposal has been made that would give all families with children in public schools a wider choice, by allowing them to enroll their children in private schools instead, with government helping to pay the tuition. Would you favor or oppose this proposal?
Results: Support: 46%; Oppose: 36%; Neither: 18%
EdNext (targeted vouchers, choice emphasis): A proposal has been made that would give low-income families with children in public schools a wider choice, by allowing them to enroll their children in private schools instead, with government helping to pay the tuition. Would you favor or oppose this proposal?
Results: Support: 42%; Oppose: 41%: Neither: 17%.
EdNext (universal vouchers, government funding emphasis): A proposal has been made that would use government funds to pay the tuition of all students who choose to attend private schools. Would you favor or oppose this proposal?
Results: Support: 26%; Oppose: 58%; Neither support nor oppose: 15%
EdNext (targeted vouchers, government funding emphasis): A proposal has been made that would use government funds to pay the tuition of low-income students who choose to attend private schools. Would you favor or oppose this proposal?
Results: Support: 34%; Oppose: 50%; Neither support nor oppose: 16%
PDK (universal vouchers, government funding emphasis): Do you favor or oppose allowing students and parents to choose a private school to attend at public expense?
Results: Favor: 31%; Oppose: 57%; Don’t know: 12%
Responses about school vouchers differ widely depending on how the question is asked. If one emphasizes that vouchers provide families a “wider choice,” a plurality favors the idea. But if one emphasizes that vouchers will be paid for by “government funds” or “at public expense,” then pluralities or even majorities in opposition can be identified. Opinion is also influenced by whether or not the voucher proposal is for all students or for students of low-income.
By splitting its sample into four parts, EdNext shows exactly how question wording affects the public’s response when asked about this policy. PDK asks about vouchers in a way that attracts the lowest level of support.
The Bottom Line
The fact that the two polls obtain similar results when questions are identical is a testament to the scientific foundations of modern survey research. Representative samples of populations can be drawn in a systematic manner, so good polls obtain similar results when they use the same questions. But the design of survey questions remains an art form. Finding a neutral question to capture public sentiment accurately is easier said than done. For that reason it is desirable to avoid emphasizing a single consideration when asking about a policy and, whenever possible, to ask different questions to random segments of a large sample in order to shed light on how question wording influences results. And it is valuable to have multiple pollsters exploring similar topics, as the public benefits from having two good polls instead of one.
– Paul E. Peterson and Martin R. West
Paul E. Peterson is the Henry Lee Shattuck Professor of Government at Harvard University and editor-in-chief of Education Next. Martin R. West is an associate professor at the Harvard Graduate School of Education and executive editor of Education Next.
