The issue that Terry Moe raises in his article “Cooking the Questions” in the Spring 2002 issue of Education Next concerns Phi Delta Kappa’s interpretations of findings from the 2001 Phi Delta Kappa/Gallup poll of the public’s attitudes toward education. In a press release, Phi Delta Kappa concluded, “It is clear that the decade of the ’90s saw support for the use of public funds for parents and students to use in attending private and church-related schools increase, peak, and then begin what has become a significant decline.” This conclusion was based on responses to the following questions:
• “Do you favor or oppose allowing students and parents to choose a private school to attend at public expense?”
• “A proposal has been made that would allow parents to send their school-age children to any public, private, or church-related school they choose. For those
parents choosing nonpublic schools, the government would pay all or part of the tuition. Would you favor or oppose this proposal in your state?”
The first question was asked in 1993, repeated in 1995, and then repeated each year thereafter. The percentages in favor of public subsidies for private schooling have been:
The second question was first asked in 1994, repeated in 1996, and repeated each year thereafter. The percentages in favor of public subsidies for private schooling have been:
Moe refers to the first question as “biased,” while he finds the second one “actually informative and neutral, precisely the kind of item that should have been used all along.” He challenges the trend documented in the first question, but fails to note that the second question reflects precisely the same trend. It could just as easily have been used as the basis of Phi Delta Kappa’s conclusion in its press release that support for vouchers increased, peaked, and then began a significant decline during the 1990s. The most interesting thing about Moe’s challenge, however, is that it was not raised by voucher advocates during the period from 1993 through 1998, when support for vouchers as measured by both questions was climbing steadily. They were, in fact, pointing to this poll and these questions as evidence that vouchers were gaining support. It was only when support stabilized and then began to decline that the complaints started.
It is, of course, not surprising that there are differing interpretations of poll data on a topic as emotionally charged as vouchers. Terry Moe, an avowed advocate of vouchers, would be expected to look with skepticism on poll results that indicate a decline in public support for them. Phi Delta Kappa, as an organization committed to the public schools, would be expected to view the decline positively. This does not mean that Phi Delta Kappa cannot and does not conduct a poll that is fair and unbiased. Phi Delta Kappa did, in fact, routinely report and comment accurately on the data during the period when support for choosing a private school to attend “at public expense” was increasing. And, regarding the validity of the poll’s findings, Phi Delta Kappa finds some measure of confirmation in the fact that its data could have been used to predict the recent defeats of voucher proposals in both California and Michigan. Having said that, we would quickly acknowledge that results from a random sample in an opinion survey are not comparable to results in an election in which those expressing an opinion do so by choice.
Moe’s concern with the first of the poll questions above is that it fails to convey to respondents the “central purpose” of a voucher program, that being “to expand the choices available to all qualifying parents, especially those who now have kids in public schools.” However, the purpose of an opinion poll is to survey public opinion based on the information the public has at the time; it is not to educate the public. The best way to bias any question is to include a definition that will sway the respondent one way or the other. The question Moe challenges avoids that possibility. His suggested wording would make it a biased question. After all, in America, the word “choice” has always been freighted with positive value.
Moe also attacks the first question on the grounds that “it focuses entirely on private school parents and asks respondents whether the government ought to be subsidizing them.” Given this criticism, it seems necessary to repeat the question, which is: “Do you favor or oppose allowing students and parents to choose a private school to attend at public expense?” This wording places no focus on “private school parents” and does not use the word “subsidizing,” a word that would clearly convey bias.
Having validated the second question as “informative and neutral,” Moe immediately infers bias because it is presented after the first question. The Gallup organization, in its role as designer of the survey, is continuously alert to the possibility of “order bias.” While Gallup doesn’t regard that as likely in this case, it plans to test the matter in the Fall 2002 survey by splitting the sample and reversing the order of the questions with the two sub-samples. What is mystifying here is that Moe chose to taint a constructive observation by suggesting that the order, which had been consistent from day one, was determined on the basis of some political agenda.
There is a misunderstanding that seems to be at the root of Moe’s search for a conspiracy in which Phi Delta Kappa is involved. Moe seems to believe that “researchers” at Phi Delta Kappa “design” the questions used in the poll. No one at Phi Delta Kappa would pretend to have the expertise to do so. The design of the questions is the exclusive responsibility of the Gallup Organization. Gallup researchers frame the questions and attest to the fact that they are unbiased and that the conclusions drawn are verified by the data. It is interesting that Moe, in referring to “Gallup’s own surveys,” says that the questions “are well worded and so should give good measures of how people respond to the basic voucher program.” The PDK poll is a Gallup survey, and that organization takes as much pride in it as does Phi Delta Kappa.
Preserving the Trend
Moe makes a lengthy comparison of questions he indicates have been asked by the Gallup Organization in other surveys. Representatives of the Gallup Organization have indicated that they are not aware of any polling data that conflict with what is reported in the PDK/Gallup Poll. However, the issue he raises regarding the use of words such as “government” and “public expense” is interesting and is one on which the PDK/Gallup poll does have some evidence. And it is a story worth telling, for it illustrates how simple the explanations are for some of the dark conspiracies Moe seems to see around each corner.
At the press conference in 1996, during the discussion of the “at public expense” question, a cameraman suggested that we ask the same question but change the word “public” to “government.” In the 1997 poll, we took that suggestion and split the sample. Lo and behold, 44 percent expressed support when the word “public” was used, and 48 percent expressed support when we used the word “government.” This was duly reported during the press conference, along with the unsubstantiated suspicion that people regard public money as their own and government money as someone else’s. Had we tried the alternative phrase “at taxpayers’ expense,” we think we could safely predict an even lower level of support. Our choice of “at public expense” represents an effort to chart a middle course and avoid bias.
Having conducted the experiment in the 1997 poll, we returned to the previous wording in 1998—not to “cook” the percentages but to preserve the trend line the question had established. Those present at each subsequent news conference know that reference is made to the fact that you could increase the percentage in favor by substituting “government” for “public.” The emphasis on trend lines makes this a good time to point out that Phi Delta Kappa and the Gallup Organization decided in the early 1990s to avoid changes in wording on the questions having to do with the relationship between private and public schools. We also decided to repeat virtually every question every year. These decisions were made so that we would avoid even the appearance of bias.
It is somewhat ironic that these commitments have precluded our making a change we would like to make in the question Moe refers to as “informing and neutral.” We would like to eliminate the word “public” from the second of the previously referenced questions where it reads “public, private, or church-related” school. We believe that the inclusion of “public” is why the support level in the second question is somewhat higher than in the first question. The change is not made because it would open us to the charge of “cooking the books.” Perhaps, in light of the criticism, we should consider dropping all three descriptors and substitute the word “nonpublic.” This might, however, not gain us anything since, in reference to this question, Moe wrote in his book Schools, Vouchers, and the American Public, “This newer PDK item has all the properties we should look for in a well-worded question.”
In what seems to be a search to find some way to discredit the PDK/Gallup poll, Moe observes, “Due to sampling error alone, some numbers will go up and others will go down.” That is precisely why Phi Delta Kappa approached the decline in support on the two questions cautiously when it began to occur. The statement in the 1999 press conference was that “support on the two questions appears to have stabilized.” Not until the 2000 press conference did we draw the conclusion that support “had peaked and appeared to be declining.” Not until 2001, the third year of decline, did we draw the conclusion that “support is declining.”
Moe also appears to find something sinister in the fact that new questions are added to the poll from time to time. Nothing will be gained by responding to each of the charges; however, one such response seems warranted. Phi Delta Kappa surveys a broad group of interested persons each year for input on the issues and topics that should be addressed by the poll. One response to the 1997 survey expressed the view that we were asking the wrong questions when we focused on questions related to vouchers. This person suggested that we should simply ask whether the respondent wanted reform to come by reforming the existing system or by finding an alternative system.
The Gallup Organization framed the appropriate question, and it was asked in 1997—with 71 percent indicating that reform should come through the “existing system.” The question has been repeated each year since, with 72 percent of respondents favoring reform through the existing system in the 2001 poll. Subsequently, another individual suggested that respondents might not take “finding an alternative system” to include vouchers. We then added a question presenting a direct choice between “improving existing public schools” and “providing vouchers.” Seventy percent opted for “improving existing public schools.” That question has been repeated each subsequent year, with the proportion favoring improving existing public schools standing at 71 percent in 2001. For some reason, Moe sees these questions as part of a conspiracy and indicative of a desire to mislead.
Moe notes that the higher percentages favoring vouchers on what he describes as the “informative and neutral” question do not find their way into PDK’s press releases. Any member of the media who has attended the annual press conference or studied the media packet will recognize the inaccuracy of this remark. Both questions are summarized in the written materials that are distributed, and the results are described in the oral presentations. The difference in the two questions and the higher results on the second item are often topics for discussion. Having asked their questions and listened to the responses, the media may then draw their own interpretations.
Conspiracy Theory
Declining support for vouchers is not an artifact of polling
by David Moore
Terry Moe likes the second question about vouchers asked by the Gallup Organization, which – like the first question, which Moe does not like-shows a clear trend of increasing and then decreasing support for vouchers over the past decade. The fact that both questions produced similar trends directly contradicts Moe’s statement that “Gallup’s own surveys” show no decline in support since the mid-1990s. Apart from the trends in the PDK/Gallup polls, the Gallup Organization has asked no other series of voucher questions repeatedly during the 1990s.
Moe also argues that the second question would have elicited more support for vouchers had it not been asked after the question he does not like. This is a possibility, but the reason for keeping the questions in the same order year after year is to protect the integrity of the trend data they produce.
Otherwise, any changes in results could be due to a change in the order of the questions, not to any real change in public opinion. The initial order of the questions was a matter of chance, not a nefarious plot on the part of the Gallup Organization to understate support for vouchers. Gallup’s subsequent experiments should put this charge to rest.
Finally, Moe criticizes all other polling organizations because their voucher questions “fail to convey the central purpose of a voucher program.” His criticisms here are far off the mark. Polling is a means of finding out what people think about certain matters, not a way of educating them.
-David Moore is senior editor of the Gallup Poll.
The Politics of Education
In a further challenge to the PDK/Gallup questions, Moe refers to a choice offered in a non-PDK/Gallup poll. The positions were: “A: Governments should give parents more educational choices by providing taxpayer-funded vouchers to help pay for private or religious schools” or “B: Government funding should be limited to children who attend public schools.” These are, by any measure, badly phrased alternatives. The phrase “more educational choices” in the first item is designed to push respondents to a positive answer. “Choice” is an attractive characteristic, and who would not want more? “Limited” is a negative concept. Who would want to be limited? The first alternative is warm and inviting. The second is cold and restricting. And look at the length of the choices. They are not comparable. It is ironic that Moe would not recognize the obvious bias in these two statements.
As a known advocate of vouchers, it is understandable why Moe would view polls demonstrating declining support with concern and suspicion. What’s strange is his view of the public itself. Moe writes, “Research has long shown that most Americans are poorly informed about public policy and don’t have well-developed views on most issues.” Those who conduct and report the PDK/Gallup poll have consistently found that the American public is often far ahead of its leaders on public policy issues that affect such vital things as schooling.
Moe refers to Phi Delta Kappa’s “using its survey to further its own political agenda.” Phi Delta Kappa does not engage in political activity per se, does not endorse candidates, and plays no active role in political activities. It does, however, recognize that the public schools are a part of the political agenda—today more so than ever—and it makes no apology for the fact that it is constitutionally charged to be an advocate for the public schools. But that doesn’t necessarily imply that PDK or its journal, Phi Delta Kappan, are involved in an effort to quash debate or to distort data. To draw that inference, as Moe does, is far from the stance a “disinterested” academic researcher ought to take.
Indeed, Moe goes beyond denigrating the work of Phi Delta Kappa and the Gallup Organization. He indicts virtually all the other groups conducting surveys on vouchers, concluding that “their measures are inappropriate” as well. Why inappropriate? Because, he says, they do not have a “nuanced understanding of each and every issue.” The complete explanation offered by Moe follows:
Sponsoring groups like CBS, NBC/Wall Street Journal, and ABC/Washington Post attempt to measure public opinion on a great variety of issues: presidential popularity, gun control, abortion, foreign policy, and many more. Education is but a small part of this, and the voucher issue is just a part of education. However well trained these researchers may be in survey methodology, they cannot be expected to have a nuanced understanding of each and every issue. As a result, they may sometimes adopt wording that seems perfectly acceptable, but that misses the mark.
Moe’s tendency, whatever his motive, seems to be to label those who express declining support for vouchers—or even report a lack of support—as at best uninformed, at worst part of a conspiracy. In the meantime, he fails to offer explanations as to why support for vouchers is such that, following the Michigan and California experiences, no state is currently considering a comprehensive voucher proposal. Neither does he attempt to explain why, almost 50 years after Milton Friedman’s pronouncement that vouchers were the future of our public schools, fewer than 25,000 students across the United States use vouchers to attend school. Surely this lack of success cannot be explained by the conspiracy theory Moe advances.
Lowell C. Rose is executive director emeritus of Phi Delta Kappa International. Alec M. Gallup is cochairman of the Gallup Organization.
