Question Wording Matters

Today, we released a Monmouth University/Gannett New Jersey poll on same-sex marriage. I characterized the poll results – 48% in favor to 43% opposed – as showing a public divided on this issue. I have been asked why I did not frame the results in a more positive light for same-sex marriage, especially since support among registered voters was slightly higher at 50%. The political reality is that these results indicate this is an issue with little momentum right now.

First, if this issue were put to a referendum, polling support would need to be above 50% for passage (referendum polls in New Jersey tend to understate the size of the “No” vote). Second, 27% of Garden State voters “strongly” oppose same-sex marriage, including 1-in-5 Democrats and 1-in-3 independents. Considering how tight the gubernatorial pre-election polls are right now, Governor Corzine needs nearly every one of these voters if he is to be re-elected. So he can’t afford to turn this into a campaign issue.

In the end, I chose to portray the results cautiously. By comparison, Garden State Equality released its own poll last August which obtained similar results, but under the headline: “Zogby Poll: New Jersey Wants to Dump Civil Unions for Marriage Equality, and Is Ready Now”. This leads me to the larger point of this post – which is to pay attention to question wording.

I was initially skeptical when I saw who conducted the poll. John Zogby has a reputation among credible pollsters for a lack of “quality control” when it comes to his client studies. For example, his firm recently conducted a poll on behalf of a conservative activist which purportedly measured Obama voters’ “knowledge” of the presidential candidates. However, it posed questions with unequal information about the two parties’ tickets that basically got the client what he wanted to hear. You can find critiques of this poll from the Wall Street Journal’s Numbers Guy and Politico. Zogby, for his part, first defended the poll but later said he was “out of town” when his firm accepted the contract.

Back to the August poll, some news organizations in New Jersey chose to report the poll under the rationale that Zogby supposedly has a good track record at election forecasting. That’s debatable, but really immaterial to this poll. The real issue is whether he asked objective questions.

The Monmouth poll’s same-sex marriage question, a variant of one first used by the Eagleton-Rutgers Poll in 2003, is: “Do you favor or oppose allowing gays and lesbians to marry legally?”

The Zogby poll asked: “Currently, New Jersey lets same-sex couples enter only into civil unions, while California and Massachusetts give same-sex couples the freedom to marry. Do you support or oppose same-sex couples in New Jersey also getting the freedom to marry?”

On its face, this question violates a number of tenets of question wording. It uses appeals to the status quo (“currently…”), external benchmarks (“other states do it”), and loaded wording (“freedom”).

So what is the impact of all this extra verbiage? To find out, I included the Zogby wording in a split-half design experiment on our last poll (acknowledging that California voters have since banned same-sex marriage).

Among registered voters, the Zogby wording produced a 51% favor to 37% oppose result. That’s not significantly different than the Monmouth question wording’s 50% favor to 40% oppose among voters. However, a closer look at the internal numbers suggests that the Zogby question wording did indeed have an impact – but it was different for different groups. Among Democrats, support for same-sex marriage using the Zogby question stood at 66%, compared to 58% in the Monmouth question. Conversely, Republicans were less supportive of same-sex marriage in the Zogby question (31%) than they were in the Monmouth wording (37%).

My best guess is that the “other states do it” and “freedom” appeals encouraged some Democratic poll respondents to be more supportive than they would have otherwise. But the reference to Massachusetts and California – seen as bastions of liberal thinking by some – seems to have moved a similar number of Republican poll respondents in exactly the opposite direction.

I’ve posted about the importance question wording before (here). It’s a lesson worth repeating. If you are a client who insists that certain words need to be included in a question, be careful what you wish for. And if you are a journalist reporting on a poll, make sure you read – and are comfortable with – how the questions are asked.