Adventures in Campaign Message Polling, part 1

Cross-posted at PolitickerNJ.

The political blogosphere recently took note of a poll in New Jersey’s 3rd Congressional district purporting to show first-term Democratic incumbent John Adler with a surprising 17 point lead over GOP challenger Jon Runyan. The poll was conducted for the Adler campaign by their own pollster, but even campaign pollsters have to produce reliable estimates if they want to stay in business. Regardless, this poll – or to be more accurate, the memo that described the poll results – raises some red flags.

First, only 3% of likely voters say they are undecided about their choice. Really? In June, when most voters probably cannot name either party’s nominee?. Second, the vote choice question was posed as a 3-way race, including Adler, Runyan and a third candidate running under the “NJ Tea Party” banner. This candidate – whose name recognition has to be near zero – received 12% of the vote in this match-up.

The poll memo reads more like a campaign fluff piece (e.g. “[Adler’s] record of independence and accountability has put him in an excellent position to win this race.”) than an insightful polling memo. Now, I’m not saying that the poll findings were fabricated. For one, the pollster, Greenberg Quinlan Rosner, is a well-known Democratic firm that is unlikely to jeopardize their reputation by generating poll numbers out of whole cloth. In fact, I’m inclined to believe the results were probably valid. The problem is that they were reported out of context.

Unless the Adler campaign has money to burn on conducting polls solely for the purpose of leaking them to the public, these results were from a longer poll that also tested all sorts of messages and strategies for the Adler campaign. The purpose of an early summer poll is to try out a variety of messages in order to identify the most effective ones for use in the campaign.

What we don’t know about this poll is at what point in the interview this three-way vote choice question was presented to respondents. I have my doubts that this was the first time in the interview that survey respondents were asked to name their vote choice. There were likely some questions about candidate characteristics that preceded this question.

So why does the order of the questions matter? Because questions asked later in a poll allow respondents to use information they heard during the course of the interview to inform their answers. At this point, the poll results no longer reflect the mind-set of typical voters because the poll respondents now have information – i.e. messages – that most voters don’t.

This is why message testing polls rarely get released to the public. Indeed, most reputable pollsters prefer it that way. Unfortunately, their concerns are occasionally overridden by a campaign manager who sees some strategic advantage in releasing the poll results.

One reason a campaign may release an internal poll is to demonstrate to potential donors that they have a viable shot at winning. Considering the healthy state of Adler’s campaign coffers, that’s clearly not a concern. So what advantage did the Adler campaign see in selectively releasing poll results?

First, we need to consider why the campaign even bothered to include an unknown, unfunded third party candidate in one of their vote choice questions. Especially since there will be three independents candidates on the ballot for this race in November. [Side note: There were four, but the state Democrats’ executive director, Robert Asaro-Angelo, successfully challenged Robert “Weedman” Forchion’s petition. This, of course, raises questions about Angelo’s contention that he never heard of the Tea Party candidate listed on that very same ballot.] [UPDATE: Rob Angelo contacted me re this statement. He admitted that he misspoke, since he obviously reviewed the names of all independent candidate filings in June.]

So, why did the Adler campaign only test the Tea Party candidate? Because it makes sense in the current political environment. While we can make a pretty strong guess as to the “Libertarian” candidate’s likely vote total in November, the impact of running under the Tea Party banner is a big question mark.

According to the Adler poll results, a Tea Party candidate may indeed peel off votes from the Republican nominee. But this is by no means a certainty. Why? To start, no one knows who the Tea Party candidate is.

And that gets us to why these numbers were released. Remember, these poll results represent one potential outcome in a context where Adler’s pollster had complete control over information presented to voters. In other words, the message testing effects measured in campaign polls do not always play out so neatly in the real world.

More importantly, a message will certainly not work if no one knows about it. And that is the case with the Tea Party candidate. So, what’s a good way to get a candidate’s name out? Show him exceeding expectations in a poll.

The message testing poll becomes the message! The Greenberg firm issued a memo to “Interested Parties” and sure enough, the story hits the internet, including The Hill, Chris Cillizza’s “The Fix” column in the Washington Post, the National Journal’s Hotline, and PolitickerNJ.

Interestingly, the poll memo was not released to newspapers, at least not to those in Adler’s district. Did the campaign think these media outlets wouldn’t be “interested?” Doubtful. The reason why the poll was released only to internet sites geared to the chattering classes was a strategic one. The intent was to let Tea Party-inclined voters “know” they have a viable option in New Jersey’s 3rd district Congressional race and to suggest to potential GOP donors that Runyan is a shaky investment. The internet is the best way to get that buzz spread with a less critical eye, especially with the burgeoning Tea Party community.

The chosen “interested parties” did their job and disseminated the campaign’s message, any caveats in their reports notwithstanding. Of course, this may backfire in the long run. Tea Party activists have been denouncing the candidate, with stories now focused on whether the Tea Party candidate is a plant. This poll is seen by some as part of a larger Democratic plot.

However, the question remains whether this poll – or more accurately the selective results in the campaign’s memo – should have been reported in the first place. When a campaign simply claims that their candidate is ahead by 17 points, no journalist in his right mind would report it. However, when a campaign has their pollster slap together a memo that purports to show a “51 to 34 percent” lead, suddenly the information is valid.

A good rule of thumb, no poll should be reported – in any venue – unless the pollster is willing to provide the entire set of questions and responses. Otherwise, it’s little more than propaganda, or worse. It’s a little too late for this poll. The Adler campaign achieved its intent – getting out a campaign message under the guise of hard fact.

Homebuyer Credit or Taxpayer Gift?

So what’s the deal with the New Jersey homebuyer tax credit? This program would give homebuyers in the state a tax credit worth up to $15,000 spread over three years. The bill (A1678) passed the legislature with overwhelming bi-partisan support and landed on the governor’s desk on June 10. Yet, Chris Christie has yet to sign it.

On New Jersey 101.5’s “Ask the Governor” program this week, Christie said he likes the idea, but will only sign it if he can figure out where the money is coming from. Proponents of the bill say it will spur construction and create jobs, which will increase revenues in both the realty transfer fund and income taxes, which will offset the cost of the program. A look at the numbers behind the program casts serious doubt that this will be the case.

The program sets aside $100 million from the property tax relief fund as a tax credit for people who purchase a home after the program goes into effect. Of this amount, $75 million is set aside for the purchase of newly constructed homes and $25 million for existing home sales.

Wow! $100 million sounds like it will go far, doesn’t it? Not really. The program would give these purchasers a credit of $15,000 or 5% of the sales price, whichever is lower. For the sake of argument, let’s say that the typical home price is $250,000. That would give the average homebuyer a credit of $12,500.

The $100 million in the kitty would provide tax credits to just 8,000 homebuyers. Hmm, sounds less impressive now. It’s a shame the legislature didn’t know this before they passed the bill, you say. Actually, they did (or at least the Senate did). The Office of Legislative Services provided a fiscal impact statement in late May that laid out just such a scenario – in fact, theirs was even more conservative.

But wait, there’s more. Let’s see how this would actually work if the program was in place. Remember the pot is divided 75/25 between new and existing construction. Using my generous estimate, that equates to 6,000 purchasers of new homes and 2,000 purchasers of existing homes who would qualify for the credit.

There were about 12,000 building permits issued for new homes in New Jersey last year. Even if some went unsold, at least 10,000 new homes were purchased last year. There is some debate on the impact that the now-expired federal tax credit had on those sales numbers. Let’s make a hypothetical assumption that one-third of those sales were driven by the tax credit and two-thirds would have happened anyway because people needed to move. In other words, we could expect that about 6,500 newly built homes will be sold in New Jersey this year without any tax credit.

Now, proponents say that the availability of a state tax credit will spur more buyers into the marketplace and thus increase demand. Let’s accept that premise and estimate that demand doubles and 6,000 homes are sold within the first six months of the program.

Well, what happens after the credit program is shut down? Do those buyers stay in the market and keep housing demand high? If we look at the experience of the federal credit program, the answer appears to be “No.”

The numbers are even more astounding if we look at how the credit will apply to sales of existing homes. According to state figures, more than 110,000 existing houses were resold last year. That translates to about 300 a day. This means the $25 million pot for existing home sales tax credits will be snapped up within a week of the program going into effect. One week!

Considering how lengthy the sales and mortgage process is, a homebuyer’s likelihood of closing on their purchase within the seven day window this credit program is available will be more a matter of luck than planning. “Mr. and Mrs. Jones, here are the keys to your house, and congratulations - you just won $15,000!”

This analysis raises a number of doubts about the economic impact – or ripple effect – of the New Jersey homebuyer tax credit. First of all, if the purpose is to spur economic activity in the form of construction jobs, then why isn’t it limited to new home sales only?

But even if the entire pot was dedicated to new home sales, it doesn’t seem to be large enough to keep demand high for enough time to ride out the current slump in the housing market. If I were a builder, I don’t think I would be hiring more workers on the dubious potential of this program.

And finally, why is the program so generous at $15,000 a pop? The federal program gave an $8,000 credit for first-time homebuyers and a $5,000 credit for current owners. These amounts appeared to be adequate to keep the housing market from dipping lower than it did.

Of course, none of this says anything about whether a tax credit of any sort is particularly smart economics. Some analysts claim that the New Jersey housing market is still overvalued and that prices need to come down further before buyers will return. Basically, they argue that the tax credit program has interfered with the open market and delayed a necessary price correction. Perhaps that’s why the eight Assembly members who voted against the bill happen to be among the most ideologically conservative in the legislature.

In any event, it’s difficult to see economic benefit in taking $100 million dollars in taxpayer money and giving it to 8,000 homebuyers for doing what they would probably have done anyway. Given New Jersey’s current fiscal crisis, maybe the state would be better off using the money to build a few hundred houses itself and sell them for a profit.

Are Nate Silver’s Pollster Ratings “Done Right”

This originally appeared as a guest column on Pollster.com.

The motto of Nate Silver’s website, www.fiverthirtyeight.com, is “Politics Done Right.” I’m not sure that his latest round of pollster ratings lives up to that moniker.

As most poll followers know, Nate shot to fame during the 2008 election, taking the statistical skills he developed to predict baseball outcomes and applying them to election forecasting. His approach was pretty accurate in that presidential race (although it’s worth noting that other poll aggregators were similarly accurate – see here and here).

Nate recently released a new set of pollster ratings that has raised some concerns among the polling community.

First, there are some questions about the accuracy of the underlying data he uses. Nate claims to have culled his results from 10 different sources, but he seems to not to have cross-checked those sources or searched original sources for verification.

I asked for Monmouth University’s poll data and found errors in the 17 poll entries he attributes to us – including six polls that were actually conducted by another pollster before we partnered with the New Jersey Gannett newspapers, one omitted poll that should have been included, two incorrect election results, and one incorrect candidate margin. [Nate emailed me that he will correct these errors in his update later this summer.]

Mark Blumenthal also noted errors and omissions in the data used to arrive at Research2000’s rating. I found evidence that suggest these errors may be fairly widespread.

In the case of prolific pollsters, like Research2000, these errors may not have a major impact on the ratings. But just one or two database errors could significantly affect the vast majority of pollsters with relatively limited track records – such as the 157 pollsters out of 262 pollsters on his list who have fewer than 5 polls to their credit.

Some observers have called on Nate to demonstrate transparency in his own methods by releasing that database. Nate has refused to do this (with a dubious rationale that the information may be proprietary) - but he does now have a process in place for pollsters to verify their own data. [If you do, make sure to check the accuracy of the actual election results as well.]

I’d be interested to see how many other pollsters find errors in their data. But the issue that has really generated buzz in our field is Nate’s claim that pollsters who either were members of the National Council on Public Polls or had committed to the American Association for Public Opinion Research (AAPOR) Transparency Initiative by June 1, 2010 exhibit superior polling performance. For these pollsters, he awards a very sizable “transparency bonus” in his latest ratings.

One of the obvious problems with his use of the bonus is that the June 1 cut-off is arbitrary. Those pollsters who signed onto the initiative by June 1, 2010 were either involved in the planning or happened to attend the AAPOR national conference in May. A general call to support the initiative did not go out until June 7 – the day after Nate’s ratings were published.

Thus, the theoretical claim regarding a transparency bonus is at least partially dependent on there also being a relationship between pollster accuracy and AAPOR conference attendance. Others have remarked on the apparent arbitrariness of this “transparency bonus” cutoff date Nate claims that regardless of how a pollster made it onto the list, there is statistical evidence these pollsters are simply better at election forecasting. I don’t quite see it.

His methodology statement includes a regression analysis of pollster ratings that is presented as evidence for using the bonus.

The problem is that even in this equation, the transparency score just misses most researcher’s threshold for being significant (p<.05). More to the point, his model – using dummy variables to identify “transparent” pollsters, partisan pollsters, and internet pollsters – is incomplete. The adjusted R-square is .03. In other words, 3% of total variance in pollster raw scores (i.e. error) is predicted by the model.

Interestingly of the three variables – transparency, partisan, and internet – only partisan polling shows a significant relationship. He decided to calculate different benchmarks that award transparent polls and penalize internet polls (even though the latter was based on only 4 cases and not statistically significant). And oddly, he does not treat partisan pollsters any differently than other pollsters, even though this was the only variable with a significant relationship to rawscore.

I decided to look at this another way, using a simple means analysis. The average error among all pollsters is +.54 (positive error is bad, negative is good). Among “transparent” pollsters it is -.63 (se=.23) and among other pollsters it is +.68 (se=.28).

But let’s isolate the more prolific pollsters, say the 63 organizations with at least 10 polls to their names who are included in Nate’s first chart. Among these pollsters, the 19 “transparent” ones have an average score of -.32 (se=.23) and the other 44 pollsters average +.03 (se=.17). The difference is not so stark now.

Firms with fewer than 10 polls to their credit have an average error score of -1.38 (se=.73) if they are “transparent” (all 8 of them) and a mean of +.83 (se=.28) if they are not. That’s a much larger difference.

I also ran some ANOVA tests for the effect of the transparency variable on pollster raw scores for various levels of polling output (e.g. pollsters with more than 10 polls, pollsters with only 1 or 2 polls, etc.). The F values for this test range from only 1.2 to 3.6, and none were significant at p<.05. In other words, there is more error variance within the two separate groups of transparent versus non-transparent pollsters than there is between the two groups.

I can only surmise that the barely significant relationship between the arbitrary transparency designation and polling accuracy is pointing to other more significant factors, including pollster output.

Consider this - 70% of “transparent” pollsters on Nate’s list are have 10 or more polls to their credit, whereas only 19% of the “non-transparent” ones do. In other words, Nate’s “bonus” is actually a sizable penalty levied against more prolific pollsters in the latter group. “Non-transparent pollsters happen to be affiliated with a large number of organizations with only a handful of polls to their name – i.e. pollsters who are prone to greater error.

For comparison, re-ran Nate’s PIE (Pollster Introduced Error) calculation using a level playing field for all 262 pollsters on the list. I set the error mean at +.50 (which is approximately the mean error among all pollsters).

Comparing the relative pollster ranking between the two lists produced some intriguing results. The vast majority of pollster ranks (175) did not change by more than 10 spots on the table. Another 67 had rank changes between 11 to 40 spots on the two lists; 11 shifted by 41 to 100 spots, and 9 pollsters gained more than 100 spots in the rankings because of the transparency bonus. Of this latter group, only 2 of the 9 had more than 15 polls recorded in the database.

Nate says that the main purpose of his project is not to rate pollsters’ past performance but to determine probable accuracy going forward. But one wonders if he needs to go this particular route to get there. Other aggregators use less elaborate methods – including straightforward mean scores – and seem to be just as accurate.

His methodology statement is about 4,800 words (with 18 footnotes). It reminds me of a lot of the techies I have worked with over the years – the kind of person who will make three left turns to go right.

This time I think Nate may have taken one left turn to many. We’ll know in November.