Friday, June 18, 2010

Are Nate Silver’s Pollster Ratings “Done Right”

This originally appeared as a guest column on Pollster.com.

The motto of Nate Silver’s website, www.fiverthirtyeight.com, is “Politics Done Right.” I’m not sure that his latest round of pollster ratings lives up to that moniker.

As most poll followers know, Nate shot to fame during the 2008 election, taking the statistical skills he developed to predict baseball outcomes and applying them to election forecasting. His approach was pretty accurate in that presidential race (although it’s worth noting that other poll aggregators were similarly accurate – see here and here).

Nate recently released a new set of pollster ratings that has raised some concerns among the polling community.

First, there are some questions about the accuracy of the underlying data he uses. Nate claims to have culled his results from 10 different sources, but he seems to not to have cross-checked those sources or searched original sources for verification.

I asked for Monmouth University’s poll data and found errors in the 17 poll entries he attributes to us – including six polls that were actually conducted by another pollster before we partnered with the New Jersey Gannett newspapers, one omitted poll that should have been included, two incorrect election results, and one incorrect candidate margin. [Nate emailed me that he will correct these errors in his update later this summer.]

Mark Blumenthal also noted errors and omissions in the data used to arrive at Research2000’s rating. I found evidence that suggest these errors may be fairly widespread.

In the case of prolific pollsters, like Research2000, these errors may not have a major impact on the ratings. But just one or two database errors could significantly affect the vast majority of pollsters with relatively limited track records – such as the 157 pollsters out of 262 pollsters on his list who have fewer than 5 polls to their credit.

Some observers have called on Nate to demonstrate transparency in his own methods by releasing that database. Nate has refused to do this (with a dubious rationale that the information may be proprietary) - but he does now have a process in place for pollsters to verify their own data. [If you do, make sure to check the accuracy of the actual election results as well.]

I’d be interested to see how many other pollsters find errors in their data. But the issue that has really generated buzz in our field is Nate’s claim that pollsters who either were members of the National Council on Public Polls or had committed to the American Association for Public Opinion Research (AAPOR) Transparency Initiative by June 1, 2010 exhibit superior polling performance. For these pollsters, he awards a very sizable “transparency bonus” in his latest ratings.

One of the obvious problems with his use of the bonus is that the June 1 cut-off is arbitrary. Those pollsters who signed onto the initiative by June 1, 2010 were either involved in the planning or happened to attend the AAPOR national conference in May. A general call to support the initiative did not go out until June 7 – the day after Nate’s ratings were published.

Thus, the theoretical claim regarding a transparency bonus is at least partially dependent on there also being a relationship between pollster accuracy and AAPOR conference attendance. Others have remarked on the apparent arbitrariness of this “transparency bonus” cutoff date Nate claims that regardless of how a pollster made it onto the list, there is statistical evidence these pollsters are simply better at election forecasting. I don’t quite see it.

His methodology statement includes a regression analysis of pollster ratings that is presented as evidence for using the bonus.

The problem is that even in this equation, the transparency score just misses most researcher’s threshold for being significant (p<.05). More to the point, his model – using dummy variables to identify “transparent” pollsters, partisan pollsters, and internet pollsters – is incomplete. The adjusted R-square is .03. In other words, 3% of total variance in pollster raw scores (i.e. error) is predicted by the model.

Interestingly of the three variables – transparency, partisan, and internet – only partisan polling shows a significant relationship. He decided to calculate different benchmarks that award transparent polls and penalize internet polls (even though the latter was based on only 4 cases and not statistically significant). And oddly, he does not treat partisan pollsters any differently than other pollsters, even though this was the only variable with a significant relationship to rawscore.

I decided to look at this another way, using a simple means analysis. The average error among all pollsters is +.54 (positive error is bad, negative is good). Among “transparent” pollsters it is -.63 (se=.23) and among other pollsters it is +.68 (se=.28).

But let’s isolate the more prolific pollsters, say the 63 organizations with at least 10 polls to their names who are included in Nate’s first chart. Among these pollsters, the 19 “transparent” ones have an average score of -.32 (se=.23) and the other 44 pollsters average +.03 (se=.17). The difference is not so stark now.

Firms with fewer than 10 polls to their credit have an average error score of -1.38 (se=.73) if they are “transparent” (all 8 of them) and a mean of +.83 (se=.28) if they are not. That’s a much larger difference.

I also ran some ANOVA tests for the effect of the transparency variable on pollster raw scores for various levels of polling output (e.g. pollsters with more than 10 polls, pollsters with only 1 or 2 polls, etc.). The F values for this test range from only 1.2 to 3.6, and none were significant at p<.05. In other words, there is more error variance within the two separate groups of transparent versus non-transparent pollsters than there is between the two groups.

I can only surmise that the barely significant relationship between the arbitrary transparency designation and polling accuracy is pointing to other more significant factors, including pollster output.

Consider this - 70% of “transparent” pollsters on Nate’s list are have 10 or more polls to their credit, whereas only 19% of the “non-transparent” ones do. In other words, Nate’s “bonus” is actually a sizable penalty levied against more prolific pollsters in the latter group. “Non-transparent pollsters happen to be affiliated with a large number of organizations with only a handful of polls to their name – i.e. pollsters who are prone to greater error.

For comparison, re-ran Nate’s PIE (Pollster Introduced Error) calculation using a level playing field for all 262 pollsters on the list. I set the error mean at +.50 (which is approximately the mean error among all pollsters).

Comparing the relative pollster ranking between the two lists produced some intriguing results. The vast majority of pollster ranks (175) did not change by more than 10 spots on the table. Another 67 had rank changes between 11 to 40 spots on the two lists; 11 shifted by 41 to 100 spots, and 9 pollsters gained more than 100 spots in the rankings because of the transparency bonus. Of this latter group, only 2 of the 9 had more than 15 polls recorded in the database.

Nate says that the main purpose of his project is not to rate pollsters’ past performance but to determine probable accuracy going forward. But one wonders if he needs to go this particular route to get there. Other aggregators use less elaborate methods – including straightforward mean scores – and seem to be just as accurate.

His methodology statement is about 4,800 words (with 18 footnotes). It reminds me of a lot of the techies I have worked with over the years – the kind of person who will make three left turns to go right.

This time I think Nate may have taken one left turn to many. We’ll know in November.

Sunday, June 13, 2010

New Jersey Property Taxes Far and Away the No. 1 Pet Peeve

This column originally appeared as an Op-Ed in the Gannett New Jersey newspapers on June 13, including the Asbury Park Press, Courier News, Courier-Post, Daily Record, and Home News Tribune.

What do New Jerseyans answer when asked to name the most unfair tax they pay? If you said property taxes, you are correct. If you said something else, I’d like to be among the first to welcome you to our state.

In public opinion polls stretching back nearly 20 years, New Jersey’s property tax consistently ranks as the most detestable when stacked up against the federal income tax, state income tax, and state sales tax. In a September 2009 Monmouth University/Gannett New Jersey Poll, 59 percent of state residents chose property taxes as being the least fair. This is similar to the 61 percent who said the same in 2005, which was up slightly from the prior decade – 54 percent in 1994 and 47 percent in 1991.

The bottom line is that no other tax in New Jersey comes close to raising the public’s ire as much as the property tax. In fact, property taxes are the number one reason people give for wanting to move out of New Jersey.

In a poll we conducted earlier this year, 80 percent of homeowners told us that, if they had to choose, they would like to see their property tax bill reduced rather than their state income tax bill. And that even included a majority of those who currently pay more in state income taxes than they do in local property taxes!

To be fair, we in New Jersey are not alone in detesting our property taxes. An Elon University poll of residents in supposedly low-tax North Carolina last year found 49 percent who said that local property taxes in that state were unfair. This result was nominally higher than the 46 percent who rated North Carolina’s personal income tax as unfair. And this sentiment even extends overseas. A YouGov poll taken in Great Britain in 2007 found that 67 percent rated their local council tax – their property tax equivalent – as unfair, compared to 41 percent who said the same about the income tax.

At its core, there is something about how property taxes are levied that just irks people. While income taxes are based on wealth, property taxes do not take into account one’s ability to pay. And that just strikes people as unfair, regardless of how much their property tax bill is.

The big difference between New Jersey and other places seems to be the all-consuming nature of the issue. During last year’s campaign for governor, the number one issue Garden State voters wanted addressed was – you guessed it – property taxes. I doubt that is true in any other state.

It may be useful to compare New Jersey public opinion today to California in the late 1970s. Back then, a Field Poll found that 70 percent of Golden State residents said their state and local taxes were too high, with 60 percent laying most of the blame on property taxes. In 1978, California voters passed Proposition 13, which placed a 2 percent cap on property tax increases. Two years later, 70 percent of the state still said their taxes were too high, but this time they tabbed the state income tax as the primary offender.

The message from the California experience seems to be that you’ve got to either reduce spending on services or wind up shifting the burden somewhere else. And it’s interesting to note that even 30 years after Proposition 13 capped annual property tax increases at 2 percent, 29 percent of California residents still say their property taxes are too high.

This is certainly food for thought as Governor Christie and others push for a 2.5 percent property tax cap here in New Jersey.

Thursday, June 10, 2010

New Jersey's Primary: What's it Mean for November?

So what does the outcome of New Jersey’s Congressional primaries mean? The Tea Party movement certainly made a statement, but there is a lot more to the message voters sent on Tuesday. Basically, voters of all stripes are frustrated, but that frustration was voiced in different ways by Republicans and Democrats.

On the GOP side - assuming Anna Little holds on in the 6th district - the Tea Party can claim only one clear winner. However, nearly all their candidates had stronger outings than are typical for challengers to the party organization’s anointed picks.

Typically, a House incumbent facing a primary challenge in New Jersey will garner a majority of 85% or better. This year, Rodney Frelinghuysen (R-11) and Frank LoBiondo (R-2) were held to about three-quarters of the vote, while Chris Smith (R-4) fell just shy of 70%, and freshman incumbent Leonard Lance (R-7) eked out a 56% majority against three challengers. Scott Garrett (R-5) was the only Republican incumbent to avoid a challenge.

In nearly every other Republican contest, the party-line candidate was held to 60% or less of the vote: 1st - Dale Glading 55%; 3rd – Jon Runyan 60%; 6th – Diane Gooch 50%, 12th – Scott Sipprelle 54%. Only Roland Straten, a sacrificial lamb in the 8th, exceeded 80%.

Clearly there is some disagreement within the Republican Party base about who should bear their standard. But, contrary to most media reports from Tuesday, these near-upsets were not due to low turnout.

More than 245,000 New Jersey voters cast ballots for Republican House candidates on June 8. To put that in perspective, votes cast in GOP House primaries between 2002 and 2008 ranged from 132,000 to 193,000. [The larger number was driven by a hotly contested 3-way race for the U.S. Senate nomination in 2002.] This was, by most measures, a very strong Republican turnout for a primary without a statewide office at stake. [See NJ House turnout trend tables below.]

The fact that 10 out of 13 GOP House primaries were contested (including 4 out of 5 incumbents) is just another indication that voters want to send a message. In a typical year, only one or two GOP Congressional primaries attract more than one candidate (although that number did jump to 7 in 2008). The increased turnout seems largely attributable to these challengers.

For example, Chris Smith typically garners 16,000 to 20,000 votes in an uncontested race. Faced with a challenger, he was only able to increase his total to just over 21,000, while Alan Bateman tallied nearly 10,000 votes.

And, while Leonard Lance increased his vote total from 10,000 in 2008 (when 7 Republicans were vying for an open seat) to 17,000 this year, his predecessor, Mike Ferguson, garnered similar numbers in his uncontested primary races.

On the Democratic side, the story is much different. Only 3 incumbents faced challengers, and 2 won with typically large 86% margins – Rob Andrews (D-1) and Albio Sires (D-13) – while freshman John Adler (D-3) was held to 75%. The remaining five Democratic officeholders ran unopposed.

But there still were a total of 5 contested Democratic House primaries this year, when 2 or 3 is the norm. And yet, Democrats could only manage to get 158,000 of their voters to the polls. Democratic turnout for House races in the prior four cycles ranged between 184,000 and 278,000 (the higher number coming in 2008).

This is exactly the kind of disappointing primary turnout that foreshadowed Jon Corzine’s defeat in last year’s race for governor. After the euphoria of 2008, Democratic voters seem forlorn.

One interesting example is in the 1st district. Between 2002 and 2006, Rob Andrews could count on at least 18,000 votes when he ran unopposed. With a challenger this year, the vote total – including Andrews and his challenger – was less than 16,000. Steve Rothman’s (D-9) 14,000 votes was also a few thousand shy of what he typically gets.

Other Democratic incumbents, Frank Pallone (D-6), Donald Payne (D-10), and Rush Holt (D-12), pulled in numbers similar to their prior races. Only Bill Pascrell (D-8) at 13,000 votes performed slightly better than in past contests. [One caveat: Without a statewide office at stake, uncontested House primary turnout can be driven by down-ballot races, which generate varying levels of interest by district]

So what does this all mean for November?

On one hand, the Democratic base shows very little enthusiasm. Advantage Republicans.

On the other hand, the Republican base appears angry and divided. If that rift can’t be healed and those with Tea Party affinities can’t be persuaded to support the GOP nominees, these voters might sit on their hands this fall. That would be good news for the Democrats.

The bottom line is that a lot of voters – for widely different reasons – are simply unhappy with the performance of their government. The question is: Do they voice their frustration at the ballot box, or do they throw up their hands and refuse to participate in a system they see as unresponsive?

That unpredictability could to lead to a wild fall campaign.

Ironically, though, it could just as easily lead to maintaining the status quo.

At least, that’s what New Jersey’s 13 Congressional incumbents hope will be the case.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

REPUBLICAN HOUSE PRIMARIES – BALLOTS CAST
CD20102008200620042002
19,2747,139nc8,134nc
224,41818,05114,44013,37725,335
327,79925,71916,22816,01024,884
431,37716,81816,10914,35019,667
529,05319,91427,50721,26536,080
613,3996,7065,3095,6136,505
730,57725,57716,42313,26220,244
87,0275,8593,9824,4156,437
99,3785,7976,1692057,336
101,000ncncnc2,005
1142,65328,03526,46124,41029,691
1216,5449,7189,3838,28411,902
132,3061,7392,7693,0913,420
Total244,805171,072144,780132,416193,506

DEMOCRAT HOUSE PRIMARIES – BALLOTS CAST
CD20102008200620042002
115,81638,67218,41820,43118,362
25,24316,4659,97110,7079,182
315,00418,1309,96011,73110,431
47,90813,1148,3689,5398,589
56,91215,36510,3418,8766,365
610,83418,60910,93414,38211,005
77,84315,7768,6318,9066,857
813,08619,94811,08311,80010,462
913,90425,41818,51317,71216,362
1021,46430,76422,36125,39740,252
118,65215,6128,2358,2676,462
1214,28423,65313,31514,7799,618
1317,41826,52734,14738,47337,357
Total158,368278,053184,277201,000191,304

        2010 numbers are unofficial election night vote counts.
        Contested races with more than one candidate on ballot are in red.
        “nc” = No candidate on the ballot.