Presidential assassination, war, video proof of something incredibly heinous (pedophilia?), etc. can absolutely lead to these outcomes. You don't even have to go that far back. Nixon and Reagan flipped states like no-one's business.
I do however agree, that 538's state-state correlation model seems weak.
California and Alabama would only flip during a wave, and that wave would consume any and all states. The fact that 538's model doesn't strongly show that pattern is a failing of it. But, it is not clear if a model that inaccurately models the unlikeliest of events (california flipping while Florida stays blue), does not necessarily mean that it is terrible predictor of it's primary target (Presidential likelihoods).
As a data scientist, I can totally understand Nate's hesitation. Do you impose strong priors on the model to reflect strong domain intuition or do build a model that best characterizes the data it is based on. In the presence of infinite data, you should abandon all domain based priors. For single digit data points, priors are essential. For any number of data in between, it is anyone's best guess.
Something could flip California and Alabama (example, Trump starts defending Roe v. Wade and in response Biden somehow manages to sound like he's opposing it). This would probably be some latent hidden variable, like whether the candidates are seen as socially conservative, which would effect all states (though California and Alabama would be the most impacted).
Constraining against them won't improve your models fit (usually by definition), and it doesn't always improve robustness (at least for situations near average)-- because they're acting to debias the model in ways that you otherwise don't have enough degrees of freedom to address.
A negative correlation here is also potentially historically supported, in the sense that sometimes DEM/GOP candidates are philosophically reversed in some way relevant to the state. As in, "The only way a GOP would get elected in X is if they had the DEM position on subject Y which would make them lose state Z, who cares as much about that subject as X but in the opposite direction."
Now-- it doesn't seem likely case in this election (e.g. Trump is not (currently) a massively pro-choice republican), so it probably shouldn't apply here-- but it's isn't hard for me to imagine how a negative correlation might show up out of the historical data.
The Economist model does exactly that, and all of their correlations are positive.
I recommend reading their methodology, they know what they're doing (I wouldn't say the same about 538). Andrew Gelman has developed some of the Bayesian methods and software that people like Nate Silver use, he's the main author of what's considered a reference book on Bayesian statistics.
It's clear that the models are tuned differently, but from Silver's replies in the PS's, it seems that he's ok with these artifacts being part of the model.
It's very interesting to see how long it takes people to do things. I am amazed that entire article took 1 hour to type up. I've spent entire afternoons trying to write shallower pieces of work.
Granted, i could just be misreading this post. :)
https://projects.economist.com/us-2020-forecast/president
You can compare this to the 538 model and see where these two teams and forecasts disagree.
The model says that Trump has a 1 in 10 chance of winning. With a fair 10-sided die it makes sense that you have a 1 in 10 chance of any given side rolling face up. But what is the die that is being rolled in these election statistics? What is the "chance" element that is being predicted?
In the dice toss scenario, we know everything relevant. In the election scenario, we don't.
A model like this is attempting to say "these are the rules we think exist. Based on the rules, and assuming the data is off by some random distribution, here's what we think could happen".
What different forecasters disagree about is what the rules are. For example, the relevance of certain demographic characteristics and the potential variance between polling (conducted prior to the election) and actual election results.
There's a huge amount of assumptions, and forecasters disagree on those assumptions. We have very little historical data (polling is very recent) and even with complete historical data, future elections do not always conform to past elections.
Once again, I’m not an expert, so I recommend looking for additional explanations, if you’re interested.
A program which randomly generates outcomes for each state, based on probability distributions inferred from the polls, and calculates who wins the election given those outcomes. They run the program repeatedly and report the proportion of simulated wins as the probability of winning. https://en.wikipedia.org/wiki/Monte_Carlo_method
So, think of it as saying these facts basically describe a ten sided die. With no other knowledge, the best you have is that you expect it to behave the same as any other ten sided die.
(There are a couple of caveats about election forecasting as opposed to weather forecasting. The first is the "October surprise," a sudden revelation that changes the election. This cycle, it was arguably Trump's covid diagnosis, although that tended if anything to push the results further in the direction they seemed to be going on their own, rather than upset any trend. The second is that, unlike with weather systems, measuring voter behavior (and widespread reporting on these measures) can change people's behavior. The effect of this is hotly contested, but one of the many explanations of Trump's victory in 2016 which hinged on turnout in a few key states is that those states were predicted wins for Clinton, so Clinton voters didn't bother voting. Despite occasional jokes to the contrary, it doesn't rain just to spite the weatherman.)
The frequentist interpretation is roughly that if I go around making my best possible predictions, and we lump together all the things that I predict at 10%, about 1 in 10 of those things happen and the rest don't. But I wouldn't be able to be more specific about which ones in that group are more likely than others.
The Bayesian interpretation is that I can really view the world as flipping coins -- I don't care whether it's due to my lack of knowledge or "true" randomness -- and as far as I can tell, the coin flip involved here is 1 in 10.
We can also use a gambling interpretation. Here's one based on security of python's random module. Imagine the following three lotteries I offer you. In lottery A, you get $100 if Trump is elected. In lottery B, you get $100 if the following python code returns true on my laptop:
random.random() <= 0.09999
random.random() <= 0.10001
Now there's an interesting extra layer to all of this because it's a model predicting, not a person. In a short space, I would basically say that we've trained models to predict in ways that are not inconsistent with any of the interpretations above, when put into situations where that is testable. Then we use them in situations where it might not be, like this.
It's different than probability (1/10 = .1)
At the bottom of the 538 page it says, " If you choose enough unlikely outcomes, we’ll eventually wind up with so few simulations remaining that we can’t produce accurate results. When that happens, we go back to our full set of simulations and run a series of regressions to see how your scenario might look if it turned up more often."
I interpret that as running a regression (linear?) and extrapolating it out to the tail where the conditioning is happening. This should eliminate the issue Andrew is seeing?
In that particular WA-MS example, if Trump suddenly took more liberal positions and somehow won WA (e.g., announces he's pro abortion), he would in fact be more at risk of losing Mississippi. The idea that these two states are in play already is fringe and would require some major idealogical (or other third variable) shifts.
Specifically, that when you get off into the weird situations like Trump winning Washington state, it's likely something incredibly weird has happened - something that likely has no historical precedent, so it may actually be a more sane thing to do to assume that now almost everything is backwards and Biden would win a bunch of states he shouldn't either.
To me, this points to a general willingness in the 538 model to just go "who knows" and build in some room for insane things to happen on the fringes. The Friday podcast episode about the 538 model specifically mentions that they have large/fat tails on their distribution that make it nearly impossible for someone to get over 95% chances of winning on a national level, and these sorts of wild results seem like the outcome of that. If you bake in an assumption that there's always a 5% chance of something crazy happening, that chance has to come from something in the data somewhere that reflects the ability of that to happen numerically, and thus will have numerical outcomes that seem impossible.
The negative correlation makes sense when we think about how difficult it is for everyone in Washington to suddenly turn conservative and everyone in Mississippi to turn liberal. Much more likely is that the crazy thing is that the candidate or circumstances changed in some way.
It makes more sense if we ask...if a candidate wins NJ what is the chance they also won AK?
It's a neat possibility to think about though. If there were enough people who did that, it would really depend on the demographics moving and where they're going. It could swing the election either way. I wonder if anyone has found numbers on this and attempted to model it.
Moving out doesn't stop you from voting. I didn't change my voter registration when I moved from San Francisco to China. Years later, back in California, I voted in San Francisco, where I was still registered, despite residing in Hayward.
For verification purposes, they asked me when I voted what my address was. I was allowed to vote despite not knowing my own apartment number.
Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016.
Gelman’s own grad student is the only person to have academically published this approach, in a paper about polling Xbox Live users.
These guys sort of make a thing that is the same in many more ways than it is different. Why not just share the code is the biggest question?
The only sensible way to predict probabilities that aren't extreme is to tell people how the model works and the figures it is currently spitting out. That's is the great thing about these kinds of blog posts, people are kicking the tyres, not just looking at the car.
Nobody predicting a one-off election with a rather special candidate would summarize a 33% chance as equivalent to having no chance.
The mistake in 2016, IMO was a) the extrapolation that came from those polls and b) people paying way too much attention to national polls, which have very little connection to electoral outcomes, given the electoral college.
Also perhaps c) the larger public not “getting” statistics in the way they’ve been presented. The NYT had, if I recall, Clinton at 90% chance of winning. That still means that in one of every ten flips of a coin is a Trump win. But people read “90% chance” as “definite win”. I don’t actually know what anyone should or could do about that.
I find this argument strange, because black turnout was unusually high in 2008. That should have a negative impact on the accuracy of statistical adjustments, not a positive one.
However, we will never know, because they never published the code.
I don't think it's likely but if those polls are indicative of what's actually happening, we're talking about potentially a 2-4 million vote swing in Trump's favor. Here's a link to estimates of voter turnout in 2016 [2].
[1] https://fivethirtyeight.com/features/trump-is-losing-ground-... [2] https://www.pewresearch.org/fact-tank/2017/05/12/black-voter...
We’ve seen that variable before.
That's a valid intuition to have but you can also clearly make the argument that if Trump wins California you're in such a weird scenario that using the traditional wisdom about correlation is dangerous. The point that 538 have tried repeatedly to make is that firstly: if you're conservative in your level of confidence you'll give a higher likelihood to outliers, and secondly: It's not particularly useful to focus on whether X has a 3% or 4% chance.
If Trump wins California, we aren't going to be talking about whether the chance was 3% or 0.3% we're going to be talking about that Nuclear explosion that wiped out 25million Californians.
For the same logic the reason that Trump winning Alaska given winning New Jersey is lower than given losing New Jersy is because your sample size is rubbish. The chance of Trump winning Alaska given losing New Jersey is an accurate number, the number of Trump winning Alaska given winning New Jersey is like saying "How likely is it Trump wins Alaska given the UK gains US statehood" it's like.... well... if that happens then we're so far outside of what the model thinks can happen then you should be that we're just gonna say it's 50:50 - because who the hell knows.
It's not like saying "Oh well if X swing state goes blue, Y will probably follow", the scenarios in this article are so bizarre that the model should rightly be very cautious and probably default to either refusing to give an answer or just default to 50:50 or the same probably ignoring that data. The implicit bias in this analysis seems to be that if NJ went Red that would be because Trump won by a big margin, but that's not a likely enough scenario to actually get numbers for, and is so unlikely that things like "The supreme court threw out all the ballots for inner city areas" start to become valid possibilities.
A statistical model only has a vague idea of context/the real world. It looks at polls (and probably not really that many polls of Alabama or Mississippi or Alaska) and sees that, statistically, Biden should win 3% of the time or so.
It doesn't have a specific world set of events in mind that would cause that, it just knows that that's how the numbers go, and thus may lead to weird circumstances in the grander results because it has to make the world match the numbers in these small corners.
So, it seems to me that the entire article is predicated on a faulty conjecture, namely that 538 uses a mixture of a normal distribution with an independent heavy-tailed one. (It's not explicitly stated what the author thinks the base model is, but I think "normal" is a reasonable guess.)
I'd be interested in seeing a reverse-engineering analysis of 538's choice of distribution parameters, and extrapolation from there to see if these pathologies still arise with (much) larger samples.
...
That said, ultimately, the choice of how fat to make the tails is a modeling decision, and how the models behave outside the regime of interest isn't as important as how they behave within the operating region. There are key ways we can evaluate goodness of fit once we have results (e.g. bias, MSE) which we can use to determine just how wrong the model was as a predictor, and chances are pretty good that we won't see, say, Trump winning NJ, so we won't actually be able to validate the tail correlation with the vote in PA. But we will be able to validate the correlation in margin between PA and NJ.
Maybe 538's tails are too fat, and every prediction in the 80-95% range ends up going as predicted. Or maybe they're not fat enough, and some races in the 99% bucket end up going the opposite way. Point is, we won't know for sure which models were the best predictors until we can verify the predictions.
(see: all models are wrong, etc. Newtonian mechanics work great as long as your objects are big and slow, for instance.)
I'm not sure why that kind of interstate correlation should impact predictions?
<incoherent rambling :D> IANAS but it feels like these correlations were added to compensate for the failure in 2016 to recognize that state A going one way implied that state B would also go that way. It "feels" like a more correct approach would be to compute some kind of error/weakness measure in a states polls by bringing in those of its geographical neighbors and incorporating the polling error of that entire block vs prior years. Or something.
The intuition I'm having difficulty conveying is that actual voting correlation is based on neighboring states only because you've got bubbles of ideology that aren't strictly cut along state lines. If strength of opinion in a bubble is going one way, then you'll see that mostly in the state at the center of the bubble, but the bubble still spreads into neighboring states, and a "stronger" bubble could push it geographically further into those neighbouring states, and/or could increase the bias in areas inside the bubble. </rambling>
[1] "Always" == most recent history
538 has low positive correlations between states on average, which actually has a big impact, it increases overall uncertainty (and therefore Trump's win probability). Why? If the states are not correlated, you usually end up with a few states going off the rails, like Trump winning Colorado without any nationwide swing.
Edit: Why the downvote? Each of the between-state correlations can be calculated from 40,000 datapoints.
Additionally, getting worked up about a 3% chance of Biden winning Alabama. I mean, what does a 3% chance even mean for a one off event, compared to a 5% chance or a .3% chance? I know fully well, that it means I should bet $100 if I can get more than $3000 payout, but the trouble is that is only if we bet often enough. (Perhaps often enough on different things.) For a one off thing, the important part is, it is with a very high degree of certainty a loss of $100. So any claims that Bidens chances of winning are too high should be regarded with high suspicion.
Also, I listened eralier to Nate Silver's model talk [0], where he discusses quite a few problems with low quality polls in some states.
[0] https://fivethirtyeight.com/features/politics-podcast-nation...
There are more than enough data points to determine the between-state error correlations, many of which seem to be very off.
> Additionally, getting worked up about a 3% chance
The weird between-state correlations actually have a large effect, they increase state and nationwide uncertainty and as a result Trump has a higher chance of winning.
What value does something like fivethirtyeight add to our democracy, if any? Is this motivation the same as that of diving deep into baseball stats or Star Wars starship engineering, just like “nerding out” for its own sake?
Contrast the voter who never looks at any of these polls with one who keeps up with them daily. Is the latter voter better off in some way? Is this just about trying to read the tea leaves so you can strut and preen later about having been correct, should the dice roll be in your favor?
My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.”
Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say.
Someone set me straight here, what is the point of all this stuff.
One possible use of polls and election models is for helping people who want to donate to candidates determine which races are closest and where their money is most likely to have an impact.
> Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say.
Because the US does not have ranked choice voting in most elections, polls are useful to determine which candidates are viable. If your preferred candidate is only polling at 5%, they are pretty unlikely to win, so you might want to vote instead for whichever of the leading candidates you find most agreeable.
In this system, there are two ways to democratically influence politics by voting:
1. Vote your preferred candidate
2. Withhold your vote from the party more closely aligned with your views, in hopes of helping shift its coalition priorities
If you fall into the second category, accurate forecasting makes a strategic difference.
On the one hand, I think I get your sentiment. On the other, I mean, we are all just solitary individuals floating through this life. Almost all of our important decisions are make at least in part (or more in some cases) dependent on the thoughts and actions of others. That's natural, right? You do otherwise in your life?
If the above is true, it makes total sense why one out of 7 billion plus people would want to understand the choices of others before making theirs.
Great question, and I share some of your concern, though I can imagine some positive framings in addition to what you wrote. For example, to use an analogy, what’s the point of trying to predict the weather, or trying to predict the stock market? There are lots of reasons including planning ahead for likely outcomes, the ability to protect against losses, and last but not least making money.
I can also imagine that the desire to talk about the potential outcomes is valuable as a social activity, and doesn’t necessarily need to meet a standard of influencing the vote, or adding to our democracy.
> My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.”
Of course if your concern is founded, this can go both ways... if the polls show the candidate you favor starting to lose, it could be a call to vote.
If polls are distracting and dissuade voters, then unfortunately election results might do exactly the same or worse. When a state has been solidly red or blue and not purple for 50 years in a row, people do (perhaps rightly so) jump to conclusions about the outcome in advance.
One question we could ask is whether, if voting were made mandatory, would election predictions go away? I’d speculate no.
Also, you can make money betting on the outcome itself. If the odds you get are underpriced relative to an accurate forecast, that's a great bet to take.
Furthermore, these forecasts influence where politicians put their focus. Let's say you're Hillary in '16 and you think Wisconsin is yours despite the forecast showing a narrow lead, maybe you should reconsider.
It's an orgy of false precision.
edit: the entire debate is based on the weird assumption that if a prediction about a particular state is wrong, then pollsters must have systematically gotten middle-class Hispanic women over 40 wrong, therefore the odds of other states will change. It's all based in the reification of particular categories that are axiomatically significant for their profession.
> It's an orgy of false precision.
The false precision is pretty obviously coming from you, not the FiveThirtyEight pages that never show more than two (or rarely three) significant figures, and emphasize in every other way they can that the numbers are approximate and uncertain. Have you seen the width of the 80% confidence intervals on their graphs?
As for falsification: all of their predictions are for testable outcomes. We'll always know soon enough who actually wins an election, and which states they won, and by what margin, and who turned out to vote. That's all public record. The only part of the post-hoc analysis that is non-trivial is figuring out how a candidate fared with specific demographic groups. It's imperfect, but between exit polling and precinct-level demographic information and election results, it certainly is possible to detect large pre-election polling errors resulting from inaccurate demographic weighting.
So, wait, you're offended by the election modellers making a prediction, and yet you yourself are making a prediction? What's yours based on? Time machine?
Treating this as a tea-leaf reading (that is, deliberately searching for meaning via free association, without investing it with a truth value) I'm reminded of the "own the libs" meme. I see folks on foxnews.com comments bragging about it; I see lefties complaining about it, but I suspect that it's overblown and not actually a driver behind people's decision-making. But that's what comes up for me when I see "NJ goes Trump" forcing "AK goes Biden".
I'm amused by the resulting thought experiment... if dems started airing "socialists for Trump" campaigns in otherwise safe GOP states, would it move the needle there? Even sillier: if you aired those ads in NJ, would it move the needle in AK?
https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Ne...
This is it in 2016:
https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Ne...
Long Island is really red there. It’s really hard to say how a democratic stronghold like NYC and something literally a 45 min train ride next to it could vote so differently. Long Islanders are not separate from NYCers, they commute to and work in the city.
To your question, could experiments work in similar situations like this across the country for either side? I think so in the next 50 years as demographics shift (and I don’t think it’s as simple as urban liberals taking over, people do become more conservative as they get older). God knows the dynamic at work between NYC and Long Island in 2016, but it’s obvious things are in flux.
I’ll make a bold prediction here. If Long Island is that red again, yeah, you better believe the typical rust belt states are staying red.
Doesn't that sound like Berkson's Paradox?
Edit: Why am I being downvoted?
Likewise asserting that California with a 3% chance of going Trump is absurd is an unreasonable degree of overconfidence. Assuming maximizing expected return, the author is implying that they would be willing to take a bet that Trump would lose California with odds >> 97::3, i.e. presumably they would take a bet where I bet $1 to every $99 they bet. To be critical of a model based on outcomes it predicts with tiny probability you need truly remarkably biased priors.
I would be more than happy to make this bet with anyone willing to take the other side - as in literally, find a modern middle-man system and I'm game.
You may want to be aware that you have provided me a nontrivial arbitrage opportunity, as the odds on predictit are closer to 7:93 : https://www.predictit.org/markets/detail/6611
However, for my use of 538, I’m perfectly happy to ignore such scenarios (such as Trump taking New Jersey). I can call the election in his favour by myself in these scenarios without needing the model.
Why is there so much fascination with polls to begin with? I understand that there are betting markets, but it seems sort of silly. If you had a 100% accurate poll, for instance, then what would be the purpose of the actual election?
Polling is useful for candidates and gives them ideas on where to target outreach and spending.
For the rest of us, it gives us something to watch. With Presidential campaigns running for almost two years in advance of the actual voting weeks, there’s a huge gap between when the thing starts and when we see results. This way, people have something to fill the time. Even now that voting has started, we are still another 10 days until the voting is done and likely another 7 after that until we have a sufficient count to know who has been elected.
That’s a long time for a populace that’s worried, distracted, and interested, especially since so many of us live in states where—due to the mechanics of a broken election system—we can’t do much to influence the national outcome.
The only way that would work is if he made people more likely to lie about their voting intentions. Now, there may be something about Trump that makes polling methodologies less accurate (notably, many pollsters have started to take into account education, which turned out to be unexpectedly important last time round) but that points just to bad methodology, not inherent unpredictability.
To run a 100% accurate poll would require you to sample every voter, so it would literally be an election.
And apparently columbia.edu does not fulfill those criteria.
538 gave Hillary a 71.4% chance of winning
https://projects.fivethirtyeight.com/2016-election-forecast/
I think that's what he's talking about.
Incidentally, this trick is something magicians sometimes do. Sometimes when a trick has gone wrong they'll make a wild guess. If they're right, the audience is impressed. If they're wrong, they'll brush it aside with some joke and the audience won't notice/mind much. This works for things card guessing tricks and puedo-psychic/cold reading stuff.
