The Signal and the Noise: Lessons for the Simulation Community from the Failure to Predict the Outcome of the 2016 Presidential Election

Don’t worry, this blog has nothing to do with politics. Instead, I wanted to spend some time trying to understand the lessons that we as engineers can learn from the failure of the prediction community to successfully forecast the outcome of the 2016 Presidential Election. For me this was an important backstory to the 2016 campaign: the pre-election poll predictions were wrong by a significant enough margin that they completely failed to forecast the outcome of the election.

This includes the prediction endorsed by celebrated American statistician Nate Silver, who famously correctly forecast the outcome of 49 out of 50 seats in Obama’s 2008 election victory (missing the 50th by a single percentage point). Although Silver’s prediction was much closer than most, many of whom gave Hillary Clinton a 95 percent or more chance of winning the election, his carefully calculated model had Donald J. Trump’s chance of victory at no more than 30 percent (although we need to acknowledge that 30 percent is a relatively big chance of any outcome, he wasn’t saying it could never happen). He, like everyone else (including possibly President-Elect Trump), was surprised by the outcome of the election.

“What does this have to do with engineering simulation?” I hear you ask. The answer, I think, is “a great deal.” CFD engineers, like psephologists*, are also interested in making predictions about the future. Whereas Silver and his peers use statistical inference (from opinion polls and other data sources) to try and predict how people will vote in an election, we use numerical models of physics to predict the future performance of a proposed product or design.

As accurate as engineering simulation is, we have to acknowledge, that just like all predictions, it rarely, if ever, gives us an “exact answer” that we can rely on without any interpretation or scrutiny. All simulations rely on modeling assumptions (for example, to do with discretization, limitations of the physics models, boundary conditions, etc.) and are subject to errors (including operator error and uncertainties in boundary conditions). It’s typically true that if you gave a sufficiently complicated simulation scenario to five different CFD engineers, the modeling choices that they would make would result in five different simulation results. Hopefully, if the engineers were sufficiently competent, the results would be within a few percentage points of each other.

This is, of course, a problem of all prediction, not just simulation (including predicting the outcome of an election). Exactly the same thing would happen if you gave the same scenario to five different experimentalists (or looked at the results of thousands of different opinion polls). All experiments involve a degree of simplification (scaling assumptions, boundary conditions, etc.) and sources of error (operator error, measurement error, sampling error, etc.).

In his excellent book The Signal and the Noise: The Art and Science of Prediction (Why So Many Predictions Fail – But Some Don’t), Silver takes a rational look at prediction in all its forms, from earthquake prediction to climate change, from sports betting to the catastrophic failure in prediction that resulted in the subprime collapse that plunged the whole world into recession. Although he doesn’t examine CFD or engineering directly, he does explore the world of weather forecasting (which let’s face it, is glorified CFD on a massive scale) and comes to the conclusion that it is one of the most accurate predictive methods available, mainly because weather forecasters have the opportunity to compare the actual weather against the previous day’s forecast on a daily basis.

Although it could be regarded as post hoc rationalization, I think that it is incredibly healthy for everyone involved in the prediction game to critically examine the results of their simulations against the outcomes of their predictions. Including acknowledging those occasions in which our predictions do a poor job of predicting the real-world performance of a product.

In the engineering simulation game, we used to call this “validation,” in which we measured against some experimental model. However, in the world of Big Data, where many of the products our simulations help to produce are tied to the “Internet of Things,” we will increasingly have the opportunity to validate our predictions against real-world usage data. This is a unique opportunity to make our predictions better.

My favorite example of this is the Tesla Model S, which on release had an active suspension system that changes the ride height of the vehicle depending on how it is driven: “As Model S accelerates, it lowers the vehicle for optimized aerodynamics and increased range.” Through examining usage data (transmitted by the vehicles back to Tesla) they were able to work out that the system wasn’t working as expected (and that some of the vehicles were hitting bumps in the road) and so they were able to issue a patch to the operating system that automatically adjusted the ride height of the entire Tesla community. It isn’t difficult to imagine a scenario in the future in which this usage data can be used to better validate the simulation models that were used to design the vehicle in the first place, and will be used to design the next generation of vehicles.

More scrutiny and better validation will always lead to better simulations, which will ultimately lead to even better products. And maybe even more accurate election predictions…

__________________

*why can’t we as “CFD engineers” have a great collective name like “psephologists”?