Does this paper, using neural nets, prove industrialization is irrelevant to global warming?

Question

Recently, a paper was published by Abbot and Morahasy (2017) - see specifically Fig. 2. The thesis of the paper is to train an artificial neural network (ANN) on the temperature time series for pre-industrial ages. Then do forecasting with the ANN to predict the temperature time series in the 20th century. Because the ANN was trained on pre-industrial time series, then we can say that industrialization is irrelevant to global warming.

This goes completely contrary to the IPCC AR5 technical summary, which shows graphs that global warming is due to anthropogenic causes. See page 74 of the latter report.

So which do you trust and why?

Answer 1

The paper is deeply flawed from both the climate science and machine learning perspectives. The most obvious being the most eye-catching claim that equilibrium climate sensitivity is approximately 0.6C, which if true would overturn our understanding of the climate system. However the paper doesn't actually explain how this figure of 0.6C is obtained from a "largest deviation" of 0.2C, it is basically just a hand-wave. Also the largest deviation is not 0.2, this is the largest average (mean absolute) deviation seen in the proxies, and you can have a mean absolute deviation without there being a trend that you could relate to increased GHG concentrations and hence estimate ECS. More importantly, this would give an estimate of transient climate sensitivity, not equilibrium climate sensitivity, and you can't reliably estimate ECS (which is global) from regional or sub-regional proxy records.

Approaches such as the one taken in this paper, which seek to see how much of the data can be attributed to "climate cycles" with the remainder being taken as the anthropogenic component are inherently biased towards low estimates of ECS. This is because of omitted variable bias; because the anthropogenic forcing signals are not included in the model, if the net effect of independent changes in the forcings is correlated with a sinusoidal component, they will be wrongly attributed to these climate cycles, when in fact they are produced by the forcings. Models like this can only be used to estimate lower bounds on ECS, likewise if you make a model using the forcings as inputs and treat the residual as being "natural variability" it will tend to over-estimate ECS, giving an upper bound estimate. The Abbot and Morahasy paper cites a similar paper by Loehle, but sadly does not also cite the comment paper (of which I was the lead author, note the corregendum). This is poor scholarship, and sadly Abbot and Morahasy have gone on to make many of the same basic mistakes (but with a more complicated model), which is a shame.

Rather than using the original datasets (many of which are freely available) the authors chose to digitise images of the datasets instead. This seems somewhat bizarre, and Gavin Schmidt points out via twitter that in at least one case the dataset has not been scaled or aligned correctly (andends at 1965, and so does not include the recent warming where anthropogenic contributions are most evident). It also transpires that Figures 5 and 9 are identical.

The paper says "However, superior fitting to the temperature proxies are obtained by using the sine wave components and composite as input data. This was established by comparing the spectral analysis composite method versus the ANN method for the training periods.". Evaluating performance on the training data is a classic error in the use of machine learning that people used to make all the time in the late 1980s and 90s, but is rarely seen today. If you have two nested models (one can be implemented as a special case of the other) of different complexities, then the more complex model will always have a lower training set error, if only because it has more capacity to memorize the random noise in the data, but that doesn't mean it is the more accurate model. For that you need out-of-sample comparisons, which are absent from the paper.

There is no handling of uncertainty in the model (for instance the periods of the cyclic components are not known exactly, cycles with slightly different periodicities will explain the observations almost as well), and likewise there will be uncertainty in the parameters of the neural net, but none of this is propagated through to give the uncertainty in the estimate of ECS. As seen in the comment on the Loehle paper, this can be substantial.

Table 13 seems to suggest that paleoclimate studies give lower ECS estimates than GCMs, which suggests a rather selective view of the paleoclimate studies, which generally indicate high ECS IIRC.

The study also has problems with too many degrees of researcher freedom (e.g. how was the particular subset of proxies chosen?) and there is a lot of (automated) exploration of model architectures and feature selection, which is often a recipe for over-fitting in model selection. It is also not clear why the observation should be a non-linear function of the cyclic variables (especially given that the cycles were obtained from the data by linear analysis).

Answer 2

Astrophysicist Ken Rice, author of the "And Then There's Physics" blog on climate science, dismisses this particular approach for, among other things, failing to address the underlying physics of the climate system:

...if you are going to use something like machine learning to make predictions about the future, you do need to be pretty confident that the data that you use to train the machine learning algorithm presents a reasonable representation of the system you’re trying to model. This requires some actual understanding of the system being considered. If you change it by, for example, pumping lots of greenhouse gases into the atmosphere, then the training data will be almost certainly not be appropriate.

Climate researcher Gavin Schmidt has also noted errors in figure 2:

Their time axis is off by ~35 years and magnitude is too large by ~10%. So their '20th C' is actually 1845-1965.

Which means that Abbot and Morahasy's analysis also doesn't include the end of the 20th century and the most recent decades of significant warming.

Answer 3

Machine learning may sound like some magical phrase, that is able to solve every problem, but it's definetly not. I don't have the paper available right now, but what I can say in general right now is the following:

What machine learning is good at, is finding patterns in any number of dimensions. If you apply machine learning to a time series (a 1-D data space) that fluctuates boringly around a mean average, as it did in pre-industrial times, then the only thing it will be able to predict is the continuation of this trend.

Furthermore, taking the ANN to predict data outside of it's training range is equivalent to just taking an extremely-high order polynomial through all your datapoints and then extrapolating into the future. The former is a classical example of data manipulation, also strongly advised against in every introductory physics or statistics class.

The last paragraph shall, without rigorous proof, just show that using ANN to predict trends in any kind of data, is not very smart and reveals a fundamental misunderstanding of how ANN work.