Signal versus noise

Everybody who is in financial markets knows about this problem.  Is the latest low US GDP data point the beginning of a new slower trend or is it a blip?  Is the stock market still going up--recent trends have been down, so is this a change in longer-term trends or what?  What we normally do to estimate trend in economics or shares is to use a moving average.  And you look at the fluctuations around the moving average.  Is there a pattern of rising highs and lows, or is each new high and each new low below the previous one?

What struck me about the charts in this article was how it's impossible to determine the trend in the unsmoothed data, but how clear the trend is in the smoothed data.  The chart shows the temperature record for central England.  Since 1945 it's been adjusted for the urban heat island effect, i.e., it has been reduced to compensate for the fact that cities are much warmer than the surrounding countryside.

Here's the chart of the unsmoothed data, with the December 2015 anomaly highlighted in the red circles:

Here's a simple 30 year moving average of the data:

Note that the moving average is only available up to 2001, because traditionally you centre the moving average at the midpoint of its span.  So the last observation on the chart above is the average from January 1986 to December 2015.  This is a problem with moving averages--you "lose" data at the beginning and end of your underlying data series.  There are some mathematical techniques you can use to get both smoother and more up to date moving averages.  One of these is the LOWESS moving average. (in these circumstances,  I use a Henderson curve, which is a bit less sophisticated and easier to calculate.)

You can see why those who don't understand statistics will be flummoxed by the big random day-to-day and month-to-month fluctuations. But the thing to focus on is the longer-term trends, and these are clearly shown by the moving averages.