Weather Forecast Accuracy

• FORUM
• Email this
  to a friend
  

• Submit Your
  Own Prices
  
  
• Sign Up to
  Our Newsletter

Weather Forecast Accuracy Summary

Which of the three commonly used UK weather forecast websites is the best? Everyday for two weeks, we noted BBC Weather's, Weather.co.uk's, & the Met Office's predictions for the up-coming days in our region and matched them up with what the weather really was in our region. All three websites performed much better than taking random guesses, you'll be pleased to know. However, out of the three websites, BBC Weather made a perfect prediction within 5 days 57% of the time, which made it superior to both the Met Office (47%) and Weather.co.uk (34%). Furthermore, when these sites did get it wrong, the BBC Weather predictions still tended to get much closer to the real weather than the other two websites' predictions. So, they're not all the same!

• Jump to Introduction
• Jump to Methodology
• Jump to Results
• Jump to Discussion
  

Introduction to British Weather (top)

Let's face it: a BBQ in the rain is a no-starter. Planning one for the up-coming weekend but really need to know the weather? You've probably noticed already that the weather websites are not all the same! What you want to know is which of the three most used websites for weather forecasts (BBC Weather, Weather.co.uk, & the Met Office) is going to be the least likely to let you down.

So, the WhatPrice geeks have taken the time to conduct some proper research on your behalf, in order that the long-asked question can be answered once and for all. What we did everyday for 2 weeks was to note each website's predictions in our region for the upcoming days, together with what the weather actually was in our humble opinions down here on the ground. In addition to this, we also made forecasts of our own, using computer code to randomly guess what the weather would be each day. With all this data it was possible to see which of the websites got the highest proportion of perfect predictions, how much each website was out by in their predictions when they got it wrong, and whether any of them were any better at predicting the weather than taking a random guess!

Weather Data Methodology (top)

Data was collected during March, 2007. Each morning for approximately 2 weeks the weather predictions for the up-coming days in our region (Cambridge) were recorded from each of the three websites. The BBC and the Met Office predicted weather up to 5 days in advance, and Weather.co.uk predicted the weather up to 10 days in advance.

There were a few differences between the websites that had to gotten around. The Met office website divided each day into multiple segments, whilst the others do not. Thus, only the predominant weather prediction for that day was noted from the Met Office site. Furthermore, as the different sites phrased their predictions differently and/or used different symbols to donate the weather, we had to use our own judgement to categorise each website's predictions into the 5 following weather categories:

• Sunny.
• Partly Cloudy
• Cloudy.
• Rain.
• Snow.

In addition to this, at the end of each day, what we personally felt was the predominant weather condition that day was noted. This allowed for a comparison between what each website predicted at each day in advance with what actually occurred.

Finally, to re-assure ourselves (or not!) against the common moan that weather predictions are so inaccurate it’s better just to toss a coin, we generated random predictions as to what the weather for the forthcoming days of the study would be using computer code. These "random guesses" were analysed just like the other websites predictions, thus providing a meaningful control.

Data Preparation

The results were analysed in two ways.

First, the absolute accuracy of each website's predictions was scored. For example, if the BBC predicted a sunny day in two days time, and the weather actually turned out to be sunny on that day, the website scored 1 for a two-day-in-advance prediction. If it turned out to be partly cloudy (or any other weather condition), the website scored zero. In this way, it was possible to work out what proportion of each websites predictions, on each day in advance, turned out to be perfect.

Second, it was assessed how far wrong each websites predictions were. For example, predicting a sunny day and their actually being a partly cloudy day is less far out than if the day turned out to be actually snowing. In order to do this, each of the five weather conditions was given an arbitrary numerical value (sunny = 5, partly cloudy = 4, cloudy = 3, rain = 2, snow = 1). For each prediction, the actual weather value was taken away from the predicted value, leaving an error value which was greater the larger the difference between the actual and predicted weather conditions. For example, predicting sun (5) but actually receiving rain (2) leaves an error score of 3 (5-2).

In order to have sufficient data points to run statistical comparisons in the accuracy and error scores of the weather websites, the scores for 1-5 days in advance were pooled into one data set, and paired t-tests were run for individual comparisons.

Weather Forecast Accuracy Results (top)

Perfect predictions

The research period turned out to be ideal for data collection, as we had at least two-three days of each of the five main weather conditions (sunny, partly cloudy, cloudy, rain, snow). Such changeable weather was the prefect test for the weather websites.

Figure 1 shows the percentage of times each website made a perfect prediction as to what the weather would be at each day in advance.

Figure 1: The percentage of times that each website made a perfect prediction as to what the weather would be, so many days in advance. The higher the bars, the better the website performed. Note that this shows absolute accuracy, so predicting sunny but there actually be some cloudy ("partly cloudy") is considered wrong for this graph. The higher the bars, the more accurate the website was. Note also that Weather.co.uk made predictions 10 days in advance, whilst the BBC and Met Office only made predictions 5 days in advance.

When comparing websites in such things, it is important to submit the data to statistical tests, which more objectively decide whether they’re really is a difference in performance between the websites. In order to do this, accuracy scores were pooled and averaged over 5 days for each website. The data for this can be seen in figure 2. Paired-samples T-tests revealed all of the websites got it perfectly right significantly more often than if they'd made a random guess (all p<.02). However, the BBC Weather website made a perfect prediction significantly more often than Weather.co.uk (t(1,57) = 1.73, p<.05). The Met Office performed somewhere between the two (though no other comparisons were significant).  

Figure 2: The % of times each website made a perfect weather prediction within 5 days. The higher the bars, the better the website performed. * The BBC made significantly more perfect predictions than Weather.co.uk (t(1,57) = 1.73, p<.05).

How far out are the weather sites?

In addition to knowing which website made a perfect prediction most often, we also wanted to know how far out they were when they got it wrong – i.e. the error values in their predictions. For example, predicting clear blue skies but receiving snow has a higher error value than predicting blue skies but experiencing partly cloudy weather. Figure 3 shows the each websites' average error in their predictions made for each day in advance.

Figure 3: How far out each websites advanced predictions were as to what the weather would be like. The higher the bars, the more wrong they were. Error bars represent one standard deviation.

Again, in order to quantify any differences statistically, the error data was pooled over 5 days for each website. The data for this can be seen in figure 4. Once more, all websites made predictions significantly closer to the actual weather than random guesses (all p<.001). The BBC's predictions were closer to the actual weather than Weather.co.uk (t(1,58) = 2.00, p<.05), and there was also a trend for the BBC predictions to be more superior than the Met Office predictions (p=.06, one-tailed). No other comparisons were significant.

Figure 4: How far out each website was in it’s predictions about what the weather would be like within 5 days. The higher the bars, the worse they performed. The error bars indicate the worst individual prediction each website made. All websites scored performed significantly better than random guesses. * Weather.co.uk were significantly further out in the accuracy of their predictions than BBC Weather (t(1,58) = 2.00, p<.05).

Conclusion (top)

So there you have it. The BBC performed better in our study than both Weather.co.uk and the Met Office. Of course, this was only one snapshot in time and the picture might be quite different if we collected data for 1 year. You'll want to bear this in mind before you drop plan B for your weekend BBQ!

Richard Wood