Weather Forecast Accuracy

Which weather forecast website is the best?

Weather Forecast Accuracy Summary

Which of the three commonly used UK weather forecast websites is the best? Everyday for two weeks, we noted BBC Weather's, Weather.co.uk's, & the Met Office's predictions for the up-coming days in our region and matched them up with what the weather really was in our region. All three websites performed much better than taking random guesses, you'll be pleased to know. However, out of the three websites, BBC Weather made a perfect prediction within 5 days 57% of the time, which made it superior to both the Met Office (47%) and Weather.co.uk (34%). Furthermore, when these sites did get it wrong, the BBC Weather predictions still tended to get much closer to the real weather than the other two websites' predictions. So, they're not all the same!

Introduction to British Weather (top)

Let's face it: a BBQ in the rain is a no-starter. Planning one for the up-coming weekend but really need to know the weather? You've probably noticed already that the weather websites are not all the same! What you want to know is which of the three most used websites for weather forecasts (BBC Weather, Weather.co.uk, & the Met Office) is going to be the least likely to let you down.

So, the WhatPrice geeks have taken the time to conduct some proper research on your behalf, in order that the long-asked question can be answered once and for all. What we did everyday for 2 weeks was to note each website's predictions in our region for the upcoming days, together with what the weather actually was in our humble opinions down here on the ground. In addition to this, we also made forecasts of our own, using computer code to randomly guess what the weather would be each day. With all this data it was possible to see which of the websites got the highest proportion of perfect predictions, how much each website was out by in their predictions when they got it wrong, and whether any of them were any better at predicting the weather than taking a random guess!

Weather Data Methodology (top)

Data was collected during March, 2007. Each morning for approximately 2 weeks the weather predictions for the up-coming days in our region (Cambridge) were recorded from each of the three websites. The BBC and the Met Office predicted weather up to 5 days in advance, and Weather.co.uk predicted the weather up to 10 days in advance.

There were a few differences between the websites that had to gotten around. The Met office website divided each day into multiple segments, whilst the others do not. Thus, only the predominant weather prediction for that day was noted from the Met Office site. Furthermore, as the different sites phrased their predictions differently and/or used different symbols to donate the weather, we had to use our own judgement to categorise each website's predictions into the 5 following weather categories:

In addition to this, at the end of each day, what we personally felt was the predominant weather condition that day was noted. This allowed for a comparison between what each website predicted at each day in advance with what actually occurred.

Finally, to re-assure ourselves (or not!) against the common moan that weather predictions are so inaccurate it’s better just to toss a coin, we generated random predictions as to what the weather for the forthcoming days of the study would be using computer code. These "random guesses" were analysed just like the other websites predictions, thus providing a meaningful control.

Data Preparation

The results were analysed in two ways.

First, the absolute accuracy of each website's predictions was scored. For example, if the BBC predicted a sunny day in two days time, and the weather actually turned out to be sunny on that day, the website scored 1 for a two-day-in-advance prediction. If it turned out to be partly cloudy (or any other weather condition), the website scored zero. In this way, it was possible to work out what proportion of each websites predictions, on each day in advance, turned out to be perfect.

Second, it was assessed how far wrong each websites predictions were. For example, predicting a sunny day and their actually being a partly cloudy day is less far out than if the day turned out to be actually snowing. In order to do this, each of the five weather conditions was given an arbitrary numerical value (sunny = 5, partly cloudy = 4, cloudy = 3, rain = 2, snow = 1). For each prediction, the actual weather value was taken away from the predicted value, leaving an error value which was greater the larger the difference between the actual and predicted weather conditions. For example, predicting sun (5) but actually receiving rain (2) leaves an error score of 3 (5-2).

In order to have sufficient data points to run statistical comparisons in the accuracy and error scores of the weather websites, the scores for 1-5 days in advance were pooled into one data set, and paired t-tests were run for individual comparisons.

Weather Forecast Accuracy Results (top)

Perfect predictions

The research period turned out to be ideal for data collection, as we had at least two-three days of each of the five main weather conditions (sunny, partly cloudy, cloudy, rain, snow). Such changeable weather was the prefect test for the weather websites.

Figure 1 shows the percentage of times each website made a perfect prediction as to what the weather would be at each day in advance.

Weather Forecast Accuracy up to five days

Figure 1: The percentage of times that each website made a perfect prediction as to what the weather would be, so many days in advance. The higher the bars, the better the website performed. Note that this shows absolute accuracy, so predicting sunny but there actually be some cloudy ("partly cloudy") is considered wrong for this graph. The higher the bars, the more accurate the website was. Note also that Weather.co.uk made predictions 10 days in advance, whilst the BBC and Met Office only made predictions 5 days in advance.

When comparing websites in such things, it is important to submit the data to statistical tests, which more objectively decide whether they’re really is a difference in performance between the websites. In order to do this, accuracy scores were pooled and averaged over 5 days for each website. The data for this can be seen in figure 2. Paired-samples T-tests revealed all of the websites got it perfectly right significantly more often than if they'd made a random guess (all p<.02). However, the BBC Weather website made a perfect prediction significantly more often than Weather.co.uk (t(1,57) = 1.73, p<.05). The Met Office performed somewhere between the two (though no other comparisons were significant).  

Weather Forecast - Accuracy in 5 days
Figure 2: The % of times each website made a perfect weather prediction within 5 days. The higher the bars, the better the website performed. * The BBC made significantly more perfect predictions than Weather.co.uk (t(1,57) = 1.73, p<.05).

How far out are the weather sites?

In addition to knowing which website made a perfect prediction most often, we also wanted to know how far out they were when they got it wrong – i.e. the error values in their predictions. For example, predicting clear blue skies but receiving snow has a higher error value than predicting blue skies but experiencing partly cloudy weather. Figure 3 shows the each websites' average error in their predictions made for each day in advance.

Weather Forecast - Looking at Days in Advance

Figure 3: How far out each websites advanced predictions were as to what the weather would be like. The higher the bars, the more wrong they were. Error bars represent one standard deviation.

Again, in order to quantify any differences statistically, the error data was pooled over 5 days for each website. The data for this can be seen in figure 4. Once more, all websites made predictions significantly closer to the actual weather than random guesses (all p<.001). The BBC's predictions were closer to the actual weather than Weather.co.uk (t(1,58) = 2.00, p<.05), and there was also a trend for the BBC predictions to be more superior than the Met Office predictions (p=.06, one-tailed). No other comparisons were significant.

Weather Forecast - How Wrong Can it be?
Figure 4: How far out each website was in it’s predictions about what the weather would be like within 5 days. The higher the bars, the worse they performed. The error bars indicate the worst individual prediction each website made. All websites scored performed significantly better than random guesses. * Weather.co.uk were significantly further out in the accuracy of their predictions than BBC Weather (t(1,58) = 2.00, p<.05).

Conclusion (top)

So there you have it. The BBC performed better in our study than both Weather.co.uk and the Met Office. Of course, this was only one snapshot in time and the picture might be quite different if we collected data for 1 year. You'll want to bear this in mind before you drop plan B for your weekend BBQ!

Richard Wood



If you found this page useful please click the +1 button below to tell Google that its a great page!

Please share this page with others, and leave a comment, we value all feedback!

Was this page useful? Do you have something to add? Do you disagree?

If your comments meet our guidelines then we will publish them (you do not need to register!)



Ttradesman - click here to join our network to receive leads from customers in your area

"How can the weather be predicted when it is being affected every day by chemtrails. Sunny days predicted recently have all been 'cloudy'!!"

Heather

"I cant believe only one other person pointed this out, but a study based on only 2 weeks data has no value whatsoever. You're wasting my time."

ed

"Very interesting. Thank you for a well-made comparison. If you ever consider redoing it, would you care to include the common assumption that "if you guess the weather tomorrow will be the same as today you'll be more accurate than the weather forecasts"? I.e. a slightly more orderly form of the random observations."

Elias Hedberg

"To base a "study" on only two weeks of data is outright ridiculous and meaningless !"

Finn Brandt

"Since it says "BBC Weather in association with the Met Office" why are their data different?"

JW, Bristol

"The random guess may be a control, but I would like to see figures based on simply forecasting that there will be no change in the weather. I read somewhere that such forecasting can be around 75% accurate since patterns do not change daily."

Mr. Gutenplatz

"This seams a bit odd, given that the BBC uses weather forecast information from the metoffice. Surekly if your experiment was correct, the BBC and the metoffice would come out the same?"

Jake

"Hello, great article, But would you explain me more how do evaluate the accuracy. You compute the error by subtract the actual and predicted indexes. (5-2=3). Figure 1 shows the percentage of times each website made a perfect prediction as to what the weather would be at each day in advance. What is the accepted accuracy for forecasting system that I can depend on? in other words, I will consider this system as bad forecast system if the percentage below x%. Please provide more information of references on Paired-samples T-tests. How do you compute p? -- Motaz K. Saad motaz.saad@gmail.com "

Motaz K. Saad

"The data seems to suggest all the sites are better at predicting four days ahead rather than 1 day ahead - i.e. the br for perfect predictions is highest at th four days ahead mark - surely this can't be correct?"

Philip

"Great article, the business of weather forecasting is either an outrage or just down right funny. I think the level of accuracy may depend on ones geographic location and micro climates. Weather predictions in the middle of the Sahara, may be more reliable. Here in the Pacific Northwest of the United States, we would be thrilled if the predictions were anywhere near 57% accurate. In fact it is so bad, than you can almost predict that the weather may be sunny, it may be cloudy, or it may rain, but it won’t be what was predicted. I have a little barometer in my house with graphics of sun, rain, storms, etc. and it is MUCH more accurate than even our NOAA weather service. Predicting the weather is big business, perhaps when the sham is exposed the government can bail them out."

Dao Jones

"Excellent study - the only problem I can see is the BBC publish differing weather forecasts for the same region depending on which part of their website you are viewing. Is this how they obtained the higher score!!!!"

Ed








FREE QUOTATION SERVICE