Tools to know long range weather prediction accuracy (3 day 5 day 7 day or longer) of each source.
As an engineer-nerd I had been told once that due to the complex nature of weather variables that predictions of over 3 days (or maybe he said 5 days) where simply no better and would never be better then rolling dice/guessing. Now I have never investigated this and would love to know based on analysis of each of your sources(looking back) how much "faith" say I could put in planning a weekend party; say should I wait till Thurs because it has been %95 accurate vs say %25 if I used my Tues info. to plan. Now rain/light rain/ or no rain & approx. temp. (say +/- 10F) would be OK. Love your Site!
We intend to add some form of analysis of the forecasting accuracy and provide that information in some form – precisely what and how is still up in the air.
I'm still waiting for a functional forecasting accuracy measurement tool. Is it still coming? I just signed up for the $20 annual membership.
I really need to be convinced that it is difficult to do, because I just can't convince myself that there is any legitimate technical reason why it can't happen. Just construct an algorithm, that's it!
All we need is to gather the predicted variable for each day that a prediction is made, then measure the observed variable and record the difference.
Then graph this difference. It could be organized by "x day-ahead" forecast which would be the average of all previous x days-ahead forecasts. It could be stated at the 90% confidence level. For example, 2-day ahead forecast accuracy for high temperature would be 4 degree, while 3-day ahead forecast accuracy would be 7 degrees. This would be most useful to people.
Or to make it more exciting and likely even more accurate, you could divide it up by averaging the predicted accuracy of any individual month (such as taking only all the Februarys in the record and do this statistical study using only the average accuracy for all the Februarys).
It is likely that only winter and summer patterns are more predictable, whereas the transition seasons of Spring and Autumn are harder to forecast. We need to know this and quantify this difference and no one is providing it.
WeatherSpark has had the best visual weather website by far for a long time. I would love to see you put this type of data on there. I think that you would dramatically increase your popularity if you did.
Unfortunately, I don't have much programming skills so can't really help you in that way, but reach out to the public for help. Someone may devote themselves to this cause to provide the world with this information which no one else is providing. I think this is a niche market that is waiting to be tapped.
David Otazu commented
I would love for such data to be made available!
Forgive my potential ignorance on the topic, but I think it a piece of cake to just record the daily predictions made for each future day in the forecasts, and assign accuracy probabilities (for a given variable, like high temperature) for each future day in a forecast.
For example, after a significant data set is compiled, you could have forecast accuracy probabilities (at the 90% confidence level) saying something like 1-day ahead forecast is 95% accurate, 2-day ahead forecast is 80% accurate, 3-day ahead forecast is 60% accurate, etc.
Could you please convince me that it is not this easy?
AdminJacob N (Founder, WeatherSpark) commented
We've added pastcasts to allow you to view past forecasts and compare them to the actual outcome.
Click on the little nubbins in the History graph header section to toggle the pastcasts.
Ziyad Saeed commented
for historical data i would love to know the accuracy of the various forecast stations. Which of the stations are better in providing accurate forecast for a particular location.
For my location WWO says it is going to be 50C but met.no says 44C that's a huge difference which one should i believe?
I took a look at forecastadvisor.com They were giving data for predictions only 1 to 3 days out with numbers like %68.05 accurate[not just %68 but %68.05(!)]. Now if I just flip a coin and ask will it rain tomorrow, then am I not %50.0000 accurate %100.0000 of the time?
I NEED MORE USEFUL INFO!
MY IDEA REFINED: (simple version V1.1, do it yourself version)
I would like to be able to "Compare" say using the compare button, the actual measured weather data and/or the current forecast/prediction graph with say the NOAA/met/WWO -24hr, -48hr, -72hr, -96hr... [reads as: NOAA Minus 24/48/72... hours] prediction data curves or maybe even overlay up to 4 curves, say NOAA -24hr and -72hr and WWO -72hr prediction along with the actual weather data curves(so Friday's prediction lines up with Friday's weather) for say the last 2 weeks of real data (2 weeks back minimum data, ideally all the predictions, at least the years, so it/we could look back and see if big snow and rain storms(significant weather events) of the past were predicted well and if so, how early ).
THIS TOOL IDEA EXAMPLE: I am looking at weather data for March 25th and I see the actual measured weather data, THEN I CAN ask for "compare" NOAA -48hr and I will get a curve that was the prediction curve of the NOAA button as it looked 48 hours prior (March 23rd same time of day).
We might need to understand when and how the prediction curves are updated or changed in the online help[maybe it is already there?]
I do not know how long you or anyone keeps the previous prediction data but it would give us good tools to actually understand how well we can trust these multi-day forecast predictions.
THE PREDICTION FLIP/FLOP ISSUE: I have felt that the forecast predictions have been changing just a few days out, very often (annoying), like I was going to move stuff around so I could do a thing outside on Thursday(said no rain//now it says rain). So I might very well like to see how much flip/flop the NOAA and WWO forecasts make from day to day also! Example "compare 4 curves; NOAA -24hr and -72hr and WWO -24hr and -72hr. And also to compare them with the current prediction (-0hrs, the default or current prediction state)
Thank you//great site!
QUESTION 1: So if I just remember some of my weather limericks like 'Red sky at night ==> sailors delight, red sky morning ==> sailors take warning' plus I flip my "lucky" %50/%50 accurate rain "coin" am I nearly as good as my %68.05 accurate sources; in other words, I want to know for sure how much of a waste of my time it is to use the forecast predictions to look say 3 days out(maybe it is good in some situations and not good in others[season/type of prediction/...??] and maybe just 2 days out or less is still a crap shoot %X of the time, and these kind of tools will help us all understand and shine light on a very important thing; namely what we can trust and what we should not trust in these important weather predictions!
NOTE 2: I was using the words 'forecast' and 'prediction' interchangeably in this note, I do not know if that is linguistically correct, but they seem the same to me at least on this topic.
Marty Alchin commented
For what it's worth, there is a service that does this already: http://www.forecastadvisor.com/. It's not very granular, though, so I think there's still value in having that information available within WeatherSpark. I'm not sure how it could be included with much detail without getting too noisy, though. Maybe just fade the forecast based on confidence, similar to what a2brute said?
AdminJacob N (Founder, WeatherSpark) commented
Very good points - fading or widening the trace (both?) would be nice ways to indicate certainty
You could extend the traces for a particular datasource into the future as far as you are confident of the data. That would show the level of confidence in each readings. Alternately, you could fade and thicken the lines as they move forward depending on their quality.
Scott Gartner commented
And the ability to compare different weather services predictions and whether they are more accurate in the winter/summer/etc. would be exceptionally helpful. At this point there are no weather services that compare any of the predicted data with what really happened (and, I suspect, for good reason since it doesn't seem very accurate).
(Me Again) If the stats said this source has been %xx accurate in 5 day (3or7 or whatever you choose) predictions for this year or maybe say for the last 5 summers or whatever I wonder if seasonal variations would be more helpful ... I do not have enough background in weather science currently but the goal of the idea is to give a good confidence factor in knowing how far out to look. Maybe the line could be dark black(thin?) in the high confidence range and then get lighter and lighter(dashed) or maybe start fanning out as the confidence of the longer range prediction goes down as the number of days in the future have historically been predicted compared to the actual real world weather.