A record of the accuracy of my local cloud forecasts

[Second Time Around]

I mentioned a few weeks ago that I'd started recording the accuracy of 6 of my local cloud forecasts here in East Kent.  The forecasts were recorded roughly at sunset when I'd put my 10 inch Dob out to cool down if clear skies were suggested.  The accuracy was recorded over a roughly 30 minute period from nautical twilight when it was dark enough to observe.  This is approx 75 to 90 minutes later than sunset at this time of year.

These are the results for April 2021 plus a few comments below.

 

1)  This is so far a very small sample and there's insufficient data to be statistically significant. 

2)  The methodology isn't scientific, but hopefully the results will be of interest, both for comparison purposes and also as a long term record of clear nights.

2)  We have our own micro climate here, and the weather is often different than just a few miles away. 

3)  We were barely influenced by the jetstream due to a blocking high that affected our weather for the entire period.  Indeed, we had absolutely no rain at all in April.  Results are likely to be very different when the jetstream reappears.

4)  Because of the differences in the data presented by each site (some are very general) the cloudiness is just shown as clear, part cloudy or cloudy.  These are represented by white, grey and black shading respectively.  I'll comment more on the weather websites at a later date.

5)  There was little difference in the overall accuracy of all 6 forecasts with none standing out.  I've scored +1 point with green shading if the forecast was accurate, 0 points with yellow shading if partly accurate, and -1 point with red shading if totally wrong.  As you can see, on this basis Weather Underground was fractionally on top with 23 out of 30 forecasts being correct.  Metcheck equalled this, but when they were wrong they were sometimes very wrong, so achieved a lower score.  Again, I'd refer you to comments 1 and 3.

 

 

Edited May 2, 2021 by Second Time Around

[Second Time Around]

Here's the May results.

This time Wunderground were poor with the Met Office being best.  The conclusion I take from the data is that none of the forecasts is statistically better or worse than any other.  The best option is to look at several each evening.

 

[Zermelo]

Thanks Steve, very interesting. I missed the April data when you put it up.

I assume that some of these sites use the same source data, I recall some discussion about that previously. Do you know? I can see some correlations between different sites on some days; that could be either because they're using the same sources or because the forecasting on those days was particularly "easy".

For the CO data, were you just taking the main forecast? (i.e. not the ones in their "experimental features")

 

[Second Time Around]

I gather that it works in the following way.  Each site uses similar source data but there are a number of such sources.  Different sites use a different combination of sources.  Even if each site used exactly the same data, they combine them in different ways to get a series of model runs.  From these a forecast is derived.

Other factors include how often the published forecasts are updated.  Then the sites vary in their presentation - I use percentage cloud cover where shown rather than symbols as below.  

BBC gives just cloudy, partly cloudy and clear.

Met Office gives just overcast, cloudy, partly cloudy or clear.  

Weather Underground gives percentage total cloud cover.

Meteoblue gives percentage low, medium and high cloud cover.

Metcheck and Clear Outside give percentage total cloud cover, plus percentage low, medium and high cloud cover.

Because of this I have to use just cloudy, part cloudy and clear on my spreadsheet.  Where total cloud cover is shown I interpret 0-25% as clear, 26 to 74% as part cloudy, and 75 to 100% as cloudy.  This is because when cloud cover is shown as a symbol on a weather map it's based on oktas, i.e. eigths.  I know it's not precise, but it's why I said last month the methodology isn't scientific.

As for Clear Outside I do have the experimental features turned on but don't use them for this purpose.

Edited June 4, 2021 by Second Time Around

[Second Time Around]

Here's the June chart.

I don't think there's anything new to learn as there's still little between the forecasts.  However, I'm going to continue to post the results if only as a record of the clear nights.  They might be of use to someone in the future.

 

 

 

 

Edited July 5, 2021 by Second Time Around

[Second Time Around]

Here's the July chart.

The Met Office is still top, with the BBC 6th and last.  The reason for the poor showing by the BBC may be because many nights it forecasts part cloudy.  Their definition of part cloudy may be different to the other forecasts and also to mine. See earlier post for what I count as part cloudy and why.

 

Edited August 7, 2021 by Second Time Around

[StuartT]

Thanks so much for sharing your work with us. 

[Zermelo]

Not wishing to hijack your thread, Steve, but checking the forecasts for last night here, I saw an opportunity for an unscientific one-off test.

I refreshed the forecasts for three websites at 20.00 and saw that there was a dramatic discrepancy:

CO had this prediction for the night ahead:

Whereas Metcheck had this*:

Meteoblue had a similar forecast to Metcheck, perhaps a little less cloud, but I didn't manage to capture it at the time.

Well as it happens, at 20.00 it was perfectly clear here and had been for some time, so I took CO with a pinch of salt, set up anyway and had a great session. I did have some cloud come over later on, but it passed quickly and I carried on until gone 02.00.

This is the second time that I've noticed a large discrepancy between the providers, and on the previous occasion CO was also the one furthest from the actuality. I find it interesting that oftentimes CO will have had a very different forecast earlier on in the same day, and that earlier forecast turned out to have been more accurate than the later one that replaced it. I believe CO is updated more often than some other forecasters, but sometimes that actually seems to work against it. As with last evening, it's not unusual for CO to have even the current situation wrong (in terms of cloud cover, at least), in spite of the frequent updates.

 

* note the anomolous figures for 20.00 - low/medium/high figures are all zero, but the combined total is somehow 12% !
I've seen this sort of discrepancy before, but not found any specific explanation; though there was a general note to the effect that sometimes "rogue data" makes its way in, and is corrected if spotted.

 

 

Edited August 26, 2021 by Zermelo

[Second Time Around]

For me yesterday at just after 8pm there were 3 forecasts giving part cloudy, 3 giving cloudy.  I checked the sky 3 times over half an hour or so from just after nautical sunset and in each case there was complete cloud cover.

As I've mentioned before, it pays to look at several forecasts.  The problem is, as you've found, there isn't always a consensus!  In fact, this disagreement between forecasts isn't unusual at all. 

[Second Time Around]

Here's the August results.

I'd add that when I've marked that I'm away I get observations from 3 local weather stations.

Now that I've got Covid and am confined to my bedroom all I can do is look out of the window. So the results may not be so accurate.  I'll mark those dates accordingly on the September results.

Edited October 1, 2021 by Second Time Around

[Zermelo]

Hope you're over it soon, Steve.

[Second Time Around]

Here's the September data.

The Met Office has pulled a little further ahead.  It'll be interesting to see if this continues through the Winter.

 

 

 

[Second Time Around]

Here's the October results.  As you can see the BBC has had a good month.  The Met Office is still well ahead, but not by so much.

 

 

Edited November 11, 2021 by Second Time Around

[Second Time Around]

Here's the November results.

This has been the worst month for observing here since I came back to astronomy 3 years ago.  It was often clear at sunset, but night after night it clouded over before it got dark!   Frustratingly, the satellite photos showed it was often clear just a few miles away!  🤯

The only properly clear night we had was on the 2nd November, and December has started in the same way.

[StuartT]

Thanks for keeping up with this. 

It’s extremely frustrating that all the weather apps I’ve tried so far are so utterly useless. I really don’t know how they can all manage to get it so wrong night after night after night.

I notice quite a few people here have the Clear Outside widget in their sig, but I don’t know why. It’s no more accurate than any of them (maybe it just seems better because it makes it’s rubbishness more granular and presents it in a fancy interface).

[Stephenstargazer]

Thanks @Second Time Around I notice that all the charts are predominantly green, with some yellow and a little red. The beauty of data in charts! So easy to see the big picture from your efforts, and not judge by personal instances, which can be frustrating.

Perhaps we should be less dismissive of forecasters and acknowledge that most of the time it is helpful. Especially  services that are free to the end user and in no way compulsory.

[jjohnson3803]

I've not done any statistics, but the astro apps I've tried have been wrong more often than right at my location.  I've switched to aviation forecasts and satellite loops.

But this does give me inspiration to do some stats and check the accuracy myself.

[Dixie]

Really good thread.  After years of using apps my only conclusion is that the only forecast that really matters is the video one either by the BBC or Met Office on the morning of the day in question.   I find the Met Office's 10 Day Trend on YouTube useful for getting a handle on the big picture. But quite often the apps get even my current weather conditions wrong.  Mind you, since I have not seen the sun here since for 10 days with no clear nights (seriously) and with a particularly poor forecast until at least the first week of Jan we are not stretching the talents of the forecasters too much. 

[Paz]

An interesting post. I don't really use apps any more, I mostly just look outside and judge it myself using "the force", and accept that I am quite often wrong.

There is one pretty good weather forecasting I use though which is if the gritting machines are out gritting the roads in the evening that bodes well for clear skies.

[Second Time Around]

Above are the December results.  Click to enlarge the image. 

Yearly analysis to follow.

Edited December 31, 2021 by Second Time Around

[Second Time Around]

The Met Office is slightly ahead, perhaps because of its consistency.  However, is this statistically significant?

Although I've an A level in pure maths and statistics it's so long ago that I've forgotten most of it.  To check whether the difference in accuracy of the forecasts is significant or not I seem to remember I'd have to do a student's t test (or is it a chi squared test?).  Anyone? 

However, the results are so similar that I think it safe to say that there's no significant difference between the various forecasts.

 

Short term vs medium term forecasts

What is interesting from my research is that there are 2 methods of deriving forecasts. 

The one most of us will be familiar with is the medium term forecasts that we see on the TV and websites.  These are computer models looking perhaps up to 14 days ahead.  They're "driven by powerful numerical weather prediction (NWP) systems. By solving physical equations, NWPs provide essential planet-scale predictions several days ahead. However, they struggle to generate high-resolution predictions for short lead times under two hours." (My bold italics.)

The other type of forecast is a nowcast.  "Nowcasting is a technique used for very short-range forecasting. The current weather is mapped and then an estimate of its speed and direction of movement is used to forecast the weather a short period ahead (assuming the weather will move without significant changes). It takes time to gather and map weather observations, so a short forecast is needed to outline what the weather is currently. Nowcasts can be used as a source of detailed guidance on the location, extent and timing of imminent, often high impact weather events."

"The Met Office produce a routine delivered service for T+0 out to T+6, for the United Kingdom, which blends our observations and UK Atmospheric Hi-Res model. At T+0 the blend is heavily weighted to observations and as time goes on the weighting of the UK Atmospheric Hi-Res model is increased, at T+6 the UK Atmospheric Hi-Res Model has the dominant weighting. The UK Atmospheric Hi-Res model is part of the Met Office flagship numerical weather prediction (NWP) model called the Unified Model. The resolution of the Nowcasting is comparable to radar data."

Nowcasting is of course what we need to decide whether to get our scopes out.  Unfortunately, nowcasting in its relative infancy.  AI is now being used to enhance its accuracy, for instance in tests that the Met Office is doing with Google's Deep Mind.  Hopefully, this will be of particular benefit to the likes of us.

In the meantime I'll continue to use these 6 cloud forecasts (with Good to Stargaze being added from next month).  

However, as I've mentioned elsewhere,  I also look at 2 other sites with satellite data.  The first is Zoom Earth for current cloud photos.  Unlike many sites it also shows cloud at night using infrared images, plus one can go back a long time.   Bear in mind though that low cloud and fog doesn't always show up on infrared images. But it often correctly shows that the only cloud in Kent is over our area, and that it's clear just a few miles away!

The other is Meteoradar as it extrapolates cloud satellite pictures to give nowcasts up to  3 hours ahead.   Unfortunately it does so only for daytime hours.  However, it's useful for soon after sunset, and also for solar observing.

 

How accurate are the current forecasts?

What will surprise many though is something I commented on in a topic where the OP was lamenting the accuracy of Clear Outside.  Even now, the current forecasts are pretty accurate.

Below are the figures for Clear Outside over the first 6 months of recording.  However, as I said, all the other forecasts will be very similar.

Correct 67%

Partly correct/partly wrong 29%

Totally wrong 4%

 

Edited December 31, 2021 by Second Time Around

[Zermelo]

Thanks Steve, very interesting. And of course, "accuracy" begs a definition, and the emphasese given by the astro community may not align with those of other interest groups.

[7170]

Thanks Steve (@Second Time Around), very interesting analysis. I had to do lots of statistics at A-Level and University but like you say I've long since forgotten a lot of it too. My personal experience is the sites change over time - presumably as models are updated.  I used to find Clear Outside pretty good, likewise the BBC too. But now find the best is the Met Office.

I'm not sure how Weather Underground forecast cloud data. The PWS upload protocol does permit cloud oktas data to be uploaded (though not all 8 steps by the looks of it) but I'm not aware of any off the shelf amateur/prosumer weather stations that actually have a cloud sensor and work out an okta number. So it is either inferred or they have commercial sources of data too. 

Are you comparing current status/forecast with conditions outside, or are you comparing the forecast from earlier in the day to actuals in the evening? I like to keep a close watch on the forecast and compare to my own weather station too. I've noticed in the few years a trend for short term forecasts to change quite considerably during the day, anecdotal data yes, but it happens enough times for me to keep noticing it - maybe I will start up a log.

Edited December 31, 2021 by 7170

[Second Time Around]

10 hours ago, 7170 said:

Thanks Steve (@Second Time Around

Are you comparing current status/forecast with conditions outside, or are you comparing the forecast from earlier in the day to actuals in the evening?

Copied from my first post on May 1st:

The forecasts were recorded roughly at sunset when I'd put my 10 inch Dob out to cool down if clear skies were suggested.  The accuracy was recorded over a roughly 30 minute period from nautical twilight when it was dark enough to observe.  This is approx 75 to 90 minutes later than sunset at this time of year.

I now use the app Sunrise Companion to sound an alarm at sunset.  If you long press on the date you also get the times of nautical sunset etc.

www.timeanddate.com also gives these times plus moonrise and moonset, but doesn't have an alarm.

 

 

Edited January 1, 2022 by Second Time Around

[The Admiral]

Thanks for compiling this Steve. As someone who needs to drag all my gear out and set up, I'm only too aware of the inadequacy of the forecasts for cloud cover. I think it is incredibly hard to predict the weather, let alone cloud cover, in this maritime country. MeteoBlue does show the results of various meteorological models and it's amazing just how much disagreement there is between them. For example:

It's worth just perusing the MeteoBlue web pages shown under 'Forecast - multimodel' and 'Forecast - multimodel ensemble', for your own location, to see what we are up against. Incidentally, in the 'Weather for Today' they do have a nowcast, for perhaps a couple of hours, but it's confined to the rain prediction. Cloud would be useful!

Clear skies

Ian

Edited January 1, 2022 by The Admiral