Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
tech.lgbt is one of the many independent Mastodon servers you can use to participate in the fediverse.
We welcome all marginalized identities. This Mastodon instance is generally for folks who are LGBTQIA+ and Allies with an interest in tech work, academics, or technology in general.

Administered by:

Server stats:

2.9K
active users

tech.lgbt: AboutStatusProfiles directoryPrivacy policyTerms of service

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.0-alpha.4+glitch

Making weather forecasting machine learning models operational!

As a team at ECMWF we have open-sourced "ai-models" and plugins for all the major open-source data-driven NWP models:

🌍 FourCastNet v2 with spherical harmonics by NVIDIA
🤖 PanguWeather 3D transformer by Huawei
🌐 GraphCast multi-mesh graph neural network by Google DeepMind

View them on the ECMWF website with the charts you know.

Or even run them yourself!
🌍 pip install ai-models-fourcastnetv2
🤖 pip install ai-models-panguweather
🌐 pip install ai-models-graphcast

These are all open-source plugins that make it easy to load data from MARS if you have access, CDS, or your own grib files.

Super proud of our work so far and that we can run these alongside our physical model now as a service to the weather community. 🌦

Also, can we talk about running, ONNX, Pytorch, and Jax for this? Now just waiting for a Tensorflow model to fill my Pokedex. 👀

(Pst, we're hiring btw! 🔥)

Sep 06, 2023, 03:53 PM·Public
6boosts·5favorites
Public

If you're curious about a proper evaluation of the first model Panguweather, we have our pre-print here:
arxiv.org/abs/2307.10128

And we've made these model predictions available on the website as charts here:

ecmwf.int/en/about/media-centr

arXiv.orgThe rise of data-driven weather forecastingData-driven modeling based on machine learning (ML) is showing enormous potential for weather forecasting. Rapid progress has been made with impressive results for some applications. The uptake of ML methods could be a game-changer for the incremental progress in traditional numerical weather prediction (NWP) known as the 'quiet revolution' of weather forecasting. The computational cost of running a forecast with standard NWP systems greatly hinders the improvements that can be made from increasing model resolution and ensemble sizes. An emerging new generation of ML models, developed using high-quality reanalysis datasets like ERA5 for training, allow forecasts that require much lower computational costs and that are highly-competitive in terms of accuracy. Here, we compare for the first time ML-generated forecasts with standard NWP-based forecasts in an operational-like context, initialized from the same initial conditions. Focusing on deterministic forecasts, we apply common forecast verification tools to assess to what extent a data-driven forecast produced with one of the recently developed ML models (PanguWeather) matches the quality and attributes of a forecast from one of the leading global NWP systems (the ECMWF IFS). The results are very promising, with comparable skill for both global metrics and extreme events, when verified against both the operational analysis and synoptic observations. Increasing forecast smoothness and bias drift with forecast lead time are identified as current drawbacks of ML-based forecasts. A new NWP paradigm is emerging relying on inference from ML models and state-of-the-art analysis and reanalysis datasets for forecast initialization and model training.
*
Public

@jesper interesting thanks! We discussed panguweather at DMI's ML meeting his month. Cc @leifdenby

Public

@Ruth_Mottram @leifdenby oh nice!

Public

Amazing work @jesper ! I'm looking forward to having a look at this. Yes, so many deep learning stacks, I wonder which one you found the easiest to get going for this?

Plenty to keep discussing @Ruth_Mottram, this will be exciting!

Public

@leifdenby @Ruth_Mottram ONNX for sure.

Quiet public

@leifdenby @Ruth_Mottram Geez, how rude of me. Also thank you so much!

*
Public

@jesper very interesting, thanks. We had a discussion about that question today at a meeting with the dynamics group at ECCC. How good are these models? How good can they be? Are they or would they be able to predict extreme events? The flooding in Greece right now? So we'll read with a lot of interest!

P.S. the link to the preprint on linkedin doesn't work for me

Public

@philippbirken oh thanks! That was me being lazy. Updated the link also it's here: arxiv.org/abs/2307.10128

I think they look pretty good. We can observe a hurricane currently for example. But they're lower resolution currently and often smooth out over longer forecast horizons (due to how they're trained).

Generally I think they look very promising. I recommend clicking around the charts a bit 😁

arXiv.orgThe rise of data-driven weather forecastingData-driven modeling based on machine learning (ML) is showing enormous potential for weather forecasting. Rapid progress has been made with impressive results for some applications. The uptake of ML methods could be a game-changer for the incremental progress in traditional numerical weather prediction (NWP) known as the 'quiet revolution' of weather forecasting. The computational cost of running a forecast with standard NWP systems greatly hinders the improvements that can be made from increasing model resolution and ensemble sizes. An emerging new generation of ML models, developed using high-quality reanalysis datasets like ERA5 for training, allow forecasts that require much lower computational costs and that are highly-competitive in terms of accuracy. Here, we compare for the first time ML-generated forecasts with standard NWP-based forecasts in an operational-like context, initialized from the same initial conditions. Focusing on deterministic forecasts, we apply common forecast verification tools to assess to what extent a data-driven forecast produced with one of the recently developed ML models (PanguWeather) matches the quality and attributes of a forecast from one of the leading global NWP systems (the ECMWF IFS). The results are very promising, with comparable skill for both global metrics and extreme events, when verified against both the operational analysis and synoptic observations. Increasing forecast smoothness and bias drift with forecast lead time are identified as current drawbacks of ML-based forecasts. A new NWP paradigm is emerging relying on inference from ML models and state-of-the-art analysis and reanalysis datasets for forecast initialization and model training.
Public

@jesper I was just wondering: might you be free for a chat on Sep 27/28? I would love to chat to you about how to maybe hack these to run for Local Area Modelling :)

Quiet public

@jesper you’re hiring? Is there an official tender out, yet?

Quiet public

@neuronenstern yes actually! 4 core positions and one on destination earth. They close on the 15th!

Explore
About