Two datasets, two different jobs

Google DeepMind opened access to two datasets behind its WeatherNext 3 forecasting system on 13 November. Researchers can now work with the satellite imagery and precipitation estimates separately from the forecasts they help produce. Both datasets have historical records and a regularly updated feed on Google Cloud Storage.

SatWeave puts observations from several weather satellites on the same grid. The satellites don’t all look at Earth in the same way or send images on the same schedule. A combined view gives researchers one starting point for tracking clouds and storms. The current feed updates hourly.

A 15-minute interval is not a 15-minute wait

PARDIG does a different job. The model learns from precipitation measurements along the narrow tracks of a NASA and JAXA satellite mission. It uses other satellite images and estimates of the atmosphere’s state to fill in the gaps. The result is a map of estimated precipitation rates, not a direct measurement at every point or a prediction of what will fall next.

PARDIG’s live feed gives a value every 15 minutes on a grid spaced about 10 kilometres apart near the equator. But those values come out in six-hour batches. Google’s documentation puts the delay at about two to eight hours, depending on when the observation fell within the batch. The Earth Engine collections are archives at launch, not the live feed.

Where the estimates need care

In a preprint, the Google researchers report that PARDIG detected precipitation better than several existing products in their comparisons with ground radar and weather stations. They also flag weaker results at some high-altitude sites and caution that polar estimates are outside the coverage of the radar used to train the model.

The longer PARDIG archive starts in 1999 and uses different inputs from the live version. Before joining the two records to study a trend, check whether an apparent change lines up with a change in the observing system. A consistent grid doesn’t necessarily mean the underlying measurements were consistent.

Sources

  1. Google DeepMind: Opening the datasets behind our AI weather modelsDated announcement, roles of SatWeave and PARDIG, hourly satellite feed and access channels.
  2. Google for Developers: PARDIG guideGrid spacing, sampling interval, batch schedule, latency, archive and version differences.
  3. Shalev et al.: PARDIG preprintTraining approach, comparison with independent observations, evaluation limits and data availability.