Elefridge.jl: Compressing atmospheric data into its real information content

Weather and climate forecasting centres worldwide produce very large amounts of data that has to be stored and shared with users. Data compression is essential to reduce file sizes sent over the internet and the demand on data archive capacity. 

The previously completed challenge within the ESoWC 2020 developed the concept of information-preserving compression by analysing the real information content in data from the Copernicus Atmospheric Monitoring Service (CAMS). Separating the false and hardly compressible information from the real information was shown to allow for high compression factors without significant information loss. 

Here, we focus on further details in the implementation of information-preserving compression for CAMS. Readily available in the current GRIB2 compression are different precision and accuracy options that can be translated to preserved information for a given data set. To implement this successfully and in an automated fashion, further improvements are necessary and the best lossless compressor available in GRIB2 that satisfies both speed and size requirements has to be found. 

This project aims to successfully implement information-preserving compression for CAMS to put this advanced compression technique into practice.

Follow the developments on GitHub