Read compressed binary file (.grib2.bz2)

Question

I have downloaded one of the files from this list https://opendata.dwd.de/weather/nwp/icon-eu/grib/03/t_2m/ (the actual filenames change every day) which are bz2 compressed.

I can read in the decompressed file using e.g.

import xarray as xr
# cfgrib + dependencies are also required
grib1 = xr.open_dataset("icon-eu_europe_regular-lat-lon_single-level_2020101212_001_ASHFL_S.grib2", engine='cfgrib')

However, I would like to read in the compressed file.

I tried things like

with bz2.open("icon-eu_europe_regular-lat-lon_single-level_2020101818_002_ASWDIFD_S.grib2.bz2", "rb") as f:
    xr.open_dataset(f, engine='cfgrib')

but this does not work.

I am looking for any way to programmatically read in the compressed file.

Answer 1

I had the same issue within processing numerical weather prediction data.

What I am doing here is to download the file and hold it as a Binary Object (e.g. with urlopen or requests). Pass this object into the following function:

import bz2, shutil
from io import BytesIO
from pathlib import Path


def bunzip_store(file: BytesIO, local_intermediate_file: Path):
    with bz2.BZ2File(file) as fr, local_intermediate_file.open(mode="wb") as fw:
        shutil.copyfileobj(fr, fw)

An unzipped file will be stored underlocal_intermediate_file. Now you should be able to open this file.