Load ODIM_H5 Volume data from German Weather Service

In this example, we obtain and read the latest 30 minutes of available volumetric radar data from German Weather Service available at opendata.dwd.de. Finally we do some plotting.

This retrieves 6 timesteps of the 10 sweeps (moments DBZH and VRADH) of the DWD volume scan of a distinct radar. This amounts to 120 data files which are combined into one volumetric Cf/Radial2 like xarray powered structure.

Exports to single file Odim_H5 and Cf/Radial2 format are shown at the end of this tutorial.

Note

The used open_odim_mfdataset implementation is based on xarray. It claims multiple data files and presents them in a simple structure. See also the notebook wradlib_odim_backend for further details.

[1]:
import wradlib as wrl
import warnings
warnings.filterwarnings('ignore')
import matplotlib.pyplot as pl
import numpy as np
import xarray as xr
try:
    get_ipython().magic("matplotlib inline")
except:
    pl.ion()
from wradlib.io import open_odim_mfdataset
[2]:
import urllib3
import os
import io
import glob
import shutil
import datetime

Download radar volumes of latest 30 minutes from server using wetterdienst

wetterdienst is a neat package for easy retrieval of data primarily from DWD. For further information have a look at their documentation.

[3]:
from wetterdienst.provider.dwd.radar import DwdRadarDataFormat, DwdRadarDataSubset, DwdRadarParameter, DwdRadarValues
from wetterdienst.provider.dwd.radar.sites import DwdRadarSite
[4]:
elevations = range(10)

end_date = datetime.datetime.utcnow()
start_date = end_date - datetime.timedelta(minutes=30)

results_velocity = []
results_reflectivity = []

for el in elevations:
    # Horizontal Doppler Velocity
    request_velocity = DwdRadarValues(
        parameter=DwdRadarParameter.SWEEP_VOL_VELOCITY_H,
        start_date=start_date,
        end_date=end_date,
        site=DwdRadarSite.ESS,
        elevation=el,
        fmt=DwdRadarDataFormat.HDF5,
        subset=DwdRadarDataSubset.POLARIMETRIC,
    )

    # Horizontal Reflectivity
    request_reflectivity = DwdRadarValues(
        parameter=DwdRadarParameter.SWEEP_VOL_REFLECTIVITY_H,
        start_date=start_date,
        end_date=end_date,
        elevation=el,
        site=DwdRadarSite.ESS,fmt=DwdRadarDataFormat.HDF5,
        subset=DwdRadarDataSubset.POLARIMETRIC,
    )

    # Submit requests.
    results_velocity.append(request_velocity.query())
    results_reflectivity.append(request_reflectivity.query())

Acquire data as memory buffer

[5]:
%%time
volume_velocity = []
for item1 in results_velocity:
    files = []
    for item2 in item1:
        files.append(item2.data)
    volume_velocity.append(files)
  2%|▎         | 6/240 [00:02<01:47,  2.17it/s]
  2%|▎         | 6/240 [00:01<01:01,  3.81it/s]
  2%|▎         | 6/240 [00:01<00:55,  4.21it/s]
  2%|▎         | 6/240 [00:01<01:03,  3.67it/s]
  2%|▎         | 6/240 [00:01<01:02,  3.75it/s]
  2%|▎         | 6/240 [00:01<00:59,  3.95it/s]
  2%|▎         | 6/240 [00:01<00:59,  3.95it/s]
  2%|▎         | 6/240 [00:01<00:53,  4.34it/s]
  2%|▎         | 6/240 [00:01<01:04,  3.64it/s]
  2%|▎         | 6/240 [00:01<00:54,  4.33it/s]
CPU times: user 5.56 s, sys: 122 ms, total: 5.68 s
Wall time: 16.5 s

[6]:
%%time
volume_reflectivity = []
for item1 in results_reflectivity:
    files = []
    for item2 in item1:
        files.append(item2.data)
    volume_reflectivity.append(files)
  2%|▎         | 6/240 [00:01<00:59,  3.94it/s]
  2%|▎         | 6/240 [00:01<00:55,  4.20it/s]
  2%|▎         | 6/240 [00:01<01:00,  3.87it/s]
  2%|▎         | 6/240 [00:01<00:54,  4.29it/s]
  2%|▎         | 6/240 [00:01<00:59,  3.92it/s]
  2%|▎         | 6/240 [00:01<00:54,  4.28it/s]
  2%|▎         | 6/240 [00:01<01:15,  3.11it/s]
  2%|▎         | 6/240 [00:01<01:00,  3.87it/s]
  2%|▎         | 6/240 [00:01<00:55,  4.21it/s]
  2%|▎         | 6/240 [00:01<01:00,  3.84it/s]
CPU times: user 5.7 s, sys: 121 ms, total: 5.82 s
Wall time: 15.3 s

Read the data into xarray powered structure

[7]:
vol = wrl.io.RadarVolume()
for r, v in zip(volume_reflectivity, volume_velocity):
    ds0 = wrl.io.open_odim_mfdataset(r, group="dataset1",
                                     concat_dim="time",
                                     combine="nested",
                                    )
    ds1 = wrl.io.open_odim_mfdataset(v, group="dataset1",
                                     concat_dim="time",
                                     combine="nested",
                                    )

    vol.append(xr.merge([ds0, ds1], combine_attrs="override"))
    vol.sort(key=lambda x: x.time.min().values)
100%|██████████| 1/1 [00:00<00:00,  2.74it/s]
100%|██████████| 1/1 [00:00<00:00,  4.45it/s]
100%|██████████| 1/1 [00:00<00:00,  4.53it/s]
100%|██████████| 1/1 [00:00<00:00,  4.47it/s]
100%|██████████| 1/1 [00:00<00:00,  4.52it/s]
100%|██████████| 1/1 [00:00<00:00,  4.44it/s]
100%|██████████| 1/1 [00:00<00:00,  4.50it/s]
100%|██████████| 1/1 [00:00<00:00,  4.38it/s]
100%|██████████| 1/1 [00:00<00:00,  4.38it/s]
100%|██████████| 1/1 [00:00<00:00,  4.42it/s]
100%|██████████| 1/1 [00:00<00:00,  5.10it/s]
100%|██████████| 1/1 [00:00<00:00,  5.17it/s]
100%|██████████| 1/1 [00:00<00:00,  4.35it/s]
100%|██████████| 1/1 [00:00<00:00,  4.41it/s]
100%|██████████| 1/1 [00:00<00:00,  4.46it/s]
100%|██████████| 1/1 [00:00<00:00,  4.49it/s]
100%|██████████| 1/1 [00:00<00:00,  4.51it/s]
100%|██████████| 1/1 [00:00<00:00,  4.54it/s]
100%|██████████| 1/1 [00:00<00:00,  4.49it/s]
100%|██████████| 1/1 [00:00<00:00,  4.34it/s]

Inspect structure

Root Group

[8]:
vol.root
[8]:
<xarray.Dataset>
Dimensions:              (sweep: 10)
Coordinates:
    sweep_mode           <U20 'azimuth_surveillance'
    longitude            float64 6.967
    altitude             float64 185.1
    latitude             float64 51.41
Dimensions without coordinates: sweep
Data variables:
    volume_number        int64 0
    platform_type        <U5 'fixed'
    instrument_type      <U5 'radar'
    primary_axis         <U6 'axis_z'
    time_coverage_start  <U20 '2021-09-02T05:10:35Z'
    time_coverage_end    <U20 '2021-09-02T05:39:02Z'
    sweep_group_name     (sweep) <U7 'sweep_0' 'sweep_1' ... 'sweep_8' 'sweep_9'
    sweep_fixed_angle    (sweep) float64 5.5 4.5 3.5 2.5 ... 8.0 12.0 17.0 25.0
Attributes:
    version:          None
    title:            None
    institution:      None
    references:       None
    source:           None
    history:          None
    comment:          im/exported using wradlib
    instrument_name:  None
    fixed_angle:      5.5
[9]:
vol.root.sweep_fixed_angle
[9]:
<xarray.DataArray 'sweep_fixed_angle' (sweep: 10)>
array([ 5.5,  4.5,  3.5,  2.5,  1.5,  0.5,  8. , 12. , 17. , 25. ])
Coordinates:
    sweep_mode  <U20 'azimuth_surveillance'
    longitude   float64 6.967
    altitude    float64 185.1
    latitude    float64 51.41
Dimensions without coordinates: sweep

Sweep Groups

[10]:
vol
[10]:
<wradlib.RadarVolume>
Dimension(s): (sweep: 10)
Elevation(s): (5.5, 4.5, 3.5, 2.5, 1.5, 0.5, 8.0, 12.0, 17.0, 25.0)
[11]:
vol[0]
[11]:
<xarray.Dataset>
Dimensions:     (azimuth: 360, time: 6, range: 720)
Coordinates:
  * azimuth     (azimuth) float64 0.5 1.5 2.5 3.5 ... 356.5 357.5 358.5 359.5
    elevation   (azimuth) float64 dask.array<chunksize=(360,), meta=np.ndarray>
    rtime       (time, azimuth) datetime64[ns] dask.array<chunksize=(1, 360), meta=np.ndarray>
  * range       (range) float32 125.0 375.0 625.0 ... 1.796e+05 1.799e+05
  * time        (time) datetime64[ns] 2021-09-02T05:10:35 ... 2021-09-02T05:3...
    sweep_mode  <U20 'azimuth_surveillance'
    longitude   float64 6.967
    latitude    float64 51.41
    altitude    float64 185.1
Data variables:
    DBZH        (time, azimuth, range) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>
    VRADH       (time, azimuth, range) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>
Attributes:
    fixed_angle:  5.5

plot sweeps

DBZH

[12]:
fig = pl.figure(figsize=(20, 30))
import matplotlib.gridspec as gridspec
gs = gridspec.GridSpec(4, 3, wspace=0.4, hspace=0.4)
for i, ts in enumerate(vol):
    swp = ts.isel(time=0)
    swp.DBZH.pipe(wrl.georef.georeference_dataset).wradlib.plot(ax=gs[i], fig=fig)
    ax = pl.gca()
    ax.set_title(vol.root.sweep_fixed_angle[i].values)
../../_images/notebooks_fileio_wradlib_load_DWD_opendata_volumes_19_0.png

VRADH

[13]:
fig = pl.figure(figsize=(20, 30))
import matplotlib.gridspec as gridspec
gs = gridspec.GridSpec(4, 3, wspace=0.4, hspace=0.4)
for i, ts in enumerate(vol):
    swp = ts.isel(time=0)
    swp.VRADH.pipe(wrl.georef.georeference_dataset).wradlib.plot(ax=gs[i], fig=fig)
    ax = pl.gca()
    ax.set_title(vol.root.sweep_fixed_angle[i].values)
../../_images/notebooks_fileio_wradlib_load_DWD_opendata_volumes_21_0.png

Plot single sweep using cartopy

[14]:
import cartopy
import cartopy.crs as ccrs
import cartopy.feature as cfeature

map_trans = ccrs.AzimuthalEquidistant(central_latitude=vol.root.latitude.values,
                                      central_longitude=vol.root.longitude.values)
[15]:
vol[-1]
[15]:
<xarray.Dataset>
Dimensions:     (azimuth: 360, time: 6, range: 240)
Coordinates:
  * azimuth     (azimuth) float64 0.5 1.5 2.5 3.5 ... 356.5 357.5 358.5 359.5
    elevation   (azimuth) float64 dask.array<chunksize=(360,), meta=np.ndarray>
    rtime       (time, azimuth) datetime64[ns] dask.array<chunksize=(1, 360), meta=np.ndarray>
  * range       (range) float32 125.0 375.0 625.0 ... 5.962e+04 5.988e+04
  * time        (time) datetime64[ns] 2021-09-02T05:13:51 ... 2021-09-02T05:3...
    sweep_mode  <U20 'azimuth_surveillance'
    longitude   float64 6.967
    latitude    float64 51.41
    altitude    float64 185.1
Data variables:
    DBZH        (time, azimuth, range) float32 dask.array<chunksize=(1, 360, 240), meta=np.ndarray>
    VRADH       (time, azimuth, range) float32 dask.array<chunksize=(1, 360, 240), meta=np.ndarray>
Attributes:
    fixed_angle:  25.0
[16]:
map_proj = ccrs.AzimuthalEquidistant(central_latitude=vol.root.latitude.values,
                                      central_longitude=vol.root.longitude.values)
pm = vol[-1].isel(time=0).DBZH.pipe(wrl.georef.georeference_dataset).wradlib.plot_ppi(proj=map_proj)
ax = pl.gca()
ax.gridlines(crs=map_proj)
print(ax)
< GeoAxes: <cartopy.crs.AzimuthalEquidistant object at 0x7f8f57c19680> >
../../_images/notebooks_fileio_wradlib_load_DWD_opendata_volumes_25_1.png
[17]:
map_proj = ccrs.Mercator(central_longitude=vol.root.longitude.values)
fig = pl.figure(figsize=(10,8))
ax = fig.add_subplot(111, projection=map_proj)
pm = vol[-1].isel(time=0).DBZH.pipe(wrl.georef.georeference_dataset).wradlib.plot_ppi(ax=ax)
ax.gridlines(draw_labels=True)
[17]:
<cartopy.mpl.gridliner.Gridliner at 0x7f8f5c078040>
../../_images/notebooks_fileio_wradlib_load_DWD_opendata_volumes_26_1.png
[18]:
fig = pl.figure(figsize=(10, 8))
proj=ccrs.AzimuthalEquidistant(central_latitude=vol.root.latitude.values,
                               central_longitude=vol.root.longitude.values)
ax = fig.add_subplot(111, projection=proj)
pm = vol[-1].isel(time=0).DBZH.pipe(wrl.georef.georeference_dataset).wradlib.plot_ppi(ax=ax)
ax.gridlines()
[18]:
<cartopy.mpl.gridliner.Gridliner at 0x7f8f5c17f580>
../../_images/notebooks_fileio_wradlib_load_DWD_opendata_volumes_27_1.png

Inspect radar moments

The DataArrays can be accessed by key or by attribute. Each DataArray inherits dimensions and coordinates of it’s parent dataset. There are attributes connected which are defined by Cf/Radial and/or ODIM_H5 standard.

[19]:
vol[-1].isel(time=0).DBZH
[19]:
<xarray.DataArray 'DBZH' (azimuth: 360, range: 240)>
dask.array<getitem, shape=(360, 240), dtype=float32, chunksize=(360, 240), chunktype=numpy.ndarray>
Coordinates:
  * azimuth     (azimuth) float64 0.5 1.5 2.5 3.5 ... 356.5 357.5 358.5 359.5
    elevation   (azimuth) float64 dask.array<chunksize=(360,), meta=np.ndarray>
    rtime       (azimuth) datetime64[ns] dask.array<chunksize=(360,), meta=np.ndarray>
  * range       (range) float32 125.0 375.0 625.0 ... 5.962e+04 5.988e+04
    time        datetime64[ns] 2021-09-02T05:13:51
    sweep_mode  <U20 'azimuth_surveillance'
    longitude   float64 6.967
    latitude    float64 51.41
    altitude    float64 185.1
Attributes:
    _Undetect:      0.0
    units:          dBZ
    long_name:      Equivalent reflectivity factor H
    standard_name:  radar_equivalent_reflectivity_factor_h
[20]:
vol[-1].isel(time=0).sweep_mode
[20]:
<xarray.DataArray 'sweep_mode' ()>
array('azimuth_surveillance', dtype='<U20')
Coordinates:
    time        datetime64[ns] 2021-09-02T05:13:51
    sweep_mode  <U20 'azimuth_surveillance'
    longitude   float64 6.967
    latitude    float64 51.41
    altitude    float64 185.1
[21]:
vol.root
[21]:
<xarray.Dataset>
Dimensions:              (sweep: 10)
Coordinates:
    sweep_mode           <U20 'azimuth_surveillance'
    longitude            float64 6.967
    altitude             float64 185.1
    latitude             float64 51.41
Dimensions without coordinates: sweep
Data variables:
    volume_number        int64 0
    platform_type        <U5 'fixed'
    instrument_type      <U5 'radar'
    primary_axis         <U6 'axis_z'
    time_coverage_start  <U20 '2021-09-02T05:10:35Z'
    time_coverage_end    <U20 '2021-09-02T05:39:02Z'
    sweep_group_name     (sweep) <U7 'sweep_0' 'sweep_1' ... 'sweep_8' 'sweep_9'
    sweep_fixed_angle    (sweep) float64 5.5 4.5 3.5 2.5 ... 8.0 12.0 17.0 25.0
Attributes:
    version:          None
    title:            None
    institution:      None
    references:       None
    source:           None
    history:          None
    comment:          im/exported using wradlib
    instrument_name:  None
    fixed_angle:      5.5

Plot Quasi Vertical Profile

[22]:
vol
[22]:
<wradlib.RadarVolume>
Dimension(s): (sweep: 10)
Elevation(s): (5.5, 4.5, 3.5, 2.5, 1.5, 0.5, 8.0, 12.0, 17.0, 25.0)
[23]:
ts = vol[-1]
ts
[23]:
<xarray.Dataset>
Dimensions:     (azimuth: 360, time: 6, range: 240)
Coordinates:
  * azimuth     (azimuth) float64 0.5 1.5 2.5 3.5 ... 356.5 357.5 358.5 359.5
    elevation   (azimuth) float64 dask.array<chunksize=(360,), meta=np.ndarray>
    rtime       (time, azimuth) datetime64[ns] dask.array<chunksize=(1, 360), meta=np.ndarray>
  * range       (range) float32 125.0 375.0 625.0 ... 5.962e+04 5.988e+04
  * time        (time) datetime64[ns] 2021-09-02T05:13:51 ... 2021-09-02T05:3...
    sweep_mode  <U20 'azimuth_surveillance'
    longitude   float64 6.967
    latitude    float64 51.41
    altitude    float64 185.1
Data variables:
    DBZH        (time, azimuth, range) float32 dask.array<chunksize=(1, 360, 240), meta=np.ndarray>
    VRADH       (time, azimuth, range) float32 dask.array<chunksize=(1, 360, 240), meta=np.ndarray>
Attributes:
    fixed_angle:  25.0
[24]:
fig = pl.figure(figsize=(10, 4))
ax = fig.add_subplot(111)
ts.DBZH.median('azimuth').plot(x='time', vmin=-10, vmax=30, ax=ax)
ax.set_title(f"{np.datetime_as_string(ts.time[0].values, unit='D')}")
ax.set_ylim(0, 20000)
[24]:
(0.0, 20000.0)
../../_images/notebooks_fileio_wradlib_load_DWD_opendata_volumes_35_1.png

Export to OdimH5

This exports the radar volume at given timestep including all moments into one ODIM_H5 compliant data file.

[25]:
vol.to_odim('dwd_odim.h5', timestep=0)

Export to Cf/Radial2

This exports the radar volume at given timestep including all moments into one Cf/Radial2 compliant data file.

[26]:
vol.to_cfradial2('dwd_cfradial2.nc', timestep=0)

Import again and check equality

[27]:
vol1 = wrl.io.open_odim_dataset('dwd_odim.h5')
vol2 = wrl.io.open_cfradial2_dataset('dwd_cfradial2.nc')
[28]:
xr.testing.assert_equal(vol1.root, vol2.root)
for i in range(len(vol1)):
    xr.testing.assert_equal(vol1[i].drop_vars("rtime"), vol2[i].drop_vars("rtime"))