Clutter detection - Satellite Cloud Images#

In this notebook we show howto identify clutter based on collocated cloudtype.

import matplotlib.pyplot as plt
import numpy as np
import wradlib as wrl
import wradlib_data
import xarray as xr
import xradar as xd
from IPython.display import display
from osgeo import osr
/home/docs/checkouts/readthedocs.org/user_builds/wradlib/conda/latest/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Read Satellite Cloud Image#

The cloud data from MSG3 satellite has some issue with the georeferencing. We need to apply a little fix, correct for that. The fix just adds the proper projection and coordinate objects to the dataset.

import warnings
import pyproj
from pyproj import CRS
from rasterio.errors import NotGeoreferencedWarning
from rasterio.transform import Affine

filename = wradlib_data.DATASETS.fetch("hdf5/SAFNWC_MSG3_CT___201304290415_BEL_________.h5")

def fix_georef(ds):
    attrs = ds.attrs
    gt = attrs['GEOTRANSFORM_GDAL_TABLE'].split(',')

    transform = Affine(
        float(gt[1]), float(gt[2]), float(attrs["XGEO_UP_LEFT"]),
        float(gt[4]), float(gt[5]), float(attrs["YGEO_UP_LEFT"])
    )

    crs = pyproj.CRS.from_proj4(attrs["PROJECTION"])
    ds = ds.rio.write_crs(crs, inplace=False)
    ds = ds.rio.write_transform(transform, inplace=False)

    ny, nx = ds.sizes[ds.rio.y_dim], ds.sizes[ds.rio.x_dim]

    xs = transform.c + (np.arange(nx) + 0.5) * transform.a
    ys = transform.f + (np.arange(ny) + 0.5) * transform.e

    ds = ds.drop_vars([ds.rio.x_dim, ds.rio.y_dim], errors="ignore")
    ds = ds.assign_coords(
        x=(ds.rio.x_dim, xs),
        y=(ds.rio.y_dim, ys),
    )

    return ds

with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("once", NotGeoreferencedWarning)
    sat = xr.open_dataset(filename, engine="rasterio", variable="CT")

    for _w in w:
        if isinstance(_w.message, NotGeoreferencedWarning):
            sat = sat.pipe(fix_georef)
            break

display(sat)
Downloading file 'hdf5/SAFNWC_MSG3_CT___201304290415_BEL_________.h5' from 'https://github.com/wradlib/wradlib-data/raw/main/data/hdf5/SAFNWC_MSG3_CT___201304290415_BEL_________.h5' to '/home/docs/.cache/wradlib-data'.
<xarray.Dataset> Size: 727kB
Dimensions:      (band: 1, y: 300, x: 600)
Coordinates:
  * band         (band) int64 8B 1
  * y            (y) float64 2kB 4.967e+06 4.964e+06 ... 4.073e+06 4.07e+06
  * x            (x) float64 5kB -6.166e+05 -6.136e+05 ... 1.178e+06 1.181e+06
    spatial_ref  int64 8B 0
Data variables:
    CT           (band, y, x) float32 720kB ...
Attributes: (12/65)
    01-PALETTE_CLASS:             PALETTE
    01-PALETTE_PAL_COLORMODEL:    RGB
    01-PALETTE_PAL_TYPE:          DIRECTINDEX
    02-PALETTE_CLASS:             PALETTE
    02-PALETTE_PAL_COLORMODEL:    RGB
    02-PALETTE_PAL_TYPE:          DIRECTINDEX
    ...                           ...
    TIME_STAMP_LOW_LINE:          20130429042542
    TIME_STAMP_UP_LINE:           20130429042642
    XGEO_LOW_RIGHT:               1179158.51935733
    XGEO_UP_LEFT:                 -618083.091571524
    YGEO_LOW_RIGHT:               4071547.35564349
    YGEO_UP_LEFT:                 4968667.95942934

Read radar data#

We read radar data from Belgium into a DataTree.

# read the radar volume scan
filename = wradlib_data.DATASETS.fetch("hdf5/20130429043000.rad.bewid.pvol.dbzh.scan1.hdf")
pvol = xd.io.open_odim_datatree(filename)
display(pvol)
Downloading file 'hdf5/20130429043000.rad.bewid.pvol.dbzh.scan1.hdf' from 'https://github.com/wradlib/wradlib-data/raw/main/data/hdf5/20130429043000.rad.bewid.pvol.dbzh.scan1.hdf' to '/home/docs/.cache/wradlib-data'.
<xarray.DataTree>
Group: /
│   Dimensions:              (sweep: 5)
│   Coordinates:
│       latitude             float64 8B ...
│       longitude            float64 8B ...
│       altitude             float64 8B ...
│   Dimensions without coordinates: sweep
│   Data variables:
│       volume_number        int64 8B 0
│       platform_type        <U5 20B 'fixed'
│       instrument_type      <U5 20B 'radar'
│       time_coverage_start  <U20 80B '2013-04-29T04:30:00Z'
│       time_coverage_end    <U20 80B '2013-04-29T04:31:39Z'
│       sweep_group_name     (sweep) int64 40B 0 1 2 3 4
│       sweep_fixed_angle    (sweep) float64 40B 0.3 0.9 1.8 3.3 6.0
│   Attributes:
│       Conventions:      ODIM_H5/V2_2
│       instrument_name:  None
│       version:          None
│       title:            None
│       institution:      None
│       references:       None
│       source:           None
│       history:          None
│       comment:          im/exported using xradar
├── Group: /sweep_0
│       Dimensions:            (azimuth: 360, range: 960)
│       Coordinates:
│         * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
│           elevation          (azimuth) float64 3kB ...
│           time               (azimuth) datetime64[ns] 3kB 2013-04-29T04:30:00.02777...
│         * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
│       Data variables:
│           DBZH               (azimuth, range) float64 3MB ...
│           sweep_mode         <U20 80B ...
│           sweep_number       int64 8B ...
│           prt_mode           <U7 28B ...
│           follow_mode        <U7 28B ...
│           sweep_fixed_angle  float64 8B ...
│           nyquist_velocity   float64 8B ...
├── Group: /sweep_1
│       Dimensions:            (azimuth: 360, range: 960)
│       Coordinates:
│         * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
│           elevation          (azimuth) float64 3kB ...
│           time               (azimuth) datetime64[ns] 3kB 2013-04-29T04:30:20.02777...
│         * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
│       Data variables:
│           DBZH               (azimuth, range) float64 3MB ...
│           sweep_mode         <U20 80B ...
│           sweep_number       int64 8B ...
│           prt_mode           <U7 28B ...
│           follow_mode        <U7 28B ...
│           sweep_fixed_angle  float64 8B ...
│           nyquist_velocity   float64 8B ...
├── Group: /sweep_2
│       Dimensions:            (azimuth: 360, range: 960)
│       Coordinates:
│         * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
│           elevation          (azimuth) float64 3kB ...
│           time               (azimuth) datetime64[ns] 3kB 2013-04-29T04:30:40.02777...
│         * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
│       Data variables:
│           DBZH               (azimuth, range) float64 3MB ...
│           sweep_mode         <U20 80B ...
│           sweep_number       int64 8B ...
│           prt_mode           <U7 28B ...
│           follow_mode        <U7 28B ...
│           sweep_fixed_angle  float64 8B ...
│           nyquist_velocity   float64 8B ...
├── Group: /sweep_3
│       Dimensions:            (azimuth: 360, range: 960)
│       Coordinates:
│         * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
│           elevation          (azimuth) float64 3kB ...
│           time               (azimuth) datetime64[ns] 3kB 2013-04-29T04:31:00.02777...
│         * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
│       Data variables:
│           DBZH               (azimuth, range) float64 3MB ...
│           sweep_mode         <U20 80B ...
│           sweep_number       int64 8B ...
│           prt_mode           <U7 28B ...
│           follow_mode        <U7 28B ...
│           sweep_fixed_angle  float64 8B ...
│           nyquist_velocity   float64 8B ...
└── Group: /sweep_4
        Dimensions:            (azimuth: 360, range: 960)
        Coordinates:
          * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
            elevation          (azimuth) float64 3kB ...
            time               (azimuth) datetime64[ns] 3kB 2013-04-29T04:31:20.02777...
          * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
        Data variables:
            DBZH               (azimuth, range) float64 3MB ...
            sweep_mode         <U20 80B ...
            sweep_number       int64 8B ...
            prt_mode           <U7 28B ...
            follow_mode        <U7 28B ...
            sweep_fixed_angle  float64 8B ...
            nyquist_velocity   float64 8B ...

Georeference#

Here we add two different georeferenced coordinates, xp, yp, zp for the radar azimuthal equidistant projection, and x, y, z for the satellite data projection.

pvol1 = pvol.match("sweep*")
vol = []
for sweep in pvol1:
    swp = pvol[sweep].to_dataset(inherit="all_coords").wrl.georef.georeference()
    swp = swp.rename(x="xp", y="yp", z="zp")
    swp = swp.wrl.georef.georeference(crs=sat.spatial_ref)
    vol.append(swp)
vol = xr.concat(vol, dim="tilt")
vol = vol.assign_coords(sweep_mode=vol.sweep_mode)
display(vol)
<xarray.Dataset> Size: 113MB
Dimensions:            (tilt: 5, azimuth: 360, range: 960)
Coordinates: (12/18)
    sweep_mode         (tilt) <U20 400B 'azimuth_surveillance' ... 'azimuth_s...
  * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
    elevation          (tilt, azimuth) float64 14kB 0.3 0.3 0.3 ... 6.0 6.0 6.0
    time               (tilt, azimuth) datetime64[ns] 14kB 2013-04-29T04:30:0...
  * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
    xp                 (tilt, azimuth, range) float64 14MB 1.091 ... -2.075e+03
    ...                 ...
    y                  (tilt, azimuth, range) float64 14MB 4.541e+06 ... 4.65...
    z                  (tilt, azimuth, range) float64 14MB 592.7 ... 2.901e+04
    latitude           float64 8B 49.91
    longitude          float64 8B 5.506
    altitude           float64 8B 592.0
    crs_wkt            int64 8B 0
Dimensions without coordinates: tilt
Data variables:
    DBZH               (tilt, azimuth, range) float64 14MB -32.0 -32.0 ... -32.0
    sweep_number       (tilt) int64 40B 0 1 2 3 4
    prt_mode           (tilt) <U7 140B 'not_set' 'not_set' ... 'not_set'
    follow_mode        (tilt) <U7 140B 'not_set' 'not_set' ... 'not_set'
    sweep_fixed_angle  (tilt) float64 40B 0.3 0.9 1.8 3.3 6.0
    nyquist_velocity   (tilt) float64 40B 7.98 7.98 7.98 7.98 7.98

Construct collocated satellite data#

Here we interpolate the satellite data into the radar grid (nearest neighbour) and assign it to the volume. See wradlib.ipol.IpolMethods.interpolate and Interpolate data on cartesian coordinates to polar coordinates.

ct = sat.isel(band=0).CT.wrl.ipol.interpolate(vol, method="map_coordinates_nearest")
vol = vol.assign(CT=ct)
display(vol)
<xarray.Dataset> Size: 120MB
Dimensions:            (tilt: 5, azimuth: 360, range: 960)
Coordinates: (12/21)
  * tilt               (tilt) int64 40B 0 1 2 3 4
    sweep_mode         (tilt) <U20 400B 'azimuth_surveillance' ... 'azimuth_s...
  * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
    elevation          (tilt, azimuth) float64 14kB 0.3 0.3 0.3 ... 6.0 6.0 6.0
    time               (tilt, azimuth) datetime64[ns] 14kB 2013-04-29T04:30:0...
  * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
    ...                 ...
    latitude           float64 8B 49.91
    longitude          float64 8B 5.506
    altitude           float64 8B 592.0
    crs_wkt            int64 8B 0
    band               int64 8B 1
    spatial_ref        int64 8B 0
Data variables:
    DBZH               (tilt, azimuth, range) float64 14MB -32.0 -32.0 ... -32.0
    sweep_number       (tilt) int64 40B 0 1 2 3 4
    prt_mode           (tilt) <U7 140B 'not_set' 'not_set' ... 'not_set'
    follow_mode        (tilt) <U7 140B 'not_set' 'not_set' ... 'not_set'
    sweep_fixed_angle  (tilt) float64 40B 0.3 0.9 1.8 3.3 6.0
    nyquist_velocity   (tilt) float64 40B 7.98 7.98 7.98 7.98 7.98
    CT                 (tilt, azimuth, range) float32 7MB 6.0 6.0 ... 10.0 10.0
print("Top-left CT center (x/y):", sat.x[0].values, sat.y[0].values)
print("Top-left radar (x/y):", vol.isel(tilt=0).x[0,0].values, vol.isel(tilt=0).y[0,0].values)
Top-left CT center (x/y): -616582.889893024 4967167.757750841
Top-left radar (x/y): 371141.201125636 4540645.9263388645

Estimate localisation errors#

timelag = 9 * 60
wind = 10
error = np.absolute(timelag) * wind

Identify clutter based on collocated cloudtype#

Then, we call wradlib.classify.ClassifyMethods.filter_cloudtype to derive the clutter map. The it is assiged to the volume.

clutter = vol.DBZH.wrl.classify.filter_cloudtype(vol.CT, smoothing=error)
vol = vol.assign(CMAP=clutter)
display(vol)
<xarray.Dataset> Size: 122MB
Dimensions:            (tilt: 5, azimuth: 360, range: 960)
Coordinates: (12/21)
  * tilt               (tilt) int64 40B 0 1 2 3 4
    sweep_mode         (tilt) <U20 400B 'azimuth_surveillance' ... 'azimuth_s...
  * azimuth            (azimuth) float32 1kB 0.5 1.5 2.5 ... 357.5 358.5 359.5
    elevation          (tilt, azimuth) float64 14kB 0.3 0.3 0.3 ... 6.0 6.0 6.0
    time               (tilt, azimuth) datetime64[ns] 14kB 2013-04-29T04:30:0...
  * range              (range) float32 4kB 125.0 375.0 ... 2.396e+05 2.399e+05
    ...                 ...
    latitude           float64 8B 49.91
    longitude          float64 8B 5.506
    altitude           float64 8B 592.0
    crs_wkt            int64 8B 0
    band               int64 8B 1
    spatial_ref        int64 8B 0
Data variables:
    DBZH               (tilt, azimuth, range) float64 14MB -32.0 -32.0 ... -32.0
    sweep_number       (tilt) int64 40B 0 1 2 3 4
    prt_mode           (tilt) <U7 140B 'not_set' 'not_set' ... 'not_set'
    follow_mode        (tilt) <U7 140B 'not_set' 'not_set' ... 'not_set'
    sweep_fixed_angle  (tilt) float64 40B 0.3 0.9 1.8 3.3 6.0
    nyquist_velocity   (tilt) float64 40B 7.98 7.98 7.98 7.98 7.98
    CT                 (tilt, azimuth, range) float32 7MB 6.0 6.0 ... 10.0 10.0
    CMAP               (tilt, azimuth, range) bool 2MB False False ... False

Plot the results#

fig = plt.figure(figsize=(16, 8))

tilt = 0

ax = fig.add_subplot(131)
pm = vol.DBZH[tilt].wrl.vis.plot(x="xp", y="yp", ax=ax)
# plt.colorbar(pm, shrink=0.5)
plt.title("Radar reflectivity")

ax = fig.add_subplot(132)
pm = vol.CT[tilt].wrl.vis.plot(x="xp", y="yp", ax=ax)
# plt.colorbar(pm, shrink=0.5)
plt.title("Satellite cloud classification")

ax = fig.add_subplot(133)
pm = vol.CMAP[tilt].wrl.vis.plot(x="xp", y="yp", ax=ax)
# plt.colorbar(pm, shrink=0.5)
plt.title("Detected clutter")

fig.tight_layout()
../../../_images/b95a433832d5926407b4941bebad4840fadc96915b1ac5d58c831b4c24880768.png