S1Tiling data flow

Overall processing

S1 Tiling processes by looping on all required S2 tiles within the time range.

For each S2 tile,

  1. It downloads the necessary S1 images that intersect the S2 tile, within the specified time range, that are not already available in input data cache

  2. Then, for each polarisation,

    1. It calibrates, cuts and orthorectifies all the S1 images onto the S2 grid

    2. It superposes (concatenates) the orthorectified images into a single S2 tile.

    3. It filters the orthorectified images.

    4. It builds masks, if required

Parallelization

The actual sequencing is not linear. S1 Tiling starts by building a dependencies graph expressed as a Dask Tasks graph. Dask framework then takes care of distributing the computations.

In the following processing of 33NWB and 33NWC from 2020-01-01 to 2020-01-10, only two S2 images are generated. It’s done by processing in parallel (but in any order compatible with the dependencies represented in the graph),

Complete task flow for processing 33NWB and 33NWC

Tasks for processing 33NWB and 33NWC

The processings

Downloading of S1 products

The downloading of S1 products is optional and done only if [DataSource].download option is set to True.

S1 products are downloaded with eodag. See [DataSource].eodag_config regarding its configuration.

Downloaded files are stored into the directory specified by [Paths].s1_images option. If the directory doesn’t exist, it’s created on the fly.

Warning

In case a S1 product could not be downloaded in time (because it’s OFFLINE, or because there is a network error, or a provider error…) for an intersecting and requested S2 tile, the associated S2 product won’t be generated – even if the other S1 product that intersects the S2 tile is present (correctly downloaded or on disk).

Note

There is no way to know that a S1 product is missing when working only from the disk (i.e. when [DataSource].download option is set to False). In that case incomplete products may be generated. Their time stamp won’t contain txxxxxx but the actual time at which the S1 product starts.

Beware, in that case we may see a txxxxxx S2 product + a t{hhmmss} S2 product.

Note

While a S1 product may be impossible to download in time when processing a S2 tile, it may still be downloaded later on for a next tile. Yet S1Tiling won’t try to reprocess the first tile for which the product was initially missing.

You’ll have to run it again with the same parameters!

SAR Calibration

Input:

An original input S1 image

Output:

None: chained in memory with cutting

OTBApplication:

OTB SARCalibration application

StepFactory:

s1tiling.libs.otbwrappers.Calibrate

This step applies σ°, β°, or γ° radiometric correction. The type of calibration is controlled with [Processing].calibration option. It also permits to remove thermal noise if required.

Note

At the end of this step, no file is produced as calibration is piped in memory with cutting to produce orthorectification ready images

Note

An extra artificial step is realized just after calibration to replace null values produced by denoising with a minimal signal value > 0. This way, 0 can be used all along the way as the no data value.

Margins cutting

Input:

None: chained in memory from SAR Calibration

Output:
OTBApplication:

OTB ResetMargin application

StepFactory:

s1tiling.libs.otbwrappers.CutBorders

This step takes care of resetting margins content to 0 when too many no-data are detected within the margin. This phenomenon happens on coasts. The margins aren’t cut out like what ExtractROI would do but filled with 0’s, which permits to keeps the initial geometry.

The implemented heuristic is to:

Note

The heuristic can be overridden thanks [Processing].override_azimuth_cut_threshold_to option.

At the end of this step, orthorectification ready images may be produced. It could be interresting to cache these product as a same cut-and-calibrated S1 image can be orthorectified into several S2 grids it intersects. The default processing of these products in memory can be disabled by passing --cache-before-ortho to program:S1Processor.

Orthorectification

Input:
Output:

orthorectified S1 images

OTBApplication:

OTB OrthoRectification application

StepFactory:

s1tiling.libs.otbwrappers.OrthoRectify

This steps ortho-rectifies the cut and calibrated (aka “orthoready”) image in S1 geometry to S2 grid.

It uses the following parameters from the request configuration file:

Concatenation

Inputs:

One or two consecutive orthorectified S1 images

Output:

The main product of S1 Tiling: the final S2 tiles

OTBApplication:

OTB Synthetize application

StepFactory:

s1tiling.libs.otbwrappers.Concatenate

This step takes care of merging all the images of the orthorectified S1 products on a given S2 grid. As all orthorectified images are almost exclusive, they are concatenated by taking the first non null pixel.

This step produces the main product of S1 Tiling: the final S2 tiles.

Two orthorectified and exclusive S1 images

⤖

 The orthorectified result

Border mask generation

Inputs:

final S2 tiles

Output:

border masks

OTBApplications:
StepFactories:

If requested, border masks are generated.

The actual generation is done in two steps:

  1. OTB BandMath application is used to generate border masks by saturating non-zero data to 1’s.

  2. OTB BinaryMorphologicalOperation application is used to smooth border masks with a ball of 5x5 radius used for opening.

Data caches

Two kinds of data are cached, but only one is regularly cleaned-up by S1 Tiling. The other kind is left along as the software cannot really tell whether they could be reused later on or not.

Important

This means that you may have to regularly clean up this space.

Downloaded S1 files

S1 files are downloaded in [Paths].s1_images. directory. Whenever there are more than 1000 S1 products in that directory, only the 1000 most recent are kept. The oldest ones are automatically removed.

OrthoReady S1 Files

OrthoRectification is done on images cut, and calibrated. A same cut and calibrated Sentinel-1 image can be orthorectified onto different Sentinel-2 tiles.

This means it could be interresting to cache these intermediary products as files. Yet, this is not the default behaviour. Indeed at this time, S1-Tiling cannot know when an “OrthoReady” file is no longer required, nor organize the processing of S1 images to help deleting those temporary files as soon as possible. In other words, it’s up to you to clean these temporary files, and to make sure to not request too many S2 tiles on long time ranges.

That’s why the default behaviour is to process “OrthoReady” product in memory. Also, this is not necessarily a big performance issue. Indeed, given OTB internals, producing an orthorectified S1 image onto a S2 tile does not calibrate the whole S1 image, but only the minimal region overlapped by the S2 tile.

Note

Unless you execute S1Processor with --cache-before-ortho, cutting, calibration and orthorectification are chained in memory.

Warning

When executed with --cache-before-ortho, Cut and calibrated (aka “OrthoReady”) files are stored in %(tmp)/S1/ directory. Do not forget to regularly clean up this space.