S1Tiling data flow
Overall processing
S1 Tiling processes by looping on all required S2 tiles within the time range.
For each S2 tile,
It downloads the necessary S1 images that intersect the S2 tile, within the specified time range, that are not already available in input data cache
Then, for each polarisation,
It calibrates, cuts and orthorectifies all the S1 images onto the S2 grid
It superposes (concatenates) the orthorectified images into a single S2 tile.
It filters the orthorectified images to reduce the speckle, if requested.
It builds masks, if requested.
Parallelization
The actual sequencing is not linear. S1 Tiling starts by building a dependencies graph expressed as a Dask Tasks graph. Dask framework then takes care of distributing the computations.
In the following processing of 33NWB and 33NWC from 2020-01-01 to 2020-01-10, only two S2 images are generated. It’s done by processing in parallel (but in any order compatible with the dependencies represented in the graph),
the S1 image inputs (first column) by calibrating and cutting them to obtain…
the orthoready files (second column), which are in turn orthorectified to obtain…
the orthorectified files (third column), which are in turn concatenated to obtain…
the final S2 products (fourth column),
border masks can in turn be generated from them – not represented on the graph.
Tasks for processing 33NWB and 33NWC
The processings
Downloading of S1 products
The downloading of S1 products is optional and done only if
[DataSource].download option is set to True.
S1 products are downloaded with eodag. See [DataSource].eodag_config regarding its configuration.
Downloaded files are stored into the directory specified by [Paths].s1_images option. If the directory doesn’t exist, it’s created on the fly.
Warning
In case a S1 product could not be downloaded in time (because it’s OFFLINE, or because there is a network error, or a provider error…) for an intersecting and requested S2 tile, the associated S2 product won’t be generated – even if the other S1 product that intersects the S2 tile is present (correctly downloaded or on disk).
Note
There is no way to know that a S1 product is missing when working only
from the disk (i.e. when [DataSource].download option is set to False). In that case
incomplete products may be generated. Their time stamp won’t contain
txxxxxx but the actual time at which the S1 product starts.
Beware, in that case we may see a txxxxxx S2 product + a
t{hhmmss} S2 product.
Note
While a S1 product may be impossible to download in time when processing a S2 tile, it may still be downloaded later on for a next tile. Yet S1Tiling won’t try to reprocess the first tile for which the product was initially missing.
You’ll have to run it again with the same parameters!
SAR Calibration
- Input:
An original input S1 image
- Output:
None: chained in memory with cutting
- OTBApplication:
- StepFactory:
This step applies σ°, β°, or γ° radiometric correction. The type of calibration is controlled with [Processing].calibration option. It also permits to remove thermal noise if required.
Note
At the end of this step, no file is produced as calibration is piped in memory with cutting to produce orthorectification ready images
Note
An extra artificial step is realized just after calibration to replace null values produced by denoising with a minimal signal value > 0. This way, 0 can be used all along the way as the no data value. This extra step also makes sure that the no-data on both sides of Sentinel-1 IPF v2.90+ products is kept as no-data.
Margins cutting
- Input:
None: chained in memory from SAR Calibration
- Output:
Either chained in memory with orthorectification
- OTBApplication:
- StepFactory:
This step takes care of resetting margins content to nodata=0 when too many no-data are detected within the margin. This phenomenon happens on coasts. The margins aren’t cut out like what ExtractROI would do but filled with 0’s, which permits to keep the initial geometry.
The implemented heuristic is the following:
1600 pixels (16km) on the top (/resp on the bottom) of the image will be cut if more than 2000 NoData (NoData is assimilated with 0 here) pixels are detected on the 100th row from the top (/resp from the bottom).
On Sentinel-1 IPF < v2.90 products, 1000 pixels (2 x 10 km) will always be cut on both sides (see MPC-0243: Masking “No-value” Pixels on GRD Products generated by the Sentinel-1 ESA IPF).
Starting from IPF v 2.90, Sentinel-1 products already have bands of nodata. No other marging cutting is done on the sides.
Note
The heuristic can be overridden thanks [Processing].override_azimuth_cut_threshold_to option.
At the end of this step, orthorectification ready images may be produced. It could be interresting to cache these product as a same cut-and-calibrated S1 image can be
orthorectified into several S2 grids it intersects. The default processing of
these products in memory can be disabled by passing --cache-before-ortho to
program:S1Processor.
Orthorectification
- Input:
Either chained in memory from cutting
- Output:
- OTBApplication:
- StepFactory:
This steps ortho-rectifies the cut and calibrated (aka “orthoready”) image in S1 geometry to S2 grid.
It uses the following parameters from the request configuration file:
Concatenation
- Inputs:
One or two consecutive orthorectified S1 images
- Output:
The main product of S1 Tiling: the final S2 tiles
- OTBApplication:
- StepFactory:
This step takes care of merging all the images of the orthorectified S1 products on a given S2 grid. As all orthorectified images are almost exclusive, they are concatenated by taking the first non null pixel.
This step produces the main product of S1 Tiling: the final S2 tiles.
|
⤖ |
|
Despeckling
- Inputs:
- Output:
- OTBApplications:
- StepFactories:
If requested, speckle filtering is applied to generate filtered S2 files.
Border mask generation
- Inputs:
- Output:
- OTBApplications:
- StepFactories:
If requested, border masks are generated.
The actual generation is done in two steps:
OTB BandMath application is used to generate border masks by saturating non-zero data to 1’s.
OTB BinaryMorphologicalOperation application is used to smooth border masks with a ball of 5x5 radius used for opening.
Data caches
Two kinds of data are cached, but only one is regularly cleaned-up by S1 Tiling. The other kind is left along as the software cannot really tell whether they could be reused later on or not.
Important
This means that you may have to regularly clean up this space.
Downloaded S1 files
S1 files are downloaded in [Paths].s1_images. directory. Whenever there are more than 1000 S1 products in that directory, only the 1000 most recent are kept. The oldest ones are automatically removed.
OrthoReady S1 Files
OrthoRectification is done on images cut, and calibrated. A same cut and calibrated Sentinel-1 image can be orthorectified onto different Sentinel-2 tiles.
This means it could be interresting to cache these intermediary products as files. Yet, this is not the default behaviour. Indeed at this time, S1-Tiling cannot know when an “OrthoReady” file is no longer required, nor organize the processing of S1 images to help deleting those temporary files as soon as possible. In other words, it’s up to you to clean these temporary files, and to make sure to not request too many S2 tiles on long time ranges.
That’s why the default behaviour is to process “OrthoReady” product in memory. Also, this is not necessarily a big performance issue. Indeed, given OTB internals, producing an orthorectified S1 image onto a S2 tile does not calibrate the whole S1 image, but only the minimal region overlapped by the S2 tile.
Note
Unless you execute S1Processor with
--cache-before-ortho, cutting, calibration and orthorectification are
chained in memory.
Warning
When executed with --cache-before-ortho, Cut and
calibrated (aka “OrthoReady”) files are stored in
%(tmp)/S1/ directory.
Do not forget to regularly clean up this space.