Usage¶
Contents:
Scenarios¶
Orthorectify pairs of Sentinel-1 images on Sentinel-2 grid¶
This is the main scenario where pairs of Sentinel-1 images are:
- calibrated according to β0, γ0 or σ0 calibration
- then orthorectified onto the Sentinel-2 grid,
- to be finally concatenated.
The unique elements in this scenario are:
- the calibration option that must be
either one of
beta
,sigma
orgamma
- the main executable which is S1Processor.
All options go in a request configuration file
(e.g. MyS1ToS2.cfg
in workingdir
). Important options will be:
- the time range (first_date and last_date),
- the Sentinel-2 tiles,
- the orthorectification options (in [Processing]),
- the directories where images are downloaded, produced, etc.
- the download credentials for the chosen data provider – see eodag_config.
Then running S1Tiling is as simple as:
cd workingdir
S1Processor MyS1ToS2.cfg
Eventually,
- The S1 products will be downloaded in s1_images.
- The orthorectified tiles will be generated in output.
- Temporary files will be produced in tmp.
Note
S1 Tiling never cleans the tmp directory as its files are cached in between runs. This means you will have to watch this directory and eventually clean it.
Orthorectify pairs of Sentinel-1 images on Sentinel-2 grid with σ0RTC NORMLIM calibration¶
In this scenario, the calibration applied is the \(σ^0_{RTC}\) NORMLIM calibration described in [Small2011].
[Small2011] | D. Small, “Flattening Gamma: Radiometric Terrain Correction for SAR Imagery,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 49, no. 8, pp. 3081-3093, Aug. 2011, doi: 10.1109/TGRS.2011.2120616. |
In S1Tiling, we have chosen to precompute Local Incidence Angle (LIA) maps on Sentinel-2 grid. Given a series of Sentinel-1 images to orthorectify on a Sentinel-2 grid, we select a pair of Sentinel-1 images to compute the LIA in the geometry of these images. The LIA map is then projected, through orthorectification, on a Sentinel-2 tile.
That map will then be used for all series of pairs of Sentinel-1 images that intersect the associated S2 tile.
Regarding options, the only difference with previous scenario are:
- the calibration option that needs to be
normlim
, - the directory where LIA maps will be searched for, or produced in.
S1Tiling will then automatically take care of:
- producing, or using existing, maps of sin(LIA) for each Sentinel-2 tiles – given an orbit and it direction,
- producing intermediary products calibrated with β0 LUT.
Warning
If you wish to parallelize this scenario and dedicate a different cluster node to each date – as recommended in “Process huge quantities of data” scenario, you will NEED produce all the LIA maps beforehand. Otherwise a same file may be concurrently written to from different nodes, and it will likely end up corrupted.
Note
This scenario requires DiapOTB and NORMLIM σ0 binaries. At this times, DiapOTB binaries are shipped with OTB 7.4 (but not with OTB 8), and NORMLIM σ0 binaries need to be compiled manually. Eventually both will be guaranteed in S1Tiling docker images.
Preproduce maps of Local Incidence Angles for σ0RTC NORMLIM calibration¶
While S1Processor is able to produce the necessary LIA maps on the fly, it is not able to do so when parallelization is done manually over time ranges – as described in “Process huge quantities of data” scenario.
A different program is provided to compute the LIA maps beforehand: S1LIAMap. It takes the exact same parameter files as S1Processor. A few options will be ignored though: calibration type, masking….
cd workingdir
# Yes, the same file works!
S1LIAMap MyS1ToS2.cfg
Note
LIA maps are perfect products to be stored and reused.
Note
This scenario requires DiapOTB and NORMLIM σ0 binaries. At this times, DiapOTB binaries are shipped with OTB 7.4 (but not with OTB 8), and NORMLIM σ0 binaries need to be compiled manually. Eventually both will be guaranteed in S1Tiling docker images.
Note
To run S1LIAMap from the official S1Tiling docker, use --lia
as the first parameter to the docker execution (just before the the
request configuration file and other S1LIAMap related parameters). See
Using S1LIAMap with a docker.
Generate masks on final products¶
Pixel masks of valid data can be produced in all S1Processor
scenarios when the option generate_border_mask is True
.
Process huge quantities of data¶
This use case concerns people that:
- have a lot of images to process over many tiles and over a consequent time-range,
- and have access to computing resources like HPC clusters
In that case, S1Tiling will be much more efficient if the parallelization is done time-wise. We recommended to cut the full time range in smaller subranges, and to distribute each subrange (with all S2 tiles) to a different node – with jobarrays for instances.
Warning
This scenario is not compatible with normlim
calibration where the LIA
maps would be computed on-the-fly. For normlim
calibration, it’s
imperative to precompute (and store LIA maps) before going massively
parallel.
Request Configuration file¶
The request configuration file passed to S1Processor
is in .ini
format.
It is expected to contain the following entries.
You can use this this template
, as a starting point.
[PATHS]
section¶
Option | Description |
---|---|
s1_images |
Where S1 images are downloaded thanks to EODAG.
S1Tiling will automatically take care to keep at most 1000 products in that directory – the 1000 last that have been downloaded. This enables to cache downloaded S1 images in beteen runs. |
output |
Where products are generated. |
lia |
Where Local Incidence Maps and sin(LIA) products are generated. Its
default value is {output}/_LIA . |
tmp |
Where intermediary files are produced, and sometimes cached for longer periods. |
geoid_file |
Path to Geoid model. If left unspecified, it’ll point automatically to the geoid resource shipped with S1 Tiling. |
srtm |
Path to SRTM files. |
[DataSource]
section¶
Option | Description |
---|---|
download |
If True , activates the downloading from specified data provider for
the ROI, otherwise only local S1 images already in s1_images will be processed. |
eodag_config |
Designates where the EODAG configuration file is expected to be found.
From S1Tiling point of view, EODAG configuration file will list the
authentification credentials for the know providers and their respective
priorities.
For instance, given a PEPS account, peps:
auth:
credentials:
username: THEUSERNAME
password: THEPASSWORD
|
nb_parallel_downloads |
Number of parallel downloads (+ unzip) of source products. Warning Don’t abuse this setting as the data provider may not support too many parallel requests. |
roi_by_tiles |
The Region of Interest (ROI) for downloading is specified in roi_by_tiles
which will contain a list of MGRS tiles. If [DataSource]
roi_by_tiles : 33NWB
|
platform_list |
Defines the list of platforms from where come the products to download
and process.
Valid values are expected in the form of S1* . |
polarisation |
Defines the polarisation mode of the products to download and process.
Only six values are valid: HH-HV , VV-VH , VV , VH , HV ,
and HH . |
orbit_direction |
Download only the products acquired in ascending ( Warning Each relative orbit is exclusive to one orbit direction, orbit_direction and relative_orbit_list shall be considered as exclusive. |
relative_orbit_list |
Download only the products from the specified relative orbits. By default (when left unspecified), no filter is applied. Warning Each relative orbit is exclusive to one orbit direction, orbit_direction and relative_orbit_list shall be considered as exclusive. |
first_date |
Initial date in YYYY-MM-DD format. |
last_date |
Final date in YYYY-MM-DD format. |
tile_to_product_overlap_ratio |
Percentage of tile area to be covered for a single or a pair of Sentinel-1 products to be retained. The number is expected as an integer in the [1..100] range. |
[Mask]
section¶
Option | Description |
---|---|
generate_border_mask |
This option allows you to choose if you want to generate border masks of the S2 image file produced. |
[Processing]
section¶
Option | Description | |||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
cache_srtm_by |
Tells whether SRTM files are copied in a temporary directory, or if symbolic links are to be created. For performance reasons with OTB 7.X, it’s better to regroup the minimal subset of the SRTM files required for processing. Symbolic links work fine most of the time, however if the files are on a remote shared filesystem (GPFS, NAS…), performances will be degraded. In those cases, it’s better to copy the required SRTM files on a local filesystem. Two values are supported for this option: |
|||||||||||||||||||||||||||||||||||||||||||||
calibration |
Defines the calibration type: gamma , beta , sigma , or
normlim . |
|||||||||||||||||||||||||||||||||||||||||||||
remove_thermal_noise |
Shall the thermal noise be removed? Important This feature requires a version of OTB >= 7.4.0 |
|||||||||||||||||||||||||||||||||||||||||||||
lower_signal_value |
Noise removal may set some pixel values to 0. However, 0, is currently reserved by S1Tiling chain as a “nodata” value introduced by Margin Cutting and Orthorectification. This parameter defines which value to use instead of 0 when noise is removed. By default: 1e-7 will be used. |
|||||||||||||||||||||||||||||||||||||||||||||
output_spatial_resolution |
Pixel size (in meters) of the output images | |||||||||||||||||||||||||||||||||||||||||||||
tiles_shapefile |
Path and filename of the tile shape definition (ESRI Shapefile). If left unspecified, it’ll point automatically to the Features.shp shapefile resource shipped with S1 Tiling. | |||||||||||||||||||||||||||||||||||||||||||||
orthorectification_gridspacing |
Grid spacing (in meters) for the interpolator in the orthorectification process for more information, please consult the OTB OrthoRectification application. A nice value is 4 x output_spatial_resolution |
|||||||||||||||||||||||||||||||||||||||||||||
orthorectification_interpolation_method |
Interpolation method used in the orthorectification process for more information, please consult the OTB OrthoRectification application. Default value is set to nearest neighbor interpolation (nn) to keep compatibilty with previous results By the way linear method could be more interesting. Note that the bco method is not currently supported |
|||||||||||||||||||||||||||||||||||||||||||||
tiles , tiles_list_in_file |
Tiles to be processed. The tiles can be given as a list:
|
|||||||||||||||||||||||||||||||||||||||||||||
mode |
Running mode:
Ex.: mode : debug logging
|
|||||||||||||||||||||||||||||||||||||||||||||
nb_parallel_processes |
Number of processes to be running in parallel
Note For optimal performances, |
|||||||||||||||||||||||||||||||||||||||||||||
ram_per_process |
RAM allowed per OTB application pipeline, in MB. | |||||||||||||||||||||||||||||||||||||||||||||
nb_otb_threads |
Numbers of threads used by each OTB application. Note For optimal performances, |
|||||||||||||||||||||||||||||||||||||||||||||
produce_lia_map |
When LIA sine map is produced, we may also desire the angle values in degrees (x100). Possible values are:
Note This option will be ignored when no LIA sine map is required. The LIA
sine map is produced by S1LIAMap program ,
or when calibration mode is
|
|||||||||||||||||||||||||||||||||||||||||||||
override_azimuth_cut_threshold_to |
Permits to override the analysis on whether top/bottom lines shall be
forced to 0 in cutting step. Possible values are:
Warning This option is not meant to be used. It only makes sense in some very specific scenarios like tests. |
|||||||||||||||||||||||||||||||||||||||||||||
fname_fmt.* |
Set of filename format templates that permits to override the default filename formats used to generate filenames. The filename formats can be overridden for both intermediary and final products. Only the final products are documented here. Filename formats for intermediary products are best left alone. If you change any, make sure to not introduce ambiguity by removing a field that would be used to distinguish two unrelated products. Available fields comme from
|
|||||||||||||||||||||||||||||||||||||||||||||
fname_fmt.concatenation |
File format pattern for concatenation products,
for β°, σ° and γ° calibrations.
{flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}.tif |
|||||||||||||||||||||||||||||||||||||||||||||
fname_fmt.s2_lia_corrected |
File format pattern for concatenation products
when NORMLIM calibrated.
{flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}_NormLim.tif |
|||||||||||||||||||||||||||||||||||||||||||||
fname_fmt.lia_product |
File format pattern for LIA and sin(LIA) files
{LIA_kind}_{flying_unit_code}_{tile_name}_{orbit_direction}_{orbit}.tif |
|||||||||||||||||||||||||||||||||||||||||||||
fname_fmt.filtered |
File format pattern for filtered files
{flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}_filtered.tif
for β°, σ° and γ° calibrations,
{flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}_NormLim_filtered.tif when NORMLIM calibrated. |
[Filtering]
section¶
Note
Multitemporal filtering is not yet integrated in S1Tiling.
Option | Description |
---|---|
filter |
If none or empty, then no filtering is done. Otherwise the following
spatial speckling filter methods from OTB Despeckle application are supported: Lee , Frost ,
Gammamap , Kuan . |
window_radius |
Sets the window radius for the spatial filtering. Take care that it is a radius, i.e. radius=1 means the filter does an 3x3 pixels averaging. |
deramp |
Deramp factor – for Frost filter only. Factor use to control the exponential function used to weight effect of the distance between the central pixel and its neighborhood. Increasing the deramp parameter will lead to take more into account pixels farther from the center and therefore increase the smoothing effects. |
nblooks |
Number of looks – for all but Frost => Lee, Gammamap and Kuan |
keep_non_filtered_products |
If not caring for non-filtered product (and if filter method is
specified), then the orthorectified and concatenated products won’t be
considered as mandatory and they will not be kept at the end of the
processing.
This (exclusion) feature cannot be used alongside
[Mask].generate_border_mask (i.e.
Warning Note: This feature is only supported after LIA calibration as of V1.0 of S1Tiling. See Issue #118. |
Log configuration¶
Default logging configuration is provided in S1Tiling
installing directory.
It can be overridden by dropping a file similar to
../s1tiling/logging.conf.yaml
in the same directory as the one
where the request configuration file is. The file
is expected to follow logging configuration
file
syntax.
Warning
This software expects the specification of:
When mode contains logging
, we make sure that
file
and important
handlers
are added to
the handlers of root
and distributed.worker
loggers
. Note that this is the default configuration.
When mode contains debug
the DEBUG
logging
level is forced into root
logger, and $OTB_LOGGER_LEVEL
environment
variable is set to DEBUG
.
Working on clusters¶
Todo
By default S1Tiling works on single machines. Internally it relies on
distributed.LocalCluster
a small adaptation would be required to
work on a multi-nodes cluster.
Warning
When executing multiple instances of S1Tiling simultaneously, make sure to use different directories for:
- logs – running S1Tiling in different directories, like
$TMPDIR/
on TREX, should be enough - storing input files, like for instance
$TMPDIR/data_raw/
on HAL/TREX for instance.
Process return code¶
The following exit code are produced when S1Processor returns:
Exit code | Description |
---|---|
0 | Execution successful |
66 | Some OTB tasks could not be executed properly. See the final report in the main log. |
67 | Downloading error. See the log produced. |
68 | When offline S1 data could not be retrieved before the configured timeout, the associated S2 products will not be generated and this exit code will be used. See the log produced. If more critical errors occur, this exit will be superceded. |
69 | Todo Output disk full |
70 | Todo Cache disk full (when using option |
71 | An empty data safe has been found and needs to be removed so it can be fetched again. See the log produced. |
72 | Error detected in the configuration file. See the log produced. |
73 | While ALL Sentinel-2 tiles for which there exist an overlapping
Sentinel-1 product have been requested,
no Sentinel-1 product has been found in the requested time range. See the log produced. |
74 | No Sentinel-1 product has been found that intersects the requested Sentinel-2 tiles within the requested time range. If downloading has been disabled, S1 products are searched in the local input directory. See the log produced. |
75 | Cannot find all the SRTM products that cover the requested Sentinel-2 tiles. See the log produced. |
76 | Geoid file is missing or the specified path is incorrect. See the log produced. |
77 | Some processing cannot be done because external applications cannot be executed. Likelly OTB and/or NORMLIM related applications aren’t correctly installed. See the log produced. |
any other | Unknown error. It could be related to Bash or to Python reserved error codes. |