Configuration

Request Configuration file

The request configuration file passed to S1Processor is in .ini format. It is expected to contain the following entries.

Note

S1LIAMap and S1IAMap work with a subset of the following configuration keys. Unsupported keys will simply be ignored.

You can use this this template, as a starting point.

[PATHS] section

Option

Description

s1_images

Where S1 images are downloaded thanks to EODAG.
S1Tiling will automatically take care to keep at most 1000 products in that directory – the 1000 last products that have been downloaded.
This enables to cache downloaded S1 images in beteen runs.

output

Where products are generated.

ia

Where (non Local) Incidence Maps and sin(IA) products are generated. Its default value is {output}/_IA.

lia

Where Local Incidence Maps and sin(LIA) products are generated. Its default value is {output}/_LIA.

tmp

Where intermediary files are produced, and sometimes cached for longer periods.

geoid_file

Path to Geoid model. If left unspecified, it’ll point automatically to the geoid resource shipped with S1 Tiling.

dem_database

Path to DEM (.gpkg) database.
By default points to the internal shapefile/srtm_tiles.gpkg file which knows the geometry of SRTM 30 DEM files.

dem_dir

Path to DEM files.

dem_format

Filename format string to locate the DEM file associated to an identifier within the [PATHS].dem_dir directory.
By default associates the id key of tiles found in the DEM database to {id}.hgt. One may want to use the keys from eotile DEM database like for instance {Product10}.tif for Copernicus 30m DEM.

dem_info

DEM identifier to inject in the products GeoTIFF metadata under DEM_INFO key. If not defined, the last part (basename) of [Paths].dem_dir will be used.

srtm

(deprecated) Use [PATHS].dem_dir. Path to SRTM files.

eof_dir

Where precise orbit orbit files (EOF) are expected to be found, or where they would be downloaded on the fly. Default value is {output}/_EOF.

See also Q: How can I configure precise orbit files retrieval?.

[DataSource] section

Option

Description

download

If True, activates the downloading from specified data provider for the ROI, otherwise only local S1 images already in s1_images will be processed.

eodag_config

Designates where the EODAG configuration file is expected to be found.
Default value: %(HOME)s/.config/eodag/eodag.yml.

From S1Tiling point of view, EODAG configuration file will list the authentification credentials for the know providers and their respective priorities.
See EODAG § on Configure EODAG

For instance, given a PEPS account, $HOME/.config/eodag/eodag.yml could contain

peps:
    auth:
        credentials:
            username: THEUSERNAME
            password: THEPASSWORD

nb_parallel_downloads

Number of parallel downloads (+ unzip) of source products.

Warning

Don’t abuse this setting as the data provider may not support too many parallel requests.

roi_by_tiles

The Region of Interest (ROI) for downloading is specified in roi_by_tiles which will contain a list of MGRS tile names. If ALL is specified, the software will download all images needed for the processing (see [Processing] section)

[DataSource]
roi_by_tiles : 33NWB

platform_list

Defines the list of platforms from where come the products to download and process. Valid values are S1A or S1B.

Warning

A single value is expected in NORMLIM scenarios.

polarisation

Defines the polarisation mode of the products to download and process. Only six values are valid: HH-HV, VV-VH, VV, VH, HV, and HH.

orbit_direction

Download only the products acquired in ascending (ASC) or in descending (DES) order. By default (when left unspecified), no filter is applied.

Warning

Each relative orbit is exclusive to one orbit direction, orbit_direction and relative_orbit_list shall be considered as exclusive.

relative_orbit_list

Download only the products from the specified relative orbits. By default (when left unspecified), no filter is applied.

Warning

Each relative orbit is exclusive to one orbit direction, orbit_direction and relative_orbit_list shall be considered as exclusive.

Warning

A single value is expected in NORMLIM and Ellipsoid Incide Angle scenarios.

first_date

Initial date in YYYY-MM-DD format.

last_date

Final date in YYYY-MM-DD format.

tile_to_product_overlap_ratio

Percentage of tile area to be covered for a single or a pair of Sentinel-1 products to be retained.

The number is expected as an integer in the [1..100] range.

[Mask] section

Option

Description

generate_border_mask

This option allows you to choose if you want to generate border masks of the S2 image files produced. Values are True or False.

[Processing] section

Option

Description

cache_dem_by

Tells whether DEM and Geoid files are copied in a temporary directory, or if symbolic links are to be created.

For performance reasons with OTB, it’s better to regroup the minimal subset of the DEM files required for processing. Symbolic links work fine most of the time, however if the files are on a remote shared filesystem (GPFS, NAS…), performances will be degraded. In those cases, it’s better to copy the required DEM files on a local filesystem.

Geoid file will be also copied (or symlinked), but in {tmp}/geoid/. It won’t be removed automatically. You can also do it manually before running S1Tiling.

Two values are supported for this option: copy and symlink. (default: symlink).

calibration

Defines the calibration type: gamma, beta, sigma, or normlim.

remove_thermal_noise

Activate the thermal noise removal in the images. Values are True or False.

lower_signal_value

Noise removal may set some pixel values to 0. However, 0, is currently reserved by S1Tiling chain as a “no-data” value introduced by Margin Cutting and Orthorectification.

This parameter defines which value to use instead of 0 when noise is removed. By default: 1e-7 will be used.

nodata.IA

No-data value to use in IA files

nodata.LIA

No-data value to use in LIA files

output_spatial_resolution

Pixel size (in meters) of the output images

tiles_shapefile

Path and filename of the tile shape definition (ESRI Shapefile). If left unspecified, it’ll point automatically to the Features.shp shapefile resource shipped with S1 Tiling.

orthorectification_gridspacing

Grid spacing (in meters) for the interpolator in the orthorectification process. For more information, please consult the OTB OrthoRectification application.

A nice value is 4 x output_spatial_resolution

orthorectification_interpolation_method

Interpolation method used in the orthorectification process. For more information, please consult the OTB OrthoRectification application.

Default value is set to nearest neighbor interpolation (nn) to keep compatibilty with previous results ; Linear method could be more interesting. Note that the bco method is not currently supported.

tiles, tiles_list_in_file

Tiles to be processed. The tiles can be given as a list:

  • tiles: list of tiles (comma separated). Ex:

    tiles: 33NWB, 33NWC

  • tiles_list_in_file: tile list in a ASCII file. Ex:

    tiles_list_in_file : ~/MyListOfTiles.txt

mode

Running mode:

  • Normal: prints normal, warning and errors on screen

  • debug: also prints debug messages, and forces $OTB_LOGGER_LEVEL=DEBUG

  • logging: saves logs to files

Ex.:

mode : debug logging

nb_parallel_processes

Number of processes to be running in parallel
This number defines the number of Dask Tasks (and indirectly of OTB applications) to be executed in parallel.

Note

For optimal performances, nb_parallel_processes*nb_otb_threads should be ≤ to the number of cores on the machine.

ram_per_process

RAM allowed per OTB application pipeline, in MB.

nb_otb_threads

Numbers of threads used by each OTB application.

Note

For optimal performances, nb_parallel_processes*nb_otb_threads should be ≤ to the number of cores on the machine.

produce_lia_map

When LIA sine map are produced, we may also desire the angle values in degrees (x100).

Possible values are:

True:

Do generate the angle map in degrees x 100.

False:

Don’t generate the angle map in degrees x 100.

Note

This option will be ignored when no LIA sine map is required. The LIA sine map is produced by S1LIAMap program, or when calibration mode is "normlim".

ia_maps_to_produce

By default, S1IAMap program produce a map of the incidence angle to the WGS84 ellipsoid in degrees x 100. This option permis to select which of the 4 IA maps will be generated.

deg:

map in degrees x 100

cos:

cosine map

sin:

sine map

tan:

tangent map

dem_warp_resampling_method

DEM files projected on S2 tiles are required to produce LIA maps. This parameters permits to select the resampling method that gdalwarp will use.

The possible values are: near, bilinear, cubic, cubicspline, lanczos, average, rms, mode, max, min, med, q1, q3 and qum.

override_azimuth_cut_threshold_to

Permits to override the analysis on whether top/bottom lines shall be forced to 0 in cutting step.

Possible values are:

True:

Force cutting at the 1600th upper and the 1600th lower lines.

False:

Force to keep every line.

not set/None:

Default analysis heuristic is used.

Warning

This option is not meant to be used. It only makes sense in some very specific scenarios like tests.

fname_fmt.*

Set of filename format templates that permits to override the default filename formats used to generate filenames.

The filename formats can be overridden for both intermediary and final products. Only the final products are documented here. Filename formats for intermediary products are best left alone.

If you change any, make sure to not introduce ambiguity by removing a field that would be used to distinguish two unrelated products.

Available fields come from internal metadata. The main ones of interest are:

Field

Content

Applies to geometry

flying_unit_code

s1a, s1b

S1/S2

tile_name

ex: 33NWB

S2

polarisation

hh, hv, vh, vv

S1/S2

orbit_direction

ASC/DES

S1/S2

orbit

5-digits number that identifies the S1 orbit

S1/S2

acquisition_time

the full timestamp (yymmddthhmmss)

S1/S2

acquisition_day

only the day (yymmddtxxxxxx)

S1/S2

acquisition_stamp

either the full timestamp (yymmddthhmmss), or the day (yymmddtxxxxxx)

S1/S2

IA_kind

IA/cos_IA/sin_IA/tan_IA

S2

LIA_kind

LIA/sin_LIA

S2

basename

Filename of initial S1 image.

S1

rootname

basename without the file extension.

S1

calibration_type

beta/gamma/sigma/dn/Normlim

S1/S2

polarless_basename

Same as basename (with file extension), but without polarisation field. Used when the product only depends on the S1 image geometry and not its content.

S1

polarless_rootname

Same as rootname (without file extension), but without polarisation field. Used when the product only depends on the S1 image geometry and not its content.

S1

fname_fmt.concatenation

File format pattern for concatenation products, for β°, σ° and γ° calibrations.

Default value: {flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}.tif

fname_fmt.s2_lia_corrected

File format pattern for concatenation products when NORMLIM calibrated.

Default value: {flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}_NormLim.tif

fname_fmt.ia_product

File format pattern for IA cos(IA), sin(IA) and tan(IA) files

Default value: {IA_kind}_{flying_unit_code}_{tile_name}_{orbit}.tif

fname_fmt.lia_product

File format pattern for LIA and sin(LIA) files

Default value: {LIA_kind}_{flying_unit_code}_{tile_name}_{orbit}.tif

fname_fmt.filtered

File format pattern for filtered files

Default value: {flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}_filtered.tif for β°, σ° and γ° calibrations,

Default value: {flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}_NormLim_filtered.tif when NORMLIM calibrated.

dname_fmt.*

Set of directory format templates that permits to override the default directories where products are generated.

The directory formats can only be overridden for final products.

The only fields available are:

Field

Reference to

{tile_name}

Name of the related tile.

{out_dir}

[PATHS].output

{tmp_dir}

[PATHS].tmp

{ia_dir}

[PATHS].ia

{lia_dir}

[PATHS].lia

Products from

Option dname_fmt.?

Default value

(β°/σ°/γ°/NORMLIM) Final tiled product

.tiled

{out_dir}/{tile_name}

Masks

.mask

{out_dir}/{tile_name}

degree(IA), cos(IA), sin(IA) and tan(IA)

.ia_product

{ia_dir}

degree(LIA) and sin(LIA)

.lia_product

{lia_dir}

Filtering

.filtered

{out_dir}/filtered/{tile_name}

creation_options.*

Set of extra options to create certain products. Creation options take a first and optional pixel type (uint8, float64…) and a list of GDAL creation options.

Products from

Option creation_options.?

Default value

Orthorectification, (β°/σ°/γ°/NORMLIM) Concatenation…

.tiled

COMPRESS=DEFLATE&gdal:co:PREDICTOR=3

Filtering

.filtered

COMPRESS=DEFLATE&gdal:co:PREDICTOR=3

Masks

.mask

uint8 COMPRESS=DEFLATE

IA (in degrees * 100)

.ia_deg

uint16 COMPRESS=DEFLATE&gdal

cos(IA)

.ia_cos

COMPRESS=DEFLATE&gdal:co:PREDICTOR=3

sin(IA)

.ia_sin

COMPRESS=DEFLATE&gdal:co:PREDICTOR=3

tan(IA)

.ia_tan

COMPRESS=DEFLATE&gdal:co:PREDICTOR=3

LIA (in degrees * 100)

.lia_deg

uint16 COMPRESS=DEFLATE&gdal

sin(LIA)

.lia_sin

COMPRESS=DEFLATE&gdal:co:PREDICTOR=3

[Filtering] section

Note

Multitemporal filtering is not yet integrated in S1Tiling.

Option

Description

filter

The following spatial speckling filter methods from OTB Despeckle application are supported: Lee, Frost, Gammamap, Kuan. If none or empty, then no filtering is done.

window_radius

Sets the window radius for the spatial filtering.
Be cautious: this does expect a radius, i.e. radius=1 means the filter does an 3x3 pixels averaging.

deramp

Deramp factor – for Frost filter only.
Factor use to control the exponential function used to weight effect of the distance between the central pixel and its neighborhood. Increasing the deramp parameter will lead to take more into account pixels farther from the center and therefore increase the smoothing effects.

nblooks

Number of looks – for all but Frost => Lee, Gammamap and Kuan

keep_non_filtered_products

If not caring for non-filtered product (and if filter method is specified), then the orthorectified and concatenated products won’t be considered as mandatory and they will not be kept at the end of the processing. This (exclusion) feature cannot be used alongside [Mask].generate_border_mask (i.e. keep_non_filtered_products cannot be False if generate_border_mask is True)

Warning

Note: This feature is only supported after LIA calibration as of V1.0 of S1Tiling. See Issue #118.

[Metadata] section

You can place in this section any extra key : value information that you want written in the GeoTIFF metadata of S1Tiling products.

Log configuration

Default logging configuration is provided in S1Tiling installing directory.

It can be overridden by dropping a file similar to ../s1tiling/logging.conf.yaml in the same directory as the one where the request configuration file is. The file is expected to follow logging configuration file syntax.

Warning

This software expects the specification of:

When mode contains logging, we make sure that file and important handlers are added to the handlers of root and distributed.worker loggers. Note that this is the default configuration.

When mode contains debug the DEBUG logging level is forced into root logger, and $OTB_LOGGER_LEVEL environment variable is set to DEBUG.

Working on clusters

Todo

By default, S1Tiling works on single machines. Internally it relies on distributed.LocalCluster a small adaptation would be required to work on a multi-nodes cluster.

Warning

When executing multiple instances of S1Tiling simultaneously, make sure to use different directories for:

  • logs – running S1Tiling in different directories, like $TMPDIR/ on TREX, should be enough

  • storing input files, like for instance $TMPDIR/data_raw/ on TREX for instance.