s1tiling.libs.otbpipeline.PipelineDescriptionSequence

class s1tiling.libs.otbpipeline.PipelineDescriptionSequence(cfg: Configuration, dryrun: bool, debug_caches: bool)

Bases: object

This class is the main entry point to describe pipelines.

Internally, it can be seen as a list of PipelineDescription objects.

New methods & Specialized methods

__init__	Constructor.
_build_dependencies	Runs the inputs through all pipeline descriptions to build the full list of intermediary and final products and what they require to be built.
_build_tasks_from_dependencies	Generates the actual list of tasks for dask.client.get().
_check_static_task_requirements	Check all tasks have their requirement fulfilled for being generated.
_prepare_inputs	Takes care of instanciating all FirstSteps with the registered FirstStepFactories.
generate_tasks	Generates the minimal list of tasks that can be passed to Dask
register_extra_parameters_for_input_factories	Registers extra parameters that will be passed to all the for FirstStep factories registered.
register_inputs	Registers a FirstStepFactory that will act as a source of FirstSteps.
register_pipeline	Registers a pipeline description from:

generate_tasks(do_watch_ram=False) → Tuple[Dict[str, Tuple | FirstStep], List[str], List[Outcome]]

Generates the minimal list of tasks that can be passed to Dask

Parameters:

do_watch_ram (bool) – Debug oriented parameter used to watch RAM usage.

Returns:

A tuple made of:

1. the dictionary of tasks

2. the list of expected final products

3. a list of observed errors (that could happen while instanciating FirstSteps

Todo:

Move into another dedicated class instead of PipelineDescriptionSequence

register_extra_parameters_for_input_factories(**extra) → None

Registers extra parameters that will be passed to all the for FirstStep factories registered.

e.g.

pipelines.register_extra_parameters_for_input_factories(
    dag=dag,
    s1_file_manager=s1_file_manager,
    dryrun=dryrun,
)

register_inputs(kind: str, first_steps_factory: FirstStepFactory) → None

Registers a FirstStepFactory that will act as a source of FirstSteps.

Parameters:

• kind (str) – FirstStep source name

• first_steps_factory (FirstStepFactory) – Hook that’ll build FirstSteps on the fly from the registered Configuration and the registered extra parameters.

Note

This will permit to extend the list of starting inputs without having to modify main source code.

register_pipeline(factory_steps: List[Type], *args, **kwargs) → PipelineDescription

Registers a pipeline description from:

Parameters:

• factory_steps (list(type)) – List of non-instanciated StepFactory classes.

• name (Optional[str]) – Optional name for the pipeline.

• product_required (bool) – Tells whether the pipeline product is expected as a final product – and not an intermediary product.

• is_name_incremental (bool) – Tells whether expected filename needs evaluations of each intermediary steps of whether it can be directly deduced from the last step.

Returns:

The PipelineDescription built.