eolearn.core.eoexecution

The module handles execution and monitoring of workflows. It enables executing a workflow multiple times and in parallel. It monitors execution times and handles any error that might occur in the process. At the end it generates a report which contains summary of the workflow and process of execution.

All this is implemented in EOExecutor class.

class eolearn.core.eoexecution.EOExecutor(workflow, execution_kwargs, *, execution_names=None, save_logs=False, logs_folder='.', filesystem=None, logs_filter=None, logs_handler_factory=<class 'logging.FileHandler'>)[source]

Bases: object

Simultaneously executes a workflow with different input arguments. In the process it monitors execution and handles errors. It can also save logs and create a html report about each execution.

Parameters
  • workflow (eolearn.core.eoworkflow.EOWorkflow) – A prepared instance of EOWorkflow class

  • execution_kwargs (Sequence[Dict[eolearn.core.eonode.EONode, Dict[str, object]]]) – A list of dictionaries where each dictionary represents execution inputs for the workflow. EOExecutor will execute the workflow for each of the given dictionaries in the list. The content of such dictionary will be used as input_kwargs parameter in EOWorkflow.execution method. Check EOWorkflow.execution for definition of a dictionary structure.

  • execution_names (Optional[List[str]]) – A list of execution names, which will be shown in execution report

  • save_logs (bool) – Flag used to specify if execution log files should be saved locally on disk

  • logs_folder (str) – A folder where logs and execution report should be saved. If filesystem parameter is defined the folder path should be relative to the filesystem.

  • filesystem (Optional[fs.base.FS]) – A filesystem object for saving logs and a report.

  • logs_filter (Optional[logging.Filter]) – An instance of a custom filter object that will filter certain logs from being written into logs. It works only if save_logs parameter is set to True.

  • logs_handler_factory (Union[Callable[[str], logging.Handler], eolearn.core.eoexecution._HandlerWithFsFactoryType]) –

    A callable class or function that initializes an instance of a logging Handler object. Its signature should support one of the following options:

    • A single parameter describing a full path to the log file.

    • Parameters path and filesystem where path to the log file is relative to the given filesystem object.

    The 2nd option is chosen only if filesystem parameter exists in the signature.

REPORT_FILENAME = 'report.html'
STATS_START_TIME = 'start_time'
STATS_END_TIME = 'end_time'
run(workers=1, multiprocess=True, **tqdm_kwargs)[source]

Runs the executor with n workers.

Parameters
  • workers (Optional[int]) – Maximum number of workflows which will be executed in parallel. Default value is 1 which will execute workflows consecutively. If set to None the number of workers will be the number of processors of the system.

  • multiprocess (bool) – If True it will use concurrent.futures.ProcessPoolExecutor which will distribute workflow executions among multiple processors. If False it will use concurrent.futures.ThreadPoolExecutor which will distribute workflow among multiple threads. However, even when multiprocess=False, tasks from workflow could still be using multiple processors. This parameter is used especially because certain task cannot run with concurrent.futures.ProcessPoolExecutor. In case of workers=1 this parameter is ignored and workflows will be executed consecutively.

  • tqdm_kwargs (Any) – Keyword arguments that will be propagated to tqdm progress bar.

Returns

A list of EOWorkflow results

Return type

List[eolearn.core.eoworkflow.WorkflowResults]

get_successful_executions()[source]

Returns a list of IDs of successful executions. The IDs are integers from interval [0, len(execution_kwargs) - 1], sorted in increasing order.

Returns

List of successful execution IDs

Return type

List[int]

get_failed_executions()[source]

Returns a list of IDs of failed executions. The IDs are integers from interval [0, len(execution_kwargs) - 1], sorted in increasing order.

Returns

List of failed execution IDs

Return type

List[int]

get_report_path(full_path=True)[source]

Returns the filename and file path of the report.

Parameters

full_path (bool) – A flag to specify if it should return full absolute paths or paths relative to the filesystem object.

Returns

Report filename

Return type

str

make_report(include_logs=True)[source]

Makes a html report and saves it into the same folder where logs are stored.

Parameters

include_logs (bool) – If True log files will be loaded into the report file. If False they will be just referenced with a link to a log file. In case of a very large number of executions it is recommended that this parameter is set to False to avoid compiling a too large report file.

Return type

EOExecutorVisualization

get_log_paths(full_path=True)[source]

Returns a list of file paths containing logs.

Parameters

full_path (bool) – A flag to specify if it should return full absolute paths or paths relative to the filesystem object.

Returns

A list of paths to log files.

Return type

List[str]

read_logs()[source]

Loads the content of log files if logs have been saved.

Return type

List[Optional[str]]