eolearn.core.utils.parallelize

Utilities for a basic Python parallelization that build on top of concurrent.futures module from Standard Python library.

eolearn.core.utils.parallelize.parallelize(function, *params, workers, multiprocess=True, **tqdm_kwargs)

Parallelizes the function on given parameters using the specified number of workers.

Parameters:

• function (Callable[[...], OutputType]) – A function to be parallelized.

• params (Iterable[Any]) – Sequences of parameters to be given to the function. It uses the same logic as Python map function.

• workers (int | None) – Maximum number of time the function will be executed in parallel.

• multiprocess (bool) – If True it will use concurrent.futures.ProcessPoolExecutor which will distribute workflow executions among multiple processors. If False it will use concurrent.futures.ThreadPoolExecutor which will distribute workflow among multiple threads. In case of workers=1 this parameter is ignored and workflows will be executed consecutively.

• tqdm_kwargs (Any) – Keyword arguments that will be propagated to tqdm progress bar.

Returns:

A list of function results.

Return type:

list[~OutputType]

eolearn.core.utils.parallelize.execute_with_mp_lock(function, *args, **kwargs)

A helper utility function that executes a given function with multiprocessing lock if the process is being executed in a multiprocessing mode.

Parameters:

• function (Callable[[...], OutputType]) – A function

• args (Any) – Function’s positional arguments

• kwargs (Any) – Function’s keyword arguments

Return type:

OutputType

eolearn.core.utils.parallelize.submit_and_monitor_execution(executor, function, *params, **tqdm_kwargs)

Performs the execution parallelization and monitors the process using a progress bar.

Parameters:

• executor (Executor) – An object that performs parallelization.

• function (Callable[[...], OutputType]) – A function to be parallelized.

• params (Iterable[Any]) – Each element in a sequence are parameters for a single call of function.

• tqdm_kwargs (Any) –

Returns:

A list of results in the same order as input parameters given by executor_params.

Return type:

list[~OutputType]

eolearn.core.utils.parallelize.join_futures(futures, **tqdm_kwargs)

Resolves futures, monitors progress, and returns a list of results.

Parameters:

• futures (list[concurrent.futures._base.Future]) – A list of futures to be joined. Note that this list will be reduced into an empty list as a side effect of this function. This way future objects will get cleared from memory already during the execution which will free some extra memory. But this can be achieved only if future objects aren’t kept in memory outside futures list.

• tqdm_kwargs (Any) – Keyword arguments that will be propagated to tqdm progress bar.

Returns:

A list of results in the order that corresponds with the order of the given input futures.

Return type:

list[Any]

eolearn.core.utils.parallelize.join_futures_iter(futures, update_interval=0.5, **tqdm_kwargs)

Resolves futures, monitors progress, and serves as an iterator over results.

Parameters:

• futures (list[concurrent.futures._base.Future]) – A list of futures to be joined. Note that this list will be reduced into an empty list as a side effect of this function. This way future objects will get cleared from memory already during the execution which will free some extra memory. But this can be achieved only if future objects aren’t kept in memory outside futures list.

• update_interval (float) – A number of seconds to wait between consecutive updates of a progress bar.

• tqdm_kwargs (Any) – Keyword arguments that will be propagated to tqdm progress bar.

Returns:

A generator that will be returning pairs (index, result) where index will define the position of future in the original list to which result belongs to.

Return type:

Generator[tuple[int, Any], None, None]