eolearn.core.utils.fs

A module implementing utilities for working with different filesystems

eolearn.core.utils.fs.get_filesystem(path, create=False, config=None, **kwargs)[source]

A utility function for initializing any type of filesystem object with PyFilesystem2 package.

Parameters:
  • path (str | Path) – A filesystem path

  • create (bool) – If the filesystem path doesn’t exist this flag indicates to either create it or raise an error

  • config (SHConfig | None) – A configuration object with AWS credentials

  • kwargs (Any) – Any keyword arguments to be passed forward

Returns:

A filesystem object

Return type:

FS

eolearn.core.utils.fs.get_base_filesystem_and_path(*path_parts, **kwargs)[source]

Parses multiple strings that define a filesystem path and returns a filesystem object with a relative path on the filesystem.

Parameters:
  • path_parts (str) – One or more strings defining a filesystem path

  • kwargs (Any) – Parameters passed to get_filesystem function

Returns:

A filesystem object and a relative path

Return type:

tuple[fs.base.FS, str]

eolearn.core.utils.fs.load_s3_filesystem(path, strict=False, config=None, aws_profile=None, **kwargs)[source]

Loads AWS s3 filesystem from a path.

Parameters:
  • path (str) – A path to a folder on s3 bucket that will be the base folder in this filesystem

  • strict (bool) – If True the filesystem will be making additional checks to the s3. Default is False.

  • config (SHConfig | None) – A configuration object with AWS credentials. By default, is set to None and in this case the default configuration will be taken.

  • aws_profile (str | None) – A name of AWS profile. If given, AWS credentials will be taken from there.

  • kwargs (Any) – Additional keyword arguments that will be used to initialize fs_s3fs.S3FS object, e.g. acl, upload_args, download_args, etc.

Returns:

A S3 filesystem object

Return type:

S3FS

eolearn.core.utils.fs.get_aws_credentials(aws_profile, config=None)[source]

Collects credentials from AWS profile and adds them to an instance of SHConfig.

Parameters:
  • aws_profile (str) – A name of AWS profile

  • config (SHConfig | None) – If existing config object is given credentials will be added to its copy, otherwise a new config object will be created.

Returns:

A config object with AWS credentials that have been loaded from AWS profile.

Return type:

SHConfig

eolearn.core.utils.fs.pickle_fs(filesystem)[source]

By default, a filesystem object cannot be pickled because it contains a thread lock and optionally some other unserializable objects. This function removes all unserializable objects before pickling the filesystem object.

Parameters:

filesystem (FS) – A filesystem object to pickle.

Returns:

A pickle of a filesystem object in bytes.

Return type:

bytes

eolearn.core.utils.fs.unpickle_fs(pickled_filesystem)[source]

An inverse function of pickle_fs function. After unpickling a filesystem object it adds back all unserializable objects that were previously removed by pickle_fs.

Parameters:

pickled_filesystem (bytes) – Bytes that represent a pickled filesystem object.

Returns:

Unpickled filesystem object.

Return type:

FS

eolearn.core.utils.fs.get_full_path(filesystem, relative_path)[source]

Given a filesystem object and a path, relative to the filesystem it provides a full path.

Parameters:
  • filesystem (FS) –

  • relative_path (str) –

Return type:

str

eolearn.core.utils.fs.join_path(*path_parts)[source]

A utility function for joining a path that is either local or S3.

Parameters:

path_parts (str) – Partial paths where the first part will be used to decide if it is an S3 path or a local path

Returns:

Joined path that is also normalized and absolute.

Return type:

str

eolearn.core.utils.fs.split_all_extensions(path)[source]

Similar to os.path.splitext but takes into account double extensions such as .json.gz.

Parameters:

path (str) –

Return type:

tuple[str, str]

eolearn.core.utils.fs.is_s3_path(path)[source]

Returns True if the path points to a S3 bucket, False otherwise.

Parameters:

path (str) –

Return type:

bool