eolearn.core.utils.fs
A module implementing utilities for working with different filesystems
- eolearn.core.utils.fs.get_filesystem(path, create=False, config=None, **kwargs)[source]
A utility function for initializing any type of filesystem object with PyFilesystem2 package.
- Parameters:
path (str | Path) – A filesystem path
create (bool) – If the filesystem path doesn’t exist this flag indicates to either create it or raise an error
config (SHConfig | None) – A configuration object with AWS credentials
kwargs (Any) – Any keyword arguments to be passed forward
- Returns:
A filesystem object
- Return type:
FS
- eolearn.core.utils.fs.get_base_filesystem_and_path(*path_parts, **kwargs)[source]
Parses multiple strings that define a filesystem path and returns a filesystem object with a relative path on the filesystem.
- Parameters:
path_parts (str) – One or more strings defining a filesystem path
kwargs (Any) – Parameters passed to get_filesystem function
- Returns:
A filesystem object and a relative path
- Return type:
tuple[fs.base.FS, str]
- eolearn.core.utils.fs.load_s3_filesystem(path, strict=False, config=None, aws_profile=None, **kwargs)[source]
Loads AWS s3 filesystem from a path.
- Parameters:
path (str) – A path to a folder on s3 bucket that will be the base folder in this filesystem
strict (bool) – If True the filesystem will be making additional checks to the s3. Default is False.
config (SHConfig | None) – A configuration object with AWS credentials. By default, is set to None and in this case the default configuration will be taken.
aws_profile (str | None) – A name of AWS profile. If given, AWS credentials will be taken from there.
kwargs (Any) – Additional keyword arguments that will be used to initialize fs_s3fs.S3FS object, e.g. acl, upload_args, download_args, etc.
- Returns:
A S3 filesystem object
- Return type:
S3FS
- eolearn.core.utils.fs.get_aws_credentials(aws_profile, config=None)[source]
Collects credentials from AWS profile and adds them to an instance of SHConfig.
- Parameters:
aws_profile (str) – A name of AWS profile
config (SHConfig | None) – If existing config object is given credentials will be added to its copy, otherwise a new config object will be created.
- Returns:
A config object with AWS credentials that have been loaded from AWS profile.
- Return type:
SHConfig
- eolearn.core.utils.fs.pickle_fs(filesystem)[source]
By default, a filesystem object cannot be pickled because it contains a thread lock and optionally some other unserializable objects. This function removes all unserializable objects before pickling the filesystem object.
- Parameters:
filesystem (FS) – A filesystem object to pickle.
- Returns:
A pickle of a filesystem object in bytes.
- Return type:
bytes
- eolearn.core.utils.fs.unpickle_fs(pickled_filesystem)[source]
An inverse function of pickle_fs function. After unpickling a filesystem object it adds back all unserializable objects that were previously removed by pickle_fs.
- Parameters:
pickled_filesystem (bytes) – Bytes that represent a pickled filesystem object.
- Returns:
Unpickled filesystem object.
- Return type:
FS
- eolearn.core.utils.fs.get_full_path(filesystem, relative_path)[source]
Given a filesystem object and a path, relative to the filesystem it provides a full path.
- Parameters:
filesystem (FS) –
relative_path (str) –
- Return type:
str
- eolearn.core.utils.fs.join_path(*path_parts)[source]
A utility function for joining a path that is either local or S3.
- Parameters:
path_parts (str) – Partial paths where the first part will be used to decide if it is an S3 path or a local path
- Returns:
Joined path that is also normalized and absolute.
- Return type:
str