mpu.aws¶
Convenience functions for AWS interactions.
-
class
mpu.aws.
ExistsStrategy
[source]¶ Bases:
enum.Enum
Strategies what to do when a file already exists.
-
ABORT
= 'abort'¶
-
RAISE
= 'raise'¶
-
REPLACE
= 'replace'¶
-
-
class
mpu.aws.
S3Path
(bucket_name, key)¶ Bases:
tuple
-
property
bucket_name
¶ Alias for field number 0
-
property
key
¶ Alias for field number 1
-
property
-
mpu.aws.
list_files
(bucket, prefix='', profile_name=None)[source]¶ List up to 1000 files in a bucket.
- Parameters
bucket (str) –
profile_name (str, optional) – AWS profile
- Returns
s3_paths
- Return type
List[str]
-
mpu.aws.
s3_download
(source, destination=None, exists_strategy=<ExistsStrategy.RAISE: 'raise'>, profile_name=None)[source]¶ Copy a file from an S3 source to a local destination.
- Parameters
source (str) – Path starting with s3://, e.g. ‘s3://bucket-name/key/foo.bar’
destination (str, optional) – If none is given, a temporary file is created
exists_strategy ({'raise', 'replace', 'abort'}) – What is done when the destination already exists? * ExistsStrategy.RAISE means a RuntimeError is raised, * ExistsStrategy.REPLACE means the local file is replaced, * ExistsStrategy.ABORT means the download is not done.
profile_name (str, optional) – AWS profile
- Returns
download_path – Path of the downloaded file.
- Return type
str
- Raises
botocore.exceptions.NoCredentialsError – Botocore is not able to find your credentials. Either specify profile_name or add the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. See https://boto3.readthedocs.io/en/latest/guide/configuration.html
-
mpu.aws.
s3_read
(source, profile_name=None)[source]¶ Read a file from an S3 source.
- Parameters
source (str) – Path starting with s3://, e.g. ‘s3://bucket-name/key/foo.bar’
profile_name (str, optional) – AWS profile
- Returns
content
- Return type
bytes
- Raises
botocore.exceptions.NoCredentialsError – Botocore is not able to find your credentials. Either specify profile_name or add the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. See https://boto3.readthedocs.io/en/latest/guide/configuration.html