mpu.aws

Convenience functions for AWS interactions.

class mpu.aws.ExistsStrategy(value)[source]

Bases: enum.Enum

Strategies what to do when a file already exists.

ABORT = 'abort'
RAISE = 'raise'
REPLACE = 'replace'
class mpu.aws.S3Path(bucket_name, key)

Bases: tuple

bucket_name

Alias for field number 0

key

Alias for field number 1

mpu.aws.list_files(bucket: str, prefix: str = '', profile_name: Optional[str] = None) → List[str][source]

List up to 1000 files in a bucket.

Parameters

  • bucket (str) –

  • prefix (str) –

  • profile_name (str, optional) – AWS profile

Returns

s3_paths

Return type

List[str]

mpu.aws.s3_download(source: str, destination: Optional[str] = None, exists_strategy: mpu.aws.ExistsStrategy = ExistsStrategy.RAISE, profile_name: Optional[str] = None) → Optional[str][source]

Copy a file from an S3 source to a local destination.

Parameters

  • source (str) – Path starting with s3://, e.g. ‘s3://bucket-name/key/foo.bar’

  • destination (str, optional) – If none is given, a temporary file is created

  • exists_strategy ({'raise', 'replace', 'abort'}) – What is done when the destination already exists? * ExistsStrategy.RAISE means a RuntimeError is raised, * ExistsStrategy.REPLACE means the local file is replaced, * ExistsStrategy.ABORT means the download is not done.

  • profile_name (str, optional) – AWS profile

Returns

download_path – Path of the downloaded file, if any was downloaded.

Return type

Optional[str]

Raises

botocore.exceptions.NoCredentialsError – Botocore is not able to find your credentials. Either specify profile_name or add the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. See https://boto3.readthedocs.io/en/latest/guide/configuration.html

mpu.aws.s3_read(source: str, profile_name: Optional[str] = None) → bytes[source]

Read a file from an S3 source.

Parameters

  • source (str) – Path starting with s3://, e.g. ‘s3://bucket-name/key/foo.bar’

  • profile_name (str, optional) – AWS profile

Returns

content

Return type

bytes

Raises

botocore.exceptions.NoCredentialsError – Botocore is not able to find your credentials. Either specify profile_name or add the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. See https://boto3.readthedocs.io/en/latest/guide/configuration.html

mpu.aws.s3_upload(source: str, destination: str, profile_name: Optional[str] = None) → None[source]

Copy a file from a local source to an S3 destination.

Parameters

  • source (str) –

  • destination (str) – Path starting with s3://, e.g. ‘s3://bucket-name/key/foo.bar’

  • profile_name (str, optional) – AWS profile