abtools.s3: Backup data to S3

abtools.s3.compress_and_upload(data, compressed_file, s3_path, multipart_chunk_size_mb=500, method='gz', delete=False, access_key=None, secret_key=None)

Compresses data and uploads to S3.

S3 upload uses s3cmd, so you must either:

  1. Manually configure s3cmd prior to use (typically using s3cmd --configure).
  2. Configure s3cmd using s3.configure().
  3. Pass your access key and secret key to compress_and_upload, which will automatically configure s3cmd.
Parameters:
  • data –

    Can be one of three things:

    1. Path to a single file
    2. Path to a directory
    3. A list of one or more paths to files or directories
  • compressed_file (str) – Path to the compressed file. Required.
  • s3_path (str) –

    The S3 path, with the filename omitted. The S3 filename will be the basename of the compressed_file. For example:

    compress_and_upload(data='/path/to/data',
                    compressed_file='/path/to/compressed.tar.gz',
                    s3_path='s3://my_bucket/path/to/')

    will result in an uploaded S3 path of s3://my_bucket/path/to/compressed.tar.gz

  • method (str) – Compression method. Options are 'gz' (gzip) or 'bz2' (bzip2). Default is 'gz'.
  • delete (bool) – If True, the compressed_file will be deleted after upload to S3. Default is False.
  • access_key (str) – AWS access key.
  • secret_key (str) – AWS secret key.
abtools.s3.put(f, s3_path, multipart_chunk_size_mb=500, logger=None)

Uploads a single file to S3, using s3cmd.

Parameters:
  • f (str) – Path to a single file.
  • s3_path (str) –

    The S3 path, with the filename omitted. The S3 filename will be the basename of the f. For example:

    put(f='/path/to/myfile.tar.gz', s3_path='s3://my_bucket/path/to/')

    will result in an uploaded S3 path of s3://my_bucket/path/to/myfile.tar.gz
abtools.s3.compress(d, output, compress='gz', logger=None)

Creates a compressed/uncompressed tar file.

Parameters:
  • d –

    Can be one of three things:

    1. the path to a single file, as a string
    2. the path to a single directory, as a string
    3. an iterable of file or directory paths
  • output (str) – Output file path.
  • compress – Compression method. Options are 'gz' (gzip), 'bz2' (bzip2) and 'none' (uncompressed). Default is 'gz'.
abtools.s3.configure(access_key=None, secret_key=None, logger=None)

Configures s3cmd prior to first use.

If no arguments are provided, you will be prompted to enter the access key and secret key interactively.

Parameters:
  • access_key (str) – AWS access key
  • secret_key (str) – AWS secret key