Skip to content

thevickypedia/s3-downloader

Repository files navigation

Versions Supported

Python

Language Stats

Language count Code coverage

Repo Stats

GitHub GitHub repo size GitHub code size

Activity

GitHub Repo created GitHub commit activity GitHub last commit

Build Status

pypi-publish pages-build-deployment

S3 Download

Python module to download objects in an S3 bucket.

Installation

pip install s3-downloader

Usage

Download objects in parallel
import s3

if __name__ == '__main__':
    wrapper = s3.Downloader(bucket_name='BUCKET_NAME')
    wrapper.run_in_parallel(threads=10)  # Defaults to 5
Download objects in sequence
import s3

if __name__ == '__main__':
    wrapper = s3.Downloader(bucket_name='BUCKET_NAME')
    wrapper.run()

Mandatory arg

  • bucket_name - Name of the s3 bucket.

Optional kwargs

  • prefix - Prefix to filter the objects based on their path. Defaults to None
  • logger - Bring your own custom pre-configured logger. Defaults to on-screen logging.
  • download_dir - Name/path of the directory where the objects have to be stored. Defaults to bucket_name at current working directory.

  • region_name - AWS region name. Defaults to the env var AWS_DEFAULT_REGION
  • profile_name - AWS profile name. Defaults to the env var PROFILE_NAME
  • aws_access_key_id - AWS access key ID. Defaults to the env var AWS_ACCESS_KEY_ID
  • aws_secret_access_key - AWS secret access key. Defaults to the env var AWS_SECRET_ACCESS_KEY

AWS values are loaded from env vars or the default config at ~/.aws/config / ~/.aws/credentials

Coding Standards

Docstring format: Google
Styling conventions: PEP 8
Clean code with pre-commit hooks: flake8 and isort

Requirement

python -m pip install gitverse

Usage

gitverse-release reverse -f release_notes.rst -t 'Release Notes'

Linting

pre-commit will ensure linting, run pytest, generate runbook & release notes, and validate hyperlinks in ALL markdown files (including Wiki pages)

Requirement

pip install sphinx==5.1.1 pre-commit recommonmark

Usage

pre-commit run --all-files

Pypi Package

pypi-module

https://pypi.org/project/s3-downloader/

License & copyright

© Vignesh Rao

Licensed under the MIT License