Object Storage API

This is a Python microservice created using FastAPI that provides a REST API to upload and download attachments and images to and from an S3 object store.

How to Run

This microservice requires a MongoDB and S3 object storage instance to run against.

Prerequisites

• Docker and Docker Compose installed (if you want to run the microservice inside Docker)
• Python 3.12, MongoDB 7.0 and MinIO installed on your machine (if you are not using Docker)
• Public key (must be OpenSSH encoded) to decode JWT access tokens (if JWT authentication/authorization is enabled)
• MongoDB Compass installed (if you want to interact with the database using a GUI)
• This repository cloned

Prerequisite Steps

1. Create a .env file alongside the .env.example file. Use the example file as a reference and modify the values accordingly.

   cp .env.example .env

2. Create a logging.ini file alongside the logging.example.ini file. Use the example file as a reference and modify it accordingly:

   cp logging.example.ini logging.ini

3. (Required only if JWT Auth is enabled) Inside the keys directory in the root of the project directory, create a copy of the public key generated by the authentication component. This is needed for decoding of JWT access tokens signed by the corresponding private key.

Inside of Docker

Ensure that Docker is installed and running on your machine before proceeding.

Using docker-compose.yml for local development

The easiest way to run the application with Docker for local development is using the docker-compose.yml file. It is configured to start

• A MongoDB instance that can be accessed at localhost:27018 using root as the username and example as the password
• A MinIO instance at localhost:9000 with a console that can be accessed at localhost:9001 using root as the username and example_password as the password
• The application in a reload mode which using the mounted object_storage_api directory means that FastAPI will watch for changes made to the code and automatically reload the application on the fly.

1. Build and start the Docker containers:

   docker compose up

   The microservice should now be running inside Docker at http://localhost:8002 and its Swagger UI could be accessed at http://localhost:8002/docs. A MongoDB instance should also be running at http://localhost:27018.

Using Dockerfile for local development

Use the Dockerfile's dev stage to run just the application itself in a container. Use this only for local development (not production)! Mounting the object_storage_api directory to the container via a volume means that FastAPI will watch for changes made to the code and automatically reload the application on the fly. The application requires a MongoDB and a MinIO instance to run. Instances of these can be started using the docker-compose.yml file.

1. Start a MongoDB and MinIO instance:

   docker compose up --detach mongo-db minio

2. Once the MongoDB and MinIO containers are running, run the container for creating the MinIO buckets:

   docker compose up minio-create-buckets

3. Build an image using the Dockerfile's dev stage from the root of the project directory:

   docker build --file Dockerfile --target dev --tag object-storage-api:dev .

4. Start the container using the image built and map it to port 8002 locally (please note that the public key volume is only needed if JWT Auth is enabled):

   docker run \
    --publish 8002:8000 \
    --name object-storage-api \
    --env-file ./.env \
    --volume ./object_storage_api:/app/object_storage_api \
    --volume ./keys/jwt-key.pub:/app/keys/jwt-key.pub \
    --volume ./logging.ini:/app/logging.ini \
    --add-host localhost:host-gateway \
    object-storage-api:dev

   The microservice should now be running inside Docker at http://localhost:8002 and its Swagger UI could be accessed at http://localhost:8002/docs.

Using Dockerfile for running all the tests

Use the Dockerfile's test stage to run all the tests in a container. Mounting the object_storage_api and test directories to the container via volumes means that any changes made to the application or test code will automatically be synced to the container next time you run the tests. The e2e tests require a MongoDB and a MinIO instance to run. Instances of these can be started using the docker-compose.yml file.

1. Start a MongoDB and a MinIO instance:

   docker compose up --detach mongo-db minio

2. Once the MongoDB and MinIO containers are running, run the container for creating the MinIO buckets:

   docker compose up minio-create-buckets

3. Build an image using the Dockerfile's test stage from the root of the project directory:

   docker build --file Dockerfile --target test --tag object-storage-api:test .

4. Run the tests using:

   docker run \
    --rm \
    --name object-storage-api-test \
    --add-host localhost:host-gateway \
    --volume ./object_storage_api:/app/object_storage_api \
    --volume ./test:/app/test \
    --volume ./logging.ini:/app/logging.ini \
    object-storage-api:test

Using Dockerfile for running the unit tests

Use the Dockerfile's test stage to run the unit tests in a container. Mounting the object_storage_api and test directories to the container via volumes means that any changes made to the application or test code will automatically be synced to the container next time you run the tests.

1. Build an image using the Dockerfile's test stage from the root of the project directory:

   docker build --file Dockerfile --target test --tag object-storage-api:test .

2. Run the tests using:

   docker run \
    --rm \
    --name object-storage-api-test \
    --volume ./object_storage_api:/app/object_storage_api \
    --volume ./test:/app/test \
    --volume ./logging.ini:/app/logging.ini \
    object-storage-api:test \
    pytest --config-file test/pytest.ini --cov object_storage_api --cov-report term-missing test/unit -v

Using Dockerfile for running the e2e tests

Use the Dockerfile's test stage to run the e2e tests in a container. Mounting the object_storage_api and test directories to the container via volumes means that any changes made to the application or test code will automatically be synced to the container next time you run the tests. These tests require a MongoDB and a MinIO instance to run. Instances of these can be started using the docker-compose.yml file.

1. Start a MongoDB and a MinIO instance:

   docker compose up --detach mongo-db minio

2. Once the MongoDB and MinIO containers are running, run the container for creating the MinIO buckets:

   docker compose up minio-create-buckets

3. Build an image using the Dockerfile's test stage from the root of the project directory:

   docker build --file Dockerfile --target test --tag object-storage-api:test .

4. Run the tests using:

   docker run \
    --rm \
    --name object-storage-api-test \
    --add-host localhost:host-gateway \
    --volume ./object_storage_api:/app/object_storage_api \
    --volume ./test:/app/test \
    --volume ./logging.ini:/app/logging.ini \
    object-storage-api:test \
    pytest --config-file test/pytest.ini test/e2e -v

Outside of Docker

Prerequisites

MongoDB

You must have access to a MongoDB instance with a database and be able to create the following indexes using either MongoDB Compass or a CLI e.g.

mongosh DATABASE_NAME --username USERNAME --password PASSWORD --authenticationDatabase=admin \
   --eval 'db.attachments.createIndex({ "entity_id": 1, "code": 1 }, { name: "attachments_file_name_uniqueness_index", unique: true })' \
   --eval 'db.images.createIndex({ "entity_id": 1, "code": 1 }, { name: "images_file_name_uniqueness_index", unique: true })'

These compound indexes ensure file names cannot be repeated within the same entity.

This needs to be done for both the development and testing databases. By default the .env.example and pyproject.toml use object-storage and test-object-storage as their names, ensure they are modified as required.

Object Storage

You must also have access to an S3 object store, such as MinIO, and have created buckets for development and testing. By default the .env.example and pyproject.toml use object-storage and test-object-storage as their names, ensure they are modified as required.

Running the api

Ensure that Python is installed on your machine before proceeding.

1. Create a Python virtual environment and activate it in the root of the project directory:

   python -m venv venv
   source venv/bin/activate

2. Install the required dependencies using pip:

   pip install .[dev]
   pip install -r requirements.txt

3. Start the application:

   fastapi dev object_storage_api/main.py --host 0.0.0.0 --port 8002

   The microservice should now be running locally at http://localhost:8002. The Swagger UI can be accessed at http://localhost:8002/docs.

Using mock data for testing [Optional]

Generating mock data

To populate the database and object storage with mock data for testing IMS first ensure both inventory-management-api and object-storage-api are running in docker and then run.

python ./scripts/dev_cli.py generate -c

This will clear the database and MinIO storage, fetch existing entities from the inventory-management-system-api and generate and add mock data for them.

You can also generate mock attachments just for specific entities by repeatedly using -e ENTITY_ID. This will ensure the entity has at least one attachment and image.

Notes

Application Configuration

The configuration for the application is handled using Pydantic Settings. It allows for loading config values from environment variables or the .env file. Please note that even when using the .env file, Pydantic will still read environment variables as well as the .env file, environment variables will always take priority over values loaded from the .env file.

Listed below are the environment variables supported by the application.

Environment Variable	Description	Mandatory	Default Value
API__TITLE	The title of the API which is added to the generated OpenAPI.	No	Object Storage Service API
API__DESCRIPTION	The description of the API which is added to the generated OpenAPI.	No	This is the API for the Object Storage Service
API__ROOT_PATH	(If using a proxy) The path prefix handled by a proxy that is not seen by the app.	No
API__ALLOWED_CORS_HEADERS	The list of headers that are allowed to be included in cross-origin requests.	Yes
API__ALLOWED_CORS_ORIGINS	The list of origins (domains) that are allowed to make cross-origin requests.	Yes
API__ALLOWED_CORS_METHODS	The list of methods that are allowed to be used to make cross-origin requests.	Yes
AUTHENTICATION__ENABLED	Whether JWT auth is enabled.	Yes
AUTHENTICATION__PUBLIC_KEY_PATH	The path to the public key to be used for decoding JWT access token signed by the corresponding private key.	If JWT auth enabled
AUTHENTICATION__JWT_ALGORITHM	The algorithm to use to decode the JWT access token.	If JWT auth enabled
DATABASE__PROTOCOL	The protocol component (i.e. mongodb) to use for the connection string for the MongoClient to connect to the database.	Yes
DATABASE__USERNAME	The database username to use for the connection string for the MongoClient to connect to the database.	Yes
DATABASE__PASSWORD	The database password to use for the connection string for the MongoClient to connect to the database.	Yes
DATABASE__HOST_AND_OPTIONS	The host (and optional port number) component as well specific options (if any) to use for the connection string for the MongoClient to connect to the database. The host component is the name or IP address of the host where the mongod instance is running, whereas the options are <name>=<value> pairs (i.e. ?authMechanism=SCRAM-SHA-256&authSource=admin) specific to the connection. For a replica set mongod instance(s), specify the hostname(s) and any options as listed in the replica set configuration - prod-mongodb-1:27017,prod-mongodb-2:27017,prod-mongodb-3:27017/?authMechanism=SCRAM-SHA-256&authSource=admin For a standalone mongod instance, specify the hostname and any options - prod-mongodb:27017/?authMechanism=SCRAM-SHA-256&authSource=admin	Yes
DATABASE__NAME	The name of the database to use for the MongoClient to connect to the database.	Yes
OBJECT_STORAGE__ENDPOINT_URL	The URL of the object storage S3 endpoint.	Yes
OBJECT_STORAGE__ACCESS_KEY	The access key to use to authenticate with the S3 object storage.	Yes
OBJECT_STORAGE__SECRET_ACCESS_KEY	The secret access key to use to authenticate with the S3 object storage.	Yes
OBJECT_STORAGE__BUCKET_NAME	The name of the S3 bucket to use for object storage.	Yes
OBJECT_STORAGE__PRESIGNED_URL_EXPIRY_SECONDS	The expiry time of presigned URLs.	Yes
ATTACHMENT__MAX_SIZE_BYTES	The maximum file size of an attachment given in bytes. Note: File systems use a factor of 1024 for GB, MB and KB instead of 1000, so here the former is expected despite them really being GiB, MiB and KiB.	Yes
ATTACHMENT__UPLOAD_LIMIT	The number of attachments that can be uploaded per entity. This is a soft limit which means that there may be cases where it may be surpassed but not by a huge margin.	Yes
IMAGE__ALLOWED_FILE_EXTENSIONS	The list of image file extensions that are allowed to be uploaded. They must be in a lowercase format.	Yes
IMAGE__THUMBNAIL_MAX_SIZE_PIXELS	The maximum width/height of generated image thumbnails. The actual width and height should maintain the original aspect ratio but neither the width nor height will exceed this value.	Yes
IMAGE__UPLOAD_LIMIT	The number of images that can be uploaded per entity. This is a soft limit which means that there may be cases where it may be surpassed but not by a huge margin.	Yes

JWT Authentication/Authorization

This microservice supports JWT authentication/authorization and this can be enabled or disabled by setting the AUTHENTICATION__ENABLED environment variable to True or False. When enabled, all the endpoints require a JWT access token to be supplied. This ensures that only authenticated and authorized users can access the resources. To decode the JWT access token, the application needs the public key that corresponding to the private key used for encoding the token. Once the JWT access token is decoded successfully, it checks that it has a username in the payload, and it has not expired. This means that any microservice can be used to generate JWT access tokens so long as it meets the above criteria. The LDAP-JWT Authentication Service is a microservice that provides user authentication against an LDAP server and returns a JWT access token.