Self-host: How to use S3 protocol

[Obeyed]

Are there any guides available for using the S3 Protocol when self-hosting?

Originally posted in #495 (comment)

When I follow the steps written in the referenced issue, I get the SignatureDoesNotMatch error:

{
    "raw": "{\"metadata\":{},\"code\":\"SignatureDoesNotMatch\",\"httpStatusCode\":403,\"userStatusCode\":400}",
    "name": "Error",
    "message": "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
    "stack": "Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n    at Object.SignatureDoesNotMatch (/app/dist/internal/errors/codes.js:140:39)\n    at Object.<anonymous> (/app/dist/http/plugins/signature-v4.js:72:34)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
  }

Standard and Resumable uploads work.

At one point I was getting the following error:

Missing S3 Protocol Access Key ID or Secret Key Environment variables

But that seems to have been my mistake with restarting the containers properly, because I'm not seeing this anymore. Now I get the SignatureDoesNotMatch error.

Any pointers on what I could be doing wrong?

Is there anything more I can share that could help?

Storage config

These are the environment variables for the storage container:

STORAGE_BACKEND: s3

# Removing this seems to have no effect
# S3_PROTOCOL_PREFIX: /storage/v1
S3_PROTOCOL_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
S3_PROTOCOL_ACCESS_KEY_SECRET: ${AWS_SECRET_ACCESS_KEY}

AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}

AWS_DEFAULT_REGION: ${S3_REGION}
REGION: ${S3_REGION}

GLOBAL_S3_BUCKET: ${S3_GLOBAL_S3_BUCKET}

TENANT_ID: ${TENANT_ENVIRONMENT}
IS_MULTITENANT: "false"

TUS_URL_PATH: /upload/resumable
TUS_URL_EXPIRY_MS: 86400000
UPLOAD_SIGNED_URL_EXPIRATION_TIME: 86400000

Kong config

My kong config for the storage-v1 path is as follows:

Notice the /storage/v1 in "Forwarded: host=$(headers.host)/storage/v1;proto=http".

## Storage routes: the storage server manages its own auth
- name: storage-v1
  _comment: "Storage: /storage/v1/* -> http://storage:5000/*"
  connect_timeout: 60
  write_timeout: 3600
  read_timeout: 3600
  url: http://storage:5000/
  routes:
    - name: storage-v1-all
      strip_path: true
      paths:
        - /storage/v1/
      request_buffering: false
      response_buffering: false
  plugins:
    - name: cors
    - name: request-transformer
      config:
        add:
          headers:
            - "Forwarded: host=$(headers.host)/storage/v1;proto=http"

Python Boto3

Example of how I'm doing the upload from a client application with Python and boto3. Couldn't find anything about using Python in the docs, so lots of guessing and trying.

import os
import boto3
from botocore.config import Config

# kong gateway url
gateway_uri = os.getenv("GATEWAY_URI")

s3_client = boto3.client(
    "s3",
    region_name="eu-west-1",
    # assuming this requires the `/storage/v1' otherwise kong can't route it
    endpoint_url=f"{gateway_uri}/storage/v1/s3",
    aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
    # assuming this is how I'd set forcePathStyle as described in the docs
    config=Config(s3={"addressing_style": "path"}),
)

# then do an upload
s3_client.upload_file(
    Filename=file_path,
    Bucket=bucket_name,
    Key=remote_file_path,
)

[Obeyed]

#533