Skip to content

Latest commit

 

History

History
256 lines (186 loc) · 10.4 KB

README.md

File metadata and controls

256 lines (186 loc) · 10.4 KB
Raw

Thunderstore

codecov

Thunderstore is a mod database and API for downloading mods.

Setup guide for development

  • Copy .env.template to .env and modify as you see fit - If you have access to the python-packages submodule & it's cloned, make sure to set BUILD_INSTALL_EXTRAS to true. Any other value will not work. Changing this setting will require rebuilding the environment (e.g. with docker compose build) for it to take effect.
  • Run docker compose up
  • Run docker compose exec django python manage.py migrate in another terminal
  • Run docker compose exec django python manage.py shell and enter the following code:
from django.contrib.sites.models import Site
Site.objects.create(domain="thunderstore.localhost", name="Thunderstore")

Make sure to substitute localhost with what you use to connect to the site! In general, you should use thunderstore.localhost as the main domain to handle auth-scoping correctly (see SESSION_COOKIE_DOMAIN later on)

You will also need to navigate to the admin panel (/djangoadmin) and configure a mapping from a site to a community. You can create a superuser account with the createsuperuser Django management command (akin to how migrate was run) to gain access to the admin panel.

To connect a site to a community, you will need to:

  1. Make sure at least one Community object exists or create one (Risk of Rain 2 should be created automatically)
  2. Make sure at least one Site object exists or create one
  3. Make the site object's domain name attribute match what you use for connecting to your development environment
  4. Create a new Community Site object, linking the two together

Test data population

There's a script for populating the local database with test data. You can run it as follows:

docker compose exec django python manage.py create_test_data

Minio

In local development, minio is used for S3 compatible file storage. You can access it via http://localhost:9000/ with thunderstore:thunderstore credentials

REST API docs

The REST API swagger documentation can be viewed from /api/docs/.

At the current moment, the only relevant API is /api/v1/package/, which lists all the active mods in the database. A specific mod can also be fetched if necessary with the /api/v1/package/{uuid4}/ endpoint, where {uuid4} is replaced with the mod's uuid4 value.

Admin site

The admin site can be accessed from /djangoadmin/. To view the admin site, you need an admin account.

Assuming docker is being used, the admin account can be created as follows:

docker compose exec django python manage.py createsuperuser

Do note that if you're running on Windows, you will need to use winpty for running that command.

Environment variable configuration for production

General variables

  • DEBUG: Should be either set to false or not at all for production
  • SECRET_KEY: A long and random string, used to hash passwords and other data. Should remain secret, as is implied by the name.
  • ALLOWED_HOSTS: Comma separated list of hostnames this server can be connected with. For example beta.thunderstore.io
  • PRIMARY_HOST: The public name of the server, such as beta.thunderstore.io
  • PROTOCOL: The protocol which to use to build URLs to the server. Either https:// or http://.
  • REPOSITORY_MAX_PACKAGE_SIZE_MB: The maximum single package size
  • REPOSITORY_MAX_PACKAGE_TOTAL_SIZE_GB: The maximum total file size used by packages

Gunicorn

  • GUNICORN_WORKER_COUNT: Used to control how many workers gunicorn will spawn
  • GUNICORN_LOG_LEVEL: Used to control gunicorn's logging level

Django

  • SESSION_COOKIE_DOMAIN: If set, allows sessions to be shared within a domain and its subdomains. For example: thunderstore.io

For local testing, recommended values are:

  • SESSION_COOKIE_DOMAIN: thunderstore.localhost

Make sure also to have the Site objects point to thunderstore.localhost or some of its subdomains, such as test.thunderstore.localhost.

Social Auth

  • SOCIAL_AUTH_SANITIZE_REDIRECTS: Set to True if you want to restrict OAuth redirect domains.
  • SOCIAL_AUTH_ALLOWED_REDIRECT_HOSTS: List allowed OAuth redirect domains, used if SOCIAL_AUTH_SANITIZE_REDIRECTS is enabled.
  • SOCIAL_AUTH_INIT_HOST: The host used for social auth initializations and callbacks, regardless of which host the user is currently on. If not set, defaults to the same value as AUTH_EXCLUSIVE_HOST. If neither are set, defaults to the host of the request.
  • AUTH_EXCLUSIVE_HOST: A hostname/domain which will exclusively be used for auth related logic, such as the social auth process. If not set, no host is treated as the exclusive auth host.

For local testing, recommended values are:

  • AUTH_EXCLUSIVE_HOST: auth.thunderstore.localhost
  • SOCIAL_AUTH_SANITIZE_REDIRECTS: auth.thunderstore.localhost,thunderstore.localhost

GitHub OAuth

To set up GitHub OAuth, head to settings on GitHub (either personal or organization settings), and from under Developer Settings select OAuth Apps.

Create a new OAuth Application, and use {AUTH_EXCLUSIVE_HOST}/auth/complete/github/ as the Authorization callback URL, where {AUTH_EXCLUSIVE_HOST} is replaced with the value that was used for the AUTH_EXCLUSIVE_HOST setting. For example for local you could use http://auth.localhost/auth/complete/github/, whereas for a live environment https://auth.thunderstore.dev/auth/complete/github/

After creating the OAuth application, you must also provide the following environment variables to the application:

  • SOCIAL_AUTH_GITHUB_KEY: The Client ID value of the OAuth application
  • SOCIAL_AUTH_GITHUB_SECRET The Client Secret value of the OAuth application

Discord OAuth

To set up a Discord OAuth, head to the Discord developer panel, and create a new OAuth application. Add a callback URL to {AUTH_EXCLUSIVE_HOST}/auth/complete/discord/, where {AUTH_EXCLUSIVE_HOST} is replaced with the value that was used for the AUTH_EXCLUSIVE_HOST setting. For example for local you could use http://auth.localhost/auth/complete/discord/, whereas for a live environment https://auth.thunderstore.dev/auth/complete/discord/

  • SOCIAL_AUTH_DISCORD_KEY: The Client ID value of the OAuth application
  • SOCIAL_AUTH_DISCORD_SECRET The Client Secret value of the OAuth application

Storage

The AWS S3 / Boto3 protocol is supported by multiple vendors and services, and such the implementation may vary depending on the provider.

Refer to https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html for more details on the implementation. Also see thunderstore/core/settings.py for what environment variables are currently implemented.

At the very least set the following variables:

  • AWS_ACCESS_KEY_ID: Auth key ID
  • AWS_SECRET_ACCESS_KEY: Auth key secret
  • AWS_S3_REGION_NAME: Storage bucket region
  • AWS_S3_ENDPOINT_URL: Storage service endpoint
  • AWS_STORAGE_BUCKET_NAME: Bucket name
  • AWS_LOCATION: Location inside the bucket where to upload files
  • AWS_S3_SECURE_URLS: Set to false to disable HTTPS, enabled by default

Usermedia storage

The usermedia APIs work by leveraging S3-compatible storage presigned URLs to handle the actual upload. As such, the usermedia backend must also be a S3 compatible storage backend. Likewise, the usermedia storage backend can be configured with environment variables:

  • USERMEDIA_S3_ENDPOINT_URL: Internally accessible storage service endpoint
  • USERMEDIA_S3_ACCESS_KEY_ID: Auth key ID
  • USERMEDIA_S3_SECRET_ACCESS_KEY: Auth key secret
  • USERMEDIA_S3_SIGNING_ENDPOINT_URL: Publicly accessible storage service endpoint
  • USERMEDIA_S3_REGION_NAME: Storage bucket region
  • USERMEDIA_S3_STORAGE_BUCKET_NAME: Storage bucket name
  • USERMEDIA_S3_LOCATION: Location inside the bucket where to upload files

The largest difference compared to the AWS S3 configuration is the addition of a USERMEDIA_S3_SIGNING_ENDPOINT_URL. If provided, this will be used when generating pre-signed URLs. Can be used to bypass the CDN domain for example.

Database

Database configuration is pretty straight forward if using a local database where no SSL is required, but remote database via SSL connections is also supported.

  • DATABASE_URL: The database URL to use for a database connection
  • DB_CLIENT_CERT: Base64 encoded client certificate to use for the database connection. Will be placed to client-cert.pem
  • DB_CLIENT_KEY: Base64 encoded client key to use for the database connection. Will be placed to client-key.pem
  • DB_SERVER_CA: Base64 encoded server CA to use for the database connection. Will be placed to server-ca.pem

The default local database configured in docker-compose.yml can be accessed:

  • From shell: docker compose exec db psql -U django
  • From browser: navigate to localhost:8080/?pgsql=db&username=django and use password django

Redis caching

You can enable caching to the redis backend by supplying a redis URL

  • REDIS_URL: The redis database URL to use for caching, e.g. redis://some-host:6379/0

Testing

Tests can be run with this command: docker compose exec django pytest If you need to recreate to database, use the following: docker compose exec django pytest --create-db --migrations

The CI pipeline checks that new PRs don't lower the test coverage. Since this process is rather slow, you might want to check the coverage locally before submitting a PR.

  • To update coverage file, run docker compose exec django coverage run -m pytest
  • To see the coverage report, run docker compose exec django coverage report -m

Test duration estimates

The test run is split across multiple workers on the CI pipeline, and the split aims to balance test across all the available workers in equal amounts of time consumption.

To be able to do so accurately, the test duration database has to be up to date. As such it's a good idea to update the test duration database every now and then.

The test duration database can be updated by running the full test suite with the --store-durations flag. So a full command example would be

docker compose exec django pytest --store-durations