compose_setup

This project contains configuration template files needed to run Dockstore using AWS Fargate. View the dockstore-deploy repository for more information about how Dockstore is setup using AWS Fargate. Log issues and see general documentation at dockstore and docs.dockstore.org respectively

If you are looking for how to run Dockstore locally as a developer, you are probably in the wrong place and should take a look at https://github.com/dockstore/dockstore/blob/develop/docker-compose.yml

Prerequisities

1. Tested on Ubuntu 20.04
2. At least 20GB of disk space, 16GB of RAM, and 4 CPUs
3. Docker setup following https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/ including the post-installation steps for running without sudo
4. The running Dockstore website will require ports 80 and 443 by default
5. A client id and client secret for each of the integrations you wish to setup, github and quay.io as a minimum probably. You will need client ids and secrets for each integration as documented on the internal wiki.

Usage

1. Call the install_bootstrap script. This templates the contents of templates using mustache to the config directory while recording your answers for future use.

2. Some additional information on the answers requested in the script

   1. Each integration requires a client id and a secret, it is worth saying that you should not check these in
   2. The discourse URL is needed to link Dockstore to a discussion forum
   3. the Google verification code and tag manager ID are used if you want to properly track visitors to Dockstore and what pages they browse to

3. After following the instructions in the bootstrap script and starting up the site with AWS Fargate, you can browse to the Dockstore site hosted at port 443 by default using https://<domain-name>.

The current setup relies upon an externally hosted container orchestration service (current AWS ECS with Fargate), externally hosted database (currently AWS RDS) and externally hosted search (currently AWS Elasticsearch).

Loading Up a Database

Loading up a database is usually not necessary since AWS RDS is persistent. Refer to https://github.com/dockstore/dockstore-deploy#database-setup

Note that database migration is run once during the startup process and is controlled via the DATABASE_GENERATED variable. Answer yes if you are working as a developer and want to start work from scratch from an empty database. Answer no if you are working as an administrator and/or wish to start Dockstore from a production or staging copy of the database.

Logging Usage

If using with logstash in a container (for development), use -f docker-compose.yml -f docker-compose.dev.yml flags after each docker compose command to merge docker-compose files (e.g. docker compose -f docker-compse.yml -f docker-compose.dev.yml build)

For example to deploy just logging

docker compose  -f docker-compose.dev.yml build
nohup docker compose -f docker-compose.dev.yml up --force-recreate --remove-orphans >/dev/null 2>&1 &
docker compose -f docker-compose.dev.yml down
docker compose -f docker-compose.dev.yml kill

Kibana Dashboard Setup

Import the export.json Dashboard from compose_setup/export.json by going to Kibana's management => saved objects => import. See https://www.elastic.co/guide/en/kibana/current/managing-saved-objects.html for more info, especially the 2nd warning.

Installing git-secrets

Dockstore uses git-secrets to help make sure that keys and private data stay out of the source tree. To install and check for git secrets:

npm ci
npm run install-git secrets

This should install git secrets into your local repository and perform a scan. If secrets are found, the run will error and output the potential secret to stdout. If you believe the scan is a false-positive, add the line glob to .gitallowed.

Handy docker-compose commands:

1. `install_bootstrap --script` will template and build everything using your previous answers (useful for quick iteration) 
2. `docker compose down` will bring all containers down safely
3. `nohup docker compose up --force-recreate --remove-orphans >/dev/null 2>&1 &` will re-create all containers known to docker-compose and delete those volumes that no longer are associated with running containers
4. `docker system prune` for cleaning out old containers and images
5. To watch the logs `docker compose logs --follow` while debugging