Dockstore provides a place for users to share tools encapsulated in Docker and described with the Common
Workflow Language (CWL) or WDL (Workflow Description Language). This enables scientists to share analytical
workflows so that they are machine readable as well as runnable in a variety of environments. While the
Dockstore is focused on serving researchers in the biosciences, the combination of Docker + CWL/WDL can be used by
anyone to describe the tools and services in their Docker images in a standardized, machine-readable way.
We hope to use this project as motivation to create a GA4GH API standard for container registries.
For the live site see dockstore.org
This repo contains the web service and CLI components for Dockstore as well as collecting documentation and the issues for the project as a whole. The usage of this is to enumerate the docker containers (from quay.io and hopefully docker hub) and the workflows (from github/bitbucket) that are available to users of Dockstore.org.
For the related web UI see the dockstore-ui project.
The following section is useful for users of Dockstore (e.g. those that want to browse, register, and launch tools).
After registering at dockstore.org, you will be able to download the Dockstore CLI at https://dockstore.org/onboarding
The CLI has the following dependencies
- Java 8 (Update 101 or newer)
- cwltool (to run CWL workflows locally)
To install CWL tool:
pip install --user cwl-runner cwltool==1.0.20170828135420 schema-salad==2.6.20170806163416 avro==1.8.1 ruamel.yaml==0.14.12 requests==2.18.4
You may need other pip installable tools like typing
or setuptools
. This depends on your python environment.
A basic Dockstore configuration file is available/should be created in ~/.dockstore/config
and contains the following
at minimum:
token = <your generated by the dockstore site>
server-url = https://www.dockstore.org:8443
- Ensure that your Java 8 version is newer than update 101.
This keeps track of breaking changes when migrating from Dockstore 1.1 to beta releases of 1.2 on the client side.
- Paths for input files are standardized to paths like
s3://test.bucket/test
,icgc://1234-efg
,https://file.org/test.txt
. This means paths likeicgc:1234-efg
will no longer work - A new version of cwltool
- The syntax for launching tools has been simplified.
--local-entry
is no longer a flag, but is an alternative to--entry
.
Unfortunately, new unique indexes enforcing better data consistency require a clean-up of unpublished workflows. Published content should remain unaffected.
delete from workflow_workflowversion ww where ww.workflowid in (select id from workflow where ispublished='f');
delete from workflow where ispublished='f';
By default, cwltool reads input files from the local filesystem. Dockstore also adds support for additional file systems such as http, https, and ftp. Through a plug-in system, Dockstore also supports the Amazon S3, Synapse, and ICGC Storage Client via plugins.
Download the above set of default plugins via:
dockstore plugin download
Configuration for plugins can be placed inside the Dockstore configuration file in the following format
token = <your generated by the dockstore site>
server-url = https://www.dockstore.org:8443
# options below this are optional
use-cache = false #set this to true to cache input files for rapid development
cache-dir = /home/<user>/.dockstore/cache #set this to determine where input files are cached (should be the same filesystem as your tool working directories)
[dockstore-file-synapse-plugin]
[dockstore-file-s3-plugin]
endpoint = #set this to point at a non AWS S3 endpoint
[dockstore-file-icgc-storage-client-plugin]
client = /media/large_volume/icgc-storage-client-1.0.23/bin/icgc-storage-client
Additional plugins can be created by taking one of the repos in plugins as a model and using pf4j as a reference. See additional documentation for more details.
The following section is useful for Dockstore developers (e.g. those that want to improve or fix the Dockstore web service and UI)
The dependency environment for Dockstore is described by our Travis-CI config. In addition to the dependencies for Dockstore users, note the setup instructions for postgres. Specifically, you will need to have postgres installed and setup with the database user specified in .travis.yml.
If you maven build in the root directory this will build not only the web service but the client tool:
mvn clean install
If you're running tests on Travis-CI (or otherwise have access to the confidential data bundle) Run them via:
mvn clean install -Ptravis-tests
If you're running in an environment that can run our slower tests, run them in addition to the confidential tests with:
mvn clean install -Pjenkins-tests
You can also run it on your local computer but will need to setup postgres separately.
- Fill in the template dockstore.yml and stash it somewhere outside the git repo (like ~/.dockstore)
- The dockstore.yml is mostly a standard Dropwizard configuration file. Refer to the linked document to setup httpClient and database.
- Start with
java -jar dockstore-webservice/target/dockstore-webservice-*.jar server ~/.dockstore/dockstore.yml
- If you need integration with GitHub.com, Quay.io. or Bitbucket for your work, you will need to follow the appropriate sections below and then fill out the corresponding fields in your dockstore.yml.
The Swagger UI is reachable while the Dockstore webservice is running. This allows you to explore available web resources.
codestyle.xml defines the coding style for Dockstore as an IntelliJ Code Style XML file that should be imported into IntelliJ IDE. We also have a matching checkstyle.xml that can be imported into other IDEs and is run during the build.
For users of Intellij or comparable IDEs, we also suggest loading the checkstyle.xml with a plugin in order to display warnings and errors while coding live rather than encountering them later when running a build.
The dockstore command line should be installed in a location in your path.
/dockstore-client/bin/dockstore
You then need to setup a ~/.dockstore/config
file with the following contents:
token: <dockstore_token_from_web_app>
server-url: http://www.dockstore.org:8080
If you are working with a custom-built or updated dockstore client you will need to update the jar in: ~/.dockstore/config/self-installs
.
Background:
- The tool-registry-schema are intended on allowing different tool registries to exchange and compare data
- Defined in swagger yaml
- We use the online swagger editor to generate a JAX-RS skeleton for implementation
- Unlike the above components, this is a server component rather than client component, thus we cannot use swagger-codegen (client-only for now?)
To regenerate the swagger client:
- Open up the yaml document for the specification in the editor.swagger.io
- Hit Generate Server and select JAX-RS
- Replace the appropriate classes in dockstore-webservice
- Unlike the client classes, we cannot separate quite as cleanly. Classes to watch out for are io.swagger.api.ToolsApi (includes DropWizard specific UnitOfWork annotations and a custom path) and io.swagger.api.impl.ToolsApiServiceImpl (includes our implementation).
- Customizations include,
@Path(DockstoreWebserviceApplication.GA4GH_API_PATH + <depends on api class>)
for Api classes,@UnitOfWork
added to resources, and@JsonNaming(PropertyNamingStrategy.KebabCaseStrategy.class)
added to model classes for GA4GH.
This is for pre-release versions that have not been released to production.
- Create a release tag and iterate pom file versions
mvn release:prepare
- Release from the tag into artifactory (you may need permissions)
mvn release:perform
- Merge to master if this is a stable release
git checkout master; git merge <your tag here>
Special note: If a test is failing during perform, but did not fail during prepare or Travis-CI builds, you may have a non-deterministic error. Skip tests during a release with mvn release:perform -Darguments="-DskipTests"
After the release to Artifactory, document the release on GitHub via the Releases page. Take a look at commits since the last release and closed pull requests for information on what goes into the changelog. Also attach the newly created Dockstore script and shaded client jar.
This is for release versions that have been released to production.
- Create a hotfix branch
mvn hf hotfix start <version>
- Iterate pom file versions to a SNAPSHOT if needed
mvn versions:set -DnewVersion=<version>-SNAPSHOT
- Prepare the release and the perform it (you may need permissions)
mvn release:prepare
andmvn release:perform
- Merge to master and develop
mvn hf hotfix finish
As with the unstable release, document the release and attach the new Dockstore script and shaded client jar.
Encrypted documents necessary for confidential testing are handled as indicated in the documents at Travis-CI for
files and environment variables.
A convenience script is provided as encrypt.sh which will compress confidential files, encrypt them, and then update an encrypted archive on GitHub. Confidential files should also be added to .gitignore to prevent accidental check-in. The unencrypted secrets.tar should be privately distributed among members of the team that need to work with confidential data.
To dump a new copy of the encrypted database from one that you have setup, use the following (or similar):
pg_dump --data-only --column-inserts webservice_test &> dockstore-integration-testing/src/test/resources/db_confidential_dump_full.sql
To add copyright headers to all files with IntelliJ
- Ensure the Copyright plugin is installed (Settings -> Plugins)
- Create a new copyright profile matching existing copyright header found on all files, name it Dockstore (Settings -> Copyright -> Copyright Profiles -> Add New)
- Set the default project copyright to Dockstore (Settings -> Copyright)
Additional documentation on developing Dockstore is available at legacy.md