The Unity SDS Processing Service facilitates large-scale data processing for scientific workflows.
This repository contains high level information (such as documentation, change log, etc.) about the U-SPS software stack. The actual U-SPS code is contained within the following set of GitHub repositories:
- U-SPS Prototype: Terraform scripts to deploy the U-SPS cluster (either the HySDS or Airflow implementations)
- U-SPS Workflows: Examples of CWL workflows that can be executed on a U-SPS cluster
- U-SPS API: The API used to manage a U-SPS cluster
- U-SPS Register Job: Implementation of the WPS-T API with respect to the supported U-SPS back-ends
- Deployment and execution of scientific data processing algorithms via OGC WPS-T API.
- CWL standard for workflow encoding.
- API management for cluster resources.
- Docker-packaged applications interacting within a Kubernetes cluster.
- U-SPS supports HySDS and Apache Airflow implementations.
This guide provides a quick way to get started with our project. Please see our docs for a more comprehensive overview.
- Docker 20.10 or higher
- Kubernetes 1.20 or higher
- Terraform 0.14 or higher
- Follow the U-SPS Setup Guide.
- Initialize the Terraform scripts.
- Apply the Terraform configuration.
- Monitor the U-SPS status on Kubernetes.
- Check data processing results.
- Tutorial: Executing the L1B workflow
- Tutorial: Using Job Labels
- Tutorial: Testing the SPS Prewarm API
N/A
N/A
See our CHANGELOG.md for a history of our changes.
Visit our releases page for versioned releases.
Questions about our project? Please see our: FAQ
Interested in contributing to our project? Please see our: CONTRIBUTING.md
See our: LICENSE
Key points of contact are:
![@dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=64&v=4){kind=small}
