Skip to content

Commit

Permalink
Add orchestration requirements
Browse files Browse the repository at this point in the history
  • Loading branch information
BerndDoser committed Sep 18, 2024
1 parent 3e8876f commit d708edb
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 12 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
.quarto/
/.quarto/
index_files/
index.html
36 changes: 25 additions & 11 deletions index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,33 @@ format:
width: 1300
---

## General Requirements
## Why do we need a workflow orchestration

- To manage the flow of data and dependencies between tasks
- To manage the execution of tasks

## How we can define a workflow

- Using a general purpose programming languages (e.g Python)
- Using a domain specific language (e.g. CWL)
- Using a graphical user interface

## Orchestration Requirements

- **Reproducibility**: Ensure your experiments are reproducible by tracking the code, data, model, and environment used in each run.
- **Resource management**: Manage resources efficiently by running tasks in parallel and optimizing resource usage.
- **Scalability**: Scale your workflows to handle large datasets and complex pipelines.
- **Monitoring**: Track the progress of workflows and monitor their performance and results.
- **Collaboration**: Enable collaboration between team members by sharing code, data, and results.


<!--
- Easy deployment of the workflow infrastructure
- Flexible pipeline definition
- Monitoring status, execution times, and results
- Fast pipeline for big data (S3 / Filesystem)
- Integration of Containers
- Integration of HPC clusters (SLURM / Apptainers)
- Integration of HPC clusters (SLURM / Apptainers) -->


## Example Workflow
Expand All @@ -36,20 +55,15 @@ format:
::: {.column width="50%"}
### Flyte
- Kubernetes and S3 storage
+ Flexible workflow using python function decorators `@task` and `@workflow`
- Flexible workflow using python function decorators `@task` and `@workflow`
- Add HPC resources to Kubernetes

**Internal use**
:::

::: {.column width="50%"}
### StreamFlow
+ Common workflow language
+ No special environment needed
+ SLURM plugin available
+ Developed by Alpha UniTO (SPACE partner)

**Workflow Distribution**
- Common workflow language (CWL)
- No special environment needed
- SLURM plugin available
:::

::::
Expand Down

0 comments on commit d708edb

Please sign in to comment.