index.qmd

## Agenda

- Introduction to Workflow Orchestration

. . .

- Solutions for Workflow Orchestration
  - StreamFlow
  - Flyte

. . .

- Infrastructure for Machine Learning Workflows
  - HPC vs AI/ML
  - File vs Object Storage


{{< include workflows.qmd >}}
{{< include streamflow.qmd >}}
{{< include flyte.qmd >}}


## Summary I {style="font-size: 90%;"}

::: {layout='[1,1]'}
::: n1
**StreamFlow**

::: {.callout-tip icon=false}
## Pros
- Common workflow language (CWL)
- Supports deployment on Slurm and Kubernetes
- No special infrastructure setup needed
:::
::: {.callout-important icon=false}
## Cons
- Input and output bindings are redundant and error-prone
- Monitoring and visualization tools are outdated and not user-friendly
:::
:::

::: n2
**Flyte**

::: {.callout-tip icon=false}
## Pros
- General-purpose workflow orchestrator leveraging Kubernetes
- Robust task interconnections with static typing
- Data pipeline driven by object storage
- Advanced monitoring and visualization features
:::
::: {.callout-important icon=false}
## Cons
- Requires migration of compute nodes from Slurm to Kubernetes
:::
:::
:::


{{< include hpc-ai.qmd >}}
{{< include storage.qmd >}}


## Summary II

- **HPC** and **AI/ML** workflows have different requirements to the infrastructure
- **Object Storage** is a scalable storage solution for unstructured data and provide fast access to large files
- **MinIO** is an open-source object storage solution that can be deployed on-premises


# Questions?