-
Notifications
You must be signed in to change notification settings - Fork 0
/
workflows.qmd
95 lines (61 loc) · 2.43 KB
/
workflows.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
## Motivated by [SPACE CoE](https://www.space-coe.eu/)
{.absolute top=-20 right=0 width=200}
::: {.fs34}
**Goal**: Astrophysical simulations for exascale computing\
**Challenge**: Analyze large-size simulation data (~ Petabytes)\
**Solution**: Efficient ML-based workflows using dimensionality reduction
:::
::: {layout='[1,2.4,0.7]'}
::: n1
**Input**
::: {.fs20}
::: {.callout-note icon=false}
## Simulation Data (~ Petabytes)
Time snapshots storing particle positions, velocities, and other properties, e.g. from OpenGADGET (IllustrisTNG)
:::
:::
:::
::: n2
**Workflow**
:::
::: n3
**Output**
::: {.fs20}
Spherinator & HiPSter
[space.h-its.org](https://space.h-its.org/)
:::
:::
:::
::: aside
K. L. Polsterer, B. Doser, A. Fehlner and S. Trujillo-Gomez [ADASS (2024)](https://arxiv.org/abs/2406.03810).
:::
<!-- {fig-align="center"} -->
## Why Workflow Orchestration?
::: {.incremental}
- **Reproducibility**: Ensure your experiments are reproducible by tracking the code, data, model, and environment.
- **Resource management**: Manage resources efficiently by running tasks in parallel and optimizing resource usage.
- **Scalability**: Scale your workflows to handle large amount of data and complex pipelines.
- **Monitoring**: Track the progress of workflows and monitor their performance and results.
- **Collaboration**: Enable collaboration between team members by sharing code, data, and results.
:::
## Requirements on Workflow Orchestration
- Define execution requirements (e.g. GPUs, CPUs, memory)
- Control runtime environment with containers
. . .
- Orchestration features
- Parallelization: Run independent tasks automatically in parallel
- Caching: Avoid recomputing successful tasks
- Nesting: Reuse workflows as tasks
- Looping: Repeat tasks based on conditions
- Scattering: Distribute data to multiple tasks
- Conditionals: Branching based on conditions
## Machine Learning Workflow Example
{fig-align="center"}
## Options to Generate a Workflow
- General purpose programming languages (e.g Python)
- Domain specific language (e.g. [Common Workflow Language](https://www.commonwl.org/))
- Graphical user interface (e.g. [ReactFlow](https://reactflow.dev/) @ [fal.ai](https://fal.ai/dashboard))
{fig-align="center"}