Skip to content

Commit

Permalink
improve motivation
Browse files Browse the repository at this point in the history
  • Loading branch information
BerndDoser committed Oct 12, 2024
1 parent 5e61188 commit 90d7362
Showing 1 changed file with 21 additions and 19 deletions.
40 changes: 21 additions & 19 deletions workflows.qmd
Original file line number Diff line number Diff line change
@@ -1,31 +1,19 @@
## Why Workflow Orchestration

::: {.incremental}

- **Reproducibility**: Ensure your experiments are reproducible by tracking the code, data, model, and environment used in each run.
- **Resource management**: Manage resources efficiently by running tasks in parallel and optimizing resource usage.
- **Scalability**: Scale your workflows to handle large datasets and complex pipelines.
- **Monitoring**: Track the progress of workflows and monitor their performance and results.
- **Collaboration**: Enable collaboration between team members by sharing code, data, and results.

:::


## Kick-off
## Motivation by [SPACE CoE](https://www.space-coe.eu/)

![](images/SPACE_logo.png){.absolute top=0 right=0 width=250}

Astrophysical Simulations for Exascale Computing ([SPACE CoE](https://www.space-coe.eu/))
**Goal**: Astrophysical Simulations for Exascale Computing\
**Challenge**: Analyze large-scale simulation data of Petabytes\
**Solution**: Efficient ML-based workflows using dimensionality reduction

::: {layout='[1,2.2,0.5]' layout-valign=center}
::: {layout='[1,2.2,0.7]' layout-valign=center}
::: n1
![](images/illustris.png)

::: {.small}
::: {.callout-note icon=false}
## Simulation Data (~ PetaBytes)
- Time snapshots storing particle positions, velocities, and other properties
- OpenGADGET (IllustrisTNG), PLUTO, RAMSES, ...
Time snapshots storing particle positions, velocities, and other properties, e.g. from OpenGADGET (IllustrisTNG), PLUTO, RAMSES, ...
:::
:::
:::
Expand All @@ -44,13 +32,27 @@ Spherinator & HiPSter
:::
:::
:::
:::{.font-size=30px}

::: aside
K. L. Polsterer, B. Doser, A. Fehlner and S. Trujillo-Gomez [ADASS (2024)]().
:::

<!-- ![](images/Space-partners-new.png){fig-align="center"} -->


## Why Workflow Orchestration?

::: {.incremental}

- **Reproducibility**: Ensure your experiments are reproducible by tracking the code, data, model, and environment used in each run.
- **Resource management**: Manage resources efficiently by running tasks in parallel and optimizing resource usage.
- **Scalability**: Scale your workflows to handle large datasets and complex pipelines.
- **Monitoring**: Track the progress of workflows and monitor their performance and results.
- **Collaboration**: Enable collaboration between team members by sharing code, data, and results.

:::


## Example Workflow

![](images/flyte-ui_mnist-workflow.png)
Expand Down

0 comments on commit 90d7362

Please sign in to comment.