Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DRAFT] Early stages / PoC of Distributed Data Movement with Gobblin-on-Temporal #3789

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[DRAFT] Early stages / PoC of Distributed Data Movement with Gobblin-on-Temporal #3789

wants to merge 12 commits into from

Commits on Sep 27, 2023

  1. Adjust Gobblin-Temporal configurability for consistency and abstraction

    @phet
    phet committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    2a0e307 View commit details
    Browse the repository at this point in the history

  2. Define Workload abstraction for Temporal workflows of unbounded siz… 

    …e through sub-workflow nesting
    @phet
    phet committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    c8c2c93 View commit details
    Browse the repository at this point in the history

  3. Add gobblin-temporal load generator for a single subsuming super-work… 

    …flow with many activities nested beneath
    @phet
    phet committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    313de5d View commit details
    Browse the repository at this point in the history

  4. Implement early pieces of Distributed Data Movement (DDM) w/ Gobblin-… 

    …on-Temporal `WorkUnit` evaluation
    @phet
    phet committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    d2258d1 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. Add JSON-ification to several foundational config-state representations

    @phet
    phet committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    16250aa View commit details
    Browse the repository at this point in the history

  2. Implement basic Distributed Data Movement (DDM) Gobblin-on-Temporal `… 

    …WorkUnit` evaluation
    @phet
    phet committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    51c3724 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. Extend WorkFulfillmentWorker deadlock detection timeout, to account… 

    … for blocking `FileSystem` calls
    @phet
    phet committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    3bc134e View commit details
    Browse the repository at this point in the history

  2. Add encapsulated convience method JobState.getJobIdFromProps

    @phet
    phet committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    e7ab48e View commit details
    Browse the repository at this point in the history

  3. Adjust work unit processing tuning for start-to-close timeout and nes… 

    …ted execution branching
    @phet
    phet committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    7cc1569 View commit details
    Browse the repository at this point in the history

  4. Rework ProcessWorkUnitImpl and fix FileSystem misuse

    @phet
    phet committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    212c176 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. Add convenience abstractions to load FileSystem, JobState, and `S… 

    …tateStore<TaskState>`
    @phet
    phet committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    350f8c8 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. Fix FileSystem resource lifecycle, uniquely name each workflow, and… 

    … drastically reduce worker concurrent task execution
    @phet
    phet committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    a8ead23 View commit details
    Browse the repository at this point in the history