Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Early stages / PoC of Distributed Data Movement with Gobblin-on-Temporal #3789

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

phet
Copy link
Contributor

@phet phet commented Sep 27, 2023

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Here are some details about my PR, including screenshots (if applicable):

ATTENTION: This PR is currently only for illustration and sharing ideas. I plan to continue iterating and then prepare the commits individually for merge.

  1. Adjust Gobblin-Temporal configurability for consistency and abstraction
  2. Define Workload abstraction for Temporal workflows of unbounded size through sub-workflow nesting
  3. Add gobblin-temporal load generator for a single subsuming super-workflow with many activities nested beneath
  4. Implement early pieces of Distributed Data Movement (DDM) w/ Gobblin-on-Temporal WorkUnit evaluation

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

manual execution for both Load-Gen and DDM via Temporal-workers-on-YARN

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant