Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support in user tools for running qualification on Dataproc GKE #612

Merged
merged 7 commits into from
Oct 13, 2023
Merged

Add support in user tools for running qualification on Dataproc GKE #612

merged 7 commits into from
Oct 13, 2023

Conversation

parthosa
Copy link
Collaborator

@parthosa parthosa commented Oct 9, 2023 •
edited
Loading

Contributes to #253. This PR adds support for Dataproc on Google Kubernetes Engine (GKE) platform in user tools.

Setting up Spark CPU job on Dataproc GKE

  1. Dataproc on GKE Overview
  2. Running Spark Jobs on Dataproc on GKE

Usage

  1. spark_rapids_user_tools: Speed Up and Cost Savings
spark_rapids_user_tools dataproc-gke qualification --cpu_cluster <cpu-cluster-name>  --eventlogs <cpu eventlogs>
  1. ascii: Speed Up and Cost Savings
ascli qualification --platform dataproc-gke --cluster <cpu-cluster-name>  --eventlogs <cpu eventlogs>
  1. ascii: Speed Up Only
ascli qualification --platform dataproc-gke --eventlogs <cpu eventlogs>
  1. ascii: Error
ascli qualification  --cluster <cpu-cluster-name>  --eventlogs <cpu eventlogs>

Tests

Created a separate PR with unit tests for better reviewing process -

@parthosa parthosa added the user_tools Scope the wrapper module running CSP, QualX, and reports (python) label Oct 9, 2023
@parthosa parthosa self-assigned this Oct 9, 2023
@parthosa parthosa changed the title Add support for Dataproc GKE in user tools Add support in user tools to allow for running qualification for Dataproc GKE environment including pricing info Oct 9, 2023
@parthosa parthosa changed the title Add support in user tools to allow for running qualification for Dataproc GKE environment including pricing info Add support in user tools for running qualification on Dataproc GKE Oct 9, 2023
@parthosa parthosa mentioned this pull request Oct 9, 2023
Closed
@parthosa
Fix pylint
557f2c7 
Signed-off-by: Partho Sarthi <[email protected]>
@cindyyuanjiang
Copy link
Collaborator

Do you have some example of Dataproc GKE qualification tool command and output, or how this is tested?

@parthosa parthosa force-pushed the spark-rapids-tools-253 branch from 2809f6f to 1c6c2a2 Compare October 11, 2023 02:08
@parthosa
Update docs and pylints
565572b 
Signed-off-by: Partho Sarthi <[email protected]>
@parthosa parthosa force-pushed the spark-rapids-tools-253 branch from 1c6c2a2 to 565572b Compare October 11, 2023 02:20
@parthosa
Update docs
30abc6e 
Signed-off-by: Partho Sarthi <[email protected]>
@parthosa parthosa marked this pull request as ready for review October 11, 2023 03:01
@parthosa
Copy link
Collaborator Author

Do you have some example of Dataproc GKE qualification tool command and output, or how this is tested?

I added references to run job on Dataproc GKE and sample commands for spark_rapids_user_tools and ascii covering different cases in the PR description.

cindyyuanjiang
cindyyuanjiang previously approved these changes Oct 12, 2023
View reviewed changes
Copy link
Collaborator

@cindyyuanjiang cindyyuanjiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

nartal1
nartal1 previously approved these changes Oct 13, 2023
View reviewed changes
Copy link
Collaborator

@nartal1 nartal1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

cindyyuanjiang
cindyyuanjiang approved these changes Oct 13, 2023
View reviewed changes
Copy link
Collaborator

@cindyyuanjiang cindyyuanjiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @parthosa

@parthosa parthosa requested a review from nartal1 October 13, 2023 21:19
@parthosa parthosa merged commit 5fd7299 into NVIDIA:dev Oct 13, 2023
8 checks passed
@parthosa parthosa deleted the spark-rapids-tools-253 branch October 17, 2023 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nartal1 nartal1 nartal1 approved these changes

@cindyyuanjiang cindyyuanjiang cindyyuanjiang approved these changes

@mattahrens mattahrens Awaiting requested review from mattahrens

Assignees

@parthosa parthosa

Labels
user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@parthosa @cindyyuanjiang @nartal1