Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DPE-4628] Add image with spark-rapids support #93

Merged
merged 28 commits into from
Jul 4, 2024
Merged

Conversation

welpaolo
Copy link
Collaborator

No description provided.

Copy link
Member

@theoctober19th theoctober19th left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really great work! (the number of Spark configurations required for GPU are 🤯) I have left some trivial comments below.

.github/workflows/integration-gpu.yaml Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These files being repeated looks a bit off to my eyes. Is there a way we could reuse these files that already exist from the spark base image directory?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 I agree with Bikalpa, I would not copy and paste, seems repetition we'd better avoid. Maybe you can also symlink the folder in the worst case

images/charmed-spark-gpu/conf/spark-defaults.conf Outdated Show resolved Hide resolved
tests/integration/integration-tests-gpu.sh Outdated Show resolved Hide resolved
tests/integration/integration-tests-gpu.sh Outdated Show resolved Hide resolved
tests/integration/integration-tests-gpu.sh Outdated Show resolved Hide resolved
tests/integration/resources/test-gpu-sql.py Outdated Show resolved Hide resolved
.github/workflows/integration-gpu.yaml Show resolved Hide resolved
Copy link
Contributor

@deusebio deusebio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good, but in general I would like to reduce the copy and paste parts. Why not just keeping a delta dockerfile only, such that we use the main rockcraft.yaml (that already has all the files and layout) and then use the Dockerfile to only custom two things:

  • GPU
  • Java versions to be used (JDK8)

Are there other things that we need in order to have a dedicated rockcraft.yaml?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 I agree with Bikalpa, I would not copy and paste, seems repetition we'd better avoid. Maybe you can also symlink the folder in the worst case

images/charmed-spark-gpu/rockcraft.yaml Outdated Show resolved Hide resolved
images/charmed-spark-gpu/rockcraft.yaml Show resolved Hide resolved
ARG BASE_IMAGE=staged-charmed-spark:latest
FROM $BASE_IMAGE
# Provide Default Entrypoint for Pebble
ENTRYPOINT [ "/bin/pebble", "enter", "--verbose", "--args", "sparkd" ]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor I believe that now rockcraft should have the primary service concept, and we could drop the entrypoint override

images/charmed-spark-gpu/rockcraft.yaml Show resolved Hide resolved
Copy link
Contributor

@deusebio deusebio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to proceed and address the re-use of base image in a subsequent PR once we figure out the JDK issue

Copy link
Member

@theoctober19th theoctober19th left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's proceed with merging this and then start working on the new java version.

@welpaolo welpaolo merged commit be00442 into 3.4-22.04/edge Jul 4, 2024
14 checks passed
@welpaolo welpaolo deleted the dpe-4628 branch July 4, 2024 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants