Skip to content

Bump Spark 4.0.0#6928

Closed
pan3793 wants to merge 1 commit intoapache:masterfrom
pan3793:spark-4.0.0
Closed

Bump Spark 4.0.0#6928
pan3793 wants to merge 1 commit intoapache:masterfrom
pan3793:spark-4.0.0

Conversation

@pan3793
Copy link
Member

@pan3793 pan3793 commented Feb 19, 2025

Why are the changes needed?

Test Spark 4.0.0 RC1
https://lists.apache.org/thread/3sx86qhnmot1p519lloyprxv9h7nt2xh

How was this patch tested?

GHA.

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added kind:infra license, community building, project builds, asf infra related, etc. kind:build labels Feb 19, 2025
@codecov-commenter
Copy link

codecov-commenter commented Feb 19, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (873503e) to head (a910169).
Report is 13 commits behind head on master.

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #6928   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         697     697           
  Lines       43192   43192           
  Branches     5854    5854           
======================================
  Misses      43192   43192           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pan3793
Copy link
Member Author

pan3793 commented Feb 20, 2025

some failures are related to SPARK-29497, need further investigation.

@pan3793 pan3793 force-pushed the spark-4.0.0 branch 2 times, most recently from 8061d40 to 704d905 Compare March 3, 2025 06:19
pan3793 added a commit that referenced this pull request May 16, 2025
### Why are the changes needed?

There were some breaking changes after we fixed compatibility for Spark 4.0.0 RC1 in #6920, but now Spark has reached 4.0.0 RC6, which has less chance to receive more breaking changes.

### How was this patch tested?

Changes are extracted from #6928, which passed CI with Spark 4.0.0 RC6

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7061 from pan3793/6920-followup.

Closes #6920

17a1bd9 [Cheng Pan] [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793 added a commit that referenced this pull request May 16, 2025
### Why are the changes needed?

There were some breaking changes after we fixed compatibility for Spark 4.0.0 RC1 in #6920, but now Spark has reached 4.0.0 RC6, which has less chance to receive more breaking changes.

### How was this patch tested?

Changes are extracted from #6928, which passed CI with Spark 4.0.0 RC6

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7061 from pan3793/6920-followup.

Closes #6920

17a1bd9 [Cheng Pan] [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit e366b09)
Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793 added a commit that referenced this pull request May 20, 2025
There were some breaking changes after we fixed compatibility for Spark 4.0.0 RC1 in #6920, but now Spark has reached 4.0.0 RC6, which has less chance to receive more breaking changes.

Changes are extracted from #6928, which passed CI with Spark 4.0.0 RC6

No.

Closes #7061 from pan3793/6920-followup.

Closes #6920

17a1bd9 [Cheng Pan] [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
@pan3793 pan3793 marked this pull request as ready for review May 23, 2025 16:50
@pan3793 pan3793 requested a review from turboFei May 23, 2025 16:50
@pan3793 pan3793 self-assigned this May 23, 2025
@pan3793 pan3793 added this to the v1.11.0 milestone May 23, 2025
@pan3793
Copy link
Member Author

pan3793 commented May 23, 2025

Artifacts of Spark 4.0, both binary distribution tarball and jars, are ready now, I made this PR a simple version bump, and Spark 4.0 extension support will be submitted independently later.

@pan3793 pan3793 requested a review from cxzl25 May 23, 2025 16:53
@pan3793 pan3793 requested a review from wForget May 26, 2025 04:04
@pan3793 pan3793 closed this in 526110d May 26, 2025
@pan3793
Copy link
Member Author

pan3793 commented May 26, 2025

Merged to master

pan3793 added a commit that referenced this pull request Aug 4, 2025
### Why are the changes needed?

Test Spark 4.0.0 RC1
https://lists.apache.org/thread/3sx86qhnmot1p519lloyprxv9h7nt2xh

### How was this patch tested?

GHA.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6928 from pan3793/spark-4.0.0.

Closes #6928

a910169 [Cheng Pan] Bump Spark 4.0.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
@pan3793 pan3793 modified the milestones: v1.11.0, v1.10.3 Aug 4, 2025
@pan3793
Copy link
Member Author

pan3793 commented Aug 4, 2025

also ported to branch-1.10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:build kind:infra license, community building, project builds, asf infra related, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants