Skip to content

[KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0#7061

Closed
pan3793 wants to merge 1 commit intoapache:masterfrom
pan3793:6920-followup
Closed

[KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0#7061
pan3793 wants to merge 1 commit intoapache:masterfrom
pan3793:6920-followup

Conversation

@pan3793
Copy link
Member

@pan3793 pan3793 commented May 15, 2025

Why are the changes needed?

There were some breaking changes after we fixed compatibility for Spark 4.0.0 RC1 in #6920, but now Spark has reached 4.0.0 RC6, which has less chance to receive more breaking changes.

How was this patch tested?

Changes are extracted from #6928, which passed CI with Spark 4.0.0 RC6

Was this patch authored or co-authored using generative AI tooling?

No.

@pan3793 pan3793 requested review from cfmcgrady and cxzl25 May 15, 2025 08:02
@pan3793 pan3793 self-assigned this May 15, 2025
@pan3793 pan3793 added this to the v1.9.4 milestone May 15, 2025
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 0% with 168 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (2ec8b46) to head (17a1bd9).

Files with missing lines Patch % Lines
...kyuubi/spark/connector/hive/HiveTableCatalog.scala 0.00% 121 Missing ⚠️
...uubi/spark/connector/hive/HiveConnectorUtils.scala 0.00% 33 Missing ⚠️
...g/apache/spark/sql/kyuubi/SparkDatasetHelper.scala 0.00% 8 Missing ⚠️
...g/apache/spark/sql/execution/SparkPlanHelper.scala 0.00% 5 Missing ⚠️
...rk/sql/execution/arrow/KyuubiArrowConverters.scala 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master   #7061    +/-   ##
=======================================
  Coverage    0.00%   0.00%            
=======================================
  Files         696     697     +1     
  Lines       43032   43185   +153     
  Branches     5849    5851     +2     
=======================================
- Misses      43032   43185   +153     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

HiveTable(sparkSession, catalog.getTableMetadata(ident.asTableIdentifier), this)
}

// scalastyle:off
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to disable the style check for this method?

Copy link
Member Author

@pan3793 pan3793 May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because the arg list is too long

@pan3793 pan3793 requested a review from cfmcgrady May 16, 2025 02:32
@pan3793 pan3793 closed this in e366b09 May 16, 2025
pan3793 added a commit that referenced this pull request May 16, 2025
### Why are the changes needed?

There were some breaking changes after we fixed compatibility for Spark 4.0.0 RC1 in #6920, but now Spark has reached 4.0.0 RC6, which has less chance to receive more breaking changes.

### How was this patch tested?

Changes are extracted from #6928, which passed CI with Spark 4.0.0 RC6

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7061 from pan3793/6920-followup.

Closes #6920

17a1bd9 [Cheng Pan] [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit e366b09)
Signed-off-by: Cheng Pan <chengpan@apache.org>
@pan3793 pan3793 modified the milestones: v1.9.4, v1.10.2 May 16, 2025
@pan3793
Copy link
Member Author

pan3793 commented May 16, 2025

Merged to master/1.10

@pan3793 pan3793 deleted the 6920-followup branch May 16, 2025 06:56
pan3793 added a commit that referenced this pull request May 20, 2025
There were some breaking changes after we fixed compatibility for Spark 4.0.0 RC1 in #6920, but now Spark has reached 4.0.0 RC6, which has less chance to receive more breaking changes.

Changes are extracted from #6928, which passed CI with Spark 4.0.0 RC6

No.

Closes #7061 from pan3793/6920-followup.

Closes #6920

17a1bd9 [Cheng Pan] [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants