Skip to content

[SDP] [SPARK-52576] Drop/recreate on full refresh and MV update #51280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
+11 −8
Open

[SDP] [SPARK-52576] Drop/recreate on full refresh and MV update #51280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1a00003
drop on full refresh
sryza Jun 25, 2025
60f7918
remove commented-out code
sryza Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,12 +178,13 @@ object DatasetManager extends Logging {
}

// Wipe the data if we need to
if ((isFullRefresh || !table.isStreamingTableOpt.get) && existingTableOpt.isDefined) {
context.spark.sql(s"TRUNCATE TABLE ${table.identifier.quotedString}")
val dropTable = (isFullRefresh || !table.isStreamingTableOpt.get) && existingTableOpt.isDefined
if (dropTable) {
catalog.dropTable(identifier)
}

// Alter the table if we need to
if (existingTableOpt.isDefined) {
if (existingTableOpt.isDefined && !dropTable) {
val existingSchema = existingTableOpt.get.schema()

val targetSchema = if (table.isStreamingTableOpt.get && !isFullRefresh) {
Expand All @@ -198,7 +199,7 @@ object DatasetManager extends Logging {
}

// Create the table if we need to
if (existingTableOpt.isEmpty) {
if (dropTable || existingTableOpt.isEmpty) {
catalog.createTable(
identifier,
new TableInfo.Builder()
Expand Down
10 changes: 6 additions & 4 deletions ...ipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/MaterializeTablesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,8 +446,9 @@ class MaterializeTablesSuite extends BaseCoreExecutionTest {

val table2 = catalog.loadTable(identifier)
assert(
table2.columns() sameElements CatalogV2Util
.structTypeToV2Columns(new StructType().add("y", IntegerType).add("x", BooleanType))
table2.columns().toSet == CatalogV2Util
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordering of columns does not appear to be deterministic (at least across different catalog implementations). Is that unexpected?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for a table, the column order matters. I think we should keep the test as it is and fix the issues we found.

Copy link
Contributor Author

@sryza sryza Jun 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we able to get some help with fixing this? What I'm observing is that, with Hive, when I create a table using the following:

      catalog.createTable(
        identifier,
        new TableInfo.Builder()
          .withProperties(mergedProperties.asJava)
          .withColumns(CatalogV2Util.structTypeToV2Columns(outputSchema))
          .withPartitions(partitioning.toArray)
          .build()
      )

and then later fetch the columns using

catalog.loadTable(identifier).columns()

The columns are returned in a different order than they appear in outputSchema.

This happens only with Hive, not the default catalog.

Copy link
Contributor

@szehon-ho szehon-ho Jun 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strange, i can take a look

I ran the test case in HiveDDLSuite a few times and cant reproduce it.

    val catalog = spark.sessionState.catalogManager.currentCatalog.asInstanceOf[TableCatalog]
    withTable("t1") {
      val identifier = Identifier.of(Array("default"), "t1")
      val outputSchema = new StructType()
        .add("a", IntegerType, true, "comment1")
        .add("b", IntegerType, true, "comment2")
        .add("c", IntegerType, true, "comment3")
        .add("d", IntegerType, true, "comment4")
      catalog.createTable(
        identifier,
        new TableInfo.Builder()
          .withProperties(Map.empty.asJava)
          .withColumns(CatalogV2Util.structTypeToV2Columns(outputSchema))
          .withPartitions(Array.empty)
          .build()
      )
      val cols = catalog.loadTable(identifier).columns()
      assert(cols.length == 4)
      assert(cols(0).name() == "a")
      assert(cols(1).name() == "b")
      assert(cols(2).name() == "c")
      assert(cols(3).name() == "d")
    }

Is it reproducible with this pr itself?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the issue is that, the hive catalog always puts partition columns at the end of the table schema. For example, a, b, c PARTITIONED BY b will end up with column order a, c, b.

Can we tweak this test to avoid this hive limitation?

Copy link
Contributor Author

@sryza sryza Jun 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it sounds like the order of the columns returned here is not defined in a catalog-agnostic way? Objections to using non-ordered comparison here in that case?

An alternative could be for MaterializeTablesSuite to have a column-order method that the upcoming Hive-specific subclass overrides. Though not sure the added complexity is worth it given that column ordering is tangential to the main behaviors we're trying to test in this case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice find. Does the suite run against hive/non-hive? For me, Im ok either way, maybe add a short comment in the test to explain.

.structTypeToV2Columns(new StructType().add("x", BooleanType).add("y", IntegerType))
.toSet
)
assert(table2.partitioning().toSeq == Seq(Expressions.identity("x")))

Expand All @@ -468,8 +469,9 @@ class MaterializeTablesSuite extends BaseCoreExecutionTest {

val table3 = catalog.loadTable(identifier)
assert(
table3.columns() sameElements CatalogV2Util
.structTypeToV2Columns(new StructType().add("y", IntegerType).add("x", BooleanType))
table3.columns().toSet == CatalogV2Util
.structTypeToV2Columns(new StructType().add("x", BooleanType).add("y", IntegerType))
.toSet
)
assert(table3.partitioning().toSeq == Seq(Expressions.identity("x")))
}
Expand Down