Skip to content

[SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable - #57975

Open
holdenk wants to merge 2 commits into
apache:masterfrom
holdenk:SPARK-58627-mark-raise-error-throwable
Open

[SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable#57975
holdenk wants to merge 2 commits into
apache:masterfrom
holdenk:SPARK-58627-mark-raise-error-throwable

Conversation

@holdenk

@holdenk holdenk commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Corrects the Expression.throwable metadata on two expressions:

  1. Overrides throwable to true on RaiseError.
  2. Makes Sequence's existing override fall back to its children rather than discarding the inherited default.

Expression.throwable (added in SPARK-46707) is opt-in metadata that tells the optimizer an expression may raise a runtime error, so a predicate containing it must not be relocated to a position where it runs on rows the original plan would not have evaluated it on. RaiseError always throws when evaluated but never declared the flag, so it inherited children.exists(_.throwable) -- false for the usual case of a literal error class and parameters.

Sequence was the only expression in the tree overriding the flag, and it did so as stepOpt.isDefined, dropping the inherited children.exists(_.throwable) term entirely. A throwing child under a stepless sequence(...) therefore reported non-throwable, which would also have masked the new flag on RaiseError.

Why are the changes needed?

Without the flag, CombineFilters and PushPredicateThroughJoin treat a predicate containing raise_error (or assert_true, which is rewritten to If(cond, null, RaiseError(...))) as freely movable. Pushing such a predicate below a selective join, or merging it into a filter that would have removed the offending rows, can make a query fail at runtime that previously succeeded.

The join below matches no rows, so the predicate should never be evaluated. But the predicate has no column references, so references.subsetOf(left.outputSet) holds trivially and it is pushed onto the left side, where it fires on the first row of t1:

CREATE OR REPLACE TEMP VIEW t1 AS SELECT * FROM VALUES (1), (2), (3) AS t(a);
CREATE OR REPLACE TEMP VIEW t2 AS SELECT * FROM VALUES (4), (5), (6) AS t(b);
SELECT * FROM t1 JOIN t2 ON t1.a = t2.b WHERE raise_error('boom') IS NULL;
-- [USER_RAISED_EXCEPTION] boom SQLSTATE: P0001

With this change the query returns an empty result.

Does this PR introduce any user-facing change?

Yes, as a bug fix. A predicate containing raise_error or assert_true is no longer pushed through a join, pushed into a join condition, or combined with an adjacent filter, so the error is raised only on the rows the unoptimized plan would have evaluated it on. Queries that previously failed spuriously now succeed. The same now holds for a throwing expression nested under a stepless sequence(...).

How was this patch tested?

Added UTs:

  • MiscExpressionsSuite: asserts the flag on RaiseError
  • FilterPushdownSuite: a raise_error predicate is not pushed through a join and not combined with an adjacent filter, each paired with a non-throwing predicate of the same shape that still is.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5) human review and cleanup after by Holden

sfc-gh-hkarau and others added 2 commits August 7, 2026 18:51
### What changes were proposed in this pull request?

Corrects the `Expression.throwable` metadata on two expressions:

1. Overrides `throwable` to `true` on `RaiseError`.
2. Makes `Sequence`'s existing override fall back to its children rather than
   discarding the inherited default.

`Expression.throwable` (added in SPARK-46707) is opt-in metadata that tells the
optimizer an expression may raise a runtime error, so a predicate containing it
must not be relocated to a position where it runs on rows the original plan
would not have evaluated it on. `RaiseError` always throws when evaluated but
never declared the flag, so it inherited `children.exists(_.throwable)` -- false
for the usual case of a literal error class and parameters.

`Sequence` was the only expression in the tree overriding the flag, and it did so
as `stepOpt.isDefined`, dropping the inherited `children.exists(_.throwable)`
term entirely. A throwing child under a stepless `sequence(...)` therefore
reported non-throwable, which would also have masked the new flag on
`RaiseError`.

### Why are the changes needed?

Without the flag, `CombineFilters` and `PushPredicateThroughJoin` treat a
predicate containing `raise_error` (or `assert_true`, which is rewritten to
`If(cond, null, RaiseError(...))`) as freely movable. Pushing such a predicate
below a selective join, or merging it into a filter that would have removed the
offending rows, can make a query fail at runtime that previously succeeded.

The join below matches no rows, so the predicate should never be evaluated. But
the predicate has no column references, so `references.subsetOf(left.outputSet)`
holds trivially and it is pushed onto the left side, where it fires on the first
row of `t1`:

```sql
CREATE OR REPLACE TEMP VIEW t1 AS SELECT * FROM VALUES (1), (2), (3) AS t(a);
CREATE OR REPLACE TEMP VIEW t2 AS SELECT * FROM VALUES (4), (5), (6) AS t(b);
SELECT * FROM t1 JOIN t2 ON t1.a = t2.b WHERE raise_error('boom') IS NULL;
-- [USER_RAISED_EXCEPTION] boom SQLSTATE: P0001
```

With this change the query returns an empty result.

### Does this PR introduce _any_ user-facing change?

Yes, as a bug fix. A predicate containing `raise_error` or `assert_true` is no
longer pushed through a join, pushed into a join condition, or combined with an
adjacent filter, so the error is raised only on the rows the unoptimized plan
would have evaluated it on. Queries that previously failed spuriously now
succeed. The same now holds for a throwing expression nested under a stepless
`sequence(...)`.

### How was this patch tested?

Added UTs:
- `MiscExpressionsSuite`: asserts the flag on `RaiseError` and its propagation
  through `AssertTrue`'s replacement.
- `FilterPushdownSuite`: a `raise_error` predicate is not pushed through a join
  and not combined with an adjacent filter, each paired with a non-throwing
  predicate of the same shape that still is; plus a `raise_error` nested under a
  stepless `sequence(...)`, covering the `Sequence` override.
- `ColumnExpressionSuite`: end-to-end, the query above returns an empty result,
  with a control that still raises once the join does produce rows.

All five new tests fail without the source changes, the end-to-end one with
exactly the `[USER_RAISED_EXCEPTION] boom` shown above.

Also ran the full `catalyst` suite (375 suites, 10460 tests) -- all green -- and
the full `sql` suite. In `sql`, every failure is a pre-existing environment
defect in my local setup, where the RocksDB native library cannot be loaded
(`UnsatisfiedLinkError: librocksdbjni*.so: libstdc++.so.6`); this takes out the
state-store and stateful-streaming suites and leaks a SparkContext that aborts
unrelated suites sharing the forked JVM. Re-running those aborted suites in
isolation, they pass. Relevant to this change, all plan-shape-sensitive suites
are green: `SQLQueryTestSuite` (golden files), the eight `TPCDS*`/`TPCH*`
`PlanStability*` suites, `TPCDSQuerySuite`, `ExplainSuite`, `SubquerySuite` and
`DataFrameJoinSuite`. Relying on CI for the state-store coverage.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)
Co-authored-by: Holden Karau <holden@pigscanfly.ca>
@holdenk holdenk changed the title Spark 58627 mark raise error throwable [SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants