[SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable - #57975
Open
holdenk wants to merge 2 commits into
Open
[SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable#57975holdenk wants to merge 2 commits into
holdenk wants to merge 2 commits into
Conversation
### What changes were proposed in this pull request?
Corrects the `Expression.throwable` metadata on two expressions:
1. Overrides `throwable` to `true` on `RaiseError`.
2. Makes `Sequence`'s existing override fall back to its children rather than
discarding the inherited default.
`Expression.throwable` (added in SPARK-46707) is opt-in metadata that tells the
optimizer an expression may raise a runtime error, so a predicate containing it
must not be relocated to a position where it runs on rows the original plan
would not have evaluated it on. `RaiseError` always throws when evaluated but
never declared the flag, so it inherited `children.exists(_.throwable)` -- false
for the usual case of a literal error class and parameters.
`Sequence` was the only expression in the tree overriding the flag, and it did so
as `stepOpt.isDefined`, dropping the inherited `children.exists(_.throwable)`
term entirely. A throwing child under a stepless `sequence(...)` therefore
reported non-throwable, which would also have masked the new flag on
`RaiseError`.
### Why are the changes needed?
Without the flag, `CombineFilters` and `PushPredicateThroughJoin` treat a
predicate containing `raise_error` (or `assert_true`, which is rewritten to
`If(cond, null, RaiseError(...))`) as freely movable. Pushing such a predicate
below a selective join, or merging it into a filter that would have removed the
offending rows, can make a query fail at runtime that previously succeeded.
The join below matches no rows, so the predicate should never be evaluated. But
the predicate has no column references, so `references.subsetOf(left.outputSet)`
holds trivially and it is pushed onto the left side, where it fires on the first
row of `t1`:
```sql
CREATE OR REPLACE TEMP VIEW t1 AS SELECT * FROM VALUES (1), (2), (3) AS t(a);
CREATE OR REPLACE TEMP VIEW t2 AS SELECT * FROM VALUES (4), (5), (6) AS t(b);
SELECT * FROM t1 JOIN t2 ON t1.a = t2.b WHERE raise_error('boom') IS NULL;
-- [USER_RAISED_EXCEPTION] boom SQLSTATE: P0001
```
With this change the query returns an empty result.
### Does this PR introduce _any_ user-facing change?
Yes, as a bug fix. A predicate containing `raise_error` or `assert_true` is no
longer pushed through a join, pushed into a join condition, or combined with an
adjacent filter, so the error is raised only on the rows the unoptimized plan
would have evaluated it on. Queries that previously failed spuriously now
succeed. The same now holds for a throwing expression nested under a stepless
`sequence(...)`.
### How was this patch tested?
Added UTs:
- `MiscExpressionsSuite`: asserts the flag on `RaiseError` and its propagation
through `AssertTrue`'s replacement.
- `FilterPushdownSuite`: a `raise_error` predicate is not pushed through a join
and not combined with an adjacent filter, each paired with a non-throwing
predicate of the same shape that still is; plus a `raise_error` nested under a
stepless `sequence(...)`, covering the `Sequence` override.
- `ColumnExpressionSuite`: end-to-end, the query above returns an empty result,
with a control that still raises once the join does produce rows.
All five new tests fail without the source changes, the end-to-end one with
exactly the `[USER_RAISED_EXCEPTION] boom` shown above.
Also ran the full `catalyst` suite (375 suites, 10460 tests) -- all green -- and
the full `sql` suite. In `sql`, every failure is a pre-existing environment
defect in my local setup, where the RocksDB native library cannot be loaded
(`UnsatisfiedLinkError: librocksdbjni*.so: libstdc++.so.6`); this takes out the
state-store and stateful-streaming suites and leaks a SparkContext that aborts
unrelated suites sharing the forked JVM. Re-running those aborted suites in
isolation, they pass. Relevant to this change, all plan-shape-sensitive suites
are green: `SQLQueryTestSuite` (golden files), the eight `TPCDS*`/`TPCH*`
`PlanStability*` suites, `TPCDSQuerySuite`, `ExplainSuite`, `SubquerySuite` and
`DataFrameJoinSuite`. Relying on CI for the state-store coverage.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Co-authored-by: Holden Karau <holden@pigscanfly.ca>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Corrects the
Expression.throwablemetadata on two expressions:throwabletotrueonRaiseError.Sequence's existing override fall back to its children rather than discarding the inherited default.Expression.throwable(added in SPARK-46707) is opt-in metadata that tells the optimizer an expression may raise a runtime error, so a predicate containing it must not be relocated to a position where it runs on rows the original plan would not have evaluated it on.RaiseErroralways throws when evaluated but never declared the flag, so it inheritedchildren.exists(_.throwable)-- false for the usual case of a literal error class and parameters.Sequencewas the only expression in the tree overriding the flag, and it did so asstepOpt.isDefined, dropping the inheritedchildren.exists(_.throwable)term entirely. A throwing child under a steplesssequence(...)therefore reported non-throwable, which would also have masked the new flag onRaiseError.Why are the changes needed?
Without the flag,
CombineFiltersandPushPredicateThroughJointreat a predicate containingraise_error(orassert_true, which is rewritten toIf(cond, null, RaiseError(...))) as freely movable. Pushing such a predicate below a selective join, or merging it into a filter that would have removed the offending rows, can make a query fail at runtime that previously succeeded.The join below matches no rows, so the predicate should never be evaluated. But the predicate has no column references, so
references.subsetOf(left.outputSet)holds trivially and it is pushed onto the left side, where it fires on the first row oft1:With this change the query returns an empty result.
Does this PR introduce any user-facing change?
Yes, as a bug fix. A predicate containing
raise_errororassert_trueis no longer pushed through a join, pushed into a join condition, or combined with an adjacent filter, so the error is raised only on the rows the unoptimized plan would have evaluated it on. Queries that previously failed spuriously now succeed. The same now holds for a throwing expression nested under a steplesssequence(...).How was this patch tested?
Added UTs:
MiscExpressionsSuite: asserts the flag onRaiseErrorFilterPushdownSuite: araise_errorpredicate is not pushed through a join and not combined with an adjacent filter, each paired with a non-throwing predicate of the same shape that still is.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5) human review and cleanup after by Holden