-
Notifications
You must be signed in to change notification settings - Fork 28.6k
[SPARK-52159][SQL] Properly handle table existence check for jdbc dialects #50835
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
I will create SPARK jira ticket shortly |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@milastdbx Could you fix the test failure, please. I think it is related to your changes:
[info] JDBCTableCatalogSuite:
[info] - show tables (61 milliseconds)
[info] - drop a table and test whether the table exists *** FAILED *** (86 milliseconds)
[info] "[SCHEMA]_NOT_FOUND" did not equal "[TABLE_OR_VIEW]_NOT_FOUND" (SparkFunSuite.scala:346)
[info] Analysis:
[info] "[SCHEMA]_NOT_FOUND" -> "[TABLE_OR_VIEW]_NOT_FOUND"
Also add the [SQL]
tag to PR's title.
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to add comments with the relevant error code links in the override function like here
It would be nice to have the links but they can become outdated, unfortunately, and this is out of our control. +1, LGTM. Merging to master. |
What changes were proposed in this pull request?
In this PR, I propose that we rethrow exception when we are doing table existence check in jdbc dialect, if exception is not related to table/schema not being found. I propose this because currently all exceptions get swallowed and method returns false. From the perspective of the system its as if table doesn't exist, which is a wrong message (e.g, we can get table does not exist even if it was network failure).
This issue is mostly exposed when TableCatalog API is used,
Error code links:
How its implemented today, tableExist cannot throw anything, so every exception gets converted to noSuchTableError which is wrong.
Does this PR introduce any user-facing change?
Customers will get proper error messages.
How was this patch tested?
Tests
Was this patch authored or co-authored using generative AI tooling?
Generetad by: COPILOT