[SPARK-58738][SQL] Add parse_sql function for SQL statement JSON analysis

[srielau]

What changes were proposed in this pull request?

Add a built-in scalar SQL function parse_sql(sqlStmt) that parse-only analyzes a SQL statement string (via SparkSqlParser, matching the session parser surface) and returns a compact JSON description:

• parse_success and, on failure, a nested STANDARD-format error object (errorClass, messageTemplate, sqlState, messageParameters, plus line / position when available). User-facing parse/scripting errors become JSON; unexpected internal failures propagate.
• Statement classification from ISO/IEC 9075-2:2023 Table 39 (statement_identifier, statement_code). Spark-only statements use product-specific identifiers with append-only negative codes. Unknown plans map to Unrecognized (code 0); known query shapes are allowlisted as SELECT.
• Multipart table_references and function_references for lineage (CTE names and correlation aliases omitted; tables inside CTE bodies included).
• select_list entries with multipart name only (no expression text).
• parameter_markers (named and unnamed_count).

The expression lives in sql/core (needs SparkSqlParser) and is registered when building session state.

Why are the changes needed?

Dialect-migration and SQL analysis tooling need an efficient, batchable way to verify that translated SQL parses and to extract statement kind / lineage references without requiring catalog objects to exist.

Does this PR introduce any user-facing change?

Yes. New SQL function (@since 5.0.0, master-only):

SELECT parse_sql('SELECT a, b FROM t');
-- {"parse_success":true,"statement_identifier":"SELECT","statement_code":21,
--  "table_references":[["t"]],"function_references":[],
--  "select_list":[{"name":["a"]},{"name":["b"]}],
--  "parameter_markers":{"named":[],"unnamed_count":0}}

SELECT get_json_object(parse_sql('SELEC'), '$.error.errorClass');
-- PARSE_SYNTAX_ERROR

How was this patch tested?

• ParseSqlResultSuite / ParseSqlSuite (sql/core)
• Golden file sql-tests/inputs/parse-sql.sql (+ analyzer results)
• ExpressionInfoSuite / ExpressionsSchemaSuite

build/sbt 'sql/testOnly *ParseSqlResultSuite *ParseSqlSuite'
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly *SQLQueryTestSuite -- -z parse-sql.sql'
build/sbt 'sql/testOnly *ExpressionsSchemaSuite *ExpressionInfoSuite'

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Grok 4.5

[srielau]

SQL Expert Review (SPARK-58738)

Domain review of the new parse_command surface (parser helpers + scalar function + goldens).

Summary

Severity	Count
Critical	2
High	4
Medium	2
Low	2
Total	10

Top actions

1. Parse with the same surface as production (SparkSqlParser / session parser).
2. Freeze the JSON contract deliberately (versioning + Expression.sql stability).
3. Do not map unknown non-commands to SELECT; sync the PR description.
4. Close CreateView / deep-walk gaps; clarify CTE vs table refs.
5. Fix @since to 4.4.0 (or document master-only).

Inline comments below cover each finding.

[srielau]

SQL Expert Re-Review (SPARK-58738 / 617940c)

Prior Criticals and most Highs from the first review look addressed (SparkSqlParser, rename to parse_sql, classification allowlist, CreateView/select_list, CTE lineage filter, private collectors, @since 5.0.0, PR description).

Verdict: Request changes — remaining Highs below should be fixed before merge.

Severity	Count
Critical	0
High	4
Medium	2
Low	2
Total	8

Top actions

1. Narrow the SparkThrowable catch so internals fail the task.
2. Extend isQueryPlan for TABLE / VALUES / time travel.
3. Stop blanket UnresolvedIdentifier lineage collection.
4. Add schema_version; document stock parser vs session extensions.

[cloud-fan]

1 blocking, 1 non-blocking, 1 nit.
The core design is coherent, but positional markers inside SQL scripts can be silently overcounted and should be fixed before merge.

Correctness (1)

• Blocking: sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala:183: Avoid re-traversing SingleStatement children so positional parameter markers are counted exactly once. -- see inline

Suggestions (1)

• Non-blocking: sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala:103: Combine table, function, and parameter collection after the required CTE-name pass to avoid four deep walks per row. -- see inline

Nits: 1 minor item (see inline comments).

Verification

I traced SingleStatement.children, QueryPlan.foreachWithSubqueries, the explicit parsedPlan recursion, and positional-marker accumulation. The first traversal visits the wrapped plan's children; the recursive traversal then visits those children again, while each PosParameter visit increments unnamedCount. I also verified the four independent deep-plan walks and the repository's ASCII-comment rule. I did not run tests.

PR metadata suggestions

• Update the successful JSON example: the implementation omits empty function_references and parameter_markers, but the PR body currently shows them as populated empty structures.

[cloud-fan]

Blocking:

This re-traverses children that SingleStatement.children already exposed to the outer foreachWithSubqueries walk. Named markers hide the duplicate visit in a set, but every PosParameter increments unnamedCount, so BEGIN SELECT * FROM t WHERE a = ?; END reports too many markers. Please visit the wrapped root without revisiting its children, and add a scripting test with ? below the root.

[cloud-fan]

Non-blocking:

Each input row currently performs four full deep-plan walks: two for tables, one for functions, and one here for parameters. Since this API is intended for batch evaluation, please keep the CTE-name pass and collect tables, functions, and parameters together in one subsequent walk while preserving first-seen order.

[cloud-fan]

Nit:

The repository guidance requires ASCII punctuation in code comments.

Suggested change

	// Table/view DDL targets only — not CreateFunction / CreateVariable names.
	// Table/view DDL targets only -- not CreateFunction / CreateVariable names.