[SPARK-57927][SQL] Add json_valid function for JSON syntax validation

[marcuslin123]

What changes were proposed in this pull request?

Add a new built-in SQL function json_valid(jsonString) -> boolean that returns true if the input is a valid JSON string, false otherwise, and null for null input.

SELECT json_valid('{"a": 1}');        -- true
SELECT json_valid('[1, 2, 3]');       -- true
SELECT json_valid('invalid');         -- false
SELECT json_valid('{"a":1} garbage'); -- false (trailing content)
SELECT json_valid('');                -- false
SELECT json_valid(null);              -- null

The function is exposed across all surfaces: SQL, the Scala DataFrame API (json_valid(col)), and PySpark (F.json_valid(col)), including Spark Connect.

Implementation: Uses a streaming parser (JsonParser + skipChildren) via the existing shared JsonExpressionUtils pattern (same as json_array_length / json_object_keys), rather than materializing a parse tree. Validation confirms the input is exactly one complete JSON value with no trailing content; empty/whitespace-only input returns false.

On strict vs. lenient semantics: This implementation reuses Spark's existing SharedFactory.jsonFactory() (which enables ALLOW_SINGLE_QUOTES and ALLOW_UNESCAPED_CONTROL_CHARS for Hive compatibility), keeping json_valid consistent with from_json and get_json_object. This diverges from MySQL's strict JSON_VALID. Happy to switch to strict RFC 8259 semantics if reviewers prefer — flagging this as an open design decision per the JIRA.

Why are the changes needed?

Spark SQL has no built-in JSON syntax validation. Users currently resort to get_json_object(col, '$') IS NOT NULL, which is semantically different — it returns null for some malformed JSON without proper validation. Other databases provide this (MySQL 8.0+ JSON_VALID(), PostgreSQL). It is a common data-quality check in ETL pipelines and performance-critical for large-scale ingestion, which is why a built-in beats a UDF.

Does this PR introduce any user-facing change?

Yes. A new json_valid function is available in SQL, the Scala DataFrame API, and PySpark (classic and Connect).

How was this patch tested?

• Catalyst unit test in JsonExpressionsSuite covering null, valid values (objects, arrays, scalars), and invalid inputs (empty, whitespace, malformed, trailing content).
• End-to-end test in JsonFunctionsSuite exercising both the SQL string form and the Scala function.
• SQL golden-file cases in json-functions.sql (regenerated .sql.out).
• PySpark doctest.
• Spark Connect plan golden files regenerated.
• Expression schema golden file regenerated.

Was this patch authored or co-authored using generative AI tooling?

Generative AI tooling (Claude Code) was used as an assistive tool for implementation guidance.

[yadavay-amzn]

Nice, clean function PR, a couple of small notes

Thanks for this @marcuslin123. The wiring looks complete and follows the json_array_length / json_object_keys template closely, so interpreted eval and codegen both come for free through RuntimeReplaceable + StaticInvoke. I checked the null / scalar / trailing-content behavior against Jackson with the same SharedFactory settings and it lines up with the description and the golden files. A few small inline notes, none blocking.

On the strict-vs-lenient question you raised: I think reusing SharedFactory is the right call since it keeps json_valid consistent with get_json_object and the rest of the JSON family. The one thing I'd suggest is calling that leniency out in the docs so nobody's surprised that single-quoted input passes.

What I verified:

• null->null, ""/" "->false, scalars (123, "foo", true, null)->true, and trailing garbage->false, all consistent with the tests and the golden files.
• Number tokens are genuinely validated (123abc, 12.3.4, 01 all false), so it's not just a first-token check.
• Deeply nested JSON (>1000 levels) returns false via the Jackson stream-read limit, same as the other JSON functions.
• All surfaces are present: expression + registry + Scala/PySpark/Connect + all golden files. R is omitted, which matches json_array_length/json_object_keys, so that seems fine. @since 4.3.0 looks right.

[yadavay-amzn]

Since this reuses SharedFactory, it inherits ALLOW_SINGLE_QUOTES and ALLOW_UNESCAPED_CONTROL_CHARS, so something like json_valid("{'a':1}") returns true. That seems like a reasonable choice for consistency with get_json_object and the rest of the JSON family, but "valid JSON string" reads as strict RFC to most people. Might be worth a sentence here (and in the Scala/PySpark docstrings) noting it follows the same lenient parsing as the other JSON functions, so the single-quote / control-char behavior isn't a surprise.

[yadavay-amzn]

I think this null check might be unreachable, since the replacement builds the StaticInvoke with the default propagateNull = true, so a null input gets short-circuited to null before this method runs. lengthOfJsonArray and jsonObjectKeys rely on that and skip the check, so it could go for consistency with its neighbors. Not harmful either way.

[yadavay-amzn]

Since the single-quote leniency is the one intentional departure from strict JSON, it might be worth a case here that pins it, e.g. ("{'a':1}", true), so the decision doesn't silently regress later.