[SPARK-52153][SQL] Fix from_json and to_json with variant

[chenhao-db]

What changes were proposed in this pull request?

It fixes two minor issues with from_json(variant) and to_json(variant).

• from_json(variant) currently ignores any JSON options. This is inconsistent with from_json(nested type containing variant), which respects JSON options.
• to_json(variant), when the variant contains special floating-point values (Infinity, NaN), the output is currently not wrapped in quotes. This is inconsistent with to_json(nested type containing floating points).
  • For example, the result of to_json(named_struct('a', cast('NaN' as double))) is {"a":"NaN"}, while the result of to_json(to_variant_object(named_struct('a', cast('NaN' as double)))) is {"a":NaN}
  • Although {"a":NaN} can be parsed by from_json when the allowNonNumericNumbers option is true, it is still not a valid JSON according to the spec. to_json should produce valid JSON.

Why are the changes needed?

This makes variant-related JSON handling more consistent with non-variant JSON handling.

Does this PR introduce any user-facing change?

Yes, as stated above.

How was this patch tested?

Unit test.

Was this patch authored or co-authored using generative AI tooling?

No.

[chenhao-db]

@cloud-fan Could you help review? Thanks!