[SPARK-52449][CONNECT][PYTHON][ML] Make datatypes for Expression.Literal.Map/Array optional

[heyihong]

What changes were proposed in this pull request?

This PR makes data types optional for Expression.Literal.Array and Expression.Literal.Map in Spark Connect protocol buffers. The key changes include:

1. Type Inference Logic: Added logic to infer array element types and map key/value types from the actual data when types are not explicitly provided
2. Converter Updates: Updated conversion logics in both Scala and Python to handle optional data types

The implementation allows Spark Connect to infer types from the first element in arrays and first key-value pair in maps when data types are not explicitly specified, reducing the overhead of type specification while maintaining backward compatibility.

Why are the changes needed?

Currently, Spark Connect requires explicit data type specification for array and map literals, even when the types can be easily inferred from the contained elements. This creates unnecessary overhead in:

1. Performance: Redundant type information increases message size and processing time
2. Usability: Developers must explicitly specify types that could be automatically inferred

By making data types optional with type inference, we can improve both performance and developer experience while maintaining backward compatibility.

Does this PR introduce any user-facing change?

No - This PR does not introduce any user-facing changes.
The change is backward compatible and existing connect clients will continue to work unchanged.

How was this patch tested?

build/sbt "connect/testOnly *LiteralExpressionProtoConverterSuite"

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor 1.2.4

[heyihong]

@hvanhovell @HyukjinKwon @beliefer

[HyukjinKwon]

cc @zhengruifeng too

[zhengruifeng]

I am not sure whether it is worthwhile to just optimize out the element_type.
For large arrays of primitive types, e.g. large dense matrix for ML, we introduced SpecializedArray.

[zhengruifeng]

For a Array[Array[Int]] case, how to infer the nullability ?

[heyihong]

You mean the nullable field is missing in the array literal? I was thinking of deprecating element_type and introducing a new DataType.Array field so that each array literal includes the nullable field within DataType.Array, for example:

   message Array {
      DataType element_type = 1; [deprecated=true]
      repeated Literal elements = 2;
      DataType.Array data_type_array = 3;
  }

[heyihong]

@zhengruifeng This change optimizes out both arrays and maps, and also applies to non-primitive types. Also, the reduction in size of function_lit_array.json seems obvious.

[heyihong]

I created a separate ticket to track the Protobuf message change: https://issues.apache.org/jira/browse/SPARK-52930

[zhengruifeng]

cc @WeichenXu123 for the ml side

[heyihong]

friendly ping @hvanhovell @HyukjinKwon @beliefer @zhengruifeng @WeichenXu123

Update: No need to review at the moment — I need to finish SPARK-52930 first.