Support recursive schema compatibility validation for container types wrapping evolved structs

[kosiew]

Which issue does this PR close?

• Part of Support additive schema evolution for List<Struct> / nested container types in Parquet scans #20835

Rationale for this change

DataFusion currently implements additive schema-evolution compatibility for Struct columns using validate_struct_compatibility. However, this logic only applies when the column itself is a Struct. When a Struct is wrapped inside container types such as List, LargeList, FixedSizeList, or Map, the planner falls back to Arrow's can_cast_types.

This behavior treats the container as opaque and causes legitimate schema evolutions (for example, adding a nullable field to a struct) to be rejected if the struct is nested inside a container.

This PR introduces a recursive datatype compatibility validator that recognizes when container types wrap a Struct and applies the same additive schema-evolution semantics to the nested structure. This ensures consistent behavior between top-level structs and structs nested inside supported container types.

What changes are included in this PR?

Recursive compatibility validation

• Introduce validate_data_type_compatibility to recursively validate compatibility between source and target datatypes.

• Support recursive validation for:

  • Struct
  • List
  • LargeList
  • FixedSizeList
  • Map

• Add helper requires_recursive_compatibility_validation to determine when recursive validation should be applied.

Planner integration

• Update schema_rewriter.rs to use the new recursive compatibility validation during physical expression adaptation.
• Preserve existing Arrow casting behavior for datatypes that do not require recursive validation.

Runtime casting support for containers

• Extend nested column casting to support:

  • List
  • LargeList
  • FixedSizeList
  • Map

• Introduce reusable helper cast_container to unify container casting logic.

• Add specific casting helpers:

  • cast_list_column
  • cast_fixed_size_list_column
  • cast_map_column

Struct casting refactor

• Simplify cast_struct_column implementation by iterating over target fields and mapping source fields by name.
• Preserve struct null buffers while filling missing fields with null arrays when allowed.

Error message improvements

• Standardize error messages from "Cannot cast struct field" to "Cannot cast field" for consistency across nested contexts.

Tests

Added comprehensive tests covering:

• Recursive compatibility validation

  • List<Struct> with additive schema evolution
  • Map<_, Struct> nested struct compatibility
  • FixedSizeList size mismatch detection

• Nested casting behavior

  • Casting List<Struct> where the target struct adds a nullable field
  • Ensuring new fields are filled with null values
  • Numeric type promotion inside nested structs

• Planner integration

  • Expression rewrite behavior for List<Struct> compatibility
  • Failure scenarios for incompatible nested field casts

• Parquet integration tests

  • End-to-end schema evolution validation when reading List<Struct> from Parquet

Are these changes tested?

Yes.

The PR adds both unit tests and integration tests:

• Unit tests in nested_struct.rs validating recursive compatibility logic and container casting behavior.
• Planner-level tests in schema_rewriter.rs ensuring correct expression rewriting.
• Parquet integration tests verifying that schema evolution works correctly when reading nested container types such as List<Struct>.

These tests cover both compatible additive schema evolution and failure scenarios.

Are there any user-facing changes?

Yes, but they are improvements to existing functionality rather than breaking changes.

DataFusion will now correctly support additive schema evolution for structs nested inside container types such as:

• List<Struct>
• LargeList<Struct>
• FixedSizeList<Struct>
• Map<_, Struct>

Previously these cases could fail validation even when the schema change was valid.

No API changes are introduced.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.