Fix BigDiffy NPE on parquet schema mismatch in array fields

[monzalo14]

Problem

When running BigDiffy in parquet mode, parquet can produce null values for non-nullable array/map fields. This happens both when:

1. The reader schema has fields missing from the writer (schema evolution)
2. The parquet data itself contains nulls in non-nullable columns

In either case, Avro's GenericDatumWriter.getArraySize calls ((Collection) null).size() with no null guard, causing a NullPointerException.

--nullableCoders does not help -- that flag operates at the Scio coder boundary, not inside Avro record serialization.

Solution

After selecting the compatible schema in getCompatibleSchemaForFiles, wrap ALL non-nullable array and map fields in a union with null via makeCollectionFieldsNullable. The original type is placed first in the union to preserve existing default values (e.g., [] for arrays), and null is added second so the Avro coder can serialize null values without NPE. The transformation recurses into nested record fields.

Also adds an early return when both schemas are identical and no collection fields need fixing.

Test plan

• 3 unit tests for makeCollectionFieldsNullable in ParquetIOTest: arrays+maps, already-nullable fields, no-collection-fields
• 1 integration test in BigDiffyTest: writes parquet files with mismatched schemas, verifies the compatible schema wraps collection fields in nullable unions, and confirms AvroCoder can encode/decode records with null array fields
• Reproduced the exact NPE against the user's actual endpoints (InteractionFact.v2.sampled, partition 2026-07-20T10) using a local ratatool-internal docker image; verified the NPE is gone with this fix
• All 29 existing + new tests pass on both 2.12 and 2.13

[codecov]

Codecov Report

❌ Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.04%. Comparing base (630873d) to head (8a08982).

Files with missing lines	Patch %	Lines
...main/scala/com/spotify/ratatool/io/ParquetIO.scala	86.20%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #902       +/-   ##
===========================================
- Coverage   70.75%   42.04%   -28.72%     
===========================================
  Files          44       44               
  Lines        1939     1967       +28     
  Branches      335      331        -4     
===========================================
- Hits         1372      827      -545     
- Misses        567     1140      +573     

Flag	Coverage Δ
ratatoolCli	2.71% <0.00%> (-0.06%)	⬇️
ratatoolCommon	0.00% <ø> (ø)
ratatoolDiffy	32.61% <86.20%> (+0.93%)	⬆️
ratatoolExamples	16.41% <0.00%> (-0.26%)	⬇️
ratatoolSampling	?
ratatoolScalacheck	76.48% <ø> (ø)
ratatoolShapeless	3.89% <0.00%> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.