AVRO-4296: [python] Bound zero-byte collection elements per datum, not per collection

[iemejia]

What changes were proposed in this pull request?

Follow-up to #3861 (AVRO-4296). That change capped the number of zero-byte-encoded collection elements (null, a zero-length fixed, or a record whose fields are all zero-byte) that a decoder will allocate, since such elements consume no input and so cannot be bounded by the "bytes remaining" check.

However, the cap was enforced per collection: read_array/read_map each start counting from zero. Because an Avro container file carries its own schema, an attacker can declare a record with many array<null> fields, each block individually under the limit but jointly unbounded. A tiny payload therefore still drives a huge allocation:

• 16 array<null> fields of ~10M each → an ~80-byte record that raises MemoryError
• 8 array<null> fields → an ~40-byte record that burns tens of seconds of CPU while allocating ~80M None references

This PR makes the zero-byte-element budget cumulative across a single decoded datum rather than per collection:

• The DatumReader tracks self._zero_byte_items_read, reset at the start of each top-level read() — the boundary DataFileReader uses per record.
• _ensure_collection_available checks the running total for zero-byte elements.
• Positive-size elements are unchanged: they are naturally bounded per collection because decoding consumes input and the bytes-remaining check shrinks as the position advances. As a side benefit, nested collections such as array<array<null>> are now also bounded in aggregate instead of getting a fresh budget per inner array.

How was this patch tested?

• Added test_record_of_array_of_null_fields_cumulative_across_datum (a multi-field record rejected once its combined zero-byte count exceeds the cap) and test_record_of_array_of_null_fields_within_datum_limit_reads (a within-limit record still decodes, and the budget resets between datums).
• Existing TestDatumReaderCollectionSizeLimit and test_datafile suites pass.
• Manually reproduced the reported amplification at the default 10M limit: 8- and 16-field records are now rejected at the second field before allocating, while a single legitimate 9,999,999-element array<null> still reads.

An equivalent fix for the Java SDK is tracked separately.

[iemejia]

The Java counterpart is #3927 (AVRO-4241).

[Copilot]

Pull request overview

This PR strengthens the Python Avro DatumReader’s protection against amplification attacks by making the “zero-byte collection element” allocation cap cumulative across a single decoded datum (record), rather than resetting per collection field.

Changes:

• Track cumulative zero-byte collection elements read per top-level DatumReader.read() via self._zero_byte_items_read, and enforce the cap across the entire datum.
• Update _ensure_collection_available to apply the zero-byte cap using the reader’s running total instead of a per-collection count.
• Add unit tests covering multi-field record amplification and verifying the budget resets between separate datums.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
lang/py/avro/io.py	Makes zero-byte element limiting cumulative per decoded datum by adding reader state and enforcing it in _ensure_collection_available.
lang/py/avro/test/test_io.py	Adds tests for multi-field record scenarios to ensure cumulative enforcement and reset-per-datum behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.