fix: reject truncated BinaryRow serialized bytes instead of panicking#364
fix: reject truncated BinaryRow serialized bytes instead of panicking#364tonghuaroot wants to merge 1 commit into
Conversation
`BinaryRow::from_serialized_bytes` only validated the 4-byte arity prefix and then handed the remaining bytes straight to `from_bytes`. A buffer that carries a valid arity prefix but a body shorter than the null bitmap therefore decoded "successfully" and panicked later, when a reader called `is_null_at` and indexed the missing null-bitmap byte (e.g. via `format_partition_value`, predicate evaluation, or `get_datum`). Add a body-length check after reading the arity: reject inputs whose body is shorter than `cal_fix_part_size_in_bytes(arity)` (null bitmap + fixed part), and reject a negative arity. The required size is computed in i64 so an absurd arity in malformed input cannot overflow. As a second layer of defense, make `is_null_at` index the null bitmap with `get` so a short buffer reports not-null and the typed field readers return a graceful error rather than panicking. Add regression tests covering a truncated body, a too-short body, a negative arity, a short-buffer `is_null_at`, and a well-formed control.
| source: None, | ||
| }); | ||
| } | ||
| Ok(Self::from_bytes(arity, body.to_vec())) |
There was a problem hiding this comment.
One more hardening edge case: this validates the required body size with i64, but after the check passes from_bytes recomputes null_bits_size_in_bytes via cal_bit_set_width_in_bytes(arity), which still does the arithmetic in i32. For a corrupt row with an extremely large positive arity and a sufficiently large body, that unchecked helper can still overflow/panic. Could we either cap arity before this point or avoid recomputing the width with the i32 helper?
| // in i64 so an absurd arity in malformed input cannot overflow. | ||
| let bit_set_width = ((arity as i64 + 63 + Self::HEADER_SIZE_IN_BYTES as i64) / 64) * 8; | ||
| let fix_part_size = bit_set_width + 8 * arity as i64; | ||
| if (body.len() as i64) < fix_part_size { |
There was a problem hiding this comment.
Separate from the large-arity overflow case above, this also rejects a currently valid zero-arity roundtrip. BinaryRow::new(0).to_serialized_bytes() emits only the 4-byte arity prefix because data is empty, but this check requires an 8-byte body for arity == 0, so from_serialized_bytes no longer accepts the bytes produced by to_serialized_bytes.
Could we special-case arity == 0 to allow an empty body, or canonicalize zero-arity rows to serialize with the expected body size? Please also add a roundtrip test for BinaryRow::new(0).to_serialized_bytes().
What
BinaryRow::from_serialized_bytescurrently validates only the 4-byte arity prefix and then passes the remaining bytes straight tofrom_bytes. A buffer that carries a valid arity prefix but a body shorter than the null bitmap therefore decodes "successfully", and the panic surfaces later when a reader touches the missing null-bitmap byte:from_serialized_bytesis used throughout the manifest / stats / partition read path (e.g.stats.rs,table_scan.rs,partition_listing.rs, the DataFusion system tables), and the decoded row is typically handed to a consumer that callsis_null_atfirst —format_partition_value, predicate evaluation,get_datum— so a truncated or malformed serializedBinaryRowaborts the reader with a bounds panic instead of a recoverable error.Change
from_serialized_bytes, after reading the arity, reject inputs whose body is shorter thancal_fix_part_size_in_bytes(arity)(null bitmap + fixed part), returning the crate's existingError::UnexpectedErrorrather than constructing a row that will panic on read. A negative arity is also rejected, and the required size is computed ini64so an absurd arity in malformed input cannot overflow thei32size math.is_null_atindex the null bitmap withget, so a short buffer reports the field as not-null and the typed field readers (already bounds-checked viaread_slice/read_byte_at) return a gracefulErrinstead ofis_null_atpanicking.Tests
Added regression tests in the
binary_rowtest module:test_from_serialized_bytes_truncated_body— valid arity prefix but empty / too-short body →Err, no panic.test_from_serialized_bytes_negative_arity— negative arity →Err.test_is_null_at_short_buffer_does_not_panic—is_null_aton a buffer lacking the null bitmap does not panic; the typed reader then returnsErr.test_from_serialized_bytes_well_formed_decodes— negative control: a correctly sized body still decodes and reads back.cargo test -p paimon,cargo build -p paimon,cargo fmt --all -- --check, andcargo clippy -p paimon --all-targets -- -D warningsall pass locally.