What happened?
A DOUBLE field rejects any integral JSON literal outside int range, even when a double holds it exactly:
// RowJsonValueExtractors.java:138, inside doubleValueExtractor()'s validator
|| (jsonNode.isIntegralNumber()
&& jsonNode.canConvertToLong()
&& jsonNode.asLong() == (long) (double) jsonNode.asInt())
asLong() on the left, asInt() on the right. asInt() truncates anything outside int range, so the two sides can never agree for a larger value and it falls through to "out of range".
Epoch millis is the everyday case — this fails today:
{"f_double": 1609459200000}
against a schema with FieldType.DOUBLE, even though 1609459200000.0d is exact.
The guard on the line above is canConvertToLong(), so the intent is clearly "an integral value that fits in a long and survives the trip through double". The asInt() is at odds with its own guard.
Anything under 2^53 is affected: epoch millis, IDs, byte counts. Values that happen to fit in an int work, and a producer that emits 1.6094592e12 works, which is presumably why this has gone unnoticed.
Note for whoever fixes this
Swapping asInt() for asLong() is not the fix. (double) Long.MAX_VALUE rounds up to 2^63, and narrowing 2^63 back to long saturates at Long.MAX_VALUE rather than overflowing, so the round-trip appears to succeed and the extractor stores 9223372036854775808 — trading an over-rejection for silent corruption. RowJsonTest's existing LONG_STRING is 2^63 − 2, which does not exercise that.
Comparing through BigDecimal is exact, and is already what the decimal branch immediately below does.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
A DOUBLE field rejects any integral JSON literal outside
intrange, even when adoubleholds it exactly:asLong()on the left,asInt()on the right.asInt()truncates anything outsideintrange, so the two sides can never agree for a larger value and it falls through to "out of range".Epoch millis is the everyday case — this fails today:
{"f_double": 1609459200000}against a schema with
FieldType.DOUBLE, even though1609459200000.0dis exact.The guard on the line above is
canConvertToLong(), so the intent is clearly "an integral value that fits in a long and survives the trip through double". TheasInt()is at odds with its own guard.Anything under 2^53 is affected: epoch millis, IDs, byte counts. Values that happen to fit in an
intwork, and a producer that emits1.6094592e12works, which is presumably why this has gone unnoticed.Note for whoever fixes this
Swapping
asInt()forasLong()is not the fix.(double) Long.MAX_VALUErounds up to 2^63, and narrowing 2^63 back tolongsaturates atLong.MAX_VALUErather than overflowing, so the round-trip appears to succeed and the extractor stores9223372036854775808— trading an over-rejection for silent corruption.RowJsonTest's existingLONG_STRINGis 2^63 − 2, which does not exercise that.Comparing through
BigDecimalis exact, and is already what the decimal branch immediately below does.Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components