feat(sdk): builtin field tables, unprojectPixel and ImageAnnotations additions (0.36.0)

[facontidavide]

Changed lines

Lines	Added	Removed	Files
Production (headers, codec, proto)	+1500	-35	16
Test	+1265	-1	4
Other (docs, CHANGELOG, VERSION, CMake)	+97	-14	8
Total	+2862	-50	28

About 1,400 of the production lines are the field tables, carried over from #202 and #203 as reviewed there.

Goals

Release 0.36.0 as the header-only part of the derived-objects SDK work, split out of #202 as agreed with its author:

• The PJ4 slices that carve the derived-objects prototype (PlotJuggler/PJ4#704) need the field tables early: the scripting, on-demand, scene, media and series slices.
• The host contract (data processors v2, evaluations, catalog snapshot v2, plot-tab tail slots, dialog scene views, manifest keys) stays in feat(sdk): host surfaces for derived objects and builtin field tables (0.36.0) #202 and becomes the next minor. It is tagged just before the PJ4 slice that bridges plugins, because every slot, flag and key is permanent once tagged.

What ships:

• Builtin field tables: field_table.hpp, field_table_registry.hpp, and one *_fields.hpp for each of the eight tabled builtins. They give one generic description of each struct's members, so the Luau and Python binders and an agent see the same fields.
• PinholeIntrinsics / unprojectPixel (depth_image_utils.hpp).
• ImageAnnotations: an optional timestamp and image_topic on the wire.
• Object-topic metadata keys: pj_snapshot and pj_derived.
• The cleanups of refactor(sdk): simplify the 0.36.0 surfaces (stacked on #202) #203 that apply to these files.

Host contract: unchanged. No new feature_floors.json entry; check_feature_floors.py passes unchanged.

Two decided changes beyond #202:

1. unprojectPixel uses the full inverse of K. It used to reject K unless the skew and projective terms were exactly 0 and 1, so a K that went through float32 was rejected (spacing near 1.0 is about 1.2e-7). It now rejects only a non-finite or singular K, a non-finite result, or a ray with z <= 0. PinholeIntrinsics keeps K⁻¹ private, so the representation is not frozen by the tag.
2. pj_derived is an open value set. The key means "produced by a host data processor", and the value names the kind ("on_demand" today; a host may stamp others, such as "markers").

How to evaluate

• The split: compared with refactor(sdk): simplify the 0.36.0 surfaces (stacked on #202) #203's final tree (7793f4d), every excluded file is byte-identical to main, and nothing shipped references a host-contract declaration. Review with git diff 7793f4d5a..HEAD: the remaining differences are the two decided changes, the review fixes below, and docs or CHANGELOG text reworded to drop host-contract references.
• Review fixes applied (from a /simplify pass, a Codex review and an Opus review, each weighed by fix cost × likelihood × severity):
  • Field-table buffer layouts report record_count = 0 when the declared size exceeds the bytes. A truncated recording, or a script setting width before data, otherwise leads a binder to read out of bounds.
  • Numeric set_number saturates. NaN becomes 0 and out-of-range values clamp to the integer or enum range, replacing undefined float-to-integer conversion (on x86 color.r = 300 used to give 44).
  • fromK requires a finite determinant.
  • The codec header and the format doc now describe when an annotation payload is empty.
  • The CHANGELOG heading has no "Unreleased" suffix.
• Review findings deliberately not applied:
  • Overflow and underflow in unproject at K or depth around 1e±100. Not reachable with pixel-scale K and metric depth.
  • An INT64_MIN annotation timestamp round-trip. This is pre-existing shared timestamp code, and the value is not realistic.
  • A reset accessor for optional fields. It is additive and header-only, so it can be added when a binder needs to clear an optional field.
• For feat(sdk): host surfaces for derived objects and builtin field tables (0.36.0) #202's rebase onto this:
  • Its @since 0.36.0 tags, its feature_floors 0.36.0 group and the 0.36.0 CHANGELOG entries move to the next minor.
  • The snapshot-v2 comment in plugin_data_api.h should drop the __markers__/ prefix test in favour of the pj_derived key.

Test plan

• ./build.sh --debug (ASan) and ./test.sh: 98/98 pass
• Red tests captured before each behaviour change: unprojectPixel (3 of 11 new tests failed before the change), buffer-layout clamping, setter saturation, determinant check
• tools/feature_floors/check_feature_floors.py: OK, no new floor entries
• pre-commit clean
• CI (abidiff additions-only for a MINOR) runs on this PR

Follow-up: the two remaining P3 findings of the Codex review

• INT64_MIN timestamp round trip (fixed). splitTimestamp writes INT64_MIN as seconds -9223372037, nanos 145224192, but combineTimestamp rejected seconds below INT64_MIN / 1e9 before adding the nanos. So the decoder refused a payload our own encoder produced. combineTimestamp now delegates to the existing PJ::combineSecondsAndNanos (time_math.hpp), which /simplify pointed out already does this. Every decoder that reads a protobuf Timestamp gets the fix.
  • Only one input range becomes newly accepted: seconds -9223372037 with nanos 145224192–999999999. Everything else behaves as before, and the wire format is unchanged.
  • New tests RoundTrip_TimestampExtremes and TimestampOutsideInt64NanosIsRejected failed before the fix.
  • Reviewed with /simplify, then Codex (no findings, "Ship") and Opus (one test-hygiene fix: the roundTrip helper now fails cleanly instead of dereferencing an empty result).
  • SDK suite: 98/98 under Debug + ASAN.
• Annotation docs contradicting the codec: already fixed in this branch. The header and image_annotations_format.md say the timestamp and image_topic are serialized.
• The optional-field reset finding stays deferred. It is recorded in PJ4's CARVING under slice 5.

🤖 Generated with Claude Code