feat(sdk): host surfaces for derived objects and builtin field tables (0.36.0)

[Alvvalencia]

Review and merge order: SDK #202 → publish plotjuggler_sdk 0.36.0 → PJ4 #697 → #698 → #699. Plugins after the SDK: #327 → #328.

Summary

The PJ4 side is three stacked PRs to release-4.1 (PlotJuggler/PJ4#697, #698, #699); the plugins side is PlotJuggler/pj-official-plugins#327 and #328. Both build and pass their tests against this branch with --sdk-local. 0.36.0 is released after this merge; then they pin it.

This is the SDK half of derived objects in PlotJuggler 4: scripts (Luau, Python) that compute point clouds, scene entities and image annotations from recorded data, evaluated at the cursor, in batch, or incrementally. 22 commits on top of v0.35.0; size breakdown below. VERSION 0.36.0, CHANGELOG [0.36.0] — Unreleased, floors updated (check_feature_floors.py OK).

Size

	Added	Removed	Files
Production code	+2 373	−34	25
Tests	+1 981	−18	13
Docs (.md, CHANGELOG)	+402	−13	8
Build, config, scripts	+23	−1	3
Total	+4 779	−66	49

Of the production lines, 1 307 are the field tables (builtin/*_fields.hpp, field_table*.hpp: headers, no ABI) and 778 are plugin_data_api.h / .hpp (the data-processor and plot-tab contract, mostly doc-comments). This PR's own CI can run (the SDK builds itself); the PJ4 and plugins PRs cannot until v0.36.0 is tagged.

What is in it, and what to decide

#	Surface	ABI impact	Question for the reviewer
1	Field tables + registry (builtin/field_table.hpp, *_fields.hpp for 8 types, describe(BuiltinObjectType))	None (headers, client side)	Is the FieldKind vocabulary (kOptionalNumber, fixed kList with list_replace, kBuffer with BufferLayout) the contract we want binders to rely on?
2	Typed data-processor requests: create_data_processor_v2, submit_evaluation / poll_evaluation / release_evaluation, PJ_DATA_PROCESSOR_TIME_FLAG_WINDOW/INSTANT	Tail slots on PJ_data_processors_host_vtable_t, gated by struct_size; new request/output/budget structs with their own struct_size; layout pinned by sentinels	Evaluation handles: is the JSON report of poll_evaluation the right boundary, and is release_evaluation mandatory for every submitted handle?
3	Catalog snapshot v2: acquire_catalog_snapshot_v2 (PJ_catalog_snapshot_v2_t, PJ_object_topic_info_t)	Tail slot on PJ_toolbox_host_vtable_t; new struct	A new struct + slot because object topics are array elements with a fixed stride. Agreed as the rule for array elements?
4	Scene 3D/2D tabs as kinds of pj.plot_tabs.v1 tabs: tail slots create_tab_v2, attach_topic, detach_topic, focus_tab	Tail slots on PJ_plot_tab_host_vtable_t (64 → 96 bytes); the 7 v1 slots and the plot tab_config JSON are unchanged; slots are NULL when the host has no scene workspace	One id namespace per plugin across kinds (same id with another kind replaces the tab; same id and kind: a plot tab is replaced by an empty plot, as v1 does, a scene tab only gets its title updated). OK? list_tab_ids does not return the kind, so a client that wants only scene tabs reads each tab_config. Worth a kind-aware listing slot now, or later if a client needs it?
5	ImageAnnotations wire: top-level timestamp and image_topic (field 6)	Additive protobuf fields; old readers skip them	Not a canonical wire-schema break, so MINOR. Agreed?
6	unprojectPixel for rectified metric depth (depth_image_utils.hpp)	None (inline header)	Today only PJ4 uses it. Keep it in the SDK for plugins, or move it to PJ4?
7	pj_snapshot object-topic metadata key	None	—
8	Plugin catalog test: loader environment isolated	Tests/CMake only	—
9	Output inference: PJ_DATA_PROCESSOR_FLAG_INFER_OUTPUTS; the report carries a root outputs array	New request flag (old hosts reject the bit)	Naming rule for unnamed returns (value, text, cloud, …) lives in the host. OK as host policy, documented here?
10	struct_size read-prefix rule for PJ_data_processor_request_t and PJ_evaluation_budget_t: hosts read the prefix they know and accept a larger struct; a new field is announced by a new flag bit	Rule change (the previous text rejected a larger struct, which would break every newer plugin on a 0.36 host)	Agreed as the rule for every future appended field?
11	out_topics for on_demand number outputs = the catalog key of the series the host writes; coverage.error in the poll report; an unknown poll state is an error in pollEvaluation	Contract text + C++ wrapper	—
12	Capability detection rule, written once: hasTypedRequests(), hasCatalogSnapshotV2(), hasSceneTabs(); flag bits are not probed; Python is probed with validateScript	C++ wrappers only	—
13	Embedded scene views in plugin dialogs: WidgetData::setSceneView / setSceneTopics / clearSceneView, dialog host bit PJ_DIALOG_HOST_EMBEDS_SCENE_VIEWS (hostHas())	Dialog-protocol keys + one capability bit (no vtable change)	—
14	Manifest: optional badge (short label shown next to what a plugin creates) and custom_topics_editor (the toolbox the host opens to create/edit derived topics)	Manifest parsing only	—
15	PlotTabHostView::createV2/focus renamed createTabV2/focusTab; kDerivedMetadataKey; @since 0.36.0 on every new surface	Wrapper rename (unreleased API)	—
16	Docs: lifetime of EPHEMERAL / HISTORY_EXEMPT / pinned per kind, per-kind WINDOW/label table, config answers the owner's own ephemerals, chart auto-zoom semantics	Docs	—

If you prefer smaller PRs

Possible cuts: (a) 1 + 6 + 7 + 8, no ABI; (b) 2 + 3, data-processor and catalog host contract; (c) 4, scene tabs; (d) 5, wire. Each cut needs its own CHANGELOG/VERSION/floors share, and PJ4 needs all of them before it can pin 0.36.0.

Test plan

• ./build.sh --debug && ./test.sh: 98/99. The only red is abi_check_test (the historical abidiff baseline), red before this branch as well.
• check_feature_floors.py, test_sdk_install.sh
• PJ4 built and tested against this branch with --sdk-local
• CI on b40bf62 (one ABI sentinel wrapped to 120 columns): pre-commit, ubuntu-22.04 and windows-2022 green; the Linux/macOS/Windows build jobs and macos-15-intel queued

Changes after the architecture review (2026-10-05)

• PJ_data_processor_output_t gains uint64_t reserved[2] (stride 48): the one array element of the typed request that could not grow is now extensible; a host rejects nonzero reserved.
• PJ_DATA_PROCESSOR_REQUEST_V1_MIN_SIZE / PJ_EVALUATION_BUDGET_V1_MIN_SIZE name the read-prefix minimums (pinned by the ABI sentinels).
• Header rules: a string value a client does not recognise in a key it knows is an error (outputs[].type "unknown" excepted); evaluation budgets and handle/report limits are host policy (no PJ4 numbers in the contract; coverage.stopped says which budget ended an evaluation); inferred output names are chosen by the host; the v1 "name:type" suffix of create_data_processor is deprecated (no client uses it).
• The data_processor_config "series" block documents optional done/total (replay progress).
• hostCapabilities() removed from the docs: it never existed; hostHas() is the API.

Tests: 98/99; the one failure is the historical abi_check_test baseline comparison, red before this branch.

[facontidavide]

Pre-tag review of 0.36.0 (head b40bf62)

I ran a correctness and ABI review and a simplify pass (reuse, simplification, efficiency, altitude) over this PR. I checked it against the host in PJ4 #697–#699 and the plugins on feat/assistant-derived-objects and feat/transform-editor-derived-objects.

Verdict: no critical defect.

• ./build.sh and ./test.sh pass (98/98), and check_feature_floors.py is OK.
• Every new layout matches its sentinel. I recomputed them for 64-bit GCC and MSVC and for wasm32.
• Renames touch only symbols added in this PR, and the released slots of pj.plot_tabs.v1 are unchanged.
• abidiff was not available, so "additions only" was checked by reading, not by the tool.

The cleanups that change no contract are in the stacked #203: helper reuse, deduced table sizes, PinholeIntrinsics, the snapshot(false) fix, and the CHANGELOG rewritten as the net change against 0.35.0.

Everything below changes the contract. Once 0.36.0 is tagged these become permanent, so each needs a decision before the tag. Line numbers are at b40bf62.

Contract decisions before the tag

1. The poll report freezes engine internals (plugin_data_api.h:1308-1330).

   • 11 of its 19 documented keys are internal, echoed or derived:
     • coverage.start_ns / end_ns: echo the request.
     • evaluated_until_ns: stale once budget_report trims the bundles, and so is evaluated.
     • cache_hits, complete: complete is stopped == "complete".
     • bundle.from_cache, bundle.revision.
     • inputs[].alias and is_object: echoes.
     • outputs.<name>.topic: documented, but the host never writes it.
   • bundle.stamp_ns is derived too: it is the max of resolved_ns.
   • The transform editor reads coverage.{stopped,error,candidates,evaluated}, bundles[].outputs.{status,value,summary,reason} and the root outputs.
   • Proposed minimum:
     {"coverage":{"candidates","evaluated","stopped","error"?,"gaps"},"bundles":[{"requested_ns","inputs":{"<alias>":ns},"outputs":{...}}],"outputs"?}
   • Optional: an SDK parseEvaluationReport(), so this prose-only schema gets one compiled implementation.

2. Two error shapes and an unused state (plugin_data_api.h:1221).

   • PJ_EVALUATION_STATE_CANCELLED is never produced: a host cancel arrives as COMPLETED with stopped:"cancelled", and release_evaluation deletes the handle.
   • FAILED returns {"error":...}, while COMPLETED with stopped:"error" carries coverage.error, so clients parse both.
   • Proposal: reserve value 3, and make FAILED carry the same report shape, or drop FAILED.

3. The v1 "<name>:<type>" on_demand route is added and deprecated in the same unreleased release (plugin_data_api.h:985-1000, createOnDemand at plugin_data_api.hpp:1840).

   • v0.35.0 has no on_demand.
   • No plugin and no PJ4 code uses it, and the SDK has no test for it.
   • Proposal: v1 create_data_processor rejects kind="on_demand" with "use create_data_processor_v2", and createOnDemand goes, or becomes a createV2 wrapper.
   • This also removes about 55 lines of OnDemandRuntimeHost::onCreate in PJ4.

4. PJ_DATA_PROCESSOR_FLAG_INFER_OUTPUTS has two meanings and negotiates nothing (plugin_data_api.h:1119-1135).

   • On a scripted submit with no outputs, the bit only turns a rejection into "infer".
   • On create, or on a submit with declared outputs, it selects lenient binding:
     • a scalar return binds to the single output;
     • a positional return binds by declaration order;
     • a keyed return binds by name.
   • Its floor equals hasTypedRequests().
   • Proposal: infer whenever a scripted submit declares no outputs, and make lenient binding the only on_demand binding (it accepts a superset).
   • That drops the bit, its floor entry, and the persisted infer_outputs recipe attribute in PJ4.

5. PJ_evaluation_budget_t.reserved is "0" but nothing rejects a nonzero value (plugin_data_api.h:1196-1212).

   • The text says a later budget field must be zero-defaultable, but the budget has no flags word, and the one spare u32 is not enforced.
   • So a 0.37 field that restricts output would be silently ignored by a 0.36 host, which is the "half-honoured" case the request rule forbids.
   • Either have hosts reject a nonzero reserved (so it can later carry flag bits), or state plainly that budgets are best-effort.

6. WINDOW + INSTANT together.

   • No kind accepts both, yet plugin_data_api.hpp:1607 says "both may be set together".
   • PJ4's validateRequestShape lets both through, and onSubmit silently uses INSTANT, against "a host rejects, never ignores".
   • Make them mutually exclusive.
   • Related: EPHEMERAL on a scripted submit is required but always implied; HISTORY_EXEMPT and label there are accepted and ignored.

7. Dialog scene views: two keys for one value (widget_data.hpp:645-676).

   • scene_topics means nothing without scene_view. The PJ4 host therefore stashes partial state in frame properties and treats "null scene_view means delete" as a convention.
   • The dialog text says a topic with an empty dataset "resolves by name", while plot tabs refuse an ambiguous one. The host applies the tab rule on both paths but only qWarnings on the dialog path.
   • Proposal:
     • one key, "scene_view": {"kind":"3d","topics":[{"topic","dataset"}]} or null, i.e. the scene part of the scene-tab config;
     • one shared SceneTopic type;
     • the kinds, accepted object types and dataset rule written once.
   • The two transports themselves (call-and-return service vs state-push panel) are rightly different.

8. custom_topics_editor adds a top-level bool for one host role (plugin_catalog.hpp:69).

   • The manifest already has a capabilities string array (:54) that nothing uses.
   • Proposal: "capabilities": ["custom_topics_editor"].
   • A smaller point: badge names one UI affordance; consider short_name.

9. pj_derived is closed to "on_demand" (object_topic_metadata.hpp:30-42).

   • "Absent or any other value means not derived", so markers need a second classifier: the docs of snapshot v2 advise a __markers__/ name-prefix test.
   • The assistant already diverges: it checks by presence only.
   • Proposal: the key means "produced by a host data processor", and its value names the kind, as an open set ("on_demand", and the host may stamp "markers").

10. hasSceneTabs() ties generic slots to scene support (plugin_data_api.h:1555, plugin_data_api.hpp:2335).

    • create_tab_v2(kind="plot") and focus_tab are useful without scenes, and one bool cannot say "3d but not 2d".
    • Proposal: the slots are present whenever they are implemented, and an unservable kind is rejected with an error naming it, as data processors already do for kinds.

11. Snapshot v2 is "ABI-VERSIONED", yet carries a host-set struct_size (plugin_data_api.h:405-411).

    • If the plugin set struct_size and the host wrote only inside it, v2 would be appendable under the same read-prefix rule, and no v3 slot would ever be needed for a top-level field.
    • Optional, for performance: a flag that skips the scalar half.
      • The editor re-acquires v2 about once a second while its panel is open.
      • PJ4 fills the scalar half field by field under the engine lock.
      • Several readers only need objectTopics().

12. unprojectPixel uses exact float equality on the skew/projective terms (depth_image_utils.hpp:37).

    • A K that went through float32 yields 1.0000000x or 1e-9, and is rejected as skewed.
    • Use a tolerance, or document "exact pinhole K required". This is behaviour, so it is not changed in refactor(sdk): simplify the 0.36.0 surfaces (stacked on #202) #203.

Additive, any later minor

• An SDK host-side decoder, readDataProcessorRequest(), the inverse of toAbiRequest. Today the read-prefix rule is re-implemented in the SDK test fake, in recording_dp_host.hpp in the plugins and in PJ4's RuntimeHostSlot.h.
• isDerivedTopic() / isSnapshotTopic() readers next to the constants. PJ4 and the assistant already disagree on the value rule, and a PJ4 scene2d test writes "pj_snapshot": true as a JSON bool.
• Design §5 of the derived-objects doc promises incremental | batch kind aliases. They are not in the ABI. Adding them is self-negotiating, since an unknown kind is rejected.

Host and plugin follow-ups (do not block the tag)

• PJ4 DataProcessorsKindRouter::onCreateV2 / onSubmit read request->kind and request->id before any struct_size check (DataProcessorsKindRouter.cpp ~L222-255). It is not UB for a legal call, but it inverts the documented order.
• PJ4 binders (object_binder.cpp:305-338, python_object_binder.cpp:90-110) rebuild the whole table or namespace tree on every field access. A for i=1,#obj.list do obj.list[i] loop is therefore O(N²). list_at is addressable, so read lazily.
• PJ4 depth_ops.cpp should call PinholeIntrinsics::fromK once per image (refactor(sdk): simplify the 0.36.0 surfaces (stacked on #202) #203) instead of validating K per pixel.
• Transform editor calls setChartAutoZoom(..., true) on every 20 Hz tick. That refits on every tick and contradicts the new widget_data.hpp guidance.

🤖 Generated with Claude Code

[facontidavide]

Follow-up: decisions on the contract items

The numbers refer to the pre-tag review above.

Decided

Unless you see a problem with one of them, these can be folded in as they are.

#	Item	Decision
2	Poll states	Reserve value 3 for CANCELLED. FAILED carries the same report shape as COMPLETED with stopped:"error", so clients parse one shape.
3	v1 "<name>:<type>" route for on_demand	Dropped. v1 create_data_processor rejects kind="on_demand" with "use create_data_processor_v2". createOnDemand goes, or becomes a createV2 wrapper.
5	PJ_evaluation_budget_t.reserved	Hosts reject a nonzero value, so the field can later carry flag bits that announce new budget fields.
6	WINDOW + INSTANT	Mutually exclusive: the host rejects a request that sets both. The comment at plugin_data_api.hpp:1607 changes to match.
8	custom_topics_editor	A tag in the manifest's existing capabilities array: "capabilities": ["custom_topics_editor"].
9	pj_derived	An open value set. The key means "produced by a host data processor", and the value names the kind ("on_demand", and the host may stamp "markers").
10	hasSceneTabs()	Tab kinds are negotiated by rejection. The slots are present whenever they are implemented, an unservable kind is rejected with an error naming it, and hasSceneTabs() means "slots present".
12	unprojectPixel	Accept a 1e-9 tolerance on the skew and projective terms.

Also decided: no incremental / batch kind aliases. The engine names stay as they are, and the ABI kinds stay transform, markers and on_demand. The design doc's §5 changes instead of the ABI.

Open, for you

1. Poll report schema (item 1). Is anything beyond the keys listed there read by the Transform Editor or the assistant?
2. INFER_OUTPUTS (item 4). Fold it into the on-demand contract: infer whenever a scripted submit declares no outputs, and make lenient binding the only binding.
3. Dialog scene views (item 7). One scene_view key holding the kind and the topics.
4. Snapshot v2 struct_size (item 11). Set by the plugin, so v2 can grow under the read-prefix rule.
5. New: a typed dataset id on the v2 request. A node reads exactly one dataset; the PJ4 spec now states that as a rule. The per-input source:topic qualifier released in 0.35.0 stays. A typed id would also cover two datasets that share a source name, which the header (~L1096) calls out as needing "a typed dataset id".

Proposal: release in two steps

• 0.36.0 now, header-only: field tables, PinholeIntrinsics and unprojectPixel, the ImageAnnotations wire additions, the metadata keys, and the refactor(sdk): simplify the 0.36.0 surfaces (stacked on #202) #203 cleanups. Items 9 and 12 above touch this part, so they land before this tag.
• The host contract as the next minor: snapshot v2, data processors v2, evaluations, the plot-tab tail slots, dialog scene views and the manifest keys. It stays in feat(sdk): host surfaces for derived objects and builtin field tables (0.36.0) #202, and is tagged just before the PJ4 slice that bridges plugins to it.

Why:

• No PJ4 build can use the host contract before that slice, so an early tag gains nothing.
• Once tagged, every slot, flag and JSON key is permanent.
• Carving the PJ4 side produces the evidence that settles the open items.

Cost: a small new PR for the header-only part, #202 rebased as the next minor, and the plugin PRs re-pinned.

PJ4 side

• The PJ4 prototype (#697–#699) stays open as the reference implementation.
• The plan is to land it on release-4.1 as small slices cut from its final code.
• Slice 0 is the documentation: requirements, target architecture, a scenario catalog seeded from its tests, and a slice tracker, in PlotJuggler/PJ4#704.
• The deviations from the prototype, with the reasons for each, are on this page: result policy, pull versus materialization, demand-gated series, the OnDemandService split, one life-cycle protocol, script isolation and TF. Your review of it would help most.

🤖 Generated with Claude Code