Try to build jetson-utils for JP 5.1#2357
Draft
PawelPeczek-Roboflow wants to merge 23 commits into
Draft
Conversation
Add an optional torch.Tensor representation to WorkflowImageData with
lazy BGR<->RGB conversion between the two backings. Layout contract:
numpy is HWC uint8 BGR (cv2 native), tensor is HWC uint8 RGB
(inference-models / torch convention). dtype is preserved; no implicit
device moves.
- __init__ accepts tensor_image; "empty" check covers the new field.
- numpy_image property: if only tensor is set, materialize via
detach().to("cpu").numpy() with a channel flip and cache the result.
- tensor_image property: mirror fallback from numpy with channel flip.
- copy_and_replace propagates tensor_image.
- create_crop_from_tensor: tensor-native sibling of create_crop with
identical metadata math.
- _read_shape_without_materialization avoids forcing device->host just
to fill parent_metadata / workflow_root_ancestor_metadata origin
coordinates when only the tensor representation is set.
Public surface is unchanged; the field is opt-in. Serialization (via
base64_image -> numpy_image -> JPEG) continues to work transparently.
New common/deserializers_tensor.py and common/serializers_tensor.py.
The numpy files are untouched per the plan's locked [ITERATE 4.A]
decision; the tensor file's deserialize_image_kind handles raw
torch.Tensor input and the dict-shape {"type": "tensor", "value": ...}
input, and delegates all other inputs (np.ndarray, base64, URL, dict
with type=base64/url) to the numpy implementation. The serializer
sibling currently re-exports the numpy functions because the lazy
tensor->numpy fallback in WorkflowImageData makes serialise_image
correct in both modes; the file exists to keep the loader's import
swap symmetric and to give future tensor-aware optimisations a
landing spot without touching the loader contract.
Mirror module per the plan's Step 5b. attach_parents_coordinates_* helpers go through ImageParentMetadata.origin_coordinates, which is populated by WorkflowImageData.parent_metadata / workflow_root_ancestor_metadata via _read_shape_without_materialization, so the tensor mirrors currently delegate to the numpy implementations. The module exists so future tensor-specific divergence has a landing spot the loader can swap to without touching the numpy file.
…path map_inference_kwargs unconditionally sets input_color_format="bgr", which is the right default for the cv2-derived numpy paths (preprocess / predict / postprocess) but breaks the new run_tensor_native_inference entry points: workflows tensor blocks pass RGB tensors per the workflows tensor-data-representation plan and need a way to opt out of the BGR override. In each of the four adapter classes that implement run_tensor_native_inference (object detection, instance segmentation, classification, semantic segmentation), pop input_color_format from kwargs before map_inference_kwargs (default None), then restore it afterwards. map_inference_kwargs itself is untouched, so every old execution path keeps the BGR default it has today. Effect: callers of run_tensor_native_inference can pass input_color_format="rgb" (or "bgr", or leave it None) and the value travels through to the underlying model unchanged.
New v3_tensor.py next to v3.py. Per the plan's locked decisions: - Manifest is verbatim (same type literal, name, version, description, fields, outputs, ui_manifest). Class name unchanged so the loader's if/else swap binds the same identifier in both branches. - run_locally calls model_manager.run_tensor_native_inference with the per-image torch tensors from WorkflowImageData.tensor_image and passes input_color_format="rgb" (adapter now respects caller's value). Skips convert_inference_detections_batch_to_sv_detections because the adapter returns sv.Detections directly. - attach_parents_coordinates_to_batch_of_sv_detections_tensor used on the local path; numpy mirror reused on the remote path per [ITERATE 6.A]. - run_remotely materialises base64_image (which lazily goes tensor -> numpy -> JPEG) and hits the same HTTP API as v3.py. Remote response is dict-shaped so the numpy converter applies. - inference_id read from sv.Detections.data when present; uuid4 fallback per image per [ITERATE 6.B].
Wire the three pieces the plan's Step 3 calls for, gated on
ENABLE_TENSOR_DATA_REPRESENTATION:
- Import env flag.
- Move serialise_image / serialise_sv_detections / serialise_rle_sv_detections
and deserialize_image_kind / deserialize_detections_kind /
deserialize_rle_detections_kind into an if/else swap that picks the
_tensor module when the flag is on. All other (de)serializer
functions stay imported from the numpy file (the tensor file only
mirrors the image/detection trio that has tensor-aware behaviour).
- Object-detection V3 block import becomes an if/else that picks
v3_tensor.py when the flag is on. Class name unchanged, so the
blocks = [...] list and load_blocks() are untouched.
load_kinds() and KINDS_SERIALIZERS / KINDS_DESERIALIZERS dict
construction are unchanged per the plan ("no new kinds").
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Related Issue(s):
Type of Change
Testing
Test details:
Checklist
Additional Context