Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
autogravity
models/*.onnx
!models/u2net-int8.onnx
!models/face_detection_yunet_2023mar.onnx
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/autogravity
/models/*.onnx
!/models/u2net-int8.onnx
!/models/face_detection_yunet_2023mar.onnx
/docs/node_modules
/docs/.output
20 changes: 11 additions & 9 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ the common development tasks:
```sh
mise install
mise run ci # formatting, vet, race-enabled tests, and build
mise run model # download and verify U²-Net
mise run model # download and verify the ONNX models
```

GitHub Actions runs `mise run ci` and validates the Docker image for both
Expand All @@ -34,19 +34,19 @@ Additional natural photographs cover a dog low in a portrait and two puppies in
grass. A licensed panda eating bamboo is also included as a regression for a
previously reported failure with a similar image.

To also run the real U²-Net model through the HTTP handler:
To also run the real YuNet and U²-Net models:

```sh
export ONNXRUNTIME_LIB=/absolute/path/to/libonnxruntime.dylib
make test-integration
```

This downloads and verifies the model, then runs race-enabled tests including
subject-location checks, mirrored-image consistency, and equivalent raw and
multipart results. The integration suite requires a working runtime and model;
it fails rather than silently skipping when they are missing. GitHub Actions
installs the pinned runtime and runs this suite on every pull request and push
to `main`. Default tests need neither the runtime nor network access.
This downloads and verifies the models, then runs race-enabled tests including
face and subject-location checks, mirrored-image consistency, and equivalent
raw and multipart results. The integration suite requires a working runtime and
models; it fails rather than silently skipping when they are missing. GitHub
Actions installs the pinned runtime and runs this suite on every pull request
and push to `main`. Default tests need neither the runtime nor network access.

Integration tests follow `MODEL_PRECISION` (default `int8`) and honor explicit
`MODEL_PATH` overrides. `make test-integration-fp32` tests the FP32 environment
Expand Down Expand Up @@ -110,8 +110,10 @@ reduce single-request latency when more CPU cores are available.

```text
cmd/autogravity/ HTTP server and lifecycle
internal/facedetection/ YuNet preprocessing, inference, and face selection
internal/imageutil/ decoding, EXIF orientation, resize, normalization
internal/ortenv/ shared ONNX Runtime lifecycle
internal/saliency/ ONNX Runtime model session and inference
internal/gravity/ saliency-weighted focal-point calculation
models/ local model location (ONNX files are gitignored)
models/ checked-in and downloaded model artifacts and licenses
```
4 changes: 3 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,10 @@ RUN mkdir -p /opt/onnxruntime \
ADD --checksum=sha256:8d10d2f3bb75ae3b6d527c77944fc5e7dcd94b29809d47a739a7a728a912b491 --chmod=0444 \
https://github.com/danielgatis/rembg/releases/download/v0.0.0/u2net.onnx /opt/models/u2net.onnx
COPY models/u2net-int8.onnx /opt/models/u2net-int8.onnx
COPY models/README.md models/U2NET_LICENSE /opt/models/
COPY models/face_detection_yunet_2023mar.onnx /opt/models/face_detection_yunet_2023mar.onnx
COPY models/README.md models/U2NET_LICENSE models/YUNET_LICENSE /opt/models/
RUN echo "b340186f56660b6665e494aab912e5f8e9adbc2317181c77fd01aa226f06553b /opt/models/u2net-int8.onnx" | sha256sum -c -
RUN echo "8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4 /opt/models/face_detection_yunet_2023mar.onnx" | sha256sum -c -

WORKDIR /src
COPY go.mod go.sum ./
Expand Down
11 changes: 8 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
FP32_MODEL_PATH := models/u2net.onnx
FP32_MODEL_URL := https://github.com/danielgatis/rembg/releases/download/v0.0.0/u2net.onnx
FP32_MODEL_SHA256 := 8d10d2f3bb75ae3b6d527c77944fc5e7dcd94b29809d47a739a7a728a912b491
FACE_MODEL_PATH := models/face_detection_yunet_2023mar.onnx
FACE_MODEL_SHA256 := 8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4
VERSION ?= dev
COMMIT ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo unknown)
LDFLAGS := -X main.version=$(VERSION) -X main.commit=$(COMMIT)

.PHONY: build run test test-integration test-integration-fp32 evaluate model model-fp32 model-int8
.PHONY: build run test test-integration test-integration-fp32 evaluate model model-fp32 model-int8 model-face

build:
go build -ldflags="$(LDFLAGS)" -o autogravity ./cmd/autogravity
Expand All @@ -19,17 +21,20 @@ test:
test-integration: model
go test -race -tags=integration ./...

test-integration-fp32: model-fp32
test-integration-fp32: model-fp32 model-face
MODEL_PATH= MODEL_PRECISION=fp32 go test -race -tags=integration ./...

evaluate: model
go test -tags=integration,evaluation -run TestEvaluateDifficultScenes -v ./cmd/autogravity

model: model-fp32 model-int8
model: model-fp32 model-int8 model-face

model-int8:
@echo "b340186f56660b6665e494aab912e5f8e9adbc2317181c77fd01aa226f06553b models/u2net-int8.onnx" | shasum -a 256 -c

model-face:
@echo "$(FACE_MODEL_SHA256) $(FACE_MODEL_PATH)" | shasum -a 256 -c

model-fp32:
@if [ ! -f "$(FP32_MODEL_PATH)" ]; then \
curl -fL --retry 3 -o "$(FP32_MODEL_PATH)" "$(FP32_MODEL_URL)"; \
Expand Down
83 changes: 51 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,18 @@

# autogravity

`autogravity` is a small Go HTTP service that finds the main visual subject in
an image. It runs U²-Net with ONNX Runtime, selects the strongest connected
salient region, and returns its weighted centroid as normalized X/Y coordinates.
It never crops, stores, or modifies the submitted image.
`autogravity` is a small Go HTTP service that finds the best crop focus in an
image. It prioritizes a confidently detected face using YuNet, then falls back
to the weighted centroid of the strongest U²-Net salient region. Results are
normalized X/Y coordinates. It never crops, stores, identifies, or modifies the
submitted image.

## Requirements

- Go 1.25 or newer
- The included INT8 U²-Net model (approximately 42 MiB). `make model` verifies it
and downloads/verifies the FP32 fallback (approximately 168 MiB).
- The included INT8 U²-Net model (approximately 42 MiB) and YuNet face detector
(approximately 230 KiB). `make model` verifies them and downloads/verifies the
FP32 U²-Net fallback (approximately 168 MiB).
- An ONNX Runtime shared library. Version 1.23.2 is used by the Docker image and
matches the pinned Go binding.

Expand Down Expand Up @@ -39,6 +41,8 @@ The server listens on `:8080`. These environment variables are available:
| `ADDR` | `:8080` | HTTP listen address |
| `MODEL_PRECISION` | `int8` | `int8` or `fp32`, on every architecture |
| `MODEL_PATH` | unset | Explicit model path; overrides `MODEL_PRECISION` |
| `FACE_MODEL_PATH` | `models/face_detection_yunet_2023mar.onnx` | YuNet face-detection model path |
| `FACE_SCORE_THRESHOLD` | `0.85` | Minimum face confidence in `(0.0, 1.0]` |
| `ONNXRUNTIME_LIB` | required | Full ONNX Runtime shared-library path |
| `MAX_CONCURRENT_ANALYSES` | available CPUs | Maximum images decoded and inferred concurrently |
| `ONNX_INTRA_OP_THREADS` | `1` | CPU threads used within each ONNX operator |
Expand Down Expand Up @@ -72,14 +76,16 @@ Measured on September 6, 2026, with macOS 26.5.2 (arm64), 18 GiB RAM, Go 1.25.14
and ONNX Runtime 1.23.2. Each result is the median of five sequential benchmark
samples using `-benchtime=3s` and the checked-in synthetic images.

Timings include decoding, orientation handling, resizing and normalization to
320 × 320, inference, and focal-point calculation. They exclude model startup,
file reads, uploads, and HTTP overhead. Performance varies with hardware and
input images; see [benchmark instructions](CONTRIBUTING.md#benchmarks) to measure
your environment.
These historical timings measure the saliency fallback: decoding, orientation
handling, resizing and normalization to 320 × 320, U²-Net inference, and
focal-point calculation. Face-selected requests skip U²-Net and are typically
substantially faster; every request still pays for the lightweight face check.
Timings exclude model startup, file reads, uploads, and HTTP overhead.
Performance varies with hardware and input images; see
[benchmark instructions](CONTRIBUTING.md#benchmarks) to measure your environment.

By default, the server runs one analysis per effective CPU using one shared
model session. With Go 1.25, this respects Linux container CPU limits through
session per model. With Go 1.25, this respects Linux container CPU limits through
the runtime's container-aware `GOMAXPROCS` setting. ONNX Runtime uses one
intra-op thread per analysis, preventing its internal worker pool from
multiplying with request concurrency. Set explicit CPU and memory limits for
Expand All @@ -88,9 +94,10 @@ the tighter constraint.

## Docker

The image bundles the verified INT8 model and downloads the verified FP32 model
and CPU-only ONNX Runtime during the build. Both models are included on both
`linux/amd64` and `linux/arm64`; selection is by environment, not architecture.
The image bundles the verified INT8 and face models and downloads the verified
FP32 model and CPU-only ONNX Runtime during the build. All models are included
on `linux/amd64` and `linux/arm64`; U²-Net precision selection is by
environment, not architecture.

```sh
docker build -t autogravity .
Expand Down Expand Up @@ -156,22 +163,31 @@ Example response:
"x": 0.68,
"y": 0.37
},
"confidence": 0.91
"confidence": 0.91,
"source": "face"
}
```

Coordinates are in `[0.0, 1.0]`, measured from the oriented image's top-left
corner. EXIF orientation is applied before analysis. Images are fitted within
the model's 320x320 input using neutral padding, without stretching or
cropping. Padding is excluded from the focal-point calculation. Pixels reaching
at least half the peak activation are grouped into connected regions, and the
region with the greatest total saliency supplies the focal point. Confidence is
the peak activation in the model's fused saliency map, clamped to `[0.0, 1.0]`.
corner. EXIF orientation is applied before analysis. A YuNet face at or above
the configured score threshold has priority, with the most prominent face's
bounding-box center supplying the focal point. If no reliable face is found,
the image is fitted within U²-Net's 320x320 input using neutral padding, without
stretching or cropping. Padding is excluded from the calculation. Pixels
reaching at least half the peak activation are grouped into connected regions,
and the region with the greatest total saliency supplies the fallback point.

`source` is `face` or `saliency`. `confidence` is the selected model's score:
YuNet's face score for `face`, or the peak fused-map activation for `saliency`.
Scores are clamped to `[0.0, 1.0]` but are not calibrated probabilities and
should not be compared across sources. Face detection does not perform identity
recognition. Blurred, obscured, or highly stylized faces may use the saliency
fallback.

Requests are limited to 10 MiB and decoded images to 20 megapixels. Separate
upload and analysis admission limits bound buffered-body and decoded-image
memory without allowing slow uploads to reserve inference capacity. The model
is loaded once at startup and its shared inference session is reused safely
memory without allowing slow uploads to reserve inference capacity. Both models
are loaded once at startup and their inference sessions are reused safely
across requests.

## Telemetry and shutdown
Expand All @@ -182,24 +198,27 @@ letters, digits, `_`, or `-` and is at most 64 characters. Logs never include
uploaded image data, filenames, query strings, or request headers.

Prometheus metrics are exposed at `/metrics`, including HTTP rates and latency,
pipeline stage latency, active inference count, outcomes, cancellations, Go
runtime statistics, process statistics, and build information. Keep this
endpoint private at the ingress layer.
pipeline stage latency, face-detection outcomes, selected gravity sources,
active inference count, cancellations, Go runtime statistics, process
statistics, and build information. Keep this endpoint private at the ingress
layer.

Every analysis has a configurable deadline. Cancellation propagates into ONNX
Runtime and terminates that request's native inference without affecting other
concurrent requests. On SIGINT or SIGTERM, readiness becomes false immediately,
new analyses receive 503 with `Retry-After`, and active requests may finish for
`SHUTDOWN_TIMEOUT`. Once the grace period expires, their contexts are cancelled,
native inference is terminated, connections are closed, and the model is then
released. Set the orchestrator termination grace period longer than
native inference is terminated, connections are closed, and the model sessions
are then released. Set the orchestrator termination grace period longer than
`SHUTDOWN_TIMEOUT`.

## Model quality

Full U²-Net fixes the person-in-room fixture previously missed by U²-NetP, but
still misses two difficult scenes. See the [fixture evaluation](internal/testimages/testdata/README.md)
for measured outputs and unchanged expected regions.
YuNet prioritizes clear faces but intentionally falls back when its confidence
is below the configured threshold. Full U²-Net fixes the person-in-room fixture
previously missed by U²-NetP, but still misses two difficult scenes. See the
[fixture evaluation](internal/testimages/testdata/README.md) for measured outputs
and unchanged expected regions.

## Contributing

Expand Down
Loading