Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,26 @@ services:
- NEO4J_PASSWORD=changeme123
- NEO4J_DATABASE=neo4j
- RENDER_ALLOW_CYPHER=true
- RENDER_JOBS_URL=http://jobs:8080
depends_on:
- neo4j

# Background jobs (render/server/jobs/index.mjs, same image as render): refreshes the dashboard
# totals (Stats nodes) every JOBS_STATS_INTERVAL_MS and runs Map rebuilds, so render only reads.
# Control API on 127.0.0.1:8081 for scripts: POST /api/stats/refresh, /api/lod/rebuild, GET /api/status.
jobs:
build: ./render
command: ["node", "server/jobs/index.mjs"]
restart: unless-stopped
ports:
- 127.0.0.1:8081:8080
environment:
- NEO4J_URI=bolt://neo4j:7687
- NEO4J_USER=neo4j
- NEO4J_PASSWORD=changeme123
- NEO4J_DATABASE=neo4j
- JOBS_STATS_INTERVAL_MS=900000
- JOBS_LOD_UPDATE_INTERVAL_MS=0
depends_on:
- neo4j

Expand Down
2 changes: 2 additions & 0 deletions docker/queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ CREATE INDEX idx_crate_downloads IF NOT EXISTS FOR (c:Crate) ON (c.downloads);
- `const_user_id` ... `idx_crate_downloads`: created by the `registry` job itself
(`db::REGISTRY_SCHEMA_CYPHER`). `idx_published_at` is a relationship range index.
- The Map build adds its own `Crate.lod_*` indexes and `LOD_DEP` (see `render/README.md`).
- The render jobs service creates `const_stats_id` on `(:Stats {id})`, the dashboard totals it
precomputes (JSON `payload`, `computed_at`); `MATCH (s:Stats) DETACH DELETE s` is always safe.

Enterprise only:

Expand Down
9 changes: 9 additions & 0 deletions render/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,12 @@ RENDER_LOD_WORKER_HEAP_MB=8192
RENDER_LOD_BUDGET=600
RENDER_LOD_MAX_LEVEL=12
RENDER_LOD_CHECKPOINT=/tmp/painter-lod-checkpoint.json

# Background jobs service (server/jobs/index.mjs, see README "Background jobs").
# render: forward stats refreshes and Map rebuilds there instead of running them itself.
#RENDER_JOBS_URL=http://127.0.0.1:8081
# jobs: dashboard totals (Stats nodes) refresh interval (0 = on request only), per-payload query timeout,
# and an optional timer for the cheap Map update (0 = off).
JOBS_STATS_INTERVAL_MS=900000
JOBS_QUERY_TIMEOUT_MS=600000
JOBS_LOD_UPDATE_INTERVAL_MS=0
3 changes: 2 additions & 1 deletion render/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ COPY server ./server
COPY --from=build /app/dist ./dist
EXPOSE 8080
USER node
# Shell form so $PORT expands: the jobs service (CMD node server/jobs/index.mjs) answers the same path.
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/api/health > /dev/null || exit 1
CMD wget -qO- "http://127.0.0.1:${PORT}/api/health" > /dev/null || exit 1
CMD ["node", "server/index.mjs"]
46 changes: 43 additions & 3 deletions render/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,45 @@ Settings are env vars read once at startup; defaults match the compose neo4j. Se
container to a host Neo4j use `bolt://host.docker.internal:7687` plus
`--add-host=host.docker.internal:host-gateway`), `NEO4J_MAX_ROWS` (row cap on every query, Cypher
panel included), `RENDER_ALLOW_CYPHER=false` (disables the Cypher panel and `POST /api/cypher`),
`RENDER_LOD_WORKER_HEAP_MB` (heap for the in-server Map build).
`RENDER_LOD_WORKER_HEAP_MB` (heap for the in-server Map build), `RENDER_JOBS_URL` (hand background
work to the jobs service below).

## Background jobs

Every dashboard's overview tiles (unsafe, build scripts, native code, diffs, registry, publisher
leaderboards, and the status bar counts) aggregate the whole graph: label scans of 3.2M versions,
whole relationship types, seconds each and tens of seconds under import load. Rather than paying for
that on a request, `server/jobs/index.mjs` runs as its own service (`jobs` in `docker-compose.yaml`,
same image, `command: node server/jobs/index.mjs`) and is the only writer of derived data:

- every `JOBS_STATS_INTERVAL_MS` (15 min; 0 = on request only) it recomputes each payload in
`server/stats.mjs` (`STATS_JOBS`: `overview`, `unsafe:{latest,all}`, `build:*`, `native:*`,
`diffs:*`, `registry`, `leaderboards`), one scan at a time, and stores it as JSON on
`(:Stats {id, payload, computed_at, elapsed_ms, bytes})` (constraint `const_stats_id`, created at
start). A timer run skips ids another instance refreshed within half the interval.
- the Map build/update worker (below) runs here when asked, and `JOBS_LOD_UPDATE_INTERVAL_MS` can
run the cheap update on a timer.

Control API on `$PORT` (compose publishes it on `127.0.0.1:8081`): `GET /api/health`,
`GET /api/status` (both runs plus the stored `computed_at` per id), `POST /api/stats/refresh`
(`{ids?: [...]}`, 202, 409 while one runs), `GET /api/lod/status`, `POST /api/lod/rebuild`,
`POST /api/lod/update`. Trigger a refresh when a painter stage finishes instead of waiting for the
timer:

```
curl -X POST http://127.0.0.1:8081/api/stats/refresh
curl -X POST http://127.0.0.1:8081/api/stats/refresh -H 'content-type: application/json' -d '{"ids":["registry","leaderboards"]}'
```

The render routes read the `Stats` node (one index seek) and report `computedAt` / `source:
"precomputed"`; the views print "totals computed N min ago". With no node yet (fresh database, no
jobs service) a route falls back to the live aggregate with its old 60 s / 10 min cache and
`source: "live"`. With `RENDER_JOBS_URL` set, render forwards `POST /api/stats/refresh` and
`POST /api/lod/{rebuild,update}` to the jobs service and shows its job status under
`/api/lod/status`, so the graph has one writer; without it, render still runs the Map worker itself
as before. `GET /api/stats/status` lists stored ages and the jobs service's view.

Locally: `npm run start:jobs` (port 8081) next to `RENDER_JOBS_URL=http://127.0.0.1:8081 npm run dev`.

## Indexes

Expand Down Expand Up @@ -121,7 +159,8 @@ interval. To keep it from starving the host:
`NODE_OPTIONS=--max-old-space-size=6144 nice -n 19 ionice -c 3 npm run lod:build -- --reuse-edges --resume`.

The UI (Rebuild/Update) and `POST /api/lod/rebuild` / `POST /api/lod/update` run the same jobs in a
worker thread, one at a time (409 if busy), heap-capped by `RENDER_LOD_WORKER_HEAP_MB`. The server
worker thread, one at a time (409 if busy), heap-capped by `RENDER_LOD_WORKER_HEAP_MB`: in the jobs
service when `RENDER_JOBS_URL` is set (render forwards the request), else inside render. The server
reloads its viewport index when a job finishes or it sees a new `built_at`.

`GET /api/lod/viewport` picks the finest grid level whose cells in the bbox fit the budget and are at
Expand All @@ -143,7 +182,8 @@ obvious ones:
| `GET /api/graph/deps/chain` | Deps view; `meta.roots/depth/truncated` |
| `GET /api/lod/*` | Map status, viewport, locate, search; viewport is 503 until a build is loaded |
| `GET /api/callers?crate&fn&match&version&depth&limit` | Callers rows; `match=exact\|contains` |
| `GET /api/{unsafe,build,native,diffs}/{stats,top,crate/:name}` | leaderboard tiles, sorted/filtered rows (`hasMore` for paging), one crate's per-version history; stats cached 60 s |
| `GET /api/{unsafe,build,native,diffs}/{stats,top,crate/:name}` | leaderboard tiles, sorted/filtered rows (`hasMore` for paging), one crate's per-version history; stats come from the jobs service's `Stats` nodes (`computedAt`), live + cached 60 s without one |
| `GET /api/stats/status`, `POST /api/stats/refresh` | stored `Stats` ages; refresh forwarded to `RENDER_JOBS_URL` (503 without one) |
| `GET /api/registry/*`, `/api/publishers/*`, `/api/releases*`, `/api/supply/*` | registry, publisher, release, and supply-chain data |
| `POST /api/cypher` | `{query, params}`, read-only, row-capped; 403 when disabled |

Expand Down
1 change: 1 addition & 0 deletions render/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"dev:web": "vite",
"build": "tsc --noEmit && vite build",
"start": "node server/index.mjs",
"start:jobs": "PORT=8081 node server/jobs/index.mjs",
"typecheck": "tsc --noEmit",
"lod:build": "node server/lod/cli.mjs build",
"lod:update": "node server/lod/cli.mjs update",
Expand Down
13 changes: 13 additions & 0 deletions render/server/config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,17 @@ export const config = {
// Positions every N iterations, so a crash can be resumed.
checkpointPath: env('RENDER_LOD_CHECKPOINT', '/tmp/painter-lod-checkpoint.json'),
},
// Base URL of the jobs service (server/jobs/index.mjs). When set, render never runs a LOD worker or a stats
// refresh itself: POST /api/lod/* and /api/stats/refresh are forwarded there, so the graph has one writer.
jobsUrl: env('RENDER_JOBS_URL', '').replace(/\/+$/, ''),
jobs: {
// Dashboard totals (Stats nodes) are recomputed this often; 0 disables the timer (manual refresh only).
statsIntervalMs: int('JOBS_STATS_INTERVAL_MS', 15 * 60_000),
// Per-payload limit: unsafe:all is ~95 s on a cold local database and slower under import load.
queryTimeoutMs: int('JOBS_QUERY_TIMEOUT_MS', 600_000),
// The cheap Map update (place crates imported since the last build); 0 = only on request.
lodUpdateIntervalMs: int('JOBS_LOD_UPDATE_INTERVAL_MS', 0),
},
};

export const describeConfig = () => ({
Expand All @@ -55,4 +66,6 @@ export const describeConfig = () => ({
depsMaxEdges: config.depsMaxEdges,
depsDefaultEdges: config.depsDefaultEdges,
lod: { iterations: config.lod.iterations, budget: config.lod.budget, maxLevel: config.lod.maxLevel },
jobsUrl: config.jobsUrl || null,
jobs: config.jobs,
});
59 changes: 59 additions & 0 deletions render/server/db.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// Driver setup shared by the render server and the jobs service, so both wait for neo4j the same way.
import neo4j from 'neo4j-driver';
import { collectGraph } from './graph.mjs';

export const createDriver = (config, { poolSize = 16 } = {}) =>
neo4j.driver(config.neo4j.uri, neo4j.auth.basic(config.neo4j.user, config.neo4j.password), {
maxConnectionPoolSize: poolSize,
connectionAcquisitionTimeout: 10_000,
disableLosslessIntegers: false,
});

// The compose stack may start the database slower than this container; 0 retries means wait forever.
export const waitForNeo4j = async (driver, config, log) => {
let attempt = 0;
for (;;) {
attempt += 1;
try {
const info = await driver.getServerInfo({ database: config.neo4j.database });
log(`connected to ${config.neo4j.uri} (${info.agent}, protocol ${info.protocolVersion})`);
return;
} catch (err) {
const retriesLeft = config.neo4j.connectRetries === 0 ? 'unlimited' : config.neo4j.connectRetries - attempt;
log(`neo4j not reachable at ${config.neo4j.uri} (attempt ${attempt}, retries left: ${retriesLeft}): ${err.message}`);
if (config.neo4j.connectRetries !== 0 && attempt >= config.neo4j.connectRetries) {
throw err;
}
await new Promise((r) => setTimeout(r, config.neo4j.connectRetryDelayMs));
}
}
};

/** Read transaction returning plain rows (driver integers and temporals converted like the API routes do). */
export const readRows = async (driver, config, cypher, params = {}, { timeoutMs = config.neo4j.queryTimeoutMs } = {}) => {
const session = driver.session({ database: config.neo4j.database, defaultAccessMode: neo4j.session.READ });
try {
const started = Date.now();
const result = await session.executeRead((tx) => tx.run(cypher, params), { timeout: timeoutMs });
const elapsedMs = Date.now() - started;
// Neo4j 5 can end a timed-out read as an empty stream instead of raising.
if (result.records.length === 0 && elapsedMs >= timeoutMs * 0.95) {
const err = new Error(`query timed out after ${elapsedMs} ms (limit ${timeoutMs} ms); the database may be busy`);
err.code = 'Neo.ClientError.Transaction.TransactionTimedOut';
throw err;
}
return collectGraph(result.records).rows;
} finally {
await session.close();
}
};

export const write = async (driver, config, cypher, params = {}, { timeoutMs = 600_000 } = {}) => {
const session = driver.session({ database: config.neo4j.database, defaultAccessMode: neo4j.session.WRITE });
try {
const result = await session.executeWrite((tx) => tx.run(cypher, params), { timeout: timeoutMs });
return result.summary.counters.updates();
} finally {
await session.close();
}
};
Loading
Loading