146 lines
6.1 KiB
Markdown
146 lines
6.1 KiB
Markdown
# Detection Pipeline — Execution Path
|
|
|
|
## Overview
|
|
|
|
A pipeline run is a sequence of named **stages** that read and write a shared
|
|
`DetectState` dict. Stages are defined in `core/detect/stages/`; the orchestrator
|
|
(`core/detect/graph/runner.py`) flattens the profile's `PipelineConfig` graph into a
|
|
linear order, runs each stage, and emits SSE events to the browser.
|
|
|
|
The full stage list is in `core/detect/graph/nodes.py`:
|
|
|
|
```
|
|
extract_frames → filter_scenes
|
|
→ field_segmentation → detect_edges
|
|
→ detect_objects → preprocess → run_ocr
|
|
→ match_brands → escalate_vlm → escalate_cloud
|
|
→ compile_report
|
|
```
|
|
|
|
See `03-detection-pipeline.svg` for the graph view.
|
|
|
|
## Profile
|
|
|
|
A `Profile` row in Postgres holds two JSONB blobs:
|
|
|
|
- `pipeline` — a `PipelineConfig` (stages + edges + routing rules) defining topology
|
|
- `configs` — `{stage_name: {...}}` per-stage parameters (fps, thresholds, prompts, ...)
|
|
|
|
Profiles are the config mechanism: **duplicate a profile and tweak it** instead of
|
|
patching defaults. `core/detect/profile.py` loads profiles by name; `_load_profile()`
|
|
in `nodes.py` merges the job's `config_overrides` on top.
|
|
|
|
## Stage runner
|
|
|
|
`PipelineRunner` (in `core/detect/graph/runner.py`) iterates the flattened stages and
|
|
between each one checks three control flags (all keyed by `job_id`):
|
|
|
|
- **cancel** — `set_cancel_check(job_id, fn)`; raises `PipelineCancelled` to abort
|
|
- **pause / resume** — a `threading.Event` per job; `_wait_if_paused()` blocks
|
|
- **step** — like resume but auto-pauses after the next stage completes
|
|
- **pause-after-stage** — toggle to step through every stage
|
|
|
|
Each stage runs inside `trace_node(state, name)` (sets a span used by tracing) and
|
|
emits `running` → `done` (or `skipped`) transitions via `core/detect/emit.py`.
|
|
|
|
## Inference: GPU-host indirection
|
|
|
|
`core/detect/graph/nodes.py` reads `INFERENCE_URL` from the environment and passes it
|
|
to every CV/ML stage:
|
|
|
|
- `INFERENCE_URL=""` (default in dev) — stages call CV/ML routines in-process
|
|
- `INFERENCE_URL=http://gpu-host:8000` — stages POST to the GPU server
|
|
(`core/gpu/server.py`) which exposes `/detect`, `/ocr`, `/preprocess`, `/vlm`,
|
|
`/detect_edges`, `/segment_field` (each with a `/debug` variant that returns
|
|
intermediate masks for the overlay viewer)
|
|
|
|
Memory note: dev and GPU machines are separate boxes on the same LAN; inference is a
|
|
network call. Heavy ML deps (`torch`, `transformers`, `paddleocr`) live only in
|
|
`core/gpu/pyproject.toml` — the API host doesn't need them.
|
|
|
|
## Browser-side CV (OpenCV WASM)
|
|
|
|
Some stages (notably the field/edge stages) can run in the browser via OpenCV WASM
|
|
(`ui/detection-app/src/cv/wasmBridge.ts`) for fast iteration without a round trip to
|
|
the GPU host. The browser UI is the test surface for the "replay loop" — change a
|
|
config, replay one stage, see the overlay. Browser CV uses OpenCV WASM directly; there
|
|
are no TypeScript ports of the algorithms.
|
|
|
|
## Cloud VLM escalation
|
|
|
|
`escalate_vlm` (local VLM on GPU host) and `escalate_cloud` (Anthropic / Gemini /
|
|
OpenAI / Groq via `core/detect/providers/`) are the last-resort branches for
|
|
unresolved candidates from `match_brands`. Skip flags:
|
|
|
|
- `SKIP_VLM=1` — emits `skipped` for `escalate_vlm`
|
|
- `SKIP_CLOUD=1` — emits `skipped` for `escalate_cloud`
|
|
|
|
## Checkpoints, StageOutput, and replay
|
|
|
|
Two tables back the replay loop:
|
|
|
|
- **Checkpoint** (`core/db/models.py:Checkpoint`) — a tree node:
|
|
`(parent_id, stage_name, config_overrides, stats)`. No blobs. Lets the UI show a
|
|
branching history of "what configs did we try at this stage?"
|
|
- **StageOutput** — a flat upsert table keyed by `(job_id, stage_name)` holding the
|
|
stage's output dict. `replay-stage` reads upstream outputs from here so a single
|
|
stage can be re-run without rerunning the whole pipeline.
|
|
|
|
API surface (`core/api/detect/replay.py`):
|
|
|
|
- `GET /checkpoints/{timeline_id}` — full tree
|
|
- `POST /replay` — clone a checkpoint into a new job, run from a chosen stage
|
|
- `POST /replay-stage` — re-run one stage in place using upstream `StageOutput` rows
|
|
- `GET /overlays/{timeline_id}/{job_id}/{stage}/{seq}` — debug overlays from MinIO
|
|
|
|
## Event flow (SSE)
|
|
|
|
Stages call `emit.transition(...)` / `emit.log(...)` / `emit.boxes(...)` etc.
|
|
(`core/detect/emit.py`). These push into Redis (`core/detect/events.py`). The SSE
|
|
endpoint `GET /detect/stream/{job_id}` (`core/api/detect/sse.py`) drains the Redis
|
|
list and writes to the open SSE response. Envoy keeps the connection open for up to
|
|
3600s (see `ctrl/k8s/base/envoy.yaml`).
|
|
|
|
```
|
|
stage code
|
|
→ emit.* (core/detect/emit.py)
|
|
→ push_detect_event → Redis RPUSH
|
|
→ [poll] /detect/stream/{job_id} → SSE chunk
|
|
→ fetch ReadableStream in detection-app
|
|
→ Pinia store update → Vue panel re-render
|
|
```
|
|
|
|
## Pipeline control endpoints
|
|
|
|
All under `core/api/detect/run.py`:
|
|
|
|
- `POST /run` — start a job from a timeline + profile
|
|
- `POST /stop/{job_id}` — cancel
|
|
- `POST /pause/{job_id}` / `POST /resume/{job_id}`
|
|
- `POST /step/{job_id}` — run one stage and pause
|
|
- `POST /pause-after-stage/{job_id}` — toggle pause-after-each-stage
|
|
- `GET /status/{job_id}` — current stage, progress
|
|
- `POST /clear/{job_id}` — discard runtime state
|
|
|
|
## Where the chunker UI fits
|
|
|
|
`ui/chunker/` is a **standalone testing utility** for the source-chunking step (split
|
|
a long source video into chunks the user picks for a Timeline). It is **not** a
|
|
pipeline stage and is not part of the detection flow. The detection pipeline reads
|
|
already-chunked sources from MinIO via `core/api/detect/sources.py`.
|
|
|
|
## Files
|
|
|
|
| Concern | File |
|
|
|---|---|
|
|
| Stage list | `core/detect/graph/nodes.py` |
|
|
| Runner (cancel/pause/resume) | `core/detect/graph/runner.py` |
|
|
| Profile loading | `core/detect/profile.py` |
|
|
| Event emission | `core/detect/emit.py`, `core/detect/events.py` |
|
|
| SSE endpoint | `core/api/detect/sse.py` |
|
|
| Replay API | `core/api/detect/replay.py` |
|
|
| Checkpoint storage | `core/detect/checkpoint/storage.py` |
|
|
| GPU server | `core/gpu/server.py` |
|
|
| Browser CV bridge | `ui/detection-app/src/cv/wasmBridge.ts` |
|
|
| Cloud VLM providers | `core/detect/providers/` |
|