mariano/mediaproc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: mariano:ddb4f17faa9f249b9049a376e2c56d66f67f7192
Branches Tags
mariano:main mariano:aws-int
...
compare: mariano:main
Branches Tags
mariano:main mariano:aws-int

2 Commits
ddb4f17faa ... main

Author SHA1 Message Date
buenosairesam
0728fc6be3 clean stale readme 2026-05-06 10:45:19 -03:00
buenosairesam
41dd488fe6 update docs 2026-05-03 03:19:19 -03:00
22 changed files with 1657 additions and 2756 deletions
Expand all files Collapse all files

161
README.md
View File

@@ -1,161 +1,10 @@
# MPR - Media Processor # MPR
A web-based media transcoding tool with Django admin, FastAPI backend, and React timeline UI. Brand and logo detection pipeline for video — extracts frames, segments the field, runs YOLO + OCR, and escalates unresolved detections to local or cloud VLMs.
## Architecture ## Docs
```
Browser (mpr.local.ar)
│
nginx:80
│
┌────┴────┐
│ │
/admin /api, /ui
│ │
Django FastAPI ◄── Timeline UI
│ │
│ ┌────┘
│ │
└───►│ (job operations)
│
gRPC Server
│
Celery Worker
```
- **Django** (`/admin`): Admin interface for data management
- **FastAPI** (`/api`): REST API and gRPC client
- **Timeline UI** (`/ui`): React app for video editing
- **gRPC Server**: Worker communication with progress streaming
- **Celery**: Job execution via FFmpeg
## Prerequisites
- Docker & Docker Compose
## Quick Start
```bash ```bash
# Add to /etc/hosts python -m http.server 8000 --directory docs
echo "127.0.0.1 mpr.local.ar" | sudo tee -a /etc/hosts # open http://localhost:8000
# Start all services
cd ctrl
cp .env.template .env
docker compose up -d
``` ```
## Access Points
| URL | Description |
|-----|-------------|
| http://mpr.local.ar/admin | Django Admin |
| http://mpr.local.ar/api/docs | FastAPI Swagger |
| http://mpr.local.ar/ui | Timeline UI |
## Commands
```bash
cd ctrl
# Start/stop
docker compose up -d
docker compose down
# Rebuild after code changes
docker compose up -d --build
# View logs
docker compose logs -f
docker compose logs -f celery
# Create admin user
docker compose exec django python admin/manage.py createsuperuser
```
## Code Generation
Models are defined as dataclasses in `core/schema/models/` and generated via `modelgen`:
- **Django ORM** models (`--include dataclasses,enums`)
- **Pydantic** schemas (`--include dataclasses,enums`)
- **TypeScript** types (`--include dataclasses,enums,api`)
- **Protobuf** definitions (`--include grpc`)
Each target only gets the model groups it needs via the `--include` flag.
```bash
# Regenerate all targets
bash ctrl/generate.sh
```
## Media Storage
MPR separates media into **input** (`MEDIA_IN`) and **output** (`MEDIA_OUT`) paths, each independently configurable. File paths are stored relative for cloud portability.
### Local Development
- Source files: `/app/media/in/video.mp4`
- Output files: `/app/media/out/video_h264.mp4`
- Served via: `http://mpr.local.ar/media/in/video.mp4` (nginx alias)
### AWS/Cloud Deployment
Input and output can be different buckets/locations:
```bash
MEDIA_IN=s3://source-bucket/media/
MEDIA_OUT=s3://output-bucket/transcoded/
```
**Scan Endpoint**: `POST /api/assets/scan` recursively scans `MEDIA_IN` and registers new files with relative paths.
See [docs/media-storage.md](docs/media-storage.md) for full details.
## Project Structure
```
mpr/
├── admin/ # Django project
│ ├── manage.py # Django management script
│ └── mpr/ # Django settings & app
│ └── media_assets/# Django app
├── core/ # Core application logic
│ ├── api/ # FastAPI + GraphQL API
│ │ └── schema/ # GraphQL types (generated)
│ ├── ffmpeg/ # FFmpeg wrappers
│ ├── rpc/ # gRPC server & client
│ │ └── protos/ # Protobuf definitions (generated)
│ ├── schema/ # Source of truth
│ │ └── models/ # Dataclass definitions
│ ├── storage/ # S3/GCP/local storage backends
│ └── task/ # Celery job execution
│ ├── executor.py # Executor abstraction
│ └── tasks.py # Celery tasks
├── ctrl/ # Docker & deployment
│ ├── docker-compose.yml
│ └── nginx.conf
├── media/
│ ├── in/ # Source media files
│ └── out/ # Transcoded output
├── modelgen/ # Code generation tool
└── ui/ # Frontend
└── timeline/ # React app
```
## Environment Variables
See `ctrl/.env.template` for all configuration options.
| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | sqlite | PostgreSQL connection string |
| `REDIS_URL` | redis://localhost:6379 | Redis for Celery |
| `GRPC_HOST` | grpc | gRPC server hostname |
| `GRPC_PORT` | 50051 | gRPC server port |
| `MPR_EXECUTOR` | local | Executor type (local/lambda) |
| `MEDIA_IN` | /app/media/in | Source media files directory |
| `MEDIA_OUT` | /app/media/out | Transcoded output directory |
| `MEDIA_BASE_URL` | /media/ | Base URL for serving media (use S3 URL for cloud) |
| `VITE_ALLOWED_HOSTS` | - | Comma-separated allowed hosts for Vite dev server |
## License
MIT

75
docs/architecture/01-architecture.dot Normal file
View File

@@ -0,0 +1,75 @@
digraph mpr_architecture {
rankdir=LR
bgcolor="#0a0e17"
fontname="Helvetica"
node [fontname="Helvetica" fontsize=11 style=filled color="#1e2a4a" fontcolor="#e8eaf0" shape=box]
edge [fontname="Helvetica" fontsize=9 fontcolor="#8892a8" color="#4a5568"]
label="System Architecture"
labelloc=t
fontsize=16
fontcolor="#0066ff"
subgraph cluster_browser {
label="Browser"
style=dashed
color="#1e2a4a"
fontcolor="#8892a8"
ui [label="detection-app\n(Vue 3 + @vue-flow)" fillcolor="#121829"]
wasm [label="OpenCV WASM\n(edge / field stages)" fillcolor="#1a1a3a" fontcolor="#0066ff"]
chunker [label="chunker UI\n(standalone test util)" fillcolor="#121829" fontcolor="#8892a8"]
}
subgraph cluster_k8s {
label="K8s cluster (Kind in dev)"
style=dashed
color="#0066ff"
fontcolor="#0066ff"
gateway [label="Envoy Gateway\nport 8080" fillcolor="#0d1a33" shape=octagon]
ui_pod [label="detection-ui pod\n(Vite :5175)" fillcolor="#121829"]
api [label="FastAPI\n:8702 /detect/*" fillcolor="#121829"]
subgraph cluster_data {
label="Data plane"
style=dashed
color="#1e2a4a"
fontcolor="#4a5568"
pg [label="PostgreSQL\njobs · profiles\ncheckpoints" fillcolor="#121829" shape=cylinder]
redis [label="Redis\n(SSE fan-out)" fillcolor="#121829" shape=cylinder]
minio [label="MinIO\nmedia · overlays" fillcolor="#121829" shape=cylinder]
}
}
subgraph cluster_gpu {
label="GPU host (LAN)"
style=dashed
color="#1e2a4a"
fontcolor="#8892a8"
gpu [label="inference server\nYOLO · OCR · VLM\nedge · field segmentation" fillcolor="#1a3a1a" fontcolor="#00c853" shape=box]
}
subgraph cluster_cloud {
label="Cloud VLM providers"
style=dashed
color="#1e2a4a"
fontcolor="#8892a8"
cloud [label="Anthropic · Gemini\nOpenAI · Groq" fillcolor="#243056" shape=octagon]
}
ui -> gateway [label="HTTP / SSE"]
chunker -> gateway [label="HTTP"]
ui -> wasm [label="worker" color="#0066ff"]
gateway -> ui_pod [label="/ /detection/*"]
gateway -> api [label="/api/*\n/api/detect/stream/*" color="#0066ff"]
api -> pg
api -> redis [label="publish events" style=dotted]
api -> minio [label="frames · overlays"]
api -> gpu [label="HTTP\nINFERENCE_URL" color="#00c853"]
api -> cloud [label="VLM escalation" style=dashed]
redis -> api [label="SSE consumer" style=dashed color="#8892a8"]
}

199
docs/architecture/01-architecture.svg Normal file
View File

@@ -0,0 +1,199 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: mpr_architecture Pages: 1 -->
<svg width="939pt" height="685pt"
viewBox="0.00 0.00 939.00 685.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 680.5)">
<title>mpr_architecture</title>
<polygon fill="#0a0e17" stroke="none" points="-4,4 -4,-680.5 935.39,-680.5 935.39,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="465.7" y="-657.3" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">System Architecture</text>
<g id="clust1" class="cluster">
<title>cluster_browser</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="8,-507 8,-641 383.51,-641 383.51,-507 8,-507"/>
<text xml:space="preserve" text-anchor="middle" x="195.75" y="-621.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Browser</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_k8s</title>
<polygon fill="#0a0e17" stroke="#0066ff" stroke-dasharray="5,2" points="225.75,-213 225.75,-499 895.21,-499 895.21,-213 225.75,-213"/>
<text xml:space="preserve" text-anchor="middle" x="560.48" y="-479.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">K8s cluster (Kind in dev)</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_data</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="762.96,-221 762.96,-463 887.21,-463 887.21,-221 762.96,-221"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-443.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#4a5568">Data plane</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_gpu</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="738.58,-113 738.58,-205 911.58,-205 911.58,-113 738.58,-113"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-185.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">GPU host (LAN)</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_cloud</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="726.77,-8 726.77,-105 923.39,-105 923.39,-8 726.77,-8"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-85.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Cloud VLM providers</text>
</g>
<!-- ui -->
<g id="node1" class="node">
<title>ui</title>
<polygon fill="#121829" stroke="#1e2a4a" points="147.12,-605 17.12,-605 17.12,-569 147.12,-569 147.12,-605"/>
<text xml:space="preserve" text-anchor="middle" x="82.12" y="-590.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">detection&#45;app</text>
<text xml:space="preserve" text-anchor="middle" x="82.12" y="-576.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(Vue 3 + @vue&#45;flow)</text>
</g>
<!-- wasm -->
<g id="node2" class="node">
<title>wasm</title>
<polygon fill="#1a1a3a" stroke="#1e2a4a" points="375.51,-605 248.51,-605 248.51,-569 375.51,-569 375.51,-605"/>
<text xml:space="preserve" text-anchor="middle" x="312.01" y="-590.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">OpenCV WASM</text>
<text xml:space="preserve" text-anchor="middle" x="312.01" y="-576.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">(edge / field stages)</text>
</g>
<!-- ui&#45;&gt;wasm -->
<g id="edge3" class="edge">
<title>ui&#45;&gt;wasm</title>
<path fill="none" stroke="#0066ff" d="M147.51,-587C175.4,-587 208.2,-587 237.08,-587"/>
<polygon fill="#0066ff" stroke="#0066ff" points="236.84,-590.5 246.84,-587 236.84,-583.5 236.84,-590.5"/>
<text xml:space="preserve" text-anchor="middle" x="191" y="-589.7" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">worker</text>
</g>
<!-- gateway -->
<g id="node4" class="node">
<title>gateway</title>
<polygon fill="#0d1a33" stroke="#1e2a4a" points="390.27,-334.9 390.27,-357.1 344.42,-372.79 279.59,-372.79 233.75,-357.1 233.75,-334.9 279.59,-319.21 344.42,-319.21 390.27,-334.9"/>
<text xml:space="preserve" text-anchor="middle" x="312.01" y="-349.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">Envoy Gateway</text>
<text xml:space="preserve" text-anchor="middle" x="312.01" y="-335.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">port 8080</text>
</g>
<!-- ui&#45;&gt;gateway -->
<g id="edge1" class="edge">
<title>ui&#45;&gt;gateway</title>
<path fill="none" stroke="#4a5568" d="M134.59,-568.56C139.39,-566 144.03,-563.15 148.25,-560 213.28,-511.53 265.01,-430.28 291.54,-383.07"/>
<polygon fill="#4a5568" stroke="#4a5568" points="294.46,-385.02 296.24,-374.58 288.33,-381.63 294.46,-385.02"/>
<text xml:space="preserve" text-anchor="middle" x="191" y="-544.3" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">HTTP / SSE</text>
</g>
<!-- chunker -->
<g id="node3" class="node">
<title>chunker</title>
<polygon fill="#121829" stroke="#1e2a4a" points="148.25,-551 16,-551 16,-515 148.25,-515 148.25,-551"/>
<text xml:space="preserve" text-anchor="middle" x="82.12" y="-536.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">chunker UI</text>
<text xml:space="preserve" text-anchor="middle" x="82.12" y="-522.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">(standalone test util)</text>
</g>
<!-- chunker&#45;&gt;gateway -->
<g id="edge2" class="edge">
<title>chunker&#45;&gt;gateway</title>
<path fill="none" stroke="#4a5568" d="M105.39,-514.73C143.45,-483.5 221.58,-419.39 269.81,-379.81"/>
<polygon fill="#4a5568" stroke="#4a5568" points="271.77,-382.72 277.28,-373.67 267.33,-377.31 271.77,-382.72"/>
<text xml:space="preserve" text-anchor="middle" x="191" y="-464.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">HTTP</text>
</g>
<!-- ui_pod -->
<g id="node5" class="node">
<title>ui_pod</title>
<polygon fill="#121829" stroke="#1e2a4a" points="622.27,-271 517.02,-271 517.02,-235 622.27,-235 622.27,-271"/>
<text xml:space="preserve" text-anchor="middle" x="569.64" y="-256.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">detection&#45;ui pod</text>
<text xml:space="preserve" text-anchor="middle" x="569.64" y="-242.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(Vite :5175)</text>
</g>
<!-- gateway&#45;&gt;ui_pod -->
<g id="edge4" class="edge">
<title>gateway&#45;&gt;ui_pod</title>
<path fill="none" stroke="#4a5568" d="M359.58,-324.11C374.93,-317.22 392.19,-309.84 408.27,-303.75 440.13,-291.68 476.26,-280.08 506.15,-271.01"/>
<polygon fill="#4a5568" stroke="#4a5568" points="506.85,-274.46 515.41,-268.23 504.83,-267.76 506.85,-274.46"/>
<text xml:space="preserve" text-anchor="middle" x="453.64" y="-306.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">/ &#160;/detection/*</text>
</g>
<!-- api -->
<g id="node6" class="node">
<title>api</title>
<polygon fill="#121829" stroke="#1e2a4a" points="618.89,-325 520.39,-325 520.39,-289 618.89,-289 618.89,-325"/>
<text xml:space="preserve" text-anchor="middle" x="569.64" y="-310.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FastAPI</text>
<text xml:space="preserve" text-anchor="middle" x="569.64" y="-296.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">:8702 /detect/*</text>
</g>
<!-- gateway&#45;&gt;api -->
<g id="edge5" class="edge">
<title>gateway&#45;&gt;api</title>
<path fill="none" stroke="#0066ff" d="M389.71,-334.3C427.93,-328.47 473.45,-321.52 509.01,-316.1"/>
<polygon fill="#0066ff" stroke="#0066ff" points="509.33,-319.59 518.69,-314.62 508.27,-312.67 509.33,-319.59"/>
<text xml:space="preserve" text-anchor="middle" x="453.64" y="-345.09" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">/api/*</text>
<text xml:space="preserve" text-anchor="middle" x="453.64" y="-333.84" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">/api/detect/stream/*</text>
</g>
<!-- pg -->
<g id="node7" class="node">
<title>pg</title>
<path fill="#121829" stroke="#1e2a4a" d="M870.21,-421.28C870.21,-424.63 849.98,-427.34 825.08,-427.34 800.18,-427.34 779.96,-424.63 779.96,-421.28 779.96,-421.28 779.96,-366.72 779.96,-366.72 779.96,-363.37 800.18,-360.66 825.08,-360.66 849.98,-360.66 870.21,-363.37 870.21,-366.72 870.21,-366.72 870.21,-421.28 870.21,-421.28"/>
<path fill="none" stroke="#1e2a4a" d="M870.21,-421.28C870.21,-417.94 849.98,-415.22 825.08,-415.22 800.18,-415.22 779.96,-417.94 779.96,-421.28"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-403.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">PostgreSQL</text>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-390.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">jobs · profiles</text>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-376.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">checkpoints</text>
</g>
<!-- api&#45;&gt;pg -->
<g id="edge6" class="edge">
<title>api&#45;&gt;pg</title>
<path fill="none" stroke="#4a5568" d="M619.25,-325.41C626.29,-328 633.45,-330.6 640.27,-333 683.23,-348.14 732.2,-364.34 768.79,-376.24"/>
<polygon fill="#4a5568" stroke="#4a5568" points="767.7,-379.57 778.29,-379.32 769.86,-372.91 767.7,-379.57"/>
</g>
<!-- redis -->
<g id="node8" class="node">
<title>redis</title>
<path fill="#121829" stroke="#1e2a4a" d="M869.46,-338.69C869.46,-341.1 849.57,-343.06 825.08,-343.06 800.6,-343.06 780.71,-341.1 780.71,-338.69 780.71,-338.69 780.71,-299.31 780.71,-299.31 780.71,-296.9 800.6,-294.94 825.08,-294.94 849.57,-294.94 869.46,-296.9 869.46,-299.31 869.46,-299.31 869.46,-338.69 869.46,-338.69"/>
<path fill="none" stroke="#1e2a4a" d="M869.46,-338.69C869.46,-336.27 849.57,-334.31 825.08,-334.31 800.6,-334.31 780.71,-336.27 780.71,-338.69"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-322.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">Redis</text>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-308.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(SSE fan&#45;out)</text>
</g>
<!-- api&#45;&gt;redis -->
<g id="edge7" class="edge">
<title>api&#45;&gt;redis</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="1,5" d="M619.3,-312.35C626.33,-312.99 633.48,-313.57 640.27,-314 683.39,-316.75 732.3,-317.99 768.82,-318.55"/>
<polygon fill="#4a5568" stroke="#4a5568" points="768.65,-322.05 778.7,-318.68 768.75,-315.05 768.65,-322.05"/>
<text xml:space="preserve" text-anchor="middle" x="678.52" y="-320.07" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">publish events</text>
</g>
<!-- minio -->
<g id="node9" class="node">
<title>minio</title>
<path fill="#121829" stroke="#1e2a4a" d="M879.21,-272.69C879.21,-275.1 854.95,-277.06 825.08,-277.06 795.22,-277.06 770.96,-275.1 770.96,-272.69 770.96,-272.69 770.96,-233.31 770.96,-233.31 770.96,-230.9 795.22,-228.94 825.08,-228.94 854.95,-228.94 879.21,-230.9 879.21,-233.31 879.21,-233.31 879.21,-272.69 879.21,-272.69"/>
<path fill="none" stroke="#1e2a4a" d="M879.21,-272.69C879.21,-270.27 854.95,-268.31 825.08,-268.31 795.22,-268.31 770.96,-270.27 770.96,-272.69"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-256.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">MinIO</text>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-242.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">media · overlays</text>
</g>
<!-- api&#45;&gt;minio -->
<g id="edge8" class="edge">
<title>api&#45;&gt;minio</title>
<path fill="none" stroke="#4a5568" d="M619.05,-289.76C626.12,-287.56 633.34,-285.47 640.27,-283.75 679.36,-274.02 723.92,-266.48 759.37,-261.3"/>
<polygon fill="#4a5568" stroke="#4a5568" points="759.64,-264.8 769.04,-259.92 758.65,-257.87 759.64,-264.8"/>
<text xml:space="preserve" text-anchor="middle" x="678.52" y="-286.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">frames · overlays</text>
</g>
<!-- gpu -->
<g id="node10" class="node">
<title>gpu</title>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="903.58,-169.25 746.58,-169.25 746.58,-120.75 903.58,-120.75 903.58,-169.25"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-154.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">inference server</text>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-141.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">YOLO · OCR · VLM</text>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-127.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">edge · field segmentation</text>
</g>
<!-- api&#45;&gt;gpu -->
<g id="edge9" class="edge">
<title>api&#45;&gt;gpu</title>
<path fill="none" stroke="#00c853" d="M612.14,-288.56C615.81,-285.99 619.26,-283.14 622.27,-280 635.32,-266.35 627.22,-255.16 640.27,-241.5 668.36,-212.08 707.41,-189.85 742.29,-174.18"/>
<polygon fill="#00c853" stroke="#00c853" points="743.33,-177.54 751.1,-170.34 740.53,-171.13 743.33,-177.54"/>
<text xml:space="preserve" text-anchor="middle" x="678.52" y="-255.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">HTTP</text>
<text xml:space="preserve" text-anchor="middle" x="678.52" y="-244.2" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">INFERENCE_URL</text>
</g>
<!-- cloud -->
<g id="node11" class="node">
<title>cloud</title>
<polygon fill="#243056" stroke="#1e2a4a" points="915.39,-31.9 915.39,-54.1 862.49,-69.79 787.67,-69.79 734.77,-54.1 734.77,-31.9 787.67,-16.21 862.49,-16.21 915.39,-31.9"/>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-46.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">Anthropic · Gemini</text>
<text xml:space="preserve" text-anchor="middle" x="825.08" y="-32.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">OpenAI · Groq</text>
</g>
<!-- api&#45;&gt;cloud -->
<g id="edge10" class="edge">
<title>api&#45;&gt;cloud</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M613.8,-288.76C616.99,-286.18 619.89,-283.27 622.27,-280 650.42,-241.25 615.31,-214.63 640.27,-173.75 665.29,-132.76 687.66,-136.87 726.77,-109 742.32,-97.91 759.51,-86.11 775.06,-75.6"/>
<polygon fill="#4a5568" stroke="#4a5568" points="776.9,-78.58 783.24,-70.09 772.99,-72.77 776.9,-78.58"/>
<text xml:space="preserve" text-anchor="middle" x="678.52" y="-176.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">VLM escalation</text>
</g>
<!-- redis&#45;&gt;api -->
<g id="edge11" class="edge">
<title>redis&#45;&gt;api</title>
<path fill="none" stroke="#8892a8" stroke-dasharray="5,2" d="M780.44,-309.86C761.04,-306.25 737.86,-302.53 716.77,-300.75 682.89,-297.89 674.23,-299.23 640.27,-300.75 637.08,-300.89 633.82,-301.07 630.53,-301.28"/>
<polygon fill="#8892a8" stroke="#8892a8" points="630.54,-297.77 620.81,-301.97 631.03,-304.75 630.54,-297.77"/>
<text xml:space="preserve" text-anchor="middle" x="678.52" y="-303.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">SSE consumer</text>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

94
docs/architecture/01a-local-architecture.dot
View File

@@ -1,94 +0,0 @@
digraph local_architecture {
rankdir=TB
node [shape=box, style=rounded, fontname="Helvetica"]
edge [fontname="Helvetica", fontsize=10]
labelloc="t"
label="MPR - Local Architecture (Celery + MinIO)"
fontsize=16
fontname="Helvetica-Bold"
graph [splines=ortho, nodesep=0.8, ranksep=0.8]
// External
subgraph cluster_external {
label="External"
style=dashed
color=gray
browser [label="Browser\nmpr.local.ar", shape=ellipse]
}
// Nginx reverse proxy
subgraph cluster_proxy {
label="Reverse Proxy"
style=filled
fillcolor="#e8f4f8"
nginx [label="nginx\nport 80"]
}
// Application layer
subgraph cluster_apps {
label="Application Layer"
style=filled
fillcolor="#f0f8e8"
django [label="Django Admin\n/admin\nport 8701"]
fastapi [label="GraphQL API\n/graphql\nport 8702"]
timeline [label="Timeline UI\n/\nport 5173"]
}
// Worker layer
subgraph cluster_workers {
label="Worker Layer"
style=filled
fillcolor="#fff8e8"
grpc_server [label="gRPC Server\nport 50051"]
celery [label="Celery Worker\nFFmpeg transcoding"]
}
// Data layer
subgraph cluster_data {
label="Data Layer"
style=filled
fillcolor="#f8e8f0"
postgres [label="PostgreSQL\nport 5436", shape=cylinder]
redis [label="Redis\nCelery queue\nport 6381", shape=cylinder]
}
// Storage
subgraph cluster_storage {
label="S3 Storage (MinIO)"
style=filled
fillcolor="#f0f0f0"
minio [label="MinIO\nS3-compatible API\nport 9000", shape=folder]
bucket_in [label="mpr-media-in/\ninput videos", shape=note]
bucket_out [label="mpr-media-out/\ntranscoded output", shape=note]
}
// Connections
browser -> nginx [label="HTTP"]
nginx -> django [xlabel="/admin"]
nginx -> fastapi [xlabel="/graphql"]
nginx -> timeline [xlabel="/"]
nginx -> minio [xlabel="/media/*"]
timeline -> fastapi [label="GraphQL"]
django -> postgres
fastapi -> postgres [label="read/write jobs"]
fastapi -> grpc_server [label="gRPC\nprogress updates"]
grpc_server -> celery [label="dispatch tasks"]
celery -> redis [label="task queue"]
celery -> postgres [label="update job status"]
celery -> minio [label="S3 API\ndownload input\nupload output"]
minio -> bucket_in [style=dotted, arrowhead=none]
minio -> bucket_out [style=dotted, arrowhead=none]
}

242
docs/architecture/01a-local-architecture.svg
View File

@@ -1,242 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: local_architecture Pages: 1 -->
<svg width="667pt" height="1095pt"
viewBox="0.00 0.00 667.00 1095.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 1090.76)">
<title>local_architecture</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-1090.76 663,-1090.76 663,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="329.5" y="-1067.56" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">MPR &#45; Local Architecture (Celery + MinIO)</text>
<g id="clust1" class="cluster">
<title>cluster_external</title>
<polygon fill="none" stroke="gray" stroke-dasharray="5,2" points="270,-947.66 270,-1051.26 424,-1051.26 424,-947.66 270,-947.66"/>
<text xml:space="preserve" text-anchor="middle" x="347" y="-1032.06" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">External</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_proxy</title>
<polygon fill="#e8f4f8" stroke="black" points="274,-819.91 274,-905.91 420,-905.91 420,-819.91 274,-819.91"/>
<text xml:space="preserve" text-anchor="middle" x="347" y="-886.71" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Reverse Proxy</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_apps</title>
<polygon fill="#f0f8e8" stroke="black" points="19,-556.16 19,-789.91 301,-789.91 301,-556.16 19,-556.16"/>
<text xml:space="preserve" text-anchor="middle" x="160" y="-770.71" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Application Layer</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_workers</title>
<polygon fill="#fff8e8" stroke="black" points="193,-302.41 193,-501.66 369,-501.66 369,-302.41 193,-302.41"/>
<text xml:space="preserve" text-anchor="middle" x="281" y="-482.46" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Worker Layer</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_data</title>
<polygon fill="#f8e8f0" stroke="black" points="8,-109.5 8,-235.16 286,-235.16 286,-109.5 8,-109.5"/>
<text xml:space="preserve" text-anchor="middle" x="147" y="-215.96" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Data Layer</text>
</g>
<g id="clust6" class="cluster">
<title>cluster_storage</title>
<polygon fill="#f0f0f0" stroke="black" points="319,-8 319,-223.95 651,-223.95 651,-8 319,-8"/>
<text xml:space="preserve" text-anchor="middle" x="485" y="-204.75" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">S3 Storage (MinIO)</text>
</g>
<!-- browser -->
<g id="node1" class="node">
<title>browser</title>
<ellipse fill="none" stroke="black" cx="347" cy="-985.71" rx="69.12" ry="30.05"/>
<text xml:space="preserve" text-anchor="middle" x="347" y="-989.66" font-family="Helvetica,sans-Serif" font-size="14.00">Browser</text>
<text xml:space="preserve" text-anchor="middle" x="347" y="-972.41" font-family="Helvetica,sans-Serif" font-size="14.00">mpr.local.ar</text>
</g>
<!-- nginx -->
<g id="node2" class="node">
<title>nginx</title>
<path fill="none" stroke="black" d="M368.5,-870.41C368.5,-870.41 325.5,-870.41 325.5,-870.41 319.5,-870.41 313.5,-864.41 313.5,-858.41 313.5,-858.41 313.5,-839.91 313.5,-839.91 313.5,-833.91 319.5,-827.91 325.5,-827.91 325.5,-827.91 368.5,-827.91 368.5,-827.91 374.5,-827.91 380.5,-833.91 380.5,-839.91 380.5,-839.91 380.5,-858.41 380.5,-858.41 380.5,-864.41 374.5,-870.41 368.5,-870.41"/>
<text xml:space="preserve" text-anchor="middle" x="347" y="-853.11" font-family="Helvetica,sans-Serif" font-size="14.00">nginx</text>
<text xml:space="preserve" text-anchor="middle" x="347" y="-835.86" font-family="Helvetica,sans-Serif" font-size="14.00">port 80</text>
</g>
<!-- browser&#45;&gt;nginx -->
<g id="edge1" class="edge">
<title>browser&#45;&gt;nginx</title>
<path fill="none" stroke="black" d="M347,-955.4C347,-955.4 347,-882.41 347,-882.41"/>
<polygon fill="black" stroke="black" points="350.5,-882.41 347,-872.41 343.5,-882.41 350.5,-882.41"/>
<text xml:space="preserve" text-anchor="middle" x="359.75" y="-917.16" font-family="Helvetica,sans-Serif" font-size="10.00">HTTP</text>
</g>
<!-- django -->
<g id="node3" class="node">
<title>django</title>
<path fill="none" stroke="black" d="M128.75,-754.41C128.75,-754.41 39.25,-754.41 39.25,-754.41 33.25,-754.41 27.25,-748.41 27.25,-742.41 27.25,-742.41 27.25,-706.66 27.25,-706.66 27.25,-700.66 33.25,-694.66 39.25,-694.66 39.25,-694.66 128.75,-694.66 128.75,-694.66 134.75,-694.66 140.75,-700.66 140.75,-706.66 140.75,-706.66 140.75,-742.41 140.75,-742.41 140.75,-748.41 134.75,-754.41 128.75,-754.41"/>
<text xml:space="preserve" text-anchor="middle" x="84" y="-737.11" font-family="Helvetica,sans-Serif" font-size="14.00">Django Admin</text>
<text xml:space="preserve" text-anchor="middle" x="84" y="-719.86" font-family="Helvetica,sans-Serif" font-size="14.00">/admin</text>
<text xml:space="preserve" text-anchor="middle" x="84" y="-702.61" font-family="Helvetica,sans-Serif" font-size="14.00">port 8701</text>
</g>
<!-- nginx&#45;&gt;django -->
<g id="edge2" class="edge">
<title>nginx&#45;&gt;django</title>
<path fill="none" stroke="black" d="M313.16,-856C242.12,-856 84,-856 84,-856 84,-856 84,-766.21 84,-766.21"/>
<polygon fill="black" stroke="black" points="87.5,-766.21 84,-756.21 80.5,-766.21 87.5,-766.21"/>
<text xml:space="preserve" text-anchor="middle" x="136.81" y="-859.25" font-family="Helvetica,sans-Serif" font-size="10.00">/admin</text>
</g>
<!-- fastapi -->
<g id="node4" class="node">
<title>fastapi</title>
<path fill="none" stroke="black" d="M281.25,-623.91C281.25,-623.91 200.75,-623.91 200.75,-623.91 194.75,-623.91 188.75,-617.91 188.75,-611.91 188.75,-611.91 188.75,-576.16 188.75,-576.16 188.75,-570.16 194.75,-564.16 200.75,-564.16 200.75,-564.16 281.25,-564.16 281.25,-564.16 287.25,-564.16 293.25,-570.16 293.25,-576.16 293.25,-576.16 293.25,-611.91 293.25,-611.91 293.25,-617.91 287.25,-623.91 281.25,-623.91"/>
<text xml:space="preserve" text-anchor="middle" x="241" y="-606.61" font-family="Helvetica,sans-Serif" font-size="14.00">GraphQL API</text>
<text xml:space="preserve" text-anchor="middle" x="241" y="-589.36" font-family="Helvetica,sans-Serif" font-size="14.00">/graphql</text>
<text xml:space="preserve" text-anchor="middle" x="241" y="-572.11" font-family="Helvetica,sans-Serif" font-size="14.00">port 8702</text>
</g>
<!-- nginx&#45;&gt;fastapi -->
<g id="edge3" class="edge">
<title>nginx&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M337.06,-827.84C337.06,-766.52 337.06,-594 337.06,-594 337.06,-594 305.04,-594 305.04,-594"/>
<polygon fill="black" stroke="black" points="305.04,-590.5 295.04,-594 305.04,-597.5 305.04,-590.5"/>
<text xml:space="preserve" text-anchor="middle" x="317.19" y="-698.16" font-family="Helvetica,sans-Serif" font-size="10.00">/graphql</text>
</g>
<!-- timeline -->
<g id="node5" class="node">
<title>timeline</title>
<path fill="none" stroke="black" d="M281,-754.41C281,-754.41 211,-754.41 211,-754.41 205,-754.41 199,-748.41 199,-742.41 199,-742.41 199,-706.66 199,-706.66 199,-700.66 205,-694.66 211,-694.66 211,-694.66 281,-694.66 281,-694.66 287,-694.66 293,-700.66 293,-706.66 293,-706.66 293,-742.41 293,-742.41 293,-748.41 287,-754.41 281,-754.41"/>
<text xml:space="preserve" text-anchor="middle" x="246" y="-737.11" font-family="Helvetica,sans-Serif" font-size="14.00">Timeline UI</text>
<text xml:space="preserve" text-anchor="middle" x="246" y="-719.86" font-family="Helvetica,sans-Serif" font-size="14.00">/</text>
<text xml:space="preserve" text-anchor="middle" x="246" y="-702.61" font-family="Helvetica,sans-Serif" font-size="14.00">port 5173</text>
</g>
<!-- nginx&#45;&gt;timeline -->
<g id="edge4" class="edge">
<title>nginx&#45;&gt;timeline</title>
<path fill="none" stroke="black" d="M313.34,-842C298.97,-842 285.44,-842 285.44,-842 285.44,-842 285.44,-766.3 285.44,-766.3"/>
<polygon fill="black" stroke="black" points="288.94,-766.3 285.44,-756.3 281.94,-766.3 288.94,-766.3"/>
<text xml:space="preserve" text-anchor="middle" x="283.94" y="-821.35" font-family="Helvetica,sans-Serif" font-size="10.00">/</text>
</g>
<!-- minio -->
<g id="node10" class="node">
<title>minio</title>
<polygon fill="none" stroke="black" points="486.38,-188.45 483.38,-192.45 462.38,-192.45 459.38,-188.45 343.62,-188.45 343.62,-128.7 486.38,-128.7 486.38,-188.45"/>
<text xml:space="preserve" text-anchor="middle" x="415" y="-171.15" font-family="Helvetica,sans-Serif" font-size="14.00">MinIO</text>
<text xml:space="preserve" text-anchor="middle" x="415" y="-153.9" font-family="Helvetica,sans-Serif" font-size="14.00">S3&#45;compatible API</text>
<text xml:space="preserve" text-anchor="middle" x="415" y="-136.65" font-family="Helvetica,sans-Serif" font-size="14.00">port 9000</text>
</g>
<!-- nginx&#45;&gt;minio -->
<g id="edge5" class="edge">
<title>nginx&#45;&gt;minio</title>
<path fill="none" stroke="black" d="M370.56,-827.73C370.56,-827.73 370.56,-200.13 370.56,-200.13"/>
<polygon fill="black" stroke="black" points="374.06,-200.13 370.56,-190.13 367.06,-200.13 374.06,-200.13"/>
<text xml:space="preserve" text-anchor="middle" x="391.56" y="-517.18" font-family="Helvetica,sans-Serif" font-size="10.00">/media/*</text>
</g>
<!-- postgres -->
<g id="node8" class="node">
<title>postgres</title>
<path fill="none" stroke="black" d="M111.75,-182.48C111.75,-185.42 90.35,-187.8 64,-187.8 37.65,-187.8 16.25,-185.42 16.25,-182.48 16.25,-182.48 16.25,-134.67 16.25,-134.67 16.25,-131.74 37.65,-129.36 64,-129.36 90.35,-129.36 111.75,-131.74 111.75,-134.67 111.75,-134.67 111.75,-182.48 111.75,-182.48"/>
<path fill="none" stroke="black" d="M111.75,-182.48C111.75,-179.55 90.35,-177.17 64,-177.17 37.65,-177.17 16.25,-179.55 16.25,-182.48"/>
<text xml:space="preserve" text-anchor="middle" x="64" y="-162.53" font-family="Helvetica,sans-Serif" font-size="14.00">PostgreSQL</text>
<text xml:space="preserve" text-anchor="middle" x="64" y="-145.28" font-family="Helvetica,sans-Serif" font-size="14.00">port 5436</text>
</g>
<!-- django&#45;&gt;postgres -->
<g id="edge7" class="edge">
<title>django&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M48.38,-694.5C48.38,-694.5 48.38,-199.71 48.38,-199.71"/>
<polygon fill="black" stroke="black" points="51.88,-199.71 48.38,-189.71 44.88,-199.71 51.88,-199.71"/>
</g>
<!-- grpc_server -->
<g id="node6" class="node">
<title>grpc_server</title>
<path fill="none" stroke="black" d="M301.5,-466.16C301.5,-466.16 222.5,-466.16 222.5,-466.16 216.5,-466.16 210.5,-460.16 210.5,-454.16 210.5,-454.16 210.5,-435.66 210.5,-435.66 210.5,-429.66 216.5,-423.66 222.5,-423.66 222.5,-423.66 301.5,-423.66 301.5,-423.66 307.5,-423.66 313.5,-429.66 313.5,-435.66 313.5,-435.66 313.5,-454.16 313.5,-454.16 313.5,-460.16 307.5,-466.16 301.5,-466.16"/>
<text xml:space="preserve" text-anchor="middle" x="262" y="-448.86" font-family="Helvetica,sans-Serif" font-size="14.00">gRPC Server</text>
<text xml:space="preserve" text-anchor="middle" x="262" y="-431.61" font-family="Helvetica,sans-Serif" font-size="14.00">port 50051</text>
</g>
<!-- fastapi&#45;&gt;grpc_server -->
<g id="edge9" class="edge">
<title>fastapi&#45;&gt;grpc_server</title>
<path fill="none" stroke="black" d="M251.88,-563.85C251.88,-563.85 251.88,-477.88 251.88,-477.88"/>
<polygon fill="black" stroke="black" points="255.38,-477.88 251.88,-467.88 248.38,-477.88 255.38,-477.88"/>
<text xml:space="preserve" text-anchor="middle" x="292" y="-525.66" font-family="Helvetica,sans-Serif" font-size="10.00">gRPC</text>
<text xml:space="preserve" text-anchor="middle" x="292" y="-512.91" font-family="Helvetica,sans-Serif" font-size="10.00">progress updates</text>
</g>
<!-- fastapi&#45;&gt;postgres -->
<g id="edge8" class="edge">
<title>fastapi&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M188.61,-594C138.18,-594 69.5,-594 69.5,-594 69.5,-594 69.5,-199.68 69.5,-199.68"/>
<polygon fill="black" stroke="black" points="73,-199.68 69.5,-189.68 66,-199.68 73,-199.68"/>
<text xml:space="preserve" text-anchor="middle" x="82.38" y="-385.16" font-family="Helvetica,sans-Serif" font-size="10.00">read/write jobs</text>
</g>
<!-- timeline&#45;&gt;fastapi -->
<g id="edge6" class="edge">
<title>timeline&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M246,-694.26C246,-694.26 246,-635.65 246,-635.65"/>
<polygon fill="black" stroke="black" points="249.5,-635.65 246,-625.65 242.5,-635.65 249.5,-635.65"/>
<text xml:space="preserve" text-anchor="middle" x="264" y="-656.16" font-family="Helvetica,sans-Serif" font-size="10.00">GraphQL</text>
</g>
<!-- celery -->
<g id="node7" class="node">
<title>celery</title>
<path fill="none" stroke="black" d="M348.62,-352.91C348.62,-352.91 213.38,-352.91 213.38,-352.91 207.38,-352.91 201.38,-346.91 201.38,-340.91 201.38,-340.91 201.38,-322.41 201.38,-322.41 201.38,-316.41 207.38,-310.41 213.38,-310.41 213.38,-310.41 348.62,-310.41 348.62,-310.41 354.62,-310.41 360.62,-316.41 360.62,-322.41 360.62,-322.41 360.62,-340.91 360.62,-340.91 360.62,-346.91 354.62,-352.91 348.62,-352.91"/>
<text xml:space="preserve" text-anchor="middle" x="281" y="-335.61" font-family="Helvetica,sans-Serif" font-size="14.00">Celery Worker</text>
<text xml:space="preserve" text-anchor="middle" x="281" y="-318.36" font-family="Helvetica,sans-Serif" font-size="14.00">FFmpeg transcoding</text>
</g>
<!-- grpc_server&#45;&gt;celery -->
<g id="edge10" class="edge">
<title>grpc_server&#45;&gt;celery</title>
<path fill="none" stroke="black" d="M262,-423.34C262,-423.34 262,-364.66 262,-364.66"/>
<polygon fill="black" stroke="black" points="265.5,-364.66 262,-354.66 258.5,-364.66 265.5,-364.66"/>
<text xml:space="preserve" text-anchor="middle" x="305.25" y="-385.16" font-family="Helvetica,sans-Serif" font-size="10.00">dispatch tasks</text>
</g>
<!-- celery&#45;&gt;postgres -->
<g id="edge12" class="edge">
<title>celery&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M201.09,-332C148.99,-332 90.62,-332 90.62,-332 90.62,-332 90.62,-199.51 90.62,-199.51"/>
<polygon fill="black" stroke="black" points="94.13,-199.51 90.63,-189.51 87.13,-199.51 94.13,-199.51"/>
<text xml:space="preserve" text-anchor="middle" x="181.38" y="-259.16" font-family="Helvetica,sans-Serif" font-size="10.00">update job status</text>
</g>
<!-- redis -->
<g id="node9" class="node">
<title>redis</title>
<path fill="none" stroke="black" d="M278.12,-192.19C278.12,-196.31 253.87,-199.66 224,-199.66 194.13,-199.66 169.88,-196.31 169.88,-192.19 169.88,-192.19 169.88,-124.97 169.88,-124.97 169.88,-120.85 194.13,-117.5 224,-117.5 253.87,-117.5 278.12,-120.85 278.12,-124.97 278.12,-124.97 278.12,-192.19 278.12,-192.19"/>
<path fill="none" stroke="black" d="M278.12,-192.19C278.12,-188.07 253.87,-184.72 224,-184.72 194.13,-184.72 169.88,-188.07 169.88,-192.19"/>
<text xml:space="preserve" text-anchor="middle" x="224" y="-171.15" font-family="Helvetica,sans-Serif" font-size="14.00">Redis</text>
<text xml:space="preserve" text-anchor="middle" x="224" y="-153.9" font-family="Helvetica,sans-Serif" font-size="14.00">Celery queue</text>
<text xml:space="preserve" text-anchor="middle" x="224" y="-136.65" font-family="Helvetica,sans-Serif" font-size="14.00">port 6381</text>
</g>
<!-- celery&#45;&gt;redis -->
<g id="edge11" class="edge">
<title>celery&#45;&gt;redis</title>
<path fill="none" stroke="black" d="M239.75,-310.09C239.75,-310.09 239.75,-211.49 239.75,-211.49"/>
<polygon fill="black" stroke="black" points="243.25,-211.49 239.75,-201.49 236.25,-211.49 243.25,-211.49"/>
<text xml:space="preserve" text-anchor="middle" x="314" y="-259.16" font-family="Helvetica,sans-Serif" font-size="10.00">task queue</text>
</g>
<!-- celery&#45;&gt;minio -->
<g id="edge13" class="edge">
<title>celery&#45;&gt;minio</title>
<path fill="none" stroke="black" d="M352.12,-310.09C352.12,-310.09 352.12,-200.39 352.12,-200.39"/>
<polygon fill="black" stroke="black" points="355.63,-200.39 352.13,-190.39 348.63,-200.39 355.63,-200.39"/>
<text xml:space="preserve" text-anchor="middle" x="441.5" y="-271.91" font-family="Helvetica,sans-Serif" font-size="10.00">S3 API</text>
<text xml:space="preserve" text-anchor="middle" x="441.5" y="-259.16" font-family="Helvetica,sans-Serif" font-size="10.00">download input</text>
<text xml:space="preserve" text-anchor="middle" x="441.5" y="-246.41" font-family="Helvetica,sans-Serif" font-size="10.00">upload output</text>
</g>
<!-- bucket_in -->
<g id="node11" class="node">
<title>bucket_in</title>
<polygon fill="none" stroke="black" points="434.75,-58.5 327.25,-58.5 327.25,-16 440.75,-16 440.75,-52.5 434.75,-58.5"/>
<polyline fill="none" stroke="black" points="434.75,-58.5 434.75,-52.5"/>
<polyline fill="none" stroke="black" points="440.75,-52.5 434.75,-52.5"/>
<text xml:space="preserve" text-anchor="middle" x="384" y="-41.2" font-family="Helvetica,sans-Serif" font-size="14.00">mpr&#45;media&#45;in/</text>
<text xml:space="preserve" text-anchor="middle" x="384" y="-23.95" font-family="Helvetica,sans-Serif" font-size="14.00">input videos</text>
</g>
<!-- minio&#45;&gt;bucket_in -->
<g id="edge14" class="edge">
<title>minio&#45;&gt;bucket_in</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M392.19,-128.27C392.19,-106.66 392.19,-78.11 392.19,-58.79"/>
</g>
<!-- bucket_out -->
<g id="node12" class="node">
<title>bucket_out</title>
<polygon fill="none" stroke="black" points="637.12,-58.5 498.88,-58.5 498.88,-16 643.12,-16 643.12,-52.5 637.12,-58.5"/>
<polyline fill="none" stroke="black" points="637.12,-58.5 637.12,-52.5"/>
<polyline fill="none" stroke="black" points="643.12,-52.5 637.12,-52.5"/>
<text xml:space="preserve" text-anchor="middle" x="571" y="-41.2" font-family="Helvetica,sans-Serif" font-size="14.00">mpr&#45;media&#45;out/</text>
<text xml:space="preserve" text-anchor="middle" x="571" y="-23.95" font-family="Helvetica,sans-Serif" font-size="14.00">transcoded output</text>
</g>
<!-- minio&#45;&gt;bucket_out -->
<g id="edge15" class="edge">
<title>minio&#45;&gt;bucket_out</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M463.56,-128.21C463.56,-92.2 463.56,-37 463.56,-37 463.56,-37 479.15,-37 498.44,-37"/>
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 18 KiB

85
docs/architecture/01b-aws-architecture.dot
View File

@@ -1,85 +0,0 @@
digraph aws_architecture {
rankdir=TB
node [shape=box, style=rounded, fontname="Helvetica"]
edge [fontname="Helvetica", fontsize=10]
labelloc="t"
label="MPR - AWS Architecture (Lambda + Step Functions)"
fontsize=16
fontname="Helvetica-Bold"
graph [splines=ortho, nodesep=0.8, ranksep=0.8]
// External
subgraph cluster_external {
label="External"
style=dashed
color=gray
browser [label="Browser\nmpr.mcrn.ar", shape=ellipse]
}
// Nginx reverse proxy
subgraph cluster_proxy {
label="Reverse Proxy"
style=filled
fillcolor="#e8f4f8"
nginx [label="nginx\nport 80"]
}
// Application layer
subgraph cluster_apps {
label="Application Layer"
style=filled
fillcolor="#f0f8e8"
django [label="Django Admin\n/admin\nport 8701"]
fastapi [label="GraphQL API\n/graphql\nport 8702"]
timeline [label="Timeline UI\n/\nport 5173"]
}
// Data layer (still local)
subgraph cluster_data {
label="Data Layer"
style=filled
fillcolor="#f8e8f0"
postgres [label="PostgreSQL\nport 5436", shape=cylinder]
}
// AWS layer
subgraph cluster_aws {
label="AWS Cloud"
style=filled
fillcolor="#fde8d0"
step_functions [label="Step Functions\nOrchestration\nstate machine"]
lambda [label="Lambda Function\nFFmpeg container\ntranscoding"]
s3 [label="S3 Buckets", shape=folder]
bucket_in [label="mpr-media-in/\ninput videos", shape=note]
bucket_out [label="mpr-media-out/\ntranscoded output", shape=note]
}
// Connections
browser -> nginx [label="HTTP"]
nginx -> django [xlabel="/admin"]
nginx -> fastapi [xlabel="/graphql"]
nginx -> timeline [xlabel="/"]
timeline -> fastapi [label="GraphQL"]
django -> postgres
fastapi -> postgres [label="read/write jobs"]
fastapi -> step_functions [label="boto3\nstart_execution()\nexecution_arn"]
step_functions -> lambda [label="invoke with\njob parameters"]
lambda -> s3 [label="download input\nupload output"]
lambda -> fastapi [label="POST /jobs/{id}/callback\nupdate status"]
fastapi -> postgres [label="callback updates\njob status"]
s3 -> bucket_in [style=dotted, arrowhead=none]
s3 -> bucket_out [style=dotted, arrowhead=none]
}

224
docs/architecture/01b-aws-architecture.svg
View File

@@ -1,224 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: aws_architecture Pages: 1 -->
<svg width="639pt" height="1081pt"
viewBox="0.00 0.00 639.00 1081.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 1077.35)">
<title>aws_architecture</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-1077.35 635.25,-1077.35 635.25,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="315.62" y="-1054.15" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">MPR &#45; AWS Architecture (Lambda + Step Functions)</text>
<g id="clust1" class="cluster">
<title>cluster_external</title>
<polygon fill="none" stroke="gray" stroke-dasharray="5,2" points="155,-934.25 155,-1037.85 315,-1037.85 315,-934.25 155,-934.25"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-1018.65" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">External</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_proxy</title>
<polygon fill="#e8f4f8" stroke="black" points="162,-806.5 162,-892.5 308,-892.5 308,-806.5 162,-806.5"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-873.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Reverse Proxy</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_apps</title>
<polygon fill="#f0f8e8" stroke="black" points="8,-542.75 8,-776.5 290,-776.5 290,-542.75 8,-542.75"/>
<text xml:space="preserve" text-anchor="middle" x="149" y="-757.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Application Layer</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_data</title>
<polygon fill="#f8e8f0" stroke="black" points="27,-372.91 27,-474.84 141,-474.84 141,-372.91 27,-372.91"/>
<text xml:space="preserve" text-anchor="middle" x="84" y="-455.64" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Data Layer</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_aws</title>
<polygon fill="#fde8d0" stroke="black" points="264,-8 264,-475.5 596,-475.5 596,-8 264,-8"/>
<text xml:space="preserve" text-anchor="middle" x="430" y="-456.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">AWS Cloud</text>
</g>
<!-- browser -->
<g id="node1" class="node">
<title>browser</title>
<ellipse fill="none" stroke="black" cx="235" cy="-972.3" rx="71.77" ry="30.05"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-976.25" font-family="Helvetica,sans-Serif" font-size="14.00">Browser</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-959" font-family="Helvetica,sans-Serif" font-size="14.00">mpr.mcrn.ar</text>
</g>
<!-- nginx -->
<g id="node2" class="node">
<title>nginx</title>
<path fill="none" stroke="black" d="M256.5,-857C256.5,-857 213.5,-857 213.5,-857 207.5,-857 201.5,-851 201.5,-845 201.5,-845 201.5,-826.5 201.5,-826.5 201.5,-820.5 207.5,-814.5 213.5,-814.5 213.5,-814.5 256.5,-814.5 256.5,-814.5 262.5,-814.5 268.5,-820.5 268.5,-826.5 268.5,-826.5 268.5,-845 268.5,-845 268.5,-851 262.5,-857 256.5,-857"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-839.7" font-family="Helvetica,sans-Serif" font-size="14.00">nginx</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-822.45" font-family="Helvetica,sans-Serif" font-size="14.00">port 80</text>
</g>
<!-- browser&#45;&gt;nginx -->
<g id="edge1" class="edge">
<title>browser&#45;&gt;nginx</title>
<path fill="none" stroke="black" d="M235,-942C235,-942 235,-869 235,-869"/>
<polygon fill="black" stroke="black" points="238.5,-869 235,-859 231.5,-869 238.5,-869"/>
<text xml:space="preserve" text-anchor="middle" x="247.75" y="-903.75" font-family="Helvetica,sans-Serif" font-size="10.00">HTTP</text>
</g>
<!-- django -->
<g id="node3" class="node">
<title>django</title>
<path fill="none" stroke="black" d="M117.75,-741C117.75,-741 28.25,-741 28.25,-741 22.25,-741 16.25,-735 16.25,-729 16.25,-729 16.25,-693.25 16.25,-693.25 16.25,-687.25 22.25,-681.25 28.25,-681.25 28.25,-681.25 117.75,-681.25 117.75,-681.25 123.75,-681.25 129.75,-687.25 129.75,-693.25 129.75,-693.25 129.75,-729 129.75,-729 129.75,-735 123.75,-741 117.75,-741"/>
<text xml:space="preserve" text-anchor="middle" x="73" y="-723.7" font-family="Helvetica,sans-Serif" font-size="14.00">Django Admin</text>
<text xml:space="preserve" text-anchor="middle" x="73" y="-706.45" font-family="Helvetica,sans-Serif" font-size="14.00">/admin</text>
<text xml:space="preserve" text-anchor="middle" x="73" y="-689.2" font-family="Helvetica,sans-Serif" font-size="14.00">port 8701</text>
</g>
<!-- nginx&#45;&gt;django -->
<g id="edge2" class="edge">
<title>nginx&#45;&gt;django</title>
<path fill="none" stroke="black" d="M201.04,-843C153.54,-843 73,-843 73,-843 73,-843 73,-752.89 73,-752.89"/>
<polygon fill="black" stroke="black" points="76.5,-752.89 73,-742.89 69.5,-752.89 76.5,-752.89"/>
<text xml:space="preserve" text-anchor="middle" x="75.09" y="-846.25" font-family="Helvetica,sans-Serif" font-size="10.00">/admin</text>
</g>
<!-- fastapi -->
<g id="node4" class="node">
<title>fastapi</title>
<path fill="none" stroke="black" d="M270.25,-610.5C270.25,-610.5 189.75,-610.5 189.75,-610.5 183.75,-610.5 177.75,-604.5 177.75,-598.5 177.75,-598.5 177.75,-562.75 177.75,-562.75 177.75,-556.75 183.75,-550.75 189.75,-550.75 189.75,-550.75 270.25,-550.75 270.25,-550.75 276.25,-550.75 282.25,-556.75 282.25,-562.75 282.25,-562.75 282.25,-598.5 282.25,-598.5 282.25,-604.5 276.25,-610.5 270.25,-610.5"/>
<text xml:space="preserve" text-anchor="middle" x="230" y="-593.2" font-family="Helvetica,sans-Serif" font-size="14.00">GraphQL API</text>
<text xml:space="preserve" text-anchor="middle" x="230" y="-575.95" font-family="Helvetica,sans-Serif" font-size="14.00">/graphql</text>
<text xml:space="preserve" text-anchor="middle" x="230" y="-558.7" font-family="Helvetica,sans-Serif" font-size="14.00">port 8702</text>
</g>
<!-- nginx&#45;&gt;fastapi -->
<g id="edge3" class="edge">
<title>nginx&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M201.11,-829C191.15,-829 182.88,-829 182.88,-829 182.88,-829 182.88,-622.1 182.88,-622.1"/>
<polygon fill="black" stroke="black" points="186.38,-622.1 182.88,-612.1 179.38,-622.1 186.38,-622.1"/>
<text xml:space="preserve" text-anchor="middle" x="163" y="-737.91" font-family="Helvetica,sans-Serif" font-size="10.00">/graphql</text>
</g>
<!-- timeline -->
<g id="node5" class="node">
<title>timeline</title>
<path fill="none" stroke="black" d="M270,-741C270,-741 200,-741 200,-741 194,-741 188,-735 188,-729 188,-729 188,-693.25 188,-693.25 188,-687.25 194,-681.25 200,-681.25 200,-681.25 270,-681.25 270,-681.25 276,-681.25 282,-687.25 282,-693.25 282,-693.25 282,-729 282,-729 282,-735 276,-741 270,-741"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-723.7" font-family="Helvetica,sans-Serif" font-size="14.00">Timeline UI</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-706.45" font-family="Helvetica,sans-Serif" font-size="14.00">/</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-689.2" font-family="Helvetica,sans-Serif" font-size="14.00">port 5173</text>
</g>
<!-- nginx&#45;&gt;timeline -->
<g id="edge4" class="edge">
<title>nginx&#45;&gt;timeline</title>
<path fill="none" stroke="black" d="M235,-814.04C235,-814.04 235,-752.97 235,-752.97"/>
<polygon fill="black" stroke="black" points="238.5,-752.97 235,-742.97 231.5,-752.97 238.5,-752.97"/>
<text xml:space="preserve" text-anchor="middle" x="233.5" y="-786.75" font-family="Helvetica,sans-Serif" font-size="10.00">/</text>
</g>
<!-- postgres -->
<g id="node6" class="node">
<title>postgres</title>
<path fill="none" stroke="black" d="M131.75,-434.03C131.75,-436.96 110.35,-439.34 84,-439.34 57.65,-439.34 36.25,-436.96 36.25,-434.03 36.25,-434.03 36.25,-386.22 36.25,-386.22 36.25,-383.29 57.65,-380.91 84,-380.91 110.35,-380.91 131.75,-383.29 131.75,-386.22 131.75,-386.22 131.75,-434.03 131.75,-434.03"/>
<path fill="none" stroke="black" d="M131.75,-434.03C131.75,-431.1 110.35,-428.72 84,-428.72 57.65,-428.72 36.25,-431.1 36.25,-434.03"/>
<text xml:space="preserve" text-anchor="middle" x="84" y="-414.07" font-family="Helvetica,sans-Serif" font-size="14.00">PostgreSQL</text>
<text xml:space="preserve" text-anchor="middle" x="84" y="-396.82" font-family="Helvetica,sans-Serif" font-size="14.00">port 5436</text>
</g>
<!-- django&#45;&gt;postgres -->
<g id="edge6" class="edge">
<title>django&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M83,-680.89C83,-680.89 83,-450.97 83,-450.97"/>
<polygon fill="black" stroke="black" points="86.5,-450.97 83,-440.97 79.5,-450.97 86.5,-450.97"/>
</g>
<!-- fastapi&#45;&gt;postgres -->
<g id="edge7" class="edge">
<title>fastapi&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M201.38,-550.41C201.38,-503.88 201.38,-420 201.38,-420 201.38,-420 143.59,-420 143.59,-420"/>
<polygon fill="black" stroke="black" points="143.59,-416.5 133.59,-420 143.59,-423.5 143.59,-416.5"/>
<text xml:space="preserve" text-anchor="middle" x="266.38" y="-499.5" font-family="Helvetica,sans-Serif" font-size="10.00">read/write jobs</text>
</g>
<!-- fastapi&#45;&gt;postgres -->
<g id="edge12" class="edge">
<title>fastapi&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M225,-550.39C225,-498.97 225,-400 225,-400 225,-400 143.64,-400 143.64,-400"/>
<polygon fill="black" stroke="black" points="143.64,-396.5 133.64,-400 143.64,-403.5 143.64,-396.5"/>
<text xml:space="preserve" text-anchor="middle" x="125.25" y="-505.88" font-family="Helvetica,sans-Serif" font-size="10.00">callback updates</text>
<text xml:space="preserve" text-anchor="middle" x="125.25" y="-493.12" font-family="Helvetica,sans-Serif" font-size="10.00">job status</text>
</g>
<!-- step_functions -->
<g id="node7" class="node">
<title>step_functions</title>
<path fill="none" stroke="black" d="M384.38,-440C384.38,-440 289.62,-440 289.62,-440 283.62,-440 277.62,-434 277.62,-428 277.62,-428 277.62,-392.25 277.62,-392.25 277.62,-386.25 283.62,-380.25 289.62,-380.25 289.62,-380.25 384.38,-380.25 384.38,-380.25 390.38,-380.25 396.38,-386.25 396.38,-392.25 396.38,-392.25 396.38,-428 396.38,-428 396.38,-434 390.38,-440 384.38,-440"/>
<text xml:space="preserve" text-anchor="middle" x="337" y="-422.7" font-family="Helvetica,sans-Serif" font-size="14.00">Step Functions</text>
<text xml:space="preserve" text-anchor="middle" x="337" y="-405.45" font-family="Helvetica,sans-Serif" font-size="14.00">Orchestration</text>
<text xml:space="preserve" text-anchor="middle" x="337" y="-388.2" font-family="Helvetica,sans-Serif" font-size="14.00">state machine</text>
</g>
<!-- fastapi&#45;&gt;step_functions -->
<g id="edge8" class="edge">
<title>fastapi&#45;&gt;step_functions</title>
<path fill="none" stroke="black" d="M282.68,-581C289.69,-581 294.51,-581 294.51,-581 294.51,-581 294.51,-451.79 294.51,-451.79"/>
<polygon fill="black" stroke="black" points="298.01,-451.79 294.51,-441.79 291.01,-451.79 298.01,-451.79"/>
<text xml:space="preserve" text-anchor="middle" x="407.25" y="-512.25" font-family="Helvetica,sans-Serif" font-size="10.00">boto3</text>
<text xml:space="preserve" text-anchor="middle" x="407.25" y="-499.5" font-family="Helvetica,sans-Serif" font-size="10.00">start_execution()</text>
<text xml:space="preserve" text-anchor="middle" x="407.25" y="-486.75" font-family="Helvetica,sans-Serif" font-size="10.00">execution_arn</text>
</g>
<!-- timeline&#45;&gt;fastapi -->
<g id="edge5" class="edge">
<title>timeline&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M235,-680.86C235,-680.86 235,-622.24 235,-622.24"/>
<polygon fill="black" stroke="black" points="238.5,-622.24 235,-612.24 231.5,-622.24 238.5,-622.24"/>
<text xml:space="preserve" text-anchor="middle" x="253" y="-642.75" font-family="Helvetica,sans-Serif" font-size="10.00">GraphQL</text>
</g>
<!-- lambda -->
<g id="node8" class="node">
<title>lambda</title>
<path fill="none" stroke="black" d="M486,-296.75C486,-296.75 368,-296.75 368,-296.75 362,-296.75 356,-290.75 356,-284.75 356,-284.75 356,-249 356,-249 356,-243 362,-237 368,-237 368,-237 486,-237 486,-237 492,-237 498,-243 498,-249 498,-249 498,-284.75 498,-284.75 498,-290.75 492,-296.75 486,-296.75"/>
<text xml:space="preserve" text-anchor="middle" x="427" y="-279.45" font-family="Helvetica,sans-Serif" font-size="14.00">Lambda Function</text>
<text xml:space="preserve" text-anchor="middle" x="427" y="-262.2" font-family="Helvetica,sans-Serif" font-size="14.00">FFmpeg container</text>
<text xml:space="preserve" text-anchor="middle" x="427" y="-244.95" font-family="Helvetica,sans-Serif" font-size="14.00">transcoding</text>
</g>
<!-- step_functions&#45;&gt;lambda -->
<g id="edge9" class="edge">
<title>step_functions&#45;&gt;lambda</title>
<path fill="none" stroke="black" d="M376.19,-380.1C376.19,-380.1 376.19,-308.38 376.19,-308.38"/>
<polygon fill="black" stroke="black" points="379.69,-308.38 376.19,-298.38 372.69,-308.38 379.69,-308.38"/>
<text xml:space="preserve" text-anchor="middle" x="407.12" y="-341.75" font-family="Helvetica,sans-Serif" font-size="10.00">invoke with</text>
<text xml:space="preserve" text-anchor="middle" x="407.12" y="-329" font-family="Helvetica,sans-Serif" font-size="10.00">job parameters</text>
</g>
<!-- lambda&#45;&gt;fastapi -->
<g id="edge11" class="edge">
<title>lambda&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M355.73,-267C306.05,-267 248.62,-267 248.62,-267 248.62,-267 248.62,-538.75 248.62,-538.75"/>
<polygon fill="black" stroke="black" points="245.13,-538.75 248.63,-548.75 252.13,-538.75 245.13,-538.75"/>
<text xml:space="preserve" text-anchor="middle" x="571.62" y="-413.38" font-family="Helvetica,sans-Serif" font-size="10.00">POST /jobs/{id}/callback</text>
<text xml:space="preserve" text-anchor="middle" x="571.62" y="-400.62" font-family="Helvetica,sans-Serif" font-size="10.00">update status</text>
</g>
<!-- s3 -->
<g id="node9" class="node">
<title>s3</title>
<polygon fill="none" stroke="black" points="473.62,-153.5 470.62,-157.5 449.62,-157.5 446.62,-153.5 380.38,-153.5 380.38,-117.5 473.62,-117.5 473.62,-153.5"/>
<text xml:space="preserve" text-anchor="middle" x="427" y="-130.82" font-family="Helvetica,sans-Serif" font-size="14.00">S3 Buckets</text>
</g>
<!-- lambda&#45;&gt;s3 -->
<g id="edge10" class="edge">
<title>lambda&#45;&gt;s3</title>
<path fill="none" stroke="black" d="M427,-236.73C427,-236.73 427,-165.27 427,-165.27"/>
<polygon fill="black" stroke="black" points="430.5,-165.27 427,-155.27 423.5,-165.27 430.5,-165.27"/>
<text xml:space="preserve" text-anchor="middle" x="464.5" y="-198.5" font-family="Helvetica,sans-Serif" font-size="10.00">download input</text>
<text xml:space="preserve" text-anchor="middle" x="464.5" y="-185.75" font-family="Helvetica,sans-Serif" font-size="10.00">upload output</text>
</g>
<!-- bucket_in -->
<g id="node10" class="node">
<title>bucket_in</title>
<polygon fill="none" stroke="black" points="379.75,-58.5 272.25,-58.5 272.25,-16 385.75,-16 385.75,-52.5 379.75,-58.5"/>
<polyline fill="none" stroke="black" points="379.75,-58.5 379.75,-52.5"/>
<polyline fill="none" stroke="black" points="385.75,-52.5 379.75,-52.5"/>
<text xml:space="preserve" text-anchor="middle" x="329" y="-41.2" font-family="Helvetica,sans-Serif" font-size="14.00">mpr&#45;media&#45;in/</text>
<text xml:space="preserve" text-anchor="middle" x="329" y="-23.95" font-family="Helvetica,sans-Serif" font-size="14.00">input videos</text>
</g>
<!-- s3&#45;&gt;bucket_in -->
<g id="edge13" class="edge">
<title>s3&#45;&gt;bucket_in</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M380.09,-136C373.1,-136 368.19,-136 368.19,-136 368.19,-136 368.19,-87.72 368.19,-58.68"/>
</g>
<!-- bucket_out -->
<g id="node11" class="node">
<title>bucket_out</title>
<polygon fill="none" stroke="black" points="582.12,-58.5 443.88,-58.5 443.88,-16 588.12,-16 588.12,-52.5 582.12,-58.5"/>
<polyline fill="none" stroke="black" points="582.12,-58.5 582.12,-52.5"/>
<polyline fill="none" stroke="black" points="588.12,-52.5 582.12,-52.5"/>
<text xml:space="preserve" text-anchor="middle" x="516" y="-41.2" font-family="Helvetica,sans-Serif" font-size="14.00">mpr&#45;media&#45;out/</text>
<text xml:space="preserve" text-anchor="middle" x="516" y="-23.95" font-family="Helvetica,sans-Serif" font-size="14.00">transcoded output</text>
</g>
<!-- s3&#45;&gt;bucket_out -->
<g id="edge14" class="edge">
<title>s3&#45;&gt;bucket_out</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M458.75,-117.02C458.75,-100.45 458.75,-76.15 458.75,-58.73"/>
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 16 KiB

83
docs/architecture/01c-gcp-architecture.dot
View File

@@ -1,83 +0,0 @@
digraph gcp_architecture {
rankdir=TB
node [shape=box, style=rounded, fontname="Helvetica"]
edge [fontname="Helvetica", fontsize=10]
labelloc="t"
label="MPR - GCP Architecture (Cloud Run Jobs + GCS)"
fontsize=16
fontname="Helvetica-Bold"
graph [splines=ortho, nodesep=0.8, ranksep=0.8]
// External
subgraph cluster_external {
label="External"
style=dashed
color=gray
browser [label="Browser\nmpr.mcrn.ar", shape=ellipse]
}
// Nginx reverse proxy
subgraph cluster_proxy {
label="Reverse Proxy"
style=filled
fillcolor="#e8f4f8"
nginx [label="nginx\nport 80"]
}
// Application layer
subgraph cluster_apps {
label="Application Layer"
style=filled
fillcolor="#f0f8e8"
django [label="Django Admin\n/admin\nport 8701"]
fastapi [label="GraphQL API\n/graphql\nport 8702"]
timeline [label="Timeline UI\n/\nport 5173"]
}
// Data layer (still local)
subgraph cluster_data {
label="Data Layer"
style=filled
fillcolor="#f8e8f0"
postgres [label="PostgreSQL\nport 5436", shape=cylinder]
}
// GCP layer
subgraph cluster_gcp {
label="Google Cloud"
style=filled
fillcolor="#e8f0fd"
cloud_run_job [label="Cloud Run Job\nFFmpeg container\ntranscoding"]
gcs [label="GCS Buckets\n(S3-compat API)", shape=folder]
bucket_in [label="mpr-media-in/\ninput videos", shape=note]
bucket_out [label="mpr-media-out/\ntranscoded output", shape=note]
}
// Connections
browser -> nginx [label="HTTP"]
nginx -> django [xlabel="/admin"]
nginx -> fastapi [xlabel="/graphql"]
nginx -> timeline [xlabel="/"]
timeline -> fastapi [label="GraphQL"]
django -> postgres
fastapi -> postgres [label="read/write jobs"]
fastapi -> cloud_run_job [label="google-cloud-run\nrun_job() + payload\nexecution_name"]
cloud_run_job -> gcs [label="S3 compat (HMAC)\ndownload input\nupload output"]
cloud_run_job -> fastapi [label="POST /jobs/{id}/callback\nupdate status"]
fastapi -> postgres [label="callback updates\njob status"]
gcs -> bucket_in [style=dotted, arrowhead=none]
gcs -> bucket_out [style=dotted, arrowhead=none]
}

210
docs/architecture/01c-gcp-architecture.svg
View File

@@ -1,210 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: gcp_architecture Pages: 1 -->
<svg width="653pt" height="957pt"
viewBox="0.00 0.00 653.00 957.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 953.35)">
<title>gcp_architecture</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-953.35 649.25,-953.35 649.25,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="322.62" y="-930.15" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">MPR &#45; GCP Architecture (Cloud Run Jobs + GCS)</text>
<g id="clust1" class="cluster">
<title>cluster_external</title>
<polygon fill="none" stroke="gray" stroke-dasharray="5,2" points="155,-810.25 155,-913.85 315,-913.85 315,-810.25 155,-810.25"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-894.65" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">External</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_proxy</title>
<polygon fill="#e8f4f8" stroke="black" points="162,-682.5 162,-768.5 308,-768.5 308,-682.5 162,-682.5"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-749.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Reverse Proxy</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_apps</title>
<polygon fill="#f0f8e8" stroke="black" points="8,-418.75 8,-652.5 290,-652.5 290,-418.75 8,-418.75"/>
<text xml:space="preserve" text-anchor="middle" x="149" y="-633.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Application Layer</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_data</title>
<polygon fill="#f8e8f0" stroke="black" points="27,-248.91 27,-350.84 141,-350.84 141,-248.91 27,-248.91"/>
<text xml:space="preserve" text-anchor="middle" x="84" y="-331.64" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Data Layer</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_gcp</title>
<polygon fill="#e8f0fd" stroke="black" points="299,-8 299,-351.5 631,-351.5 631,-8 299,-8"/>
<text xml:space="preserve" text-anchor="middle" x="465" y="-332.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Google Cloud</text>
</g>
<!-- browser -->
<g id="node1" class="node">
<title>browser</title>
<ellipse fill="none" stroke="black" cx="235" cy="-848.3" rx="71.77" ry="30.05"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-852.25" font-family="Helvetica,sans-Serif" font-size="14.00">Browser</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-835" font-family="Helvetica,sans-Serif" font-size="14.00">mpr.mcrn.ar</text>
</g>
<!-- nginx -->
<g id="node2" class="node">
<title>nginx</title>
<path fill="none" stroke="black" d="M256.5,-733C256.5,-733 213.5,-733 213.5,-733 207.5,-733 201.5,-727 201.5,-721 201.5,-721 201.5,-702.5 201.5,-702.5 201.5,-696.5 207.5,-690.5 213.5,-690.5 213.5,-690.5 256.5,-690.5 256.5,-690.5 262.5,-690.5 268.5,-696.5 268.5,-702.5 268.5,-702.5 268.5,-721 268.5,-721 268.5,-727 262.5,-733 256.5,-733"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-715.7" font-family="Helvetica,sans-Serif" font-size="14.00">nginx</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-698.45" font-family="Helvetica,sans-Serif" font-size="14.00">port 80</text>
</g>
<!-- browser&#45;&gt;nginx -->
<g id="edge1" class="edge">
<title>browser&#45;&gt;nginx</title>
<path fill="none" stroke="black" d="M235,-818C235,-818 235,-745 235,-745"/>
<polygon fill="black" stroke="black" points="238.5,-745 235,-735 231.5,-745 238.5,-745"/>
<text xml:space="preserve" text-anchor="middle" x="247.75" y="-779.75" font-family="Helvetica,sans-Serif" font-size="10.00">HTTP</text>
</g>
<!-- django -->
<g id="node3" class="node">
<title>django</title>
<path fill="none" stroke="black" d="M117.75,-617C117.75,-617 28.25,-617 28.25,-617 22.25,-617 16.25,-611 16.25,-605 16.25,-605 16.25,-569.25 16.25,-569.25 16.25,-563.25 22.25,-557.25 28.25,-557.25 28.25,-557.25 117.75,-557.25 117.75,-557.25 123.75,-557.25 129.75,-563.25 129.75,-569.25 129.75,-569.25 129.75,-605 129.75,-605 129.75,-611 123.75,-617 117.75,-617"/>
<text xml:space="preserve" text-anchor="middle" x="73" y="-599.7" font-family="Helvetica,sans-Serif" font-size="14.00">Django Admin</text>
<text xml:space="preserve" text-anchor="middle" x="73" y="-582.45" font-family="Helvetica,sans-Serif" font-size="14.00">/admin</text>
<text xml:space="preserve" text-anchor="middle" x="73" y="-565.2" font-family="Helvetica,sans-Serif" font-size="14.00">port 8701</text>
</g>
<!-- nginx&#45;&gt;django -->
<g id="edge2" class="edge">
<title>nginx&#45;&gt;django</title>
<path fill="none" stroke="black" d="M201.04,-719C153.54,-719 73,-719 73,-719 73,-719 73,-628.89 73,-628.89"/>
<polygon fill="black" stroke="black" points="76.5,-628.89 73,-618.89 69.5,-628.89 76.5,-628.89"/>
<text xml:space="preserve" text-anchor="middle" x="75.09" y="-722.25" font-family="Helvetica,sans-Serif" font-size="10.00">/admin</text>
</g>
<!-- fastapi -->
<g id="node4" class="node">
<title>fastapi</title>
<path fill="none" stroke="black" d="M270.25,-486.5C270.25,-486.5 189.75,-486.5 189.75,-486.5 183.75,-486.5 177.75,-480.5 177.75,-474.5 177.75,-474.5 177.75,-438.75 177.75,-438.75 177.75,-432.75 183.75,-426.75 189.75,-426.75 189.75,-426.75 270.25,-426.75 270.25,-426.75 276.25,-426.75 282.25,-432.75 282.25,-438.75 282.25,-438.75 282.25,-474.5 282.25,-474.5 282.25,-480.5 276.25,-486.5 270.25,-486.5"/>
<text xml:space="preserve" text-anchor="middle" x="230" y="-469.2" font-family="Helvetica,sans-Serif" font-size="14.00">GraphQL API</text>
<text xml:space="preserve" text-anchor="middle" x="230" y="-451.95" font-family="Helvetica,sans-Serif" font-size="14.00">/graphql</text>
<text xml:space="preserve" text-anchor="middle" x="230" y="-434.7" font-family="Helvetica,sans-Serif" font-size="14.00">port 8702</text>
</g>
<!-- nginx&#45;&gt;fastapi -->
<g id="edge3" class="edge">
<title>nginx&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M201.11,-705C191.15,-705 182.88,-705 182.88,-705 182.88,-705 182.88,-498.1 182.88,-498.1"/>
<polygon fill="black" stroke="black" points="186.38,-498.1 182.88,-488.1 179.38,-498.1 186.38,-498.1"/>
<text xml:space="preserve" text-anchor="middle" x="163" y="-613.91" font-family="Helvetica,sans-Serif" font-size="10.00">/graphql</text>
</g>
<!-- timeline -->
<g id="node5" class="node">
<title>timeline</title>
<path fill="none" stroke="black" d="M270,-617C270,-617 200,-617 200,-617 194,-617 188,-611 188,-605 188,-605 188,-569.25 188,-569.25 188,-563.25 194,-557.25 200,-557.25 200,-557.25 270,-557.25 270,-557.25 276,-557.25 282,-563.25 282,-569.25 282,-569.25 282,-605 282,-605 282,-611 276,-617 270,-617"/>
<text xml:space="preserve" text-anchor="middle" x="235" y="-599.7" font-family="Helvetica,sans-Serif" font-size="14.00">Timeline UI</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-582.45" font-family="Helvetica,sans-Serif" font-size="14.00">/</text>
<text xml:space="preserve" text-anchor="middle" x="235" y="-565.2" font-family="Helvetica,sans-Serif" font-size="14.00">port 5173</text>
</g>
<!-- nginx&#45;&gt;timeline -->
<g id="edge4" class="edge">
<title>nginx&#45;&gt;timeline</title>
<path fill="none" stroke="black" d="M235,-690.04C235,-690.04 235,-628.97 235,-628.97"/>
<polygon fill="black" stroke="black" points="238.5,-628.97 235,-618.97 231.5,-628.97 238.5,-628.97"/>
<text xml:space="preserve" text-anchor="middle" x="233.5" y="-662.75" font-family="Helvetica,sans-Serif" font-size="10.00">/</text>
</g>
<!-- postgres -->
<g id="node6" class="node">
<title>postgres</title>
<path fill="none" stroke="black" d="M131.75,-310.03C131.75,-312.96 110.35,-315.34 84,-315.34 57.65,-315.34 36.25,-312.96 36.25,-310.03 36.25,-310.03 36.25,-262.22 36.25,-262.22 36.25,-259.29 57.65,-256.91 84,-256.91 110.35,-256.91 131.75,-259.29 131.75,-262.22 131.75,-262.22 131.75,-310.03 131.75,-310.03"/>
<path fill="none" stroke="black" d="M131.75,-310.03C131.75,-307.1 110.35,-304.72 84,-304.72 57.65,-304.72 36.25,-307.1 36.25,-310.03"/>
<text xml:space="preserve" text-anchor="middle" x="84" y="-290.07" font-family="Helvetica,sans-Serif" font-size="14.00">PostgreSQL</text>
<text xml:space="preserve" text-anchor="middle" x="84" y="-272.82" font-family="Helvetica,sans-Serif" font-size="14.00">port 5436</text>
</g>
<!-- django&#45;&gt;postgres -->
<g id="edge6" class="edge">
<title>django&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M59.62,-556.89C59.62,-556.89 59.62,-326.97 59.62,-326.97"/>
<polygon fill="black" stroke="black" points="63.13,-326.97 59.63,-316.97 56.13,-326.97 63.13,-326.97"/>
</g>
<!-- fastapi&#45;&gt;postgres -->
<g id="edge7" class="edge">
<title>fastapi&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M177.34,-467C135.16,-467 83,-467 83,-467 83,-467 83,-327.1 83,-327.1"/>
<polygon fill="black" stroke="black" points="86.5,-327.1 83,-317.1 79.5,-327.1 86.5,-327.1"/>
<text xml:space="preserve" text-anchor="middle" x="266.38" y="-375.5" font-family="Helvetica,sans-Serif" font-size="10.00">read/write jobs</text>
</g>
<!-- fastapi&#45;&gt;postgres -->
<g id="edge11" class="edge">
<title>fastapi&#45;&gt;postgres</title>
<path fill="none" stroke="black" d="M177.57,-447C143.88,-447 106.38,-447 106.38,-447 106.38,-447 106.38,-327.15 106.38,-327.15"/>
<polygon fill="black" stroke="black" points="109.88,-327.15 106.38,-317.15 102.88,-327.15 109.88,-327.15"/>
<text xml:space="preserve" text-anchor="middle" x="125.25" y="-381.88" font-family="Helvetica,sans-Serif" font-size="10.00">callback updates</text>
<text xml:space="preserve" text-anchor="middle" x="125.25" y="-369.12" font-family="Helvetica,sans-Serif" font-size="10.00">job status</text>
</g>
<!-- cloud_run_job -->
<g id="node7" class="node">
<title>cloud_run_job</title>
<path fill="none" stroke="black" d="M505,-316C505,-316 387,-316 387,-316 381,-316 375,-310 375,-304 375,-304 375,-268.25 375,-268.25 375,-262.25 381,-256.25 387,-256.25 387,-256.25 505,-256.25 505,-256.25 511,-256.25 517,-262.25 517,-268.25 517,-268.25 517,-304 517,-304 517,-310 511,-316 505,-316"/>
<text xml:space="preserve" text-anchor="middle" x="446" y="-298.7" font-family="Helvetica,sans-Serif" font-size="14.00">Cloud Run Job</text>
<text xml:space="preserve" text-anchor="middle" x="446" y="-281.45" font-family="Helvetica,sans-Serif" font-size="14.00">FFmpeg container</text>
<text xml:space="preserve" text-anchor="middle" x="446" y="-264.2" font-family="Helvetica,sans-Serif" font-size="14.00">transcoding</text>
</g>
<!-- fastapi&#45;&gt;cloud_run_job -->
<g id="edge8" class="edge">
<title>fastapi&#45;&gt;cloud_run_job</title>
<path fill="none" stroke="black" d="M247.42,-426.41C247.42,-379.88 247.42,-296 247.42,-296 247.42,-296 363.07,-296 363.07,-296"/>
<polygon fill="black" stroke="black" points="363.07,-299.5 373.07,-296 363.07,-292.5 363.07,-299.5"/>
<text xml:space="preserve" text-anchor="middle" x="414.38" y="-388.25" font-family="Helvetica,sans-Serif" font-size="10.00">google&#45;cloud&#45;run</text>
<text xml:space="preserve" text-anchor="middle" x="414.38" y="-375.5" font-family="Helvetica,sans-Serif" font-size="10.00">run_job() + payload</text>
<text xml:space="preserve" text-anchor="middle" x="414.38" y="-362.75" font-family="Helvetica,sans-Serif" font-size="10.00">execution_name</text>
</g>
<!-- timeline&#45;&gt;fastapi -->
<g id="edge5" class="edge">
<title>timeline&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M235,-556.86C235,-556.86 235,-498.24 235,-498.24"/>
<polygon fill="black" stroke="black" points="238.5,-498.24 235,-488.24 231.5,-498.24 238.5,-498.24"/>
<text xml:space="preserve" text-anchor="middle" x="253" y="-518.75" font-family="Helvetica,sans-Serif" font-size="10.00">GraphQL</text>
</g>
<!-- cloud_run_job&#45;&gt;fastapi -->
<g id="edge10" class="edge">
<title>cloud_run_job&#45;&gt;fastapi</title>
<path fill="none" stroke="black" d="M374.7,-276C306.06,-276 212.58,-276 212.58,-276 212.58,-276 212.58,-414.88 212.58,-414.88"/>
<polygon fill="black" stroke="black" points="209.08,-414.88 212.58,-424.88 216.08,-414.88 209.08,-414.88"/>
<text xml:space="preserve" text-anchor="middle" x="585.62" y="-381.88" font-family="Helvetica,sans-Serif" font-size="10.00">POST /jobs/{id}/callback</text>
<text xml:space="preserve" text-anchor="middle" x="585.62" y="-369.12" font-family="Helvetica,sans-Serif" font-size="10.00">update status</text>
</g>
<!-- gcs -->
<g id="node8" class="node">
<title>gcs</title>
<polygon fill="none" stroke="black" points="510.25,-160 507.25,-164 486.25,-164 483.25,-160 381.75,-160 381.75,-117.5 510.25,-117.5 510.25,-160"/>
<text xml:space="preserve" text-anchor="middle" x="446" y="-142.7" font-family="Helvetica,sans-Serif" font-size="14.00">GCS Buckets</text>
<text xml:space="preserve" text-anchor="middle" x="446" y="-125.45" font-family="Helvetica,sans-Serif" font-size="14.00">(S3&#45;compat API)</text>
</g>
<!-- cloud_run_job&#45;&gt;gcs -->
<g id="edge9" class="edge">
<title>cloud_run_job&#45;&gt;gcs</title>
<path fill="none" stroke="black" d="M446,-255.95C446,-255.95 446,-171.81 446,-171.81"/>
<polygon fill="black" stroke="black" points="449.5,-171.81 446,-161.81 442.5,-171.81 449.5,-171.81"/>
<text xml:space="preserve" text-anchor="middle" x="492.12" y="-217.75" font-family="Helvetica,sans-Serif" font-size="10.00">S3 compat (HMAC)</text>
<text xml:space="preserve" text-anchor="middle" x="492.12" y="-205" font-family="Helvetica,sans-Serif" font-size="10.00">download input</text>
<text xml:space="preserve" text-anchor="middle" x="492.12" y="-192.25" font-family="Helvetica,sans-Serif" font-size="10.00">upload output</text>
</g>
<!-- bucket_in -->
<g id="node9" class="node">
<title>bucket_in</title>
<polygon fill="none" stroke="black" points="414.75,-58.5 307.25,-58.5 307.25,-16 420.75,-16 420.75,-52.5 414.75,-58.5"/>
<polyline fill="none" stroke="black" points="414.75,-58.5 414.75,-52.5"/>
<polyline fill="none" stroke="black" points="420.75,-52.5 414.75,-52.5"/>
<text xml:space="preserve" text-anchor="middle" x="364" y="-41.2" font-family="Helvetica,sans-Serif" font-size="14.00">mpr&#45;media&#45;in/</text>
<text xml:space="preserve" text-anchor="middle" x="364" y="-23.95" font-family="Helvetica,sans-Serif" font-size="14.00">input videos</text>
</g>
<!-- gcs&#45;&gt;bucket_in -->
<g id="edge12" class="edge">
<title>gcs&#45;&gt;bucket_in</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M401.25,-117.22C401.25,-100 401.25,-75.96 401.25,-58.74"/>
</g>
<!-- bucket_out -->
<g id="node10" class="node">
<title>bucket_out</title>
<polygon fill="none" stroke="black" points="617.12,-58.5 478.88,-58.5 478.88,-16 623.12,-16 623.12,-52.5 617.12,-58.5"/>
<polyline fill="none" stroke="black" points="617.12,-58.5 617.12,-52.5"/>
<polyline fill="none" stroke="black" points="623.12,-52.5 617.12,-52.5"/>
<text xml:space="preserve" text-anchor="middle" x="551" y="-41.2" font-family="Helvetica,sans-Serif" font-size="14.00">mpr&#45;media&#45;out/</text>
<text xml:space="preserve" text-anchor="middle" x="551" y="-23.95" font-family="Helvetica,sans-Serif" font-size="14.00">transcoded output</text>
</g>
<!-- gcs&#45;&gt;bucket_out -->
<g id="edge13" class="edge">
<title>gcs&#45;&gt;bucket_out</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M494.56,-117.22C494.56,-100 494.56,-75.96 494.56,-58.74"/>
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 15 KiB

101
docs/architecture/02-data-model.dot
View File

@@ -1,22 +1,99 @@
digraph data_model { digraph data_model {
rankdir=LR rankdir=LR
node [shape=record, fontname="Helvetica", fontsize=11] bgcolor="#0a0e17"
edge [fontname="Helvetica", fontsize=10] fontname="Helvetica"
node [fontname="Helvetica" fontsize=11 shape=plaintext]
edge [fontname="Helvetica" fontsize=9 fontcolor="#8892a8" color="#4a5568"]
labelloc="t" label="Data Model"
label="MPR - Data Model" labelloc=t
fontsize=16 fontsize=16
fontname="Helvetica-Bold" fontcolor="#0066ff"
graph [splines=ortho, nodesep=0.6, ranksep=1.2] MediaAsset [label=<
<table border="0" cellborder="1" cellspacing="0" color="#1e2a4a" bgcolor="#121829">
<tr><td colspan="2" bgcolor="#0d1a33"><font color="#0066ff" face="JetBrains Mono"><b>MediaAsset</b></font></td></tr>
<tr><td><font color="#8892a8">id</font></td><td><font color="#e8eaf0">UUID PK</font></td></tr>
<tr><td><font color="#8892a8">filename</font></td><td><font color="#e8eaf0">str</font></td></tr>
<tr><td><font color="#8892a8">file_path</font></td><td><font color="#e8eaf0">str (relative)</font></td></tr>
<tr><td><font color="#8892a8">duration / fps / size</font></td><td><font color="#e8eaf0">probe metadata</font></td></tr>
</table>
>]
MediaAsset [label="{MediaAsset|id: UUID (PK)\lfilename: str\lfile_path: str (S3 key)\lfile_size: int?\lstatus: pending/ready/error\lerror_message: str?\l|duration: float?\lvideo_codec: str?\laudio_codec: str?\lwidth: int?\lheight: int?\lframerate: float?\lbitrate: int?\lproperties: JSON\l|comments: str\ltags: JSON[]\l|created_at: datetime\lupdated_at: datetime\l}"] Profile [label=<
<table border="0" cellborder="1" cellspacing="0" color="#1e2a4a" bgcolor="#121829">
<tr><td colspan="2" bgcolor="#0d1a33"><font color="#0066ff" face="JetBrains Mono"><b>Profile</b></font></td></tr>
<tr><td><font color="#8892a8">name</font></td><td><font color="#e8eaf0">str</font></td></tr>
<tr><td><font color="#8892a8">pipeline</font></td><td><font color="#e8eaf0">JSONB topology</font></td></tr>
<tr><td><font color="#8892a8">configs</font></td><td><font color="#e8eaf0">JSONB per-stage</font></td></tr>
</table>
>]
TranscodePreset [label="{TranscodePreset|id: UUID (PK)\lname: str (unique)\ldescription: str\lis_builtin: bool\l|container: str\l|video_codec: str\lvideo_bitrate: str?\lvideo_crf: int?\lvideo_preset: str?\lresolution: str?\lframerate: float?\l|audio_codec: str\laudio_bitrate: str?\laudio_channels: int?\laudio_samplerate: int?\l|extra_args: JSON[]\l|created_at: datetime\lupdated_at: datetime\l}"] Timeline [label=<
<table border="0" cellborder="1" cellspacing="0" color="#1e2a4a" bgcolor="#121829">
<tr><td colspan="2" bgcolor="#0d1a33"><font color="#0066ff" face="JetBrains Mono"><b>Timeline</b></font></td></tr>
<tr><td><font color="#8892a8">id</font></td><td><font color="#e8eaf0">UUID PK</font></td></tr>
<tr><td><font color="#8892a8">source_asset_id</font></td><td><font color="#e8eaf0">FK MediaAsset</font></td></tr>
<tr><td><font color="#8892a8">chunk_paths</font></td><td><font color="#e8eaf0">str[]</font></td></tr>
<tr><td><font color="#8892a8">profile_name</font></td><td><font color="#e8eaf0">str</font></td></tr>
<tr><td><font color="#8892a8">fps / status</font></td><td><font color="#e8eaf0">cached, ready, ...</font></td></tr>
</table>
>]
TranscodeJob [label="{TranscodeJob|id: UUID (PK)\l|source_asset_id: UUID (FK)\l|preset_id: UUID? (FK)\lpreset_snapshot: JSON\l|trim_start: float?\ltrim_end: float?\l|output_filename: str\loutput_path: str? (S3 key)\loutput_asset_id: UUID? (FK)\l|status: pending/processing/...\lprogress: float (0-100)\lcurrent_frame: int?\lcurrent_time: float?\lspeed: str?\lerror_message: str?\l|celery_task_id: str?\lexecution_arn: str?\lpriority: int\l|created_at: datetime\lstarted_at: datetime?\lcompleted_at: datetime?\l}"] Job [label=<
<table border="0" cellborder="1" cellspacing="0" color="#1e2a4a" bgcolor="#121829">
<tr><td colspan="2" bgcolor="#0d1a33"><font color="#0066ff" face="JetBrains Mono"><b>Job</b></font></td></tr>
<tr><td><font color="#8892a8">id</font></td><td><font color="#e8eaf0">UUID PK</font></td></tr>
<tr><td><font color="#8892a8">timeline_id</font></td><td><font color="#e8eaf0">FK Timeline</font></td></tr>
<tr><td><font color="#8892a8">parent_id</font></td><td><font color="#e8eaf0">FK Job (replay tree)</font></td></tr>
<tr><td><font color="#8892a8">profile_name</font></td><td><font color="#e8eaf0">str</font></td></tr>
<tr><td><font color="#8892a8">config_overrides</font></td><td><font color="#e8eaf0">JSONB</font></td></tr>
<tr><td><font color="#8892a8">run_type</font></td><td><font color="#e8eaf0">initial / replay / retry</font></td></tr>
<tr><td><font color="#8892a8">status / current_stage</font></td><td><font color="#e8eaf0">runtime</font></td></tr>
</table>
>]
MediaAsset -> TranscodeJob [xlabel="1:N source_asset"] Checkpoint [label=<
TranscodePreset -> TranscodeJob [xlabel="1:N preset"] <table border="0" cellborder="1" cellspacing="0" color="#1e2a4a" bgcolor="#121829">
TranscodeJob -> MediaAsset [xlabel="1:1 output_asset", style=dashed] <tr><td colspan="2" bgcolor="#0d1a33"><font color="#0066ff" face="JetBrains Mono"><b>Checkpoint</b></font></td></tr>
<tr><td><font color="#8892a8">id</font></td><td><font color="#e8eaf0">UUID PK</font></td></tr>
<tr><td><font color="#8892a8">timeline_id</font></td><td><font color="#e8eaf0">FK Timeline</font></td></tr>
<tr><td><font color="#8892a8">job_id</font></td><td><font color="#e8eaf0">FK Job (nullable)</font></td></tr>
<tr><td><font color="#8892a8">parent_id</font></td><td><font color="#e8eaf0">FK Checkpoint (tree)</font></td></tr>
<tr><td><font color="#8892a8">stage_name</font></td><td><font color="#e8eaf0">str</font></td></tr>
<tr><td><font color="#8892a8">config_overrides / stats</font></td><td><font color="#e8eaf0">JSONB (no blobs)</font></td></tr>
</table>
>]
StageOutput [label=<
<table border="0" cellborder="1" cellspacing="0" color="#1e2a4a" bgcolor="#121829">
<tr><td colspan="2" bgcolor="#0d1a33"><font color="#0066ff" face="JetBrains Mono"><b>StageOutput</b></font></td></tr>
<tr><td><font color="#8892a8">id</font></td><td><font color="#e8eaf0">UUID PK</font></td></tr>
<tr><td><font color="#8892a8">job_id</font></td><td><font color="#e8eaf0">FK Job</font></td></tr>
<tr><td><font color="#8892a8">timeline_id</font></td><td><font color="#e8eaf0">FK Timeline</font></td></tr>
<tr><td><font color="#8892a8">stage_name</font></td><td><font color="#e8eaf0">str</font></td></tr>
<tr><td><font color="#8892a8">checkpoint_id</font></td><td><font color="#e8eaf0">FK Checkpoint (nullable)</font></td></tr>
<tr><td><font color="#8892a8">output</font></td><td><font color="#e8eaf0">JSONB (flat upsert)</font></td></tr>
</table>
>]
Brand [label=<
<table border="0" cellborder="1" cellspacing="0" color="#1e2a4a" bgcolor="#121829">
<tr><td colspan="2" bgcolor="#0d1a33"><font color="#0066ff" face="JetBrains Mono"><b>Brand</b></font></td></tr>
<tr><td><font color="#8892a8">canonical_name</font></td><td><font color="#e8eaf0">str (indexed)</font></td></tr>
<tr><td><font color="#8892a8">aliases</font></td><td><font color="#e8eaf0">str[]</font></td></tr>
<tr><td><font color="#8892a8">source</font></td><td><font color="#e8eaf0">ocr / local_vlm / cloud_llm / manual</font></td></tr>
<tr><td><font color="#8892a8">airings</font></td><td><font color="#e8eaf0">JSONB[]</font></td></tr>
</table>
>]
MediaAsset -> Timeline [label="source_asset_id"]
Timeline -> Job [label="timeline_id"]
Job -> Job [label="parent_id\n(replay tree)" style=dashed]
Profile -> Job [label="profile_name" color="#0066ff"]
Job -> Checkpoint [label="job_id"]
Timeline -> Checkpoint [label="timeline_id"]
Checkpoint -> Checkpoint [label="parent_id\n(tree)" style=dashed]
Job -> StageOutput [label="job_id"]
Checkpoint -> StageOutput [label="checkpoint_id" style=dotted]
} }

357
docs/architecture/02-data-model.svg
View File

@@ -4,125 +4,272 @@
<!-- Generated by graphviz version 14.1.2 (0) <!-- Generated by graphviz version 14.1.2 (0)
--> -->
<!-- Title: data_model Pages: 1 --> <!-- Title: data_model Pages: 1 -->
<svg width="2134pt" height="286pt" <svg width="1661pt" height="442pt"
viewBox="0.00 0.00 2134.00 286.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> viewBox="0.00 0.00 1661.00 442.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 282)"> <g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 437.5)">
<title>data_model</title> <title>data_model</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-282 2130.25,-282 2130.25,4 -4,4"/> <polygon fill="#0a0e17" stroke="none" points="-4,4 -4,-437.5 1656.75,-437.5 1656.75,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="1063.12" y="-258.8" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">MPR &#45; Data Model</text> <text xml:space="preserve" text-anchor="middle" x="826.38" y="-414.3" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">Data Model</text>
<!-- MediaAsset --> <!-- MediaAsset -->
<g id="node1" class="node"> <g id="node1" class="node">
<title>MediaAsset</title> <title>MediaAsset</title>
<polygon fill="none" stroke="black" points="118,-134 118,-250 708,-250 708,-134 118,-134"/> <polygon fill="#121829" stroke="none" points="51.12,-176.25 51.12,-276 258.12,-276 258.12,-176.25 51.12,-176.25"/>
<text xml:space="preserve" text-anchor="middle" x="157.88" y="-188.3" font-family="Helvetica,sans-Serif" font-size="11.00">MediaAsset</text> <polygon fill="#0d1a33" stroke="none" points="51.12,-254.25 51.12,-276 258.12,-276 258.12,-254.25 51.12,-254.25"/>
<polyline fill="none" stroke="black" points="197.75,-134 197.75,-250"/> <polygon fill="none" stroke="#1e2a4a" points="51.12,-254.25 51.12,-276 258.12,-276 258.12,-254.25 51.12,-254.25"/>
<text xml:space="preserve" text-anchor="start" x="205.75" y="-222.05" font-family="Helvetica,sans-Serif" font-size="11.00">id: UUID (PK)</text> <text xml:space="preserve" text-anchor="start" x="123.12" y="-263.55" font-family="JetBrains Mono" font-weight="bold" font-size="11.00" fill="#0066ff">MediaAsset</text>
<text xml:space="preserve" text-anchor="start" x="205.75" y="-208.55" font-family="Helvetica,sans-Serif" font-size="11.00">filename: str</text> <polygon fill="none" stroke="#1e2a4a" points="51.12,-234.75 51.12,-254.25 163.62,-254.25 163.62,-234.75 51.12,-234.75"/>
<text xml:space="preserve" text-anchor="start" x="205.75" y="-195.05" font-family="Helvetica,sans-Serif" font-size="11.00">file_path: str (S3 key)</text> <text xml:space="preserve" text-anchor="start" x="102.5" y="-240.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">id</text>
<text xml:space="preserve" text-anchor="start" x="205.75" y="-181.55" font-family="Helvetica,sans-Serif" font-size="11.00">file_size: int?</text> <polygon fill="none" stroke="#1e2a4a" points="163.62,-234.75 163.62,-254.25 258.12,-254.25 258.12,-234.75 163.62,-234.75"/>
<text xml:space="preserve" text-anchor="start" x="205.75" y="-168.05" font-family="Helvetica,sans-Serif" font-size="11.00">status: pending/ready/error</text> <text xml:space="preserve" text-anchor="start" x="188" y="-240.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">UUID PK</text>
<text xml:space="preserve" text-anchor="start" x="205.75" y="-154.55" font-family="Helvetica,sans-Serif" font-size="11.00">error_message: str?</text> <polygon fill="none" stroke="#1e2a4a" points="51.12,-215.25 51.12,-234.75 163.62,-234.75 163.62,-215.25 51.12,-215.25"/>
<polyline fill="none" stroke="black" points="365.25,-134 365.25,-250"/> <text xml:space="preserve" text-anchor="start" x="83.75" y="-221.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">filename</text>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-235.55" font-family="Helvetica,sans-Serif" font-size="11.00">duration: float?</text> <polygon fill="none" stroke="#1e2a4a" points="163.62,-215.25 163.62,-234.75 258.12,-234.75 258.12,-215.25 163.62,-215.25"/>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-222.05" font-family="Helvetica,sans-Serif" font-size="11.00">video_codec: str?</text> <text xml:space="preserve" text-anchor="start" x="203.38" y="-221.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str</text>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-208.55" font-family="Helvetica,sans-Serif" font-size="11.00">audio_codec: str?</text> <polygon fill="none" stroke="#1e2a4a" points="51.12,-195.75 51.12,-215.25 163.62,-215.25 163.62,-195.75 51.12,-195.75"/>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-195.05" font-family="Helvetica,sans-Serif" font-size="11.00">width: int?</text> <text xml:space="preserve" text-anchor="start" x="84.12" y="-201.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">file_path</text>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-181.55" font-family="Helvetica,sans-Serif" font-size="11.00">height: int?</text> <polygon fill="none" stroke="#1e2a4a" points="163.62,-195.75 163.62,-215.25 258.12,-215.25 258.12,-195.75 163.62,-195.75"/>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-168.05" font-family="Helvetica,sans-Serif" font-size="11.00">framerate: float?</text> <text xml:space="preserve" text-anchor="start" x="176" y="-201.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str (relative)</text>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-154.55" font-family="Helvetica,sans-Serif" font-size="11.00">bitrate: int?</text> <polygon fill="none" stroke="#1e2a4a" points="51.12,-176.25 51.12,-195.75 163.62,-195.75 163.62,-176.25 51.12,-176.25"/>
<text xml:space="preserve" text-anchor="start" x="373.25" y="-141.05" font-family="Helvetica,sans-Serif" font-size="11.00">properties: JSON</text> <text xml:space="preserve" text-anchor="start" x="54.12" y="-182.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">duration / fps / size</text>
<polyline fill="none" stroke="black" points="477.25,-134 477.25,-250"/> <polygon fill="none" stroke="#1e2a4a" points="163.62,-176.25 163.62,-195.75 258.12,-195.75 258.12,-176.25 163.62,-176.25"/>
<text xml:space="preserve" text-anchor="start" x="485.25" y="-195.05" font-family="Helvetica,sans-Serif" font-size="11.00">comments: str</text> <text xml:space="preserve" text-anchor="start" x="166.62" y="-182.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">probe metadata</text>
<text xml:space="preserve" text-anchor="start" x="485.25" y="-181.55" font-family="Helvetica,sans-Serif" font-size="11.00">tags: JSON[]</text>
<polyline fill="none" stroke="black" points="573.5,-134 573.5,-250"/>
<text xml:space="preserve" text-anchor="start" x="581.5" y="-195.05" font-family="Helvetica,sans-Serif" font-size="11.00">created_at: datetime</text>
<text xml:space="preserve" text-anchor="start" x="581.5" y="-181.55" font-family="Helvetica,sans-Serif" font-size="11.00">updated_at: datetime</text>
</g> </g>
<!-- TranscodeJob --> <!-- Timeline -->
<g id="node3" class="node"> <g id="node3" class="node">
<title>TranscodeJob</title> <title>Timeline</title>
<polygon fill="none" stroke="black" points="912,-147.5 912,-236.5 2126.25,-236.5 2126.25,-147.5 912,-147.5"/> <polygon fill="#121829" stroke="none" points="423.75,-166.5 423.75,-285.75 619.5,-285.75 619.5,-166.5 423.75,-166.5"/>
<text xml:space="preserve" text-anchor="middle" x="956" y="-188.3" font-family="Helvetica,sans-Serif" font-size="11.00">TranscodeJob</text> <polygon fill="#0d1a33" stroke="none" points="423.75,-264 423.75,-285.75 619.5,-285.75 619.5,-264 423.75,-264"/>
<polyline fill="none" stroke="black" points="1000,-147.5 1000,-236.5"/> <polygon fill="none" stroke="#1e2a4a" points="423.75,-264 423.75,-285.75 619.5,-285.75 619.5,-264 423.75,-264"/>
<text xml:space="preserve" text-anchor="start" x="1008" y="-188.3" font-family="Helvetica,sans-Serif" font-size="11.00">id: UUID (PK)</text> <text xml:space="preserve" text-anchor="start" x="498.38" y="-273.3" font-family="JetBrains Mono" font-weight="bold" font-size="11.00" fill="#0066ff">Timeline</text>
<polyline fill="none" stroke="black" points="1088,-147.5 1088,-236.5"/> <polygon fill="none" stroke="#1e2a4a" points="423.75,-244.5 423.75,-264 516.75,-264 516.75,-244.5 423.75,-244.5"/>
<text xml:space="preserve" text-anchor="start" x="1096" y="-188.3" font-family="Helvetica,sans-Serif" font-size="11.00">source_asset_id: UUID (FK)</text> <text xml:space="preserve" text-anchor="start" x="465.38" y="-250.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">id</text>
<polyline fill="none" stroke="black" points="1252.5,-147.5 1252.5,-236.5"/> <polygon fill="none" stroke="#1e2a4a" points="516.75,-244.5 516.75,-264 619.5,-264 619.5,-244.5 516.75,-244.5"/>
<text xml:space="preserve" text-anchor="start" x="1260.5" y="-195.05" font-family="Helvetica,sans-Serif" font-size="11.00">preset_id: UUID? (FK)</text> <text xml:space="preserve" text-anchor="start" x="545.25" y="-250.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">UUID PK</text>
<text xml:space="preserve" text-anchor="start" x="1260.5" y="-181.55" font-family="Helvetica,sans-Serif" font-size="11.00">preset_snapshot: JSON</text> <polygon fill="none" stroke="#1e2a4a" points="423.75,-225 423.75,-244.5 516.75,-244.5 516.75,-225 423.75,-225"/>
<polyline fill="none" stroke="black" points="1393.75,-147.5 1393.75,-236.5"/> <text xml:space="preserve" text-anchor="start" x="426.75" y="-231.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">source_asset_id</text>
<text xml:space="preserve" text-anchor="start" x="1401.75" y="-195.05" font-family="Helvetica,sans-Serif" font-size="11.00">trim_start: float?</text> <polygon fill="none" stroke="#1e2a4a" points="516.75,-225 516.75,-244.5 619.5,-244.5 619.5,-225 516.75,-225"/>
<text xml:space="preserve" text-anchor="start" x="1401.75" y="-181.55" font-family="Helvetica,sans-Serif" font-size="11.00">trim_end: float?</text> <text xml:space="preserve" text-anchor="start" x="527.62" y="-231.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK MediaAsset</text>
<polyline fill="none" stroke="black" points="1502,-147.5 1502,-236.5"/> <polygon fill="none" stroke="#1e2a4a" points="423.75,-205.5 423.75,-225 516.75,-225 516.75,-205.5 423.75,-205.5"/>
<text xml:space="preserve" text-anchor="start" x="1510" y="-201.8" font-family="Helvetica,sans-Serif" font-size="11.00">output_filename: str</text> <text xml:space="preserve" text-anchor="start" x="436.12" y="-211.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">chunk_paths</text>
<text xml:space="preserve" text-anchor="start" x="1510" y="-188.3" font-family="Helvetica,sans-Serif" font-size="11.00">output_path: str? (S3 key)</text> <polygon fill="none" stroke="#1e2a4a" points="516.75,-205.5 516.75,-225 619.5,-225 619.5,-205.5 516.75,-205.5"/>
<text xml:space="preserve" text-anchor="start" x="1510" y="-174.8" font-family="Helvetica,sans-Serif" font-size="11.00">output_asset_id: UUID? (FK)</text> <text xml:space="preserve" text-anchor="start" x="556.12" y="-211.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str[]</text>
<polyline fill="none" stroke="black" points="1671.75,-147.5 1671.75,-236.5"/> <polygon fill="none" stroke="#1e2a4a" points="423.75,-186 423.75,-205.5 516.75,-205.5 516.75,-186 423.75,-186"/>
<text xml:space="preserve" text-anchor="start" x="1679.75" y="-222.05" font-family="Helvetica,sans-Serif" font-size="11.00">status: pending/processing/...</text> <text xml:space="preserve" text-anchor="start" x="435" y="-192.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">profile_name</text>
<text xml:space="preserve" text-anchor="start" x="1679.75" y="-208.55" font-family="Helvetica,sans-Serif" font-size="11.00">progress: float (0&#45;100)</text> <polygon fill="none" stroke="#1e2a4a" points="516.75,-186 516.75,-205.5 619.5,-205.5 619.5,-186 516.75,-186"/>
<text xml:space="preserve" text-anchor="start" x="1679.75" y="-195.05" font-family="Helvetica,sans-Serif" font-size="11.00">current_frame: int?</text> <text xml:space="preserve" text-anchor="start" x="560.62" y="-192.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str</text>
<text xml:space="preserve" text-anchor="start" x="1679.75" y="-181.55" font-family="Helvetica,sans-Serif" font-size="11.00">current_time: float?</text> <polygon fill="none" stroke="#1e2a4a" points="423.75,-166.5 423.75,-186 516.75,-186 516.75,-166.5 423.75,-166.5"/>
<text xml:space="preserve" text-anchor="start" x="1679.75" y="-168.05" font-family="Helvetica,sans-Serif" font-size="11.00">speed: str?</text> <text xml:space="preserve" text-anchor="start" x="439.12" y="-172.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">fps / status</text>
<text xml:space="preserve" text-anchor="start" x="1679.75" y="-154.55" font-family="Helvetica,sans-Serif" font-size="11.00">error_message: str?</text> <polygon fill="none" stroke="#1e2a4a" points="516.75,-166.5 516.75,-186 619.5,-186 619.5,-166.5 516.75,-166.5"/>
<polyline fill="none" stroke="black" points="1851.25,-147.5 1851.25,-236.5"/> <text xml:space="preserve" text-anchor="start" x="519.75" y="-172.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">cached, ready, ...</text>
<text xml:space="preserve" text-anchor="start" x="1859.25" y="-201.8" font-family="Helvetica,sans-Serif" font-size="11.00">celery_task_id: str?</text>
<text xml:space="preserve" text-anchor="start" x="1859.25" y="-188.3" font-family="Helvetica,sans-Serif" font-size="11.00">execution_arn: str?</text>
<text xml:space="preserve" text-anchor="start" x="1859.25" y="-174.8" font-family="Helvetica,sans-Serif" font-size="11.00">priority: int</text>
<polyline fill="none" stroke="black" points="1973,-147.5 1973,-236.5"/>
<text xml:space="preserve" text-anchor="start" x="1981" y="-201.8" font-family="Helvetica,sans-Serif" font-size="11.00">created_at: datetime</text>
<text xml:space="preserve" text-anchor="start" x="1981" y="-188.3" font-family="Helvetica,sans-Serif" font-size="11.00">started_at: datetime?</text>
<text xml:space="preserve" text-anchor="start" x="1981" y="-174.8" font-family="Helvetica,sans-Serif" font-size="11.00">completed_at: datetime?</text>
</g> </g>
<!-- MediaAsset&#45;&gt;TranscodeJob --> <!-- MediaAsset&#45;&gt;Timeline -->
<g id="edge1" class="edge"> <g id="edge1" class="edge">
<title>MediaAsset&#45;&gt;TranscodeJob</title> <title>MediaAsset&#45;&gt;Timeline</title>
<path fill="none" stroke="black" d="M708.33,-192C708.33,-192 900.24,-192 900.24,-192"/> <path fill="none" stroke="#4a5568" d="M265.63,-226.12C309.46,-226.12 359.96,-226.12 404.39,-226.12"/>
<polygon fill="black" stroke="black" points="900.24,-195.5 910.24,-192 900.24,-188.5 900.24,-195.5"/> <polygon fill="#4a5568" stroke="#4a5568" points="404.33,-229.63 414.33,-226.13 404.33,-222.63 404.33,-229.63"/>
<text xml:space="preserve" text-anchor="middle" x="762.66" y="-182.5" font-family="Helvetica,sans-Serif" font-size="10.00">1:N source_asset</text> <text xml:space="preserve" text-anchor="middle" x="362.5" y="-228.82" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">source_asset_id</text>
</g> </g>
<!-- TranscodePreset --> <!-- Profile -->
<g id="node2" class="node"> <g id="node2" class="node">
<title>TranscodePreset</title> <title>Profile</title>
<polygon fill="none" stroke="black" points="0,-0.5 0,-89.5 826,-89.5 826,-0.5 0,-0.5"/> <polygon fill="#121829" stroke="none" points="449.25,-43 449.25,-123.25 594,-123.25 594,-43 449.25,-43"/>
<text xml:space="preserve" text-anchor="middle" x="53.38" y="-41.3" font-family="Helvetica,sans-Serif" font-size="11.00">TranscodePreset</text> <polygon fill="#0d1a33" stroke="none" points="449.25,-101.5 449.25,-123.25 594,-123.25 594,-101.5 449.25,-101.5"/>
<polyline fill="none" stroke="black" points="106.75,-0.5 106.75,-89.5"/> <polygon fill="none" stroke="#1e2a4a" points="449.25,-101.5 449.25,-123.25 594,-123.25 594,-101.5 449.25,-101.5"/>
<text xml:space="preserve" text-anchor="start" x="114.75" y="-61.55" font-family="Helvetica,sans-Serif" font-size="11.00">id: UUID (PK)</text> <text xml:space="preserve" text-anchor="start" x="504" y="-110.8" font-family="JetBrains Mono" font-weight="bold" font-size="11.00" fill="#0066ff">Profile</text>
<text xml:space="preserve" text-anchor="start" x="114.75" y="-48.05" font-family="Helvetica,sans-Serif" font-size="11.00">name: str (unique)</text> <polygon fill="none" stroke="#1e2a4a" points="449.25,-82 449.25,-101.5 498,-101.5 498,-82 449.25,-82"/>
<text xml:space="preserve" text-anchor="start" x="114.75" y="-34.55" font-family="Helvetica,sans-Serif" font-size="11.00">description: str</text> <text xml:space="preserve" text-anchor="start" x="458.25" y="-88.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">name</text>
<text xml:space="preserve" text-anchor="start" x="114.75" y="-21.05" font-family="Helvetica,sans-Serif" font-size="11.00">is_builtin: bool</text> <polygon fill="none" stroke="#1e2a4a" points="498,-82 498,-101.5 594,-101.5 594,-82 498,-82"/>
<polyline fill="none" stroke="black" points="225.5,-0.5 225.5,-89.5"/> <text xml:space="preserve" text-anchor="start" x="538.5" y="-88.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str</text>
<text xml:space="preserve" text-anchor="start" x="233.5" y="-41.3" font-family="Helvetica,sans-Serif" font-size="11.00">container: str</text> <polygon fill="none" stroke="#1e2a4a" points="449.25,-62.5 449.25,-82 498,-82 498,-62.5 449.25,-62.5"/>
<polyline fill="none" stroke="black" points="315.75,-0.5 315.75,-89.5"/> <text xml:space="preserve" text-anchor="start" x="452.25" y="-68.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">pipeline</text>
<text xml:space="preserve" text-anchor="start" x="323.75" y="-75.05" font-family="Helvetica,sans-Serif" font-size="11.00">video_codec: str</text> <polygon fill="none" stroke="#1e2a4a" points="498,-62.5 498,-82 594,-82 594,-62.5 498,-62.5"/>
<text xml:space="preserve" text-anchor="start" x="323.75" y="-61.55" font-family="Helvetica,sans-Serif" font-size="11.00">video_bitrate: str?</text> <text xml:space="preserve" text-anchor="start" x="502.88" y="-68.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">JSONB topology</text>
<text xml:space="preserve" text-anchor="start" x="323.75" y="-48.05" font-family="Helvetica,sans-Serif" font-size="11.00">video_crf: int?</text> <polygon fill="none" stroke="#1e2a4a" points="449.25,-43 449.25,-62.5 498,-62.5 498,-43 449.25,-43"/>
<text xml:space="preserve" text-anchor="start" x="323.75" y="-34.55" font-family="Helvetica,sans-Serif" font-size="11.00">video_preset: str?</text> <text xml:space="preserve" text-anchor="start" x="454.12" y="-49.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">configs</text>
<text xml:space="preserve" text-anchor="start" x="323.75" y="-21.05" font-family="Helvetica,sans-Serif" font-size="11.00">resolution: str?</text> <polygon fill="none" stroke="#1e2a4a" points="498,-43 498,-62.5 594,-62.5 594,-43 498,-43"/>
<text xml:space="preserve" text-anchor="start" x="323.75" y="-7.55" font-family="Helvetica,sans-Serif" font-size="11.00">framerate: float?</text> <text xml:space="preserve" text-anchor="start" x="501" y="-49.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">JSONB per&#45;stage</text>
<polyline fill="none" stroke="black" points="432.25,-0.5 432.25,-89.5"/>
<text xml:space="preserve" text-anchor="start" x="440.25" y="-61.55" font-family="Helvetica,sans-Serif" font-size="11.00">audio_codec: str</text>
<text xml:space="preserve" text-anchor="start" x="440.25" y="-48.05" font-family="Helvetica,sans-Serif" font-size="11.00">audio_bitrate: str?</text>
<text xml:space="preserve" text-anchor="start" x="440.25" y="-34.55" font-family="Helvetica,sans-Serif" font-size="11.00">audio_channels: int?</text>
<text xml:space="preserve" text-anchor="start" x="440.25" y="-21.05" font-family="Helvetica,sans-Serif" font-size="11.00">audio_samplerate: int?</text>
<polyline fill="none" stroke="black" points="573.5,-0.5 573.5,-89.5"/>
<text xml:space="preserve" text-anchor="start" x="581.5" y="-41.3" font-family="Helvetica,sans-Serif" font-size="11.00">extra_args: JSON[]</text>
<polyline fill="none" stroke="black" points="691.5,-0.5 691.5,-89.5"/>
<text xml:space="preserve" text-anchor="start" x="699.5" y="-48.05" font-family="Helvetica,sans-Serif" font-size="11.00">created_at: datetime</text>
<text xml:space="preserve" text-anchor="start" x="699.5" y="-34.55" font-family="Helvetica,sans-Serif" font-size="11.00">updated_at: datetime</text>
</g> </g>
<!-- TranscodePreset&#45;&gt;TranscodeJob --> <!-- Job -->
<g id="node4" class="node">
<title>Job</title>
<polygon fill="#121829" stroke="none" points="730,-4 730,-162.25 977.5,-162.25 977.5,-4 730,-4"/>
<polygon fill="#0d1a33" stroke="none" points="730,-140.5 730,-162.25 977.5,-162.25 977.5,-140.5 730,-140.5"/>
<polygon fill="none" stroke="#1e2a4a" points="730,-140.5 730,-162.25 977.5,-162.25 977.5,-140.5 730,-140.5"/>
<text xml:space="preserve" text-anchor="start" x="845.12" y="-149.8" font-family="JetBrains Mono" font-weight="bold" font-size="11.00" fill="#0066ff">Job</text>
<polygon fill="none" stroke="#1e2a4a" points="730,-121 730,-140.5 857.5,-140.5 857.5,-121 730,-121"/>
<text xml:space="preserve" text-anchor="start" x="788.88" y="-127.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">id</text>
<polygon fill="none" stroke="#1e2a4a" points="857.5,-121 857.5,-140.5 977.5,-140.5 977.5,-121 857.5,-121"/>
<text xml:space="preserve" text-anchor="start" x="894.62" y="-127.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">UUID PK</text>
<polygon fill="none" stroke="#1e2a4a" points="730,-101.5 730,-121 857.5,-121 857.5,-101.5 730,-101.5"/>
<text xml:space="preserve" text-anchor="start" x="764.12" y="-107.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">timeline_id</text>
<polygon fill="none" stroke="#1e2a4a" points="857.5,-101.5 857.5,-121 977.5,-121 977.5,-101.5 857.5,-101.5"/>
<text xml:space="preserve" text-anchor="start" x="885.62" y="-107.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Timeline</text>
<polygon fill="none" stroke="#1e2a4a" points="730,-82 730,-101.5 857.5,-101.5 857.5,-82 730,-82"/>
<text xml:space="preserve" text-anchor="start" x="768.25" y="-88.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">parent_id</text>
<polygon fill="none" stroke="#1e2a4a" points="857.5,-82 857.5,-101.5 977.5,-101.5 977.5,-82 857.5,-82"/>
<text xml:space="preserve" text-anchor="start" x="863.88" y="-88.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Job (replay tree)</text>
<polygon fill="none" stroke="#1e2a4a" points="730,-62.5 730,-82 857.5,-82 857.5,-62.5 730,-62.5"/>
<text xml:space="preserve" text-anchor="start" x="758.5" y="-68.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">profile_name</text>
<polygon fill="none" stroke="#1e2a4a" points="857.5,-62.5 857.5,-82 977.5,-82 977.5,-62.5 857.5,-62.5"/>
<text xml:space="preserve" text-anchor="start" x="910" y="-68.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str</text>
<polygon fill="none" stroke="#1e2a4a" points="730,-43 730,-62.5 857.5,-62.5 857.5,-43 730,-43"/>
<text xml:space="preserve" text-anchor="start" x="748.75" y="-49.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">config_overrides</text>
<polygon fill="none" stroke="#1e2a4a" points="857.5,-43 857.5,-62.5 977.5,-62.5 977.5,-43 857.5,-43"/>
<text xml:space="preserve" text-anchor="start" x="900.25" y="-49.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">JSONB</text>
<polygon fill="none" stroke="#1e2a4a" points="730,-23.5 730,-43 857.5,-43 857.5,-23.5 730,-23.5"/>
<text xml:space="preserve" text-anchor="start" x="769.75" y="-29.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">run_type</text>
<polygon fill="none" stroke="#1e2a4a" points="857.5,-23.5 857.5,-43 977.5,-43 977.5,-23.5 857.5,-23.5"/>
<text xml:space="preserve" text-anchor="start" x="860.5" y="-29.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">initial / replay / retry</text>
<polygon fill="none" stroke="#1e2a4a" points="730,-4 730,-23.5 857.5,-23.5 857.5,-4 730,-4"/>
<text xml:space="preserve" text-anchor="start" x="733" y="-10.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">status / current_stage</text>
<polygon fill="none" stroke="#1e2a4a" points="857.5,-4 857.5,-23.5 977.5,-23.5 977.5,-4 857.5,-4"/>
<text xml:space="preserve" text-anchor="start" x="896.12" y="-10.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">runtime</text>
</g>
<!-- Profile&#45;&gt;Job -->
<g id="edge4" class="edge">
<title>Profile&#45;&gt;Job</title>
<path fill="none" stroke="#0066ff" d="M601.7,-83.12C634.51,-83.12 673.61,-83.12 711.08,-83.12"/>
<polygon fill="#0066ff" stroke="#0066ff" points="710.6,-86.63 720.6,-83.13 710.6,-79.63 710.6,-86.63"/>
<text xml:space="preserve" text-anchor="middle" x="674.75" y="-85.83" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">profile_name</text>
</g>
<!-- Timeline&#45;&gt;Job -->
<g id="edge2" class="edge"> <g id="edge2" class="edge">
<title>TranscodePreset&#45;&gt;TranscodeJob</title> <title>Timeline&#45;&gt;Job</title>
<path fill="none" stroke="black" d="M767.25,-89.95C767.25,-125.61 767.25,-169.5 767.25,-169.5 767.25,-169.5 900.26,-169.5 900.26,-169.5"/> <path fill="none" stroke="#4a5568" d="M627.08,-180.88C654,-169.22 683.41,-156.48 711.85,-144.16"/>
<polygon fill="black" stroke="black" points="900.26,-173 910.26,-169.5 900.26,-166 900.26,-173"/> <polygon fill="#4a5568" stroke="#4a5568" points="713.1,-147.43 720.89,-140.24 710.32,-141.01 713.1,-147.43"/>
<text xml:space="preserve" text-anchor="middle" x="768.85" y="-160" font-family="Helvetica,sans-Serif" font-size="10.00">1:N preset</text> <text xml:space="preserve" text-anchor="middle" x="674.75" y="-174.34" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">timeline_id</text>
</g> </g>
<!-- TranscodeJob&#45;&gt;MediaAsset --> <!-- Checkpoint -->
<g id="node5" class="node">
<title>Checkpoint</title>
<polygon fill="#121829" stroke="none" points="1055.75,-116.75 1055.75,-255.5 1310,-255.5 1310,-116.75 1055.75,-116.75"/>
<polygon fill="#0d1a33" stroke="none" points="1055.75,-233.75 1055.75,-255.5 1310,-255.5 1310,-233.75 1055.75,-233.75"/>
<polygon fill="none" stroke="#1e2a4a" points="1055.75,-233.75 1055.75,-255.5 1310,-255.5 1310,-233.75 1055.75,-233.75"/>
<text xml:space="preserve" text-anchor="start" x="1151.75" y="-243.05" font-family="JetBrains Mono" font-weight="bold" font-size="11.00" fill="#0066ff">Checkpoint</text>
<polygon fill="none" stroke="#1e2a4a" points="1055.75,-214.25 1055.75,-233.75 1190.75,-233.75 1190.75,-214.25 1055.75,-214.25"/>
<text xml:space="preserve" text-anchor="start" x="1118.38" y="-220.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">id</text>
<polygon fill="none" stroke="#1e2a4a" points="1190.75,-214.25 1190.75,-233.75 1310,-233.75 1310,-214.25 1190.75,-214.25"/>
<text xml:space="preserve" text-anchor="start" x="1227.5" y="-220.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">UUID PK</text>
<polygon fill="none" stroke="#1e2a4a" points="1055.75,-194.75 1055.75,-214.25 1190.75,-214.25 1190.75,-194.75 1055.75,-194.75"/>
<text xml:space="preserve" text-anchor="start" x="1093.62" y="-200.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">timeline_id</text>
<polygon fill="none" stroke="#1e2a4a" points="1190.75,-194.75 1190.75,-214.25 1310,-214.25 1310,-194.75 1190.75,-194.75"/>
<text xml:space="preserve" text-anchor="start" x="1218.5" y="-200.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Timeline</text>
<polygon fill="none" stroke="#1e2a4a" points="1055.75,-175.25 1055.75,-194.75 1190.75,-194.75 1190.75,-175.25 1055.75,-175.25"/>
<text xml:space="preserve" text-anchor="start" x="1107.5" y="-181.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">job_id</text>
<polygon fill="none" stroke="#1e2a4a" points="1190.75,-175.25 1190.75,-194.75 1310,-194.75 1310,-175.25 1190.75,-175.25"/>
<text xml:space="preserve" text-anchor="start" x="1205.75" y="-181.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Job (nullable)</text>
<polygon fill="none" stroke="#1e2a4a" points="1055.75,-155.75 1055.75,-175.25 1190.75,-175.25 1190.75,-155.75 1055.75,-155.75"/>
<text xml:space="preserve" text-anchor="start" x="1097.75" y="-161.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">parent_id</text>
<polygon fill="none" stroke="#1e2a4a" points="1190.75,-155.75 1190.75,-175.25 1310,-175.25 1310,-155.75 1190.75,-155.75"/>
<text xml:space="preserve" text-anchor="start" x="1193.75" y="-161.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Checkpoint (tree)</text>
<polygon fill="none" stroke="#1e2a4a" points="1055.75,-136.25 1055.75,-155.75 1190.75,-155.75 1190.75,-136.25 1055.75,-136.25"/>
<text xml:space="preserve" text-anchor="start" x="1089.88" y="-142.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">stage_name</text>
<polygon fill="none" stroke="#1e2a4a" points="1190.75,-136.25 1190.75,-155.75 1310,-155.75 1310,-136.25 1190.75,-136.25"/>
<text xml:space="preserve" text-anchor="start" x="1242.88" y="-142.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str</text>
<polygon fill="none" stroke="#1e2a4a" points="1055.75,-116.75 1055.75,-136.25 1190.75,-136.25 1190.75,-116.75 1055.75,-116.75"/>
<text xml:space="preserve" text-anchor="start" x="1058.75" y="-122.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">config_overrides / stats</text>
<polygon fill="none" stroke="#1e2a4a" points="1190.75,-116.75 1190.75,-136.25 1310,-136.25 1310,-116.75 1190.75,-116.75"/>
<text xml:space="preserve" text-anchor="start" x="1203.5" y="-122.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">JSONB (no blobs)</text>
</g>
<!-- Timeline&#45;&gt;Checkpoint -->
<g id="edge6" class="edge">
<title>Timeline&#45;&gt;Checkpoint</title>
<path fill="none" stroke="#4a5568" d="M627.16,-227.73C721.11,-228.26 862.78,-226.78 985.5,-216.12 1002.13,-214.68 1019.49,-212.71 1036.71,-210.47"/>
<polygon fill="#4a5568" stroke="#4a5568" points="1037.05,-213.95 1046.5,-209.15 1036.12,-207.01 1037.05,-213.95"/>
<text xml:space="preserve" text-anchor="middle" x="853.75" y="-230.12" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">timeline_id</text>
</g>
<!-- Job&#45;&gt;Job -->
<g id="edge3" class="edge"> <g id="edge3" class="edge">
<title>TranscodeJob&#45;&gt;MediaAsset</title> <title>Job&#45;&gt;Job</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M911.86,-214.5C911.86,-214.5 719.76,-214.5 719.76,-214.5"/> <path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M826.82,-166.05C831.41,-176.99 840.39,-184.25 853.75,-184.25 862.73,-184.25 869.72,-180.97 874.74,-175.5"/>
<polygon fill="black" stroke="black" points="719.76,-211 709.76,-214.5 719.76,-218 719.76,-211"/> <polygon fill="#4a5568" stroke="#4a5568" points="877.52,-177.66 879.88,-167.33 871.59,-173.94 877.52,-177.66"/>
<text xml:space="preserve" text-anchor="middle" x="775.31" y="-205" font-family="Helvetica,sans-Serif" font-size="10.00">1:1 output_asset</text> <text xml:space="preserve" text-anchor="middle" x="853.75" y="-198.2" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">parent_id</text>
<text xml:space="preserve" text-anchor="middle" x="853.75" y="-186.95" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">(replay tree)</text>
</g>
<!-- Job&#45;&gt;Checkpoint -->
<g id="edge5" class="edge">
<title>Job&#45;&gt;Checkpoint</title>
<path fill="none" stroke="#4a5568" d="M985.45,-124.28C1002.47,-129.64 1019.99,-135.15 1037.22,-140.58"/>
<polygon fill="#4a5568" stroke="#4a5568" points="1035.87,-143.82 1046.46,-143.49 1037.98,-137.15 1035.87,-143.82"/>
<text xml:space="preserve" text-anchor="middle" x="1016.62" y="-140.41" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">job_id</text>
</g>
<!-- StageOutput -->
<g id="node6" class="node">
<title>StageOutput</title>
<polygon fill="#121829" stroke="none" points="1425,-29.75 1425,-168.5 1644.75,-168.5 1644.75,-29.75 1425,-29.75"/>
<polygon fill="#0d1a33" stroke="none" points="1425,-146.75 1425,-168.5 1644.75,-168.5 1644.75,-146.75 1425,-146.75"/>
<polygon fill="none" stroke="#1e2a4a" points="1425,-146.75 1425,-168.5 1644.75,-168.5 1644.75,-146.75 1425,-146.75"/>
<text xml:space="preserve" text-anchor="start" x="1499.62" y="-156.05" font-family="JetBrains Mono" font-weight="bold" font-size="11.00" fill="#0066ff">StageOutput</text>
<polygon fill="none" stroke="#1e2a4a" points="1425,-127.25 1425,-146.75 1505.25,-146.75 1505.25,-127.25 1425,-127.25"/>
<text xml:space="preserve" text-anchor="start" x="1460.25" y="-133.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">id</text>
<polygon fill="none" stroke="#1e2a4a" points="1505.25,-127.25 1505.25,-146.75 1644.75,-146.75 1644.75,-127.25 1505.25,-127.25"/>
<text xml:space="preserve" text-anchor="start" x="1552.12" y="-133.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">UUID PK</text>
<polygon fill="none" stroke="#1e2a4a" points="1425,-107.75 1425,-127.25 1505.25,-127.25 1505.25,-107.75 1425,-107.75"/>
<text xml:space="preserve" text-anchor="start" x="1449.38" y="-113.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">job_id</text>
<polygon fill="none" stroke="#1e2a4a" points="1505.25,-107.75 1505.25,-127.25 1644.75,-127.25 1644.75,-107.75 1505.25,-107.75"/>
<text xml:space="preserve" text-anchor="start" x="1558.12" y="-113.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Job</text>
<polygon fill="none" stroke="#1e2a4a" points="1425,-88.25 1425,-107.75 1505.25,-107.75 1505.25,-88.25 1425,-88.25"/>
<text xml:space="preserve" text-anchor="start" x="1435.5" y="-94.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">timeline_id</text>
<polygon fill="none" stroke="#1e2a4a" points="1505.25,-88.25 1505.25,-107.75 1644.75,-107.75 1644.75,-88.25 1505.25,-88.25"/>
<text xml:space="preserve" text-anchor="start" x="1543.12" y="-94.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Timeline</text>
<polygon fill="none" stroke="#1e2a4a" points="1425,-68.75 1425,-88.25 1505.25,-88.25 1505.25,-68.75 1425,-68.75"/>
<text xml:space="preserve" text-anchor="start" x="1431.75" y="-74.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">stage_name</text>
<polygon fill="none" stroke="#1e2a4a" points="1505.25,-68.75 1505.25,-88.25 1644.75,-88.25 1644.75,-68.75 1505.25,-68.75"/>
<text xml:space="preserve" text-anchor="start" x="1567.5" y="-74.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str</text>
<polygon fill="none" stroke="#1e2a4a" points="1425,-49.25 1425,-68.75 1505.25,-68.75 1505.25,-49.25 1425,-49.25"/>
<text xml:space="preserve" text-anchor="start" x="1428" y="-55.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">checkpoint_id</text>
<polygon fill="none" stroke="#1e2a4a" points="1505.25,-49.25 1505.25,-68.75 1644.75,-68.75 1644.75,-49.25 1505.25,-49.25"/>
<text xml:space="preserve" text-anchor="start" x="1508.25" y="-55.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">FK Checkpoint (nullable)</text>
<polygon fill="none" stroke="#1e2a4a" points="1425,-29.75 1425,-49.25 1505.25,-49.25 1505.25,-29.75 1425,-29.75"/>
<text xml:space="preserve" text-anchor="start" x="1447.12" y="-35.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">output</text>
<polygon fill="none" stroke="#1e2a4a" points="1505.25,-29.75 1505.25,-49.25 1644.75,-49.25 1644.75,-29.75 1505.25,-29.75"/>
<text xml:space="preserve" text-anchor="start" x="1522.88" y="-35.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">JSONB (flat upsert)</text>
</g>
<!-- Job&#45;&gt;StageOutput -->
<g id="edge8" class="edge">
<title>Job&#45;&gt;StageOutput</title>
<path fill="none" stroke="#4a5568" d="M985.05,-85.59C1077.88,-87.42 1205.62,-90.07 1318,-92.88 1346.37,-93.58 1376.84,-94.42 1405.69,-95.25"/>
<polygon fill="#4a5568" stroke="#4a5568" points="1405.47,-98.74 1415.57,-95.53 1405.67,-91.74 1405.47,-98.74"/>
<text xml:space="preserve" text-anchor="middle" x="1182.88" y="-95.58" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">job_id</text>
</g>
<!-- Checkpoint&#45;&gt;Checkpoint -->
<g id="edge7" class="edge">
<title>Checkpoint&#45;&gt;Checkpoint</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M1154.6,-259.25C1159,-270.14 1168.43,-277.5 1182.88,-277.5 1192.58,-277.5 1200.02,-274.18 1205.2,-268.69"/>
<polygon fill="#4a5568" stroke="#4a5568" points="1207.97,-270.86 1210.35,-260.53 1202.05,-267.12 1207.97,-270.86"/>
<text xml:space="preserve" text-anchor="middle" x="1182.88" y="-291.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">parent_id</text>
<text xml:space="preserve" text-anchor="middle" x="1182.88" y="-280.2" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">(tree)</text>
</g>
<!-- Checkpoint&#45;&gt;StageOutput -->
<g id="edge9" class="edge">
<title>Checkpoint&#45;&gt;StageOutput</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="1,5" d="M1317.69,-152.86C1346.78,-145.63 1377.48,-138 1406.32,-130.83"/>
<polygon fill="#4a5568" stroke="#4a5568" points="1406.95,-134.28 1415.81,-128.47 1405.26,-127.49 1406.95,-134.28"/>
<text xml:space="preserve" text-anchor="middle" x="1367.5" y="-150.53" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">checkpoint_id</text>
</g>
<!-- Brand -->
<g id="node7" class="node">
<title>Brand</title>
<polygon fill="#121829" stroke="none" points="8,-302.25 8,-402 301.25,-402 301.25,-302.25 8,-302.25"/>
<polygon fill="#0d1a33" stroke="none" points="8,-380.25 8,-402 301.25,-402 301.25,-380.25 8,-380.25"/>
<polygon fill="none" stroke="#1e2a4a" points="8,-380.25 8,-402 301.25,-402 301.25,-380.25 8,-380.25"/>
<text xml:space="preserve" text-anchor="start" x="138.12" y="-389.55" font-family="JetBrains Mono" font-weight="bold" font-size="11.00" fill="#0066ff">Brand</text>
<polygon fill="none" stroke="#1e2a4a" points="8,-360.75 8,-380.25 101.75,-380.25 101.75,-360.75 8,-360.75"/>
<text xml:space="preserve" text-anchor="start" x="11" y="-366.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">canonical_name</text>
<polygon fill="none" stroke="#1e2a4a" points="101.75,-360.75 101.75,-380.25 301.25,-380.25 301.25,-360.75 101.75,-360.75"/>
<text xml:space="preserve" text-anchor="start" x="166.25" y="-366.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str (indexed)</text>
<polygon fill="none" stroke="#1e2a4a" points="8,-341.25 8,-360.75 101.75,-360.75 101.75,-341.25 8,-341.25"/>
<text xml:space="preserve" text-anchor="start" x="35.75" y="-347.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">aliases</text>
<polygon fill="none" stroke="#1e2a4a" points="101.75,-341.25 101.75,-360.75 301.25,-360.75 301.25,-341.25 101.75,-341.25"/>
<text xml:space="preserve" text-anchor="start" x="189.5" y="-347.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">str[]</text>
<polygon fill="none" stroke="#1e2a4a" points="8,-321.75 8,-341.25 101.75,-341.25 101.75,-321.75 8,-321.75"/>
<text xml:space="preserve" text-anchor="start" x="36.5" y="-327.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">source</text>
<polygon fill="none" stroke="#1e2a4a" points="101.75,-321.75 101.75,-341.25 301.25,-341.25 301.25,-321.75 101.75,-321.75"/>
<text xml:space="preserve" text-anchor="start" x="104.75" y="-327.8" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">ocr / local_vlm / cloud_llm / manual</text>
<polygon fill="none" stroke="#1e2a4a" points="8,-302.25 8,-321.75 101.75,-321.75 101.75,-302.25 8,-302.25"/>
<text xml:space="preserve" text-anchor="start" x="36.5" y="-308.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8892a8">airings</text>
<polygon fill="none" stroke="#1e2a4a" points="101.75,-302.25 101.75,-321.75 301.25,-321.75 301.25,-302.25 101.75,-302.25"/>
<text xml:space="preserve" text-anchor="start" x="179.75" y="-308.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">JSONB[]</text>
</g> </g>
</g> </g>
</svg> </svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 28 KiB

42
docs/architecture/03-detection-pipeline.dot Normal file
View File

@@ -0,0 +1,42 @@
digraph detection_pipeline {
rankdir=TB
bgcolor="#0a0e17"
fontname="Helvetica"
node [fontname="Helvetica" fontsize=11 style=filled color="#1e2a4a" fontcolor="#e8eaf0" shape=box]
edge [fontname="Helvetica" fontsize=9 fontcolor="#8892a8" color="#4a5568"]
label="Detection Pipeline (core/detect/graph/nodes.py)"
labelloc=t
fontsize=16
fontcolor="#0066ff"
extract_frames [label="extract_frames\n(ffmpeg, fps from profile)" fillcolor="#121829"]
filter_scenes [label="filter_scenes\n(scene-change filter)" fillcolor="#121829"]
field_seg [label="field_segmentation\n(HSV mask · GPU/WASM)" fillcolor="#0d1a33" fontcolor="#0066ff"]
detect_edges [label="detect_edges\n(Canny + Hough · GPU/WASM)" fillcolor="#0d1a33" fontcolor="#0066ff"]
detect_objects [label="detect_objects\n(YOLO · GPU)" fillcolor="#1a3a1a" fontcolor="#00c853"]
preprocess [label="preprocess\n(crop · contrast · deskew)" fillcolor="#121829"]
run_ocr [label="run_ocr\n(OCR · GPU)" fillcolor="#1a3a1a" fontcolor="#00c853"]
match_brands [label="match_brands\n(rapidfuzz vs session)" fillcolor="#121829"]
escalate_vlm [label="escalate_vlm\n(local VLM · GPU)" fillcolor="#1a3a1a" fontcolor="#00c853"]
escalate_cloud [label="escalate_cloud\n(Anthropic · Gemini\nOpenAI · Groq)" fillcolor="#243056" shape=octagon]
compile_report [label="compile_report\n(timeline + brand stats)" fillcolor="#0d1a33" fontcolor="#0066ff"]
extract_frames -> filter_scenes
filter_scenes -> field_seg
filter_scenes -> detect_objects
field_seg -> detect_edges [label="masks"]
detect_edges -> detect_objects [style=dashed label="region hints"]
detect_objects -> preprocess [label="boxes"]
preprocess -> run_ocr
run_ocr -> match_brands [label="text candidates"]
match_brands -> escalate_vlm [label="unresolved"]
escalate_vlm -> escalate_cloud [label="still unresolved"]
match_brands -> compile_report
escalate_vlm -> compile_report
escalate_cloud -> compile_report
}

176
docs/architecture/03-detection-pipeline.svg Normal file
View File

@@ -0,0 +1,176 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: detection_pipeline Pages: 1 -->
<svg width="410pt" height="901pt"
viewBox="0.00 0.00 410.00 901.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 897.24)">
<title>detection_pipeline</title>
<polygon fill="#0a0e17" stroke="none" points="-4,4 -4,-897.24 406.25,-897.24 406.25,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="201.12" y="-874.04" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">Detection Pipeline (core/detect/graph/nodes.py)</text>
<!-- extract_frames -->
<g id="node1" class="node">
<title>extract_frames</title>
<polygon fill="#121829" stroke="#1e2a4a" points="349.19,-865.74 194.44,-865.74 194.44,-829.74 349.19,-829.74 349.19,-865.74"/>
<text xml:space="preserve" text-anchor="middle" x="271.82" y="-850.79" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">extract_frames</text>
<text xml:space="preserve" text-anchor="middle" x="271.82" y="-837.29" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(ffmpeg, fps from profile)</text>
</g>
<!-- filter_scenes -->
<g id="node2" class="node">
<title>filter_scenes</title>
<polygon fill="#121829" stroke="#1e2a4a" points="336.82,-792.74 206.82,-792.74 206.82,-756.74 336.82,-756.74 336.82,-792.74"/>
<text xml:space="preserve" text-anchor="middle" x="271.82" y="-777.79" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">filter_scenes</text>
<text xml:space="preserve" text-anchor="middle" x="271.82" y="-764.29" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(scene&#45;change filter)</text>
</g>
<!-- extract_frames&#45;&gt;filter_scenes -->
<g id="edge1" class="edge">
<title>extract_frames&#45;&gt;filter_scenes</title>
<path fill="none" stroke="#4a5568" d="M271.82,-829.55C271.82,-821.97 271.82,-812.84 271.82,-804.28"/>
<polygon fill="#4a5568" stroke="#4a5568" points="275.32,-804.28 271.82,-794.28 268.32,-804.28 275.32,-804.28"/>
</g>
<!-- field_seg -->
<g id="node3" class="node">
<title>field_seg</title>
<polygon fill="#0d1a33" stroke="#1e2a4a" points="292.44,-719.74 139.19,-719.74 139.19,-683.74 292.44,-683.74 292.44,-719.74"/>
<text xml:space="preserve" text-anchor="middle" x="215.82" y="-704.79" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">field_segmentation</text>
<text xml:space="preserve" text-anchor="middle" x="215.82" y="-691.29" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">(HSV mask · GPU/WASM)</text>
</g>
<!-- filter_scenes&#45;&gt;field_seg -->
<g id="edge2" class="edge">
<title>filter_scenes&#45;&gt;field_seg</title>
<path fill="none" stroke="#4a5568" d="M258.26,-756.55C251.66,-748.18 243.58,-737.94 236.25,-728.64"/>
<polygon fill="#4a5568" stroke="#4a5568" points="239.13,-726.64 230.18,-720.96 233.63,-730.98 239.13,-726.64"/>
</g>
<!-- detect_objects -->
<g id="node5" class="node">
<title>detect_objects</title>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="312.94,-553.24 216.69,-553.24 216.69,-517.24 312.94,-517.24 312.94,-553.24"/>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-538.29" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">detect_objects</text>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-524.79" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">(YOLO · GPU)</text>
</g>
<!-- filter_scenes&#45;&gt;detect_objects -->
<g id="edge3" class="edge">
<title>filter_scenes&#45;&gt;detect_objects</title>
<path fill="none" stroke="#4a5568" d="M284,-756.64C290.55,-746.44 298.05,-732.94 301.82,-719.74 316.35,-668.74 313.7,-652.93 305.82,-600.49 303.79,-587.04 303.57,-583.05 296.82,-571.24 295.07,-568.19 293.02,-565.19 290.82,-562.3"/>
<polygon fill="#4a5568" stroke="#4a5568" points="293.53,-560.09 284.4,-554.71 288.18,-564.61 293.53,-560.09"/>
</g>
<!-- detect_edges -->
<g id="node4" class="node">
<title>detect_edges</title>
<polygon fill="#0d1a33" stroke="#1e2a4a" points="296.82,-636.49 112.82,-636.49 112.82,-600.49 296.82,-600.49 296.82,-636.49"/>
<text xml:space="preserve" text-anchor="middle" x="204.82" y="-621.54" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">detect_edges</text>
<text xml:space="preserve" text-anchor="middle" x="204.82" y="-608.04" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">(Canny + Hough · GPU/WASM)</text>
</g>
<!-- field_seg&#45;&gt;detect_edges -->
<g id="edge4" class="edge">
<title>field_seg&#45;&gt;detect_edges</title>
<path fill="none" stroke="#4a5568" d="M213.48,-683.51C212.09,-673.24 210.29,-659.94 208.69,-648.13"/>
<polygon fill="#4a5568" stroke="#4a5568" points="212.2,-647.92 207.39,-638.48 205.26,-648.86 212.2,-647.92"/>
<text xml:space="preserve" text-anchor="middle" x="225.22" y="-657.19" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">masks</text>
</g>
<!-- detect_edges&#45;&gt;detect_objects -->
<g id="edge5" class="edge">
<title>detect_edges&#45;&gt;detect_objects</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M217.54,-600.26C225.6,-589.35 236.18,-575.02 245.29,-562.68"/>
<polygon fill="#4a5568" stroke="#4a5568" points="247.88,-565.07 251,-554.94 242.25,-560.91 247.88,-565.07"/>
<text xml:space="preserve" text-anchor="middle" x="265.41" y="-573.94" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">region hints</text>
</g>
<!-- preprocess -->
<g id="node6" class="node">
<title>preprocess</title>
<polygon fill="#121829" stroke="#1e2a4a" points="344.07,-469.99 185.57,-469.99 185.57,-433.99 344.07,-433.99 344.07,-469.99"/>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-455.04" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">preprocess</text>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-441.54" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(crop · contrast · deskew)</text>
</g>
<!-- detect_objects&#45;&gt;preprocess -->
<g id="edge6" class="edge">
<title>detect_objects&#45;&gt;preprocess</title>
<path fill="none" stroke="#4a5568" d="M264.82,-517.01C264.82,-506.74 264.82,-493.44 264.82,-481.63"/>
<polygon fill="#4a5568" stroke="#4a5568" points="268.32,-481.99 264.82,-471.99 261.32,-481.99 268.32,-481.99"/>
<text xml:space="preserve" text-anchor="middle" x="277.94" y="-490.69" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">boxes</text>
</g>
<!-- run_ocr -->
<g id="node7" class="node">
<title>run_ocr</title>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="306.57,-396.99 223.07,-396.99 223.07,-360.99 306.57,-360.99 306.57,-396.99"/>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-382.04" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">run_ocr</text>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-368.54" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">(OCR · GPU)</text>
</g>
<!-- preprocess&#45;&gt;run_ocr -->
<g id="edge7" class="edge">
<title>preprocess&#45;&gt;run_ocr</title>
<path fill="none" stroke="#4a5568" d="M264.82,-433.8C264.82,-426.22 264.82,-417.09 264.82,-408.53"/>
<polygon fill="#4a5568" stroke="#4a5568" points="268.32,-408.53 264.82,-398.53 261.32,-408.53 268.32,-408.53"/>
</g>
<!-- match_brands -->
<g id="node8" class="node">
<title>match_brands</title>
<polygon fill="#121829" stroke="#1e2a4a" points="333.19,-313.74 196.44,-313.74 196.44,-277.74 333.19,-277.74 333.19,-313.74"/>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-298.79" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">match_brands</text>
<text xml:space="preserve" text-anchor="middle" x="264.82" y="-285.29" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(rapidfuzz vs session)</text>
</g>
<!-- run_ocr&#45;&gt;match_brands -->
<g id="edge8" class="edge">
<title>run_ocr&#45;&gt;match_brands</title>
<path fill="none" stroke="#4a5568" d="M264.82,-360.76C264.82,-350.49 264.82,-337.19 264.82,-325.38"/>
<polygon fill="#4a5568" stroke="#4a5568" points="268.32,-325.74 264.82,-315.74 261.32,-325.74 268.32,-325.74"/>
<text xml:space="preserve" text-anchor="middle" x="300.07" y="-334.44" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">text candidates</text>
</g>
<!-- escalate_vlm -->
<g id="node9" class="node">
<title>escalate_vlm</title>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="278.82,-230.49 166.82,-230.49 166.82,-194.49 278.82,-194.49 278.82,-230.49"/>
<text xml:space="preserve" text-anchor="middle" x="222.82" y="-215.54" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">escalate_vlm</text>
<text xml:space="preserve" text-anchor="middle" x="222.82" y="-202.04" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">(local VLM · GPU)</text>
</g>
<!-- match_brands&#45;&gt;escalate_vlm -->
<g id="edge9" class="edge">
<title>match_brands&#45;&gt;escalate_vlm</title>
<path fill="none" stroke="#4a5568" d="M253.63,-277.56C250.16,-271.97 246.44,-265.67 243.32,-259.74 240.25,-253.91 237.22,-247.53 234.47,-241.41"/>
<polygon fill="#4a5568" stroke="#4a5568" points="237.69,-240.03 230.48,-232.27 231.27,-242.83 237.69,-240.03"/>
<text xml:space="preserve" text-anchor="middle" x="268.07" y="-251.19" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">unresolved</text>
</g>
<!-- compile_report -->
<g id="node11" class="node">
<title>compile_report</title>
<polygon fill="#0d1a33" stroke="#1e2a4a" points="343.19,-36 194.44,-36 194.44,0 343.19,0 343.19,-36"/>
<text xml:space="preserve" text-anchor="middle" x="268.82" y="-21.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">compile_report</text>
<text xml:space="preserve" text-anchor="middle" x="268.82" y="-7.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">(timeline + brand stats)</text>
</g>
<!-- match_brands&#45;&gt;compile_report -->
<g id="edge11" class="edge">
<title>match_brands&#45;&gt;compile_report</title>
<path fill="none" stroke="#4a5568" d="M282.53,-277.29C286.71,-272.08 290.6,-266.06 292.82,-259.74 294.47,-255.02 292.91,-253.49 292.82,-248.49 291.26,-162.01 306.72,-137.93 285.82,-54 285.21,-51.55 284.41,-49.07 283.5,-46.62"/>
<polygon fill="#4a5568" stroke="#4a5568" points="286.79,-45.4 279.57,-37.65 280.38,-48.21 286.79,-45.4"/>
</g>
<!-- escalate_cloud -->
<g id="node10" class="node">
<title>escalate_cloud</title>
<polygon fill="#243056" stroke="#1e2a4a" points="240.57,-94.74 240.57,-125.5 185.65,-147.24 107.98,-147.24 53.06,-125.5 53.06,-94.74 107.98,-73 185.65,-73 240.57,-94.74"/>
<text xml:space="preserve" text-anchor="middle" x="146.82" y="-119.92" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">escalate_cloud</text>
<text xml:space="preserve" text-anchor="middle" x="146.82" y="-106.42" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">(Anthropic · Gemini</text>
<text xml:space="preserve" text-anchor="middle" x="146.82" y="-92.92" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">OpenAI · Groq)</text>
</g>
<!-- escalate_vlm&#45;&gt;escalate_cloud -->
<g id="edge10" class="edge">
<title>escalate_vlm&#45;&gt;escalate_cloud</title>
<path fill="none" stroke="#4a5568" d="M203.52,-194.11C198.01,-188.71 192.18,-182.58 187.32,-176.49 182.39,-170.32 177.58,-163.51 173.1,-156.69"/>
<polygon fill="#4a5568" stroke="#4a5568" points="176.34,-155.27 168.01,-148.72 170.44,-159.03 176.34,-155.27"/>
<text xml:space="preserve" text-anchor="middle" x="221.07" y="-167.94" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">still unresolved</text>
</g>
<!-- escalate_vlm&#45;&gt;compile_report -->
<g id="edge12" class="edge">
<title>escalate_vlm&#45;&gt;compile_report</title>
<path fill="none" stroke="#4a5568" d="M242.63,-194.26C247.42,-189.05 251.96,-182.97 254.82,-176.49 273.25,-134.61 273.61,-80.49 271.62,-47.83"/>
<polygon fill="#4a5568" stroke="#4a5568" points="275.12,-47.66 270.89,-37.94 268.14,-48.18 275.12,-47.66"/>
</g>
<!-- escalate_cloud&#45;&gt;compile_report -->
<g id="edge13" class="edge">
<title>escalate_cloud&#45;&gt;compile_report</title>
<path fill="none" stroke="#4a5568" d="M192.59,-75.31C207.2,-64.51 223.06,-52.8 236.51,-42.86"/>
<polygon fill="#4a5568" stroke="#4a5568" points="238.21,-45.96 244.17,-37.2 234.05,-40.33 238.21,-45.96"/>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

104
docs/architecture/03-job-flow.dot
View File

@@ -1,104 +0,0 @@
digraph job_flow {
rankdir=TB
node [shape=box, style=rounded, fontname="Helvetica"]
edge [fontname="Helvetica", fontsize=10]
labelloc="t"
label="MPR - Job Flow"
fontsize=16
fontname="Helvetica-Bold"
graph [splines=ortho, nodesep=0.6, ranksep=0.6]
// API entry points
subgraph cluster_api {
label="API Entry Points"
style=dashed
color=gray
rest_create [label="POST /api/jobs/", shape=ellipse]
gql_create [label="mutation createJob", shape=ellipse]
rest_cancel [label="POST /api/jobs/{id}/cancel", shape=ellipse]
rest_callback [label="POST /api/jobs/{id}/callback", shape=ellipse]
}
// Job states
subgraph cluster_states {
label="Job States"
style=filled
fillcolor="#f8f8f8"
pending [label="PENDING", fillcolor="#ffc107", style="filled,rounded"]
processing [label="PROCESSING", fillcolor="#17a2b8", style="filled,rounded", fontcolor=white]
completed [label="COMPLETED", fillcolor="#28a745", style="filled,rounded", fontcolor=white]
failed [label="FAILED", fillcolor="#dc3545", style="filled,rounded", fontcolor=white]
cancelled [label="CANCELLED", fillcolor="#6c757d", style="filled,rounded", fontcolor=white]
}
// State transitions
pending -> processing [xlabel="worker picks up"]
processing -> completed [xlabel="success"]
processing -> failed [xlabel="error"]
pending -> cancelled [xlabel="user cancels"]
processing -> cancelled [xlabel="user cancels"]
failed -> pending [xlabel="retry"]
rest_create -> pending
gql_create -> pending
rest_cancel -> cancelled [style=dashed]
// Executor dispatch
subgraph cluster_dispatch {
label="Executor Dispatch"
style=filled
fillcolor="#fff8e8"
dispatch [label="MPR_EXECUTOR", shape=diamond]
}
pending -> dispatch
// Local path
subgraph cluster_local {
label="Local Mode (Celery)"
style=filled
fillcolor="#e8f4e8"
celery_task [label="Celery Task\n(transcode queue)"]
s3_download [label="S3 Download\n(MinIO)"]
ffmpeg_local [label="FFmpeg\ntranscode/trim"]
s3_upload [label="S3 Upload\n(MinIO)"]
db_update [label="DB Update\n(update_job_progress)"]
}
dispatch -> celery_task [xlabel="local"]
celery_task -> s3_download
s3_download -> ffmpeg_local
ffmpeg_local -> s3_upload
s3_upload -> db_update
db_update -> completed [style=dotted]
// Lambda path
subgraph cluster_lambda {
label="Lambda Mode (AWS)"
style=filled
fillcolor="#fde8d0"
sfn_start [label="Step Functions\nstart_execution"]
lambda_fn [label="Lambda\nFFmpeg container"]
s3_dl_aws [label="S3 Download\n(AWS)"]
ffmpeg_aws [label="FFmpeg\ntranscode/trim"]
s3_ul_aws [label="S3 Upload\n(AWS)"]
callback [label="HTTP Callback\nPOST /jobs/{id}/callback"]
}
dispatch -> sfn_start [xlabel="lambda"]
sfn_start -> lambda_fn
lambda_fn -> s3_dl_aws
s3_dl_aws -> ffmpeg_aws
ffmpeg_aws -> s3_ul_aws
s3_ul_aws -> callback
callback -> completed [style=dotted]
rest_callback -> completed [style=dashed, xlabel="Lambda reports"]
}

329
docs/architecture/03-job-flow.svg
View File

@@ -1,329 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: job_flow Pages: 1 -->
<svg width="1621pt" height="655pt"
viewBox="0.00 0.00 1621.00 655.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 650.5)">
<title>job_flow</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-650.5 1617,-650.5 1617,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="806.5" y="-627.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">MPR &#45; Job Flow</text>
<g id="clust1" class="cluster">
<title>cluster_api</title>
<polygon fill="none" stroke="gray" stroke-dasharray="5,2" points="297,-269.75 297,-349.25 1395,-349.25 1395,-269.75 297,-269.75"/>
<text xml:space="preserve" text-anchor="middle" x="846" y="-330.05" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">API Entry Points</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_states</title>
<polygon fill="#f8f8f8" stroke="black" points="572,-11.25 572,-261.75 939,-261.75 939,-11.25 572,-11.25"/>
<text xml:space="preserve" text-anchor="middle" x="755.5" y="-242.55" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Job States</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_dispatch</title>
<polygon fill="#fff8e8" stroke="black" points="103,-531.5 103,-611 377,-611 377,-531.5 103,-531.5"/>
<text xml:space="preserve" text-anchor="middle" x="240" y="-591.8" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Executor Dispatch</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_local</title>
<polygon fill="#e8f4e8" stroke="black" points="8,-93.5 8,-523.5 203,-523.5 203,-93.5 8,-93.5"/>
<text xml:space="preserve" text-anchor="middle" x="105.5" y="-504.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Local Mode (Celery)</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_lambda</title>
<polygon fill="#fde8d0" stroke="black" points="1403,-8 1403,-523.5 1605,-523.5 1605,-8 1403,-8"/>
<text xml:space="preserve" text-anchor="middle" x="1504" y="-504.3" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="16.00">Lambda Mode (AWS)</text>
</g>
<!-- rest_create -->
<g id="node1" class="node">
<title>rest_create</title>
<ellipse fill="none" stroke="black" cx="389" cy="-295.75" rx="84.35" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="389" y="-291.07" font-family="Helvetica,sans-Serif" font-size="14.00">POST /api/jobs/</text>
</g>
<!-- pending -->
<g id="node5" class="node">
<title>pending</title>
<path fill="#ffc107" stroke="black" d="M647.88,-226.25C647.88,-226.25 592.12,-226.25 592.12,-226.25 586.12,-226.25 580.12,-220.25 580.12,-214.25 580.12,-214.25 580.12,-202.25 580.12,-202.25 580.12,-196.25 586.12,-190.25 592.12,-190.25 592.12,-190.25 647.88,-190.25 647.88,-190.25 653.88,-190.25 659.88,-196.25 659.88,-202.25 659.88,-202.25 659.88,-214.25 659.88,-214.25 659.88,-220.25 653.88,-226.25 647.88,-226.25"/>
<text xml:space="preserve" text-anchor="middle" x="620" y="-203.57" font-family="Helvetica,sans-Serif" font-size="14.00">PENDING</text>
</g>
<!-- rest_create&#45;&gt;pending -->
<g id="edge7" class="edge">
<title>rest_create&#45;&gt;pending</title>
<path fill="none" stroke="black" d="M389,-277.61C389,-253.52 389,-214 389,-214 389,-214 568.25,-214 568.25,-214"/>
<polygon fill="black" stroke="black" points="568.25,-217.5 578.25,-214 568.25,-210.5 568.25,-217.5"/>
</g>
<!-- gql_create -->
<g id="node2" class="node">
<title>gql_create</title>
<ellipse fill="none" stroke="black" cx="620" cy="-295.75" rx="103.29" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="620" y="-291.07" font-family="Helvetica,sans-Serif" font-size="14.00">mutation createJob</text>
</g>
<!-- gql_create&#45;&gt;pending -->
<g id="edge8" class="edge">
<title>gql_create&#45;&gt;pending</title>
<path fill="none" stroke="black" d="M620,-277.62C620,-277.62 620,-238.17 620,-238.17"/>
<polygon fill="black" stroke="black" points="623.5,-238.17 620,-228.17 616.5,-238.17 623.5,-238.17"/>
</g>
<!-- rest_cancel -->
<g id="node3" class="node">
<title>rest_cancel</title>
<ellipse fill="none" stroke="black" cx="1247" cy="-295.75" rx="140.12" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="1247" y="-291.07" font-family="Helvetica,sans-Serif" font-size="14.00">POST /api/jobs/{id}/cancel</text>
</g>
<!-- cancelled -->
<g id="node9" class="node">
<title>cancelled</title>
<path fill="#6c757d" stroke="black" d="M918.62,-55.25C918.62,-55.25 843.38,-55.25 843.38,-55.25 837.38,-55.25 831.38,-49.25 831.38,-43.25 831.38,-43.25 831.38,-31.25 831.38,-31.25 831.38,-25.25 837.38,-19.25 843.38,-19.25 843.38,-19.25 918.62,-19.25 918.62,-19.25 924.62,-19.25 930.62,-25.25 930.62,-31.25 930.62,-31.25 930.62,-43.25 930.62,-43.25 930.62,-49.25 924.62,-55.25 918.62,-55.25"/>
<text xml:space="preserve" text-anchor="middle" x="881" y="-32.58" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">CANCELLED</text>
</g>
<!-- rest_cancel&#45;&gt;cancelled -->
<g id="edge9" class="edge">
<title>rest_cancel&#45;&gt;cancelled</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M1247,-277.56C1247,-218.66 1247,-37 1247,-37 1247,-37 942.64,-37 942.64,-37"/>
<polygon fill="black" stroke="black" points="942.64,-33.5 932.64,-37 942.64,-40.5 942.64,-33.5"/>
</g>
<!-- rest_callback -->
<g id="node4" class="node">
<title>rest_callback</title>
<ellipse fill="none" stroke="black" cx="915" cy="-295.75" rx="148.54" ry="18"/>
<text xml:space="preserve" text-anchor="middle" x="915" y="-291.07" font-family="Helvetica,sans-Serif" font-size="14.00">POST /api/jobs/{id}/callback</text>
</g>
<!-- completed -->
<g id="node7" class="node">
<title>completed</title>
<path fill="#28a745" stroke="black" d="M776.75,-55.25C776.75,-55.25 699.25,-55.25 699.25,-55.25 693.25,-55.25 687.25,-49.25 687.25,-43.25 687.25,-43.25 687.25,-31.25 687.25,-31.25 687.25,-25.25 693.25,-19.25 699.25,-19.25 699.25,-19.25 776.75,-19.25 776.75,-19.25 782.75,-19.25 788.75,-25.25 788.75,-31.25 788.75,-31.25 788.75,-43.25 788.75,-43.25 788.75,-49.25 782.75,-55.25 776.75,-55.25"/>
<text xml:space="preserve" text-anchor="middle" x="738" y="-32.58" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">COMPLETED</text>
</g>
<!-- rest_callback&#45;&gt;completed -->
<g id="edge24" class="edge">
<title>rest_callback&#45;&gt;completed</title>
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M783.42,-287.15C783.42,-287.15 783.42,-67.24 783.42,-67.24"/>
<polygon fill="black" stroke="black" points="786.92,-67.24 783.42,-57.24 779.92,-67.24 786.92,-67.24"/>
<text xml:space="preserve" text-anchor="middle" x="745.17" y="-180.44" font-family="Helvetica,sans-Serif" font-size="10.00">Lambda reports</text>
</g>
<!-- processing -->
<g id="node6" class="node">
<title>processing</title>
<path fill="#17a2b8" stroke="black" d="M768.75,-140.75C768.75,-140.75 685.25,-140.75 685.25,-140.75 679.25,-140.75 673.25,-134.75 673.25,-128.75 673.25,-128.75 673.25,-116.75 673.25,-116.75 673.25,-110.75 679.25,-104.75 685.25,-104.75 685.25,-104.75 768.75,-104.75 768.75,-104.75 774.75,-104.75 780.75,-110.75 780.75,-116.75 780.75,-116.75 780.75,-128.75 780.75,-128.75 780.75,-134.75 774.75,-140.75 768.75,-140.75"/>
<text xml:space="preserve" text-anchor="middle" x="727" y="-118.08" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">PROCESSING</text>
</g>
<!-- pending&#45;&gt;processing -->
<g id="edge1" class="edge">
<title>pending&#45;&gt;processing</title>
<path fill="none" stroke="black" d="M654.58,-189.87C654.58,-166.46 654.58,-129 654.58,-129 654.58,-129 661.34,-129 661.34,-129"/>
<polygon fill="black" stroke="black" points="661.34,-132.5 671.34,-129 661.34,-125.5 661.34,-132.5"/>
<text xml:space="preserve" text-anchor="middle" x="616.33" y="-159.3" font-family="Helvetica,sans-Serif" font-size="10.00">worker picks up</text>
</g>
<!-- pending&#45;&gt;cancelled -->
<g id="edge4" class="edge">
<title>pending&#45;&gt;cancelled</title>
<path fill="none" stroke="black" d="M660.36,-208C737.33,-208 897.54,-208 897.54,-208 897.54,-208 897.54,-67.04 897.54,-67.04"/>
<polygon fill="black" stroke="black" points="901.04,-67.04 897.54,-57.04 894.04,-67.04 901.04,-67.04"/>
<text xml:space="preserve" text-anchor="middle" x="819.06" y="-211.25" font-family="Helvetica,sans-Serif" font-size="10.00">user cancels</text>
</g>
<!-- dispatch -->
<g id="node10" class="node">
<title>dispatch</title>
<path fill="none" stroke="black" d="M228.12,-573.84C228.12,-573.84 122.92,-559.16 122.92,-559.16 116.98,-558.33 116.98,-556.67 122.92,-555.84 122.92,-555.84 228.12,-541.16 228.12,-541.16 234.06,-540.33 245.94,-540.33 251.88,-541.16 251.88,-541.16 357.08,-555.84 357.08,-555.84 363.02,-556.67 363.02,-558.33 357.08,-559.16 357.08,-559.16 251.88,-573.84 251.88,-573.84 245.94,-574.67 234.06,-574.67 228.12,-573.84"/>
<text xml:space="preserve" text-anchor="middle" x="240" y="-552.83" font-family="Helvetica,sans-Serif" font-size="14.00">MPR_EXECUTOR</text>
</g>
<!-- pending&#45;&gt;dispatch -->
<g id="edge10" class="edge">
<title>pending&#45;&gt;dispatch</title>
<path fill="none" stroke="black" d="M579.92,-202C483.92,-202 248.76,-202 248.76,-202 248.76,-202 248.76,-528.84 248.76,-528.84"/>
<polygon fill="black" stroke="black" points="245.26,-528.84 248.76,-538.84 252.26,-528.84 245.26,-528.84"/>
</g>
<!-- processing&#45;&gt;completed -->
<g id="edge2" class="edge">
<title>processing&#45;&gt;completed</title>
<path fill="none" stroke="black" d="M734,-104.62C734,-104.62 734,-67.16 734,-67.16"/>
<polygon fill="black" stroke="black" points="737.5,-67.16 734,-57.16 730.5,-67.16 737.5,-67.16"/>
<text xml:space="preserve" text-anchor="middle" x="714.88" y="-89.14" font-family="Helvetica,sans-Serif" font-size="10.00">success</text>
</g>
<!-- failed -->
<g id="node8" class="node">
<title>failed</title>
<path fill="#dc3545" stroke="black" d="M632,-55.25C632,-55.25 592,-55.25 592,-55.25 586,-55.25 580,-49.25 580,-43.25 580,-43.25 580,-31.25 580,-31.25 580,-25.25 586,-19.25 592,-19.25 592,-19.25 632,-19.25 632,-19.25 638,-19.25 644,-25.25 644,-31.25 644,-31.25 644,-43.25 644,-43.25 644,-49.25 638,-55.25 632,-55.25"/>
<text xml:space="preserve" text-anchor="middle" x="612" y="-32.58" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">FAILED</text>
</g>
<!-- processing&#45;&gt;failed -->
<g id="edge3" class="edge">
<title>processing&#45;&gt;failed</title>
<path fill="none" stroke="black" d="M680.25,-104.62C680.25,-77.88 680.25,-31 680.25,-31 680.25,-31 655.64,-31 655.64,-31"/>
<polygon fill="black" stroke="black" points="655.64,-27.5 645.64,-31 655.64,-34.5 655.64,-27.5"/>
<text xml:space="preserve" text-anchor="middle" x="668.62" y="-58.76" font-family="Helvetica,sans-Serif" font-size="10.00">error</text>
</g>
<!-- processing&#45;&gt;cancelled -->
<g id="edge5" class="edge">
<title>processing&#45;&gt;cancelled</title>
<path fill="none" stroke="black" d="M780.93,-123C819.44,-123 864.46,-123 864.46,-123 864.46,-123 864.46,-66.95 864.46,-66.95"/>
<polygon fill="black" stroke="black" points="867.96,-66.95 864.46,-56.95 860.96,-66.95 867.96,-66.95"/>
<text xml:space="preserve" text-anchor="middle" x="820.35" y="-126.25" font-family="Helvetica,sans-Serif" font-size="10.00">user cancels</text>
</g>
<!-- failed&#45;&gt;pending -->
<g id="edge6" class="edge">
<title>failed&#45;&gt;pending</title>
<path fill="none" stroke="black" d="M612.06,-55.55C612.06,-55.55 612.06,-178.31 612.06,-178.31"/>
<polygon fill="black" stroke="black" points="608.56,-178.31 612.06,-188.31 615.56,-178.31 608.56,-178.31"/>
<text xml:space="preserve" text-anchor="middle" x="600.44" y="-120.18" font-family="Helvetica,sans-Serif" font-size="10.00">retry</text>
</g>
<!-- celery_task -->
<g id="node11" class="node">
<title>celery_task</title>
<path fill="none" stroke="black" d="M162.75,-488C162.75,-488 43.25,-488 43.25,-488 37.25,-488 31.25,-482 31.25,-476 31.25,-476 31.25,-457.5 31.25,-457.5 31.25,-451.5 37.25,-445.5 43.25,-445.5 43.25,-445.5 162.75,-445.5 162.75,-445.5 168.75,-445.5 174.75,-451.5 174.75,-457.5 174.75,-457.5 174.75,-476 174.75,-476 174.75,-482 168.75,-488 162.75,-488"/>
<text xml:space="preserve" text-anchor="middle" x="103" y="-470.7" font-family="Helvetica,sans-Serif" font-size="14.00">Celery Task</text>
<text xml:space="preserve" text-anchor="middle" x="103" y="-453.45" font-family="Helvetica,sans-Serif" font-size="14.00">(transcode queue)</text>
</g>
<!-- dispatch&#45;&gt;celery_task -->
<g id="edge11" class="edge">
<title>dispatch&#45;&gt;celery_task</title>
<path fill="none" stroke="black" d="M142.89,-552.62C142.89,-552.62 142.89,-499.67 142.89,-499.67"/>
<polygon fill="black" stroke="black" points="146.39,-499.67 142.89,-489.67 139.39,-499.67 146.39,-499.67"/>
<text xml:space="preserve" text-anchor="middle" x="131.27" y="-529.4" font-family="Helvetica,sans-Serif" font-size="10.00">local</text>
</g>
<!-- sfn_start -->
<g id="node16" class="node">
<title>sfn_start</title>
<path fill="none" stroke="black" d="M1525.88,-488C1525.88,-488 1428.12,-488 1428.12,-488 1422.12,-488 1416.12,-482 1416.12,-476 1416.12,-476 1416.12,-457.5 1416.12,-457.5 1416.12,-451.5 1422.12,-445.5 1428.12,-445.5 1428.12,-445.5 1525.88,-445.5 1525.88,-445.5 1531.88,-445.5 1537.88,-451.5 1537.88,-457.5 1537.88,-457.5 1537.88,-476 1537.88,-476 1537.88,-482 1531.88,-488 1525.88,-488"/>
<text xml:space="preserve" text-anchor="middle" x="1477" y="-470.7" font-family="Helvetica,sans-Serif" font-size="14.00">Step Functions</text>
<text xml:space="preserve" text-anchor="middle" x="1477" y="-453.45" font-family="Helvetica,sans-Serif" font-size="14.00">start_execution</text>
</g>
<!-- dispatch&#45;&gt;sfn_start -->
<g id="edge17" class="edge">
<title>dispatch&#45;&gt;sfn_start</title>
<path fill="none" stroke="black" d="M336.81,-552.63C336.81,-533.84 336.81,-467 336.81,-467 336.81,-467 1404.18,-467 1404.18,-467"/>
<polygon fill="black" stroke="black" points="1404.18,-470.5 1414.18,-467 1404.18,-463.5 1404.18,-470.5"/>
<text xml:space="preserve" text-anchor="middle" x="809.3" y="-470.25" font-family="Helvetica,sans-Serif" font-size="10.00">lambda</text>
</g>
<!-- s3_download -->
<g id="node12" class="node">
<title>s3_download</title>
<path fill="none" stroke="black" d="M144.38,-402.5C144.38,-402.5 61.62,-402.5 61.62,-402.5 55.62,-402.5 49.62,-396.5 49.62,-390.5 49.62,-390.5 49.62,-372 49.62,-372 49.62,-366 55.62,-360 61.62,-360 61.62,-360 144.38,-360 144.38,-360 150.38,-360 156.38,-366 156.38,-372 156.38,-372 156.38,-390.5 156.38,-390.5 156.38,-396.5 150.38,-402.5 144.38,-402.5"/>
<text xml:space="preserve" text-anchor="middle" x="103" y="-385.2" font-family="Helvetica,sans-Serif" font-size="14.00">S3 Download</text>
<text xml:space="preserve" text-anchor="middle" x="103" y="-367.95" font-family="Helvetica,sans-Serif" font-size="14.00">(MinIO)</text>
</g>
<!-- celery_task&#45;&gt;s3_download -->
<g id="edge12" class="edge">
<title>celery_task&#45;&gt;s3_download</title>
<path fill="none" stroke="black" d="M103,-445.17C103,-445.17 103,-414.33 103,-414.33"/>
<polygon fill="black" stroke="black" points="106.5,-414.33 103,-404.33 99.5,-414.33 106.5,-414.33"/>
</g>
<!-- ffmpeg_local -->
<g id="node13" class="node">
<title>ffmpeg_local</title>
<path fill="none" stroke="black" d="M153,-317C153,-317 59,-317 59,-317 53,-317 47,-311 47,-305 47,-305 47,-286.5 47,-286.5 47,-280.5 53,-274.5 59,-274.5 59,-274.5 153,-274.5 153,-274.5 159,-274.5 165,-280.5 165,-286.5 165,-286.5 165,-305 165,-305 165,-311 159,-317 153,-317"/>
<text xml:space="preserve" text-anchor="middle" x="106" y="-299.7" font-family="Helvetica,sans-Serif" font-size="14.00">FFmpeg</text>
<text xml:space="preserve" text-anchor="middle" x="106" y="-282.45" font-family="Helvetica,sans-Serif" font-size="14.00">transcode/trim</text>
</g>
<!-- s3_download&#45;&gt;ffmpeg_local -->
<g id="edge13" class="edge">
<title>s3_download&#45;&gt;ffmpeg_local</title>
<path fill="none" stroke="black" d="M103,-359.67C103,-359.67 103,-328.83 103,-328.83"/>
<polygon fill="black" stroke="black" points="106.5,-328.83 103,-318.83 99.5,-328.83 106.5,-328.83"/>
</g>
<!-- s3_upload -->
<g id="node14" class="node">
<title>s3_upload</title>
<path fill="none" stroke="black" d="M138.62,-229.5C138.62,-229.5 75.38,-229.5 75.38,-229.5 69.38,-229.5 63.38,-223.5 63.38,-217.5 63.38,-217.5 63.38,-199 63.38,-199 63.38,-193 69.38,-187 75.38,-187 75.38,-187 138.62,-187 138.62,-187 144.62,-187 150.62,-193 150.62,-199 150.62,-199 150.62,-217.5 150.62,-217.5 150.62,-223.5 144.62,-229.5 138.62,-229.5"/>
<text xml:space="preserve" text-anchor="middle" x="107" y="-212.2" font-family="Helvetica,sans-Serif" font-size="14.00">S3 Upload</text>
<text xml:space="preserve" text-anchor="middle" x="107" y="-194.95" font-family="Helvetica,sans-Serif" font-size="14.00">(MinIO)</text>
</g>
<!-- ffmpeg_local&#45;&gt;s3_upload -->
<g id="edge14" class="edge">
<title>ffmpeg_local&#45;&gt;s3_upload</title>
<path fill="none" stroke="black" d="M107,-274.12C107,-274.12 107,-241.45 107,-241.45"/>
<polygon fill="black" stroke="black" points="110.5,-241.45 107,-231.45 103.5,-241.45 110.5,-241.45"/>
</g>
<!-- db_update -->
<g id="node15" class="node">
<title>db_update</title>
<path fill="none" stroke="black" d="M180.88,-144C180.88,-144 35.12,-144 35.12,-144 29.12,-144 23.12,-138 23.12,-132 23.12,-132 23.12,-113.5 23.12,-113.5 23.12,-107.5 29.12,-101.5 35.12,-101.5 35.12,-101.5 180.88,-101.5 180.88,-101.5 186.88,-101.5 192.88,-107.5 192.88,-113.5 192.88,-113.5 192.88,-132 192.88,-132 192.88,-138 186.88,-144 180.88,-144"/>
<text xml:space="preserve" text-anchor="middle" x="108" y="-126.7" font-family="Helvetica,sans-Serif" font-size="14.00">DB Update</text>
<text xml:space="preserve" text-anchor="middle" x="108" y="-109.45" font-family="Helvetica,sans-Serif" font-size="14.00">(update_job_progress)</text>
</g>
<!-- s3_upload&#45;&gt;db_update -->
<g id="edge15" class="edge">
<title>s3_upload&#45;&gt;db_update</title>
<path fill="none" stroke="black" d="M107,-186.67C107,-186.67 107,-155.83 107,-155.83"/>
<polygon fill="black" stroke="black" points="110.5,-155.83 107,-145.83 103.5,-155.83 110.5,-155.83"/>
</g>
<!-- db_update&#45;&gt;completed -->
<g id="edge16" class="edge">
<title>db_update&#45;&gt;completed</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M193.17,-117C345.61,-117 649.29,-117 649.29,-117 649.29,-117 649.29,-43 649.29,-43 649.29,-43 675.4,-43 675.4,-43"/>
<polygon fill="black" stroke="black" points="675.4,-46.5 685.4,-43 675.4,-39.5 675.4,-46.5"/>
</g>
<!-- lambda_fn -->
<g id="node17" class="node">
<title>lambda_fn</title>
<path fill="none" stroke="black" d="M1546,-402.5C1546,-402.5 1428,-402.5 1428,-402.5 1422,-402.5 1416,-396.5 1416,-390.5 1416,-390.5 1416,-372 1416,-372 1416,-366 1422,-360 1428,-360 1428,-360 1546,-360 1546,-360 1552,-360 1558,-366 1558,-372 1558,-372 1558,-390.5 1558,-390.5 1558,-396.5 1552,-402.5 1546,-402.5"/>
<text xml:space="preserve" text-anchor="middle" x="1487" y="-385.2" font-family="Helvetica,sans-Serif" font-size="14.00">Lambda</text>
<text xml:space="preserve" text-anchor="middle" x="1487" y="-367.95" font-family="Helvetica,sans-Serif" font-size="14.00">FFmpeg container</text>
</g>
<!-- sfn_start&#45;&gt;lambda_fn -->
<g id="edge18" class="edge">
<title>sfn_start&#45;&gt;lambda_fn</title>
<path fill="none" stroke="black" d="M1477,-445.17C1477,-445.17 1477,-414.33 1477,-414.33"/>
<polygon fill="black" stroke="black" points="1480.5,-414.33 1477,-404.33 1473.5,-414.33 1480.5,-414.33"/>
</g>
<!-- s3_dl_aws -->
<g id="node18" class="node">
<title>s3_dl_aws</title>
<path fill="none" stroke="black" d="M1534.38,-317C1534.38,-317 1451.62,-317 1451.62,-317 1445.62,-317 1439.62,-311 1439.62,-305 1439.62,-305 1439.62,-286.5 1439.62,-286.5 1439.62,-280.5 1445.62,-274.5 1451.62,-274.5 1451.62,-274.5 1534.38,-274.5 1534.38,-274.5 1540.38,-274.5 1546.38,-280.5 1546.38,-286.5 1546.38,-286.5 1546.38,-305 1546.38,-305 1546.38,-311 1540.38,-317 1534.38,-317"/>
<text xml:space="preserve" text-anchor="middle" x="1493" y="-299.7" font-family="Helvetica,sans-Serif" font-size="14.00">S3 Download</text>
<text xml:space="preserve" text-anchor="middle" x="1493" y="-282.45" font-family="Helvetica,sans-Serif" font-size="14.00">(AWS)</text>
</g>
<!-- lambda_fn&#45;&gt;s3_dl_aws -->
<g id="edge19" class="edge">
<title>lambda_fn&#45;&gt;s3_dl_aws</title>
<path fill="none" stroke="black" d="M1493,-359.67C1493,-359.67 1493,-328.83 1493,-328.83"/>
<polygon fill="black" stroke="black" points="1496.5,-328.83 1493,-318.83 1489.5,-328.83 1496.5,-328.83"/>
</g>
<!-- ffmpeg_aws -->
<g id="node19" class="node">
<title>ffmpeg_aws</title>
<path fill="none" stroke="black" d="M1545,-229.5C1545,-229.5 1451,-229.5 1451,-229.5 1445,-229.5 1439,-223.5 1439,-217.5 1439,-217.5 1439,-199 1439,-199 1439,-193 1445,-187 1451,-187 1451,-187 1545,-187 1545,-187 1551,-187 1557,-193 1557,-199 1557,-199 1557,-217.5 1557,-217.5 1557,-223.5 1551,-229.5 1545,-229.5"/>
<text xml:space="preserve" text-anchor="middle" x="1498" y="-212.2" font-family="Helvetica,sans-Serif" font-size="14.00">FFmpeg</text>
<text xml:space="preserve" text-anchor="middle" x="1498" y="-194.95" font-family="Helvetica,sans-Serif" font-size="14.00">transcode/trim</text>
</g>
<!-- s3_dl_aws&#45;&gt;ffmpeg_aws -->
<g id="edge20" class="edge">
<title>s3_dl_aws&#45;&gt;ffmpeg_aws</title>
<path fill="none" stroke="black" d="M1493,-274.12C1493,-274.12 1493,-241.45 1493,-241.45"/>
<polygon fill="black" stroke="black" points="1496.5,-241.45 1493,-231.45 1489.5,-241.45 1496.5,-241.45"/>
</g>
<!-- s3_ul_aws -->
<g id="node20" class="node">
<title>s3_ul_aws</title>
<path fill="none" stroke="black" d="M1532.62,-144C1532.62,-144 1469.38,-144 1469.38,-144 1463.38,-144 1457.38,-138 1457.38,-132 1457.38,-132 1457.38,-113.5 1457.38,-113.5 1457.38,-107.5 1463.38,-101.5 1469.38,-101.5 1469.38,-101.5 1532.62,-101.5 1532.62,-101.5 1538.62,-101.5 1544.62,-107.5 1544.62,-113.5 1544.62,-113.5 1544.62,-132 1544.62,-132 1544.62,-138 1538.62,-144 1532.62,-144"/>
<text xml:space="preserve" text-anchor="middle" x="1501" y="-126.7" font-family="Helvetica,sans-Serif" font-size="14.00">S3 Upload</text>
<text xml:space="preserve" text-anchor="middle" x="1501" y="-109.45" font-family="Helvetica,sans-Serif" font-size="14.00">(AWS)</text>
</g>
<!-- ffmpeg_aws&#45;&gt;s3_ul_aws -->
<g id="edge21" class="edge">
<title>ffmpeg_aws&#45;&gt;s3_ul_aws</title>
<path fill="none" stroke="black" d="M1501,-186.67C1501,-186.67 1501,-155.83 1501,-155.83"/>
<polygon fill="black" stroke="black" points="1504.5,-155.83 1501,-145.83 1497.5,-155.83 1504.5,-155.83"/>
</g>
<!-- callback -->
<g id="node21" class="node">
<title>callback</title>
<path fill="none" stroke="black" d="M1585.12,-58.5C1585.12,-58.5 1422.88,-58.5 1422.88,-58.5 1416.88,-58.5 1410.88,-52.5 1410.88,-46.5 1410.88,-46.5 1410.88,-28 1410.88,-28 1410.88,-22 1416.88,-16 1422.88,-16 1422.88,-16 1585.12,-16 1585.12,-16 1591.12,-16 1597.12,-22 1597.12,-28 1597.12,-28 1597.12,-46.5 1597.12,-46.5 1597.12,-52.5 1591.12,-58.5 1585.12,-58.5"/>
<text xml:space="preserve" text-anchor="middle" x="1504" y="-41.2" font-family="Helvetica,sans-Serif" font-size="14.00">HTTP Callback</text>
<text xml:space="preserve" text-anchor="middle" x="1504" y="-23.95" font-family="Helvetica,sans-Serif" font-size="14.00">POST /jobs/{id}/callback</text>
</g>
<!-- s3_ul_aws&#45;&gt;callback -->
<g id="edge22" class="edge">
<title>s3_ul_aws&#45;&gt;callback</title>
<path fill="none" stroke="black" d="M1501,-101.17C1501,-101.17 1501,-70.33 1501,-70.33"/>
<polygon fill="black" stroke="black" points="1504.5,-70.33 1501,-60.33 1497.5,-70.33 1504.5,-70.33"/>
</g>
<!-- callback&#45;&gt;completed -->
<g id="edge23" class="edge">
<title>callback&#45;&gt;completed</title>
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M1427.5,-58.88C1427.5,-69.48 1427.5,-80 1427.5,-80 1427.5,-80 786.08,-80 786.08,-80 786.08,-80 786.08,-67.14 786.08,-67.14"/>
<polygon fill="black" stroke="black" points="789.58,-67.14 786.08,-57.14 782.58,-67.14 789.58,-67.14"/>
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 24 KiB

181
docs/architecture/04-media-storage.md
View File

@@ -1,31 +1,24 @@
# Media Storage Architecture # Media & Artifact Storage
## Overview ## Overview
MPR uses **S3-compatible storage** everywhere. Locally via MinIO, in production via AWS S3. The same boto3 code and S3 keys work in both environments - the only difference is the `S3_ENDPOINT_URL` env var. MPR stores everything on **S3-compatible** object storage. Locally that's MinIO; in any
cloud target (AWS, GCS via HMAC, Cloudflare R2, etc.) it's the provider's S3 API. The
code in `core/storage/` uses boto3 throughout — only the endpoint URL and credentials
change between environments.
## Storage Strategy ## What goes where
### S3 Buckets | Bucket / prefix | Contents | Producer | Consumer |
|---|---|---|---|
| `mpr-media-in` | Source video files (chunks the user uploaded or device-recorded) | user / chunker UI | `extract_frames` stage, `core/api/detect/sources.py` |
| `mpr-media-out` | Per-job artifacts: extracted frame caches, debug overlays | pipeline stages, `core/api/detect/replay.py` overlays endpoints | UI panels (frame strip, overlay viewer) |
| Bucket | Env Var | Purpose | Both buckets live behind the same S3 client (`core/storage/`). DB rows store relative
|--------|---------|---------| keys (e.g. `chunks/2025-04-15/match-01.mp4`); the bucket is implicit.
| `mpr-media-in` | `S3_BUCKET_IN` | Source media files |
| `mpr-media-out` | `S3_BUCKET_OUT` | Transcoded/trimmed output |
### S3 Keys as File Paths ## Local development (MinIO)
- **Database**: Stores S3 object keys (e.g., `video1.mp4`, `subfolder/video3.mp4`)
- **Local dev**: MinIO serves these via S3 API on port 9000
- **AWS**: Real S3, same keys, different endpoint
### Why S3 Everywhere?
1. **Identical code paths** - no branching between local and cloud
2. **Seamless executor switching** - Celery and Lambda both use boto3
3. **Cloud-native** - ready for production without refactoring
## Local Development (MinIO)
### Configuration
```bash ```bash
S3_ENDPOINT_URL=http://minio:9000 S3_ENDPOINT_URL=http://minio:9000
S3_BUCKET_IN=mpr-media-in S3_BUCKET_IN=mpr-media-in
@@ -34,137 +27,49 @@ AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin
``` ```
### How It Works In the Tilt setup, MinIO runs as a k8s Deployment with port-forwards for `9000` (S3 API)
- MinIO runs as a Docker container (port 9000 API, port 9001 console) and `9001` (web console). A `minio-init` job creates the buckets on first start.
- `minio-init` container creates buckets and sets public read access on startup
- Nginx proxies `/media/in/` and `/media/out/` to MinIO buckets ## Cloud (AWS S3 / GCS / others)
- Upload files via MinIO Console (http://localhost:9001) or `mc` CLI
### Upload Files to MinIO
```bash ```bash
# Using mc CLI # AWS S3 — no endpoint URL needed
mc alias set local http://localhost:9000 minioadmin minioadmin S3_BUCKET_IN=...
mc cp video.mp4 local/mpr-media-in/ S3_BUCKET_OUT=...
# Using aws CLI with endpoint override
aws --endpoint-url http://localhost:9000 s3 cp video.mp4 s3://mpr-media-in/
```
## AWS Production (S3)
### Configuration
```bash
# No S3_ENDPOINT_URL = uses real AWS S3
S3_BUCKET_IN=mpr-media-in
S3_BUCKET_OUT=mpr-media-out
AWS_REGION=us-east-1 AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=<real-key> AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=<real-secret> AWS_SECRET_ACCESS_KEY=...
```
### Upload Files to S3 # GCS via HMAC
```bash
aws s3 cp video.mp4 s3://mpr-media-in/
aws s3 sync /local/media/ s3://mpr-media-in/
```
## GCP Production (GCS via S3 compatibility)
GCS exposes an S3-compatible API. The same `core/storage/s3.py` boto3 code works
with no changes — only the endpoint and credentials differ.
### GCS HMAC Keys
Generate under **Cloud Storage → Settings → Interoperability** in the GCP console.
These act as `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`.
### Configuration
```bash
S3_ENDPOINT_URL=https://storage.googleapis.com S3_ENDPOINT_URL=https://storage.googleapis.com
S3_BUCKET_IN=mpr-media-in AWS_ACCESS_KEY_ID=<gcs hmac access>
S3_BUCKET_OUT=mpr-media-out AWS_SECRET_ACCESS_KEY=<gcs hmac secret>
AWS_ACCESS_KEY_ID=<GCS HMAC access key>
AWS_SECRET_ACCESS_KEY=<GCS HMAC secret>
# Executor
MPR_EXECUTOR=gcp
GCP_PROJECT_ID=my-project
GCP_REGION=us-central1
CLOUD_RUN_JOB=mpr-transcode
CALLBACK_URL=https://mpr.mcrn.ar/api
CALLBACK_API_KEY=<secret>
``` ```
### Upload Files to GCS ## Database vs. object storage
```bash
gcloud storage cp video.mp4 gs://mpr-media-in/
# Or with the aws CLI via compat endpoint Heavy artifacts (frames, masks, overlays) live in MinIO/S3. The `Checkpoint` and
aws --endpoint-url https://storage.googleapis.com s3 cp video.mp4 s3://mpr-media-in/ `StageOutput` tables in Postgres (see `02-data-model.svg`) hold structured outputs
``` (detections, stats, references to S3 keys) — never blobs. Frame caches keyed by
`timeline_id` are written by the first run of `extract_frames` and reused by every
later replay on the same timeline.
### Cloud Run Job Handler ## Storage module
`core/task/gcp_handler.py` is the Cloud Run Job entrypoint. It reads the job payload
from `MPR_JOB_PAYLOAD` (injected by `GCPExecutor`), uses `core/storage` for all
GCS access (S3 compat), and POSTs the completion callback to the API.
Set the Cloud Run Job command to: `python -m core.task.gcp_handler` `core/storage/` exposes the small set of helpers callers need:
## Storage Module
`core/storage/` package provides all S3 operations:
```python ```python
from core.storage import ( from core.storage import (
get_s3_client, # boto3 client (MinIO or AWS) get_s3_client,
list_objects, # List bucket contents, filter by extension list_objects,
download_file, # Download S3 object to local path download_file,
download_to_temp, # Download to temp file (caller cleans up) download_to_temp,
upload_file, # Upload local file to S3 upload_file,
get_presigned_url, # Generate presigned URL get_presigned_url,
BUCKET_IN, # Input bucket name BUCKET_IN,
BUCKET_OUT, # Output bucket name BUCKET_OUT,
) )
``` ```
## API Endpoints Anything else (multipart, lifecycle, versioning) is the bucket's responsibility, not
the application's.
### Scan Media (REST)
```http
POST /api/assets/scan
```
Lists objects in `S3_BUCKET_IN`, registers new media files.
### Scan Media (GraphQL)
```graphql
mutation { scanMediaFolder { found registered skipped files } }
```
## Job Flow with S3
### Local Mode (Celery)
1. Celery task receives `source_key` and `output_key`
2. Downloads source from `S3_BUCKET_IN` to temp file
3. Runs FFmpeg locally
4. Uploads result to `S3_BUCKET_OUT`
5. Cleans up temp files
### Lambda Mode (AWS)
1. Step Functions invokes Lambda with S3 keys
2. Lambda downloads source from `S3_BUCKET_IN` to `/tmp`
3. Runs FFmpeg in container
4. Uploads result to `S3_BUCKET_OUT`
5. Calls back to API with result
### Cloud Run Job Mode (GCP)
1. `GCPExecutor` triggers Cloud Run Job with payload in `MPR_JOB_PAYLOAD`
2. `core/task/gcp_handler.py` downloads source from `S3_BUCKET_IN` (GCS S3 compat)
3. Runs FFmpeg in container
4. Uploads result to `S3_BUCKET_OUT` (GCS S3 compat)
5. Calls back to API with result
All three paths use the same S3-compatible bucket names and key structure.
## Supported File Types
**Video:** `.mp4`, `.mkv`, `.avi`, `.mov`, `.webm`, `.flv`, `.wmv`, `.m4v`
**Audio:** `.mp3`, `.wav`, `.flac`, `.aac`, `.ogg`, `.m4a`

290
docs/architecture/05-chunker-pipeline.md
View File

@@ -1,290 +0,0 @@
# Chunker Pipeline — Execution Path
## Overview
The chunker pipeline splits a media file into time-based segments using FFmpeg stream-copy. Events flow from worker threads through Redis and gRPC-Web streaming to the browser UI in real time.
**7 hops from worker thread to pixel:**
```
Worker thread → Pipeline._emit() → event_bridge() → Redis RPUSH
→ [50ms poll] gRPC server LRANGE → yield protobuf
→ HTTP/2 frame → Envoy (grpc-web filter)
→ HTTP/1.1 chunk → nginx (proxy_buffering off)
→ fetch ReadableStream → protobuf-ts decode
→ setEvents([...prev, evt]) → React re-render
```
---
## Step 1: Job Creation (Browser → GraphQL → Celery)
```
User clicks "Start"
→ App.tsx: handleStart(config)
→ api.ts: createChunkJob(config)
→ POST /graphql (nginx :80 → fastapi:8702)
→ graphql.py: Mutation.create_chunk_job()
→ core.db: creates ChunkJob row in Postgres
→ Celery: run_job.delay(job_type="chunk", job_id=..., payload=...)
→ Returns { id, celery_task_id } to browser
→ App.tsx: setJobId(id) — triggers gRPC stream subscription
```
**Files:** `ui/chunker/src/api.ts`, `core/api/graphql.py`, `core/jobs/task.py`
---
## Step 2: gRPC-Web Stream (Browser → nginx → Envoy → gRPC Server)
Once `jobId` is set, `useGrpcStream(jobId)` opens a server-streaming RPC:
```
useGrpcStream(jobId) fires useEffect
→ GrpcWebFetchTransport({ baseUrl: "/grpc-web" })
→ WorkerServiceClient.streamChunkPipeline({ jobId })
→ fetch() POST to /grpc-web/worker.WorkerService/StreamChunkPipeline
→ nginx :80 /grpc-web/ (proxy_pass → envoy:8090, proxy_buffering off)
→ Envoy :8090 (grpc_web filter: HTTP/1.1 grpc-web → HTTP/2 native gRPC)
→ gRPC server :50051 WorkerServicer.StreamChunkPipeline()
→ Enters Redis polling loop (Step 5)
```
**Files:** `ui/chunker/src/hooks/useGrpcStream.ts`, `ctrl/nginx.conf`, `ctrl/envoy.yaml`, `core/rpc/server.py`
**Key nginx config:** `proxy_buffering off` is critical — without it, nginx collects the entire upstream response before forwarding, defeating streaming entirely.
---
## Step 3: Celery Worker → ChunkHandler
```
Celery picks up run_job task
→ task.py: run_job(job_type="chunk", job_id, payload)
→ registry.get_handler("chunk") → ChunkHandler
→ chunk.py: ChunkHandler.process(job_id, payload)
→ download_to_temp(BUCKET_IN, source_key) — pulls source from MinIO/S3
→ Creates output_dir: /app/media/out/chunks/{job_id}/
→ Constructs event_bridge callback (bridges Pipeline events → Redis)
→ pipeline = Pipeline(source, ..., event_callback=event_bridge, output_dir=...)
→ pipeline.run()
```
**Files:** `core/jobs/task.py`, `core/jobs/handlers/chunk.py`
The `event_bridge` closure wraps every `Pipeline._emit()` call, forwarding to `push_event(job_id, event_type, data)` which writes to Redis.
---
## Step 4: Pipeline Orchestration (inside Celery worker process)
`Pipeline.run()` spawns multiple threads:
```
pipeline.run():
│
├─ Chunker(source, chunk_duration)
│ → ffprobe source file → gets duration, file_size
│ → calculates total_chunks = ceil(duration / chunk_duration)
│
├─ _emit("pipeline_start", {...}) → event_bridge → Redis
├─ _emit("pipeline_info", {file_size, duration, total_chunks}) → Redis
│
├─ Creates ChunkQueue(maxsize=10)
├─ Creates WorkerPool(num_workers=N, chunk_queue, processor, event_callback)
│
├─ pool.start() — spawns N worker threads
│
├─ MONITOR THREAD starts (_monitor_progress)
│ → Every 500ms: _emit("pipeline_progress", {elapsed, throughput_mbps}) → Redis
│
├─ PRODUCER THREAD starts (_produce_chunks)
│ → Iterates chunker.chunks() → yields Chunk(sequence, start_time, end_time)
│ → For each: chunk_queue.put(chunk)
│ → _emit("chunk_queued", {sequence, start_time, end_time, queue_size}) → Redis
│ → chunk_queue.close() when done (sends N sentinel Nones)
│
├─ WORKER THREADS (N concurrent, each runs worker.py:Worker.run())
│ │ Each worker loops:
│ │
│ ├─ chunk = chunk_queue.get(timeout=1.0)
│ ├─ _emit("chunk_processing", {sequence, state:"processing", queue_size}) → Redis
│ │
│ ├─ processor.process(chunk)
│ │ ├─ ffmpeg: runs `ffmpeg -ss start -to end -c copy chunk_NNNN.mp4`
│ │ ├─ simulated_decode: sleep(random) + checksum
│ │ └─ checksum: reads bytes, computes hash
│ │
│ ├─ On success: _emit("chunk_done", {sequence, processing_time, retries, queue_size}) → Redis
│ ├─ On failure: retries with exponential backoff (0.1s, 0.2s, 0.4s...)
│ │ └─ _emit("chunk_retry", {sequence, attempt, backoff}) → Redis
│ │ └─ _emit("chunk_error", {sequence, error, retries}) → Redis (after exhaustion)
│ │
│ └─ On sentinel (None): _emit("worker_status", {state:"stopped"}) → Redis
│
├─ pool.wait() — joins all worker threads, collects results
├─ monitor_stop.set() — stops progress monitor
│
├─ ResultCollector — reassembles results in sequence order
│ └─ _emit("chunk_collected", {sequence, buffered, emitted}) → Redis
│
├─ Writes manifest.json to output_dir
│
└─ _emit("pipeline_complete", {total_chunks, processed, failed, elapsed, throughput}) → Redis
```
**Files:** `core/chunker/pipeline.py`, `core/chunker/worker.py`, `core/chunker/pool.py`, `core/chunker/chunker.py`, `core/chunker/collector.py`
---
## Step 5: Redis — the Event Bus
```
WRITE side (Celery worker, all threads):
push_event(job_id, event_type, data)
→ json.dumps({"event": event_type, ...data})
→ Redis RPUSH to key "chunk_events:{job_id}"
→ Redis EXPIRE 3600 (1 hour TTL)
READ side (gRPC server, StreamChunkPipeline):
poll_events(job_id, cursor)
→ Redis LRANGE "chunk_events:{job_id}" cursor -1
→ Returns (parsed_events, new_cursor)
→ Called every 50ms (time.sleep(0.05) in server loop)
```
Redis acts as a decoupling layer between the Celery worker process (which runs the pipeline) and the gRPC server process (which streams to browsers). Events are appended with RPUSH and read with cursor-based LRANGE polling.
**Files:** `core/events.py`
---
## Step 6: gRPC Server → Envoy → nginx → Browser
```
server.py: StreamChunkPipeline polling loop:
while context.is_active():
events, cursor = poll_events(job_id, cursor) ← Redis LRANGE
for data in events:
yield worker_pb2.ChunkPipelineEvent( ← serialized protobuf message
job_id, event_type, sequence, worker_id,
state, queue_size, elapsed, throughput_mbps,
total_chunks, processed_chunks, failed_chunks,
error, processing_time, retries
)
if event_type in ("pipeline_complete", "pipeline_error"):
return ← ends the stream
time.sleep(0.05) ← 50ms poll interval
Each yield sends:
→ gRPC HTTP/2 DATA frame to Envoy
→ Envoy grpc_web filter: HTTP/2 → base64-encoded grpc-web-text
→ nginx proxy_pass (proxy_buffering off) → chunked HTTP/1.1 to browser
→ fetch() ReadableStream in GrpcWebFetchTransport
→ @protobuf-ts decodes protobuf → ChunkPipelineEvent TypeScript object
```
**Files:** `core/rpc/server.py`, `ctrl/envoy.yaml`, `ctrl/nginx.conf`, `ui/common/api/grpc/worker.ts`, `ui/common/api/grpc/worker.client.ts`
---
## Step 7: React State Derivation and Rendering
```
useGrpcStream.ts:
for await (const msg of stream.responses):
const evt = toEvent(msg) ← maps protobuf camelCase → snake_case PipelineEvent
setEvents(prev => [...prev, evt]) ← appends to events array
if pipeline_complete/error → setDone(true), break
App.tsx useMemo(events):
Iterates ALL events on every update, derives:
├─ chunkMap: Map<sequence, ChunkInfo> — state machine per chunk
│ pending → queued → processing → done/error/retry
├─ workerMap: Map<worker_id, WorkerInfo> — state per worker
│ idle → processing → idle → ... → stopped
├─ stats: PipelineStats
│ total_chunks, processed, failed, retries, elapsed, throughput_mbps, queue_size
├─ errors: ErrorEntry[] — every event containing an error field
└─ queueSize: number — last seen queue_size value
Renders:
├─ ChunkGrid — colored cells per chunk (pending/queued/processing/done/error)
├─ QueueGauge — current queue depth / max
├─ WorkerPanel — per-worker state + current chunk assignment
├─ StatsPanel — elapsed time, throughput, processed/failed counts
├─ ErrorLog — scrollable error list
└─ OutputFiles — download links (when done)
```
**Files:** `ui/chunker/src/hooks/useGrpcStream.ts`, `ui/chunker/src/App.tsx`
---
## Step 8: Output File Access (after pipeline completes)
```
App.tsx useEffect([done, jobId]):
→ api.ts: getChunkOutputFiles(jobId)
→ POST /graphql → graphql.py: chunk_output_files(job_id)
→ Reads /app/media/out/chunks/{job_id}/ directory listing from disk
→ Returns [{key, size, url: "/media/out/chunks/{job_id}/chunk_0001.mp4"}]
→ Browser renders download links
→ Click link → nginx /media/out/ → alias /app/media/out/ → serves file from disk
```
Chunks are written directly to `media/out/chunks/{job_id}/` by the ffmpeg processor — no MinIO upload needed for output. Nginx serves them with `autoindex on`.
**Files:** `core/api/graphql.py`, `core/jobs/handlers/chunk.py`, `ctrl/nginx.conf`
---
## Event Types Reference
| Event | Source | Key Fields |
|-------|--------|------------|
| `pipeline_start` | Pipeline.run() | source, chunk_duration, num_workers, processor_type |
| `pipeline_info` | Pipeline.run() | file_size, source_duration, total_chunks |
| `pipeline_progress` | Monitor thread (500ms) | elapsed, throughput_mbps |
| `chunk_queued` | Producer thread | sequence, start_time, end_time, duration, queue_size |
| `chunk_processing` | Worker thread | sequence, worker_id, state, queue_size |
| `chunk_done` | Worker thread | sequence, processing_time, retries, queue_size |
| `chunk_retry` | Worker thread | sequence, attempt, backoff |
| `chunk_error` | Worker thread | sequence, error, retries |
| `chunk_collected` | ResultCollector | sequence, buffered, emitted |
| `worker_status` | Worker thread | worker_id, state (idle/processing/stopped) |
| `pipeline_complete` | Pipeline.run() | total_chunks, processed, failed, elapsed, throughput_mbps |
| `pipeline_error` | Pipeline.run() | error |
---
## Thread Model (inside Celery worker)
```
Celery worker process
└─ run_job task thread
└─ Pipeline.run()
├─ Producer thread — enqueues chunks
├─ Monitor thread — emits progress every 500ms
├─ Worker thread 0 — pulls from queue, processes
├─ Worker thread 1 — pulls from queue, processes
├─ Worker thread 2 — pulls from queue, processes
└─ Worker thread 3 — pulls from queue, processes
```
All threads share the same `event_callback` → `event_bridge` → `push_event()`, which creates a new Redis connection per call. Thread-safe via Redis atomic RPUSH.
---
## Infrastructure
| Service | Port | Role |
|---------|------|------|
| nginx | 80 | Reverse proxy, static file serving |
| fastapi | 8702 | GraphQL API (Strawberry) |
| celery | — | Task worker (runs pipeline) |
| redis | 6379 | Event bus + Celery broker |
| grpc | 50051 | gRPC server (StreamChunkPipeline) |
| envoy | 8090 | gRPC-Web ↔ native gRPC translation |
| minio | 9000 | S3-compatible source media storage |
| postgres | 5432 | Job/asset metadata |

145
docs/architecture/05-detection-pipeline.md Normal file
View File

@@ -0,0 +1,145 @@
# Detection Pipeline — Execution Path
## Overview
A pipeline run is a sequence of named **stages** that read and write a shared
`DetectState` dict. Stages are defined in `core/detect/stages/`; the orchestrator
(`core/detect/graph/runner.py`) flattens the profile's `PipelineConfig` graph into a
linear order, runs each stage, and emits SSE events to the browser.
The full stage list is in `core/detect/graph/nodes.py`:
```
extract_frames → filter_scenes
→ field_segmentation → detect_edges
→ detect_objects → preprocess → run_ocr
→ match_brands → escalate_vlm → escalate_cloud
→ compile_report
```
See `03-detection-pipeline.svg` for the graph view.
## Profile
A `Profile` row in Postgres holds two JSONB blobs:
- `pipeline` — a `PipelineConfig` (stages + edges + routing rules) defining topology
- `configs` — `{stage_name: {...}}` per-stage parameters (fps, thresholds, prompts, ...)
Profiles are the config mechanism: **duplicate a profile and tweak it** instead of
patching defaults. `core/detect/profile.py` loads profiles by name; `_load_profile()`
in `nodes.py` merges the job's `config_overrides` on top.
## Stage runner
`PipelineRunner` (in `core/detect/graph/runner.py`) iterates the flattened stages and
between each one checks three control flags (all keyed by `job_id`):
- **cancel** — `set_cancel_check(job_id, fn)`; raises `PipelineCancelled` to abort
- **pause / resume** — a `threading.Event` per job; `_wait_if_paused()` blocks
- **step** — like resume but auto-pauses after the next stage completes
- **pause-after-stage** — toggle to step through every stage
Each stage runs inside `trace_node(state, name)` (sets a span used by tracing) and
emits `running` → `done` (or `skipped`) transitions via `core/detect/emit.py`.
## Inference: GPU-host indirection
`core/detect/graph/nodes.py` reads `INFERENCE_URL` from the environment and passes it
to every CV/ML stage:
- `INFERENCE_URL=""` (default in dev) — stages call CV/ML routines in-process
- `INFERENCE_URL=http://gpu-host:8000` — stages POST to the GPU server
(`core/gpu/server.py`) which exposes `/detect`, `/ocr`, `/preprocess`, `/vlm`,
`/detect_edges`, `/segment_field` (each with a `/debug` variant that returns
intermediate masks for the overlay viewer)
Memory note: dev and GPU machines are separate boxes on the same LAN; inference is a
network call. Heavy ML deps (`torch`, `transformers`, `paddleocr`) live only in
`core/gpu/pyproject.toml` — the API host doesn't need them.
## Browser-side CV (OpenCV WASM)
Some stages (notably the field/edge stages) can run in the browser via OpenCV WASM
(`ui/detection-app/src/cv/wasmBridge.ts`) for fast iteration without a round trip to
the GPU host. The browser UI is the test surface for the "replay loop" — change a
config, replay one stage, see the overlay. Browser CV uses OpenCV WASM directly; there
are no TypeScript ports of the algorithms.
## Cloud VLM escalation
`escalate_vlm` (local VLM on GPU host) and `escalate_cloud` (Anthropic / Gemini /
OpenAI / Groq via `core/detect/providers/`) are the last-resort branches for
unresolved candidates from `match_brands`. Skip flags:
- `SKIP_VLM=1` — emits `skipped` for `escalate_vlm`
- `SKIP_CLOUD=1` — emits `skipped` for `escalate_cloud`
## Checkpoints, StageOutput, and replay
Two tables back the replay loop:
- **Checkpoint** (`core/db/models.py:Checkpoint`) — a tree node:
`(parent_id, stage_name, config_overrides, stats)`. No blobs. Lets the UI show a
branching history of "what configs did we try at this stage?"
- **StageOutput** — a flat upsert table keyed by `(job_id, stage_name)` holding the
stage's output dict. `replay-stage` reads upstream outputs from here so a single
stage can be re-run without rerunning the whole pipeline.
API surface (`core/api/detect/replay.py`):
- `GET /checkpoints/{timeline_id}` — full tree
- `POST /replay` — clone a checkpoint into a new job, run from a chosen stage
- `POST /replay-stage` — re-run one stage in place using upstream `StageOutput` rows
- `GET /overlays/{timeline_id}/{job_id}/{stage}/{seq}` — debug overlays from MinIO
## Event flow (SSE)
Stages call `emit.transition(...)` / `emit.log(...)` / `emit.boxes(...)` etc.
(`core/detect/emit.py`). These push into Redis (`core/detect/events.py`). The SSE
endpoint `GET /detect/stream/{job_id}` (`core/api/detect/sse.py`) drains the Redis
list and writes to the open SSE response. Envoy keeps the connection open for up to
3600s (see `ctrl/k8s/base/envoy.yaml`).
```
stage code
→ emit.* (core/detect/emit.py)
→ push_detect_event → Redis RPUSH
→ [poll] /detect/stream/{job_id} → SSE chunk
→ fetch ReadableStream in detection-app
→ Pinia store update → Vue panel re-render
```
## Pipeline control endpoints
All under `core/api/detect/run.py`:
- `POST /run` — start a job from a timeline + profile
- `POST /stop/{job_id}` — cancel
- `POST /pause/{job_id}` / `POST /resume/{job_id}`
- `POST /step/{job_id}` — run one stage and pause
- `POST /pause-after-stage/{job_id}` — toggle pause-after-each-stage
- `GET /status/{job_id}` — current stage, progress
- `POST /clear/{job_id}` — discard runtime state
## Where the chunker UI fits
`ui/chunker/` is a **standalone testing utility** for the source-chunking step (split
a long source video into chunks the user picks for a Timeline). It is **not** a
pipeline stage and is not part of the detection flow. The detection pipeline reads
already-chunked sources from MinIO via `core/api/detect/sources.py`.
## Files
| Concern | File |
|---|---|
| Stage list | `core/detect/graph/nodes.py` |
| Runner (cancel/pause/resume) | `core/detect/graph/runner.py` |
| Profile loading | `core/detect/profile.py` |
| Event emission | `core/detect/emit.py`, `core/detect/events.py` |
| SSE endpoint | `core/api/detect/sse.py` |
| Replay API | `core/api/detect/replay.py` |
| Checkpoint storage | `core/detect/checkpoint/storage.py` |
| GPU server | `core/gpu/server.py` |
| Browser CV bridge | `ui/detection-app/src/cv/wasmBridge.ts` |
| Cloud VLM providers | `core/detect/providers/` |

209
docs/architecture/styles.css
View File

@@ -1,209 +0,0 @@
:root {
--bg-color: #1a1a2e;
--text-color: #e8e8e8;
--accent-color: #4a90d9;
--border-color: #333;
--sidebar-width: 220px;
--sidebar-bg: #151528;
}
* {
box-sizing: border-box;
margin: 0;
padding: 0;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, sans-serif;
background-color: var(--bg-color);
color: var(--text-color);
line-height: 1.6;
}
/* Sidebar navigation */
.sidebar {
position: fixed;
top: 0;
left: 0;
width: var(--sidebar-width);
height: 100vh;
background: var(--sidebar-bg);
border-right: 1px solid var(--border-color);
padding: 1.5rem 1rem;
overflow-y: auto;
z-index: 10;
}
.sidebar h2 {
font-size: 1.2rem;
color: var(--accent-color);
margin-bottom: 1.5rem;
padding-bottom: 0.5rem;
border-bottom: 1px solid var(--border-color);
}
.sidebar ul {
list-style: none;
display: flex;
flex-direction: column;
gap: 0.25rem;
}
.sidebar li {
display: block;
}
.sidebar a {
display: block;
padding: 0.4rem 0.6rem;
color: var(--text-color);
text-decoration: none;
font-size: 0.85rem;
border-radius: 4px;
transition: background 0.15s, color 0.15s;
}
.sidebar a:hover {
background: rgba(74, 144, 217, 0.15);
color: var(--accent-color);
}
/* Main content */
.content {
margin-left: var(--sidebar-width);
padding: 2rem;
}
h1 {
font-size: 2rem;
margin-bottom: 1rem;
color: var(--accent-color);
}
.content > h2 {
font-size: 1.5rem;
margin: 2rem 0 1rem;
color: var(--text-color);
border-bottom: 1px solid var(--border-color);
padding-bottom: 0.5rem;
scroll-margin-top: 1rem;
}
.diagram-container {
display: flex;
flex-wrap: wrap;
gap: 2rem;
margin-top: 1rem;
}
.diagram {
flex: 1;
min-width: 400px;
background: #252540;
border-radius: 8px;
padding: 1rem;
border: 1px solid var(--border-color);
}
.diagram h3 {
font-size: 1.1rem;
margin-bottom: 0.5rem;
color: var(--accent-color);
}
.diagram img,
.diagram object {
width: 100%;
height: auto;
background: white;
border-radius: 4px;
}
.diagram a {
display: block;
text-align: center;
margin-top: 0.5rem;
color: var(--accent-color);
text-decoration: none;
font-size: 0.9rem;
}
.diagram a:hover {
text-decoration: underline;
}
.legend {
margin-top: 2rem;
padding: 1rem;
background: #252540;
border-radius: 8px;
border: 1px solid var(--border-color);
}
.legend h3 {
margin-bottom: 0.5rem;
}
.legend ul {
list-style: none;
display: flex;
flex-wrap: wrap;
gap: 1rem;
}
.legend li {
display: flex;
align-items: center;
gap: 0.5rem;
}
.legend .color-box {
width: 16px;
height: 16px;
border-radius: 3px;
}
code {
background: #333;
padding: 0.2rem 0.4rem;
border-radius: 3px;
font-family: 'Monaco', 'Consolas', monospace;
font-size: 0.9em;
}
pre {
background: #252540;
padding: 1rem;
border-radius: 8px;
overflow-x: auto;
border: 1px solid var(--border-color);
}
pre code {
background: none;
padding: 0;
}
/* Responsive: collapse sidebar on small screens */
@media (max-width: 768px) {
.sidebar {
position: static;
width: 100%;
height: auto;
border-right: none;
border-bottom: 1px solid var(--border-color);
}
.sidebar ul {
flex-direction: row;
flex-wrap: wrap;
}
.content {
margin-left: 0;
}
.diagram {
min-width: 100%;
}
}

884
docs/index.html
View File

@@ -1,380 +1,564 @@
<!doctype html> <!DOCTYPE html>
<html lang="en"> <html lang="en">
<head> <head>
<meta charset="UTF-8" /> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>MPR - Architecture</title> <title>MPR — Detection Pipeline Architecture</title>
<link rel="stylesheet" href="architecture/styles.css" /> <style>
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap');
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: #0a0e17;
color: #e8eaf0;
font-family: 'Inter', sans-serif;
line-height: 1.6;
height: 100vh;
overflow: hidden;
display: flex;
flex-direction: column;
}
header {
padding: 16px 24px;
border-bottom: 1px solid #1e2a4a;
display: flex;
align-items: baseline;
gap: 16px;
flex-shrink: 0;
}
header h1 {
font-family: 'JetBrains Mono', monospace;
font-size: 22px;
font-weight: 600;
letter-spacing: 3px;
color: #0066ff;
}
header .subtitle {
font-size: 13px;
color: #4a5568;
letter-spacing: 1px;
text-transform: uppercase;
}
.layout {
display: flex;
flex: 1;
min-height: 0;
}
nav {
display: flex;
flex-direction: column;
gap: 0;
width: 200px;
flex-shrink: 0;
background: #121829;
border-right: 1px solid #1e2a4a;
padding: 8px 0;
overflow-y: auto;
}
nav a {
padding: 10px 20px;
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: #8892a8;
text-decoration: none;
border-left: 2px solid transparent;
transition: all 0.15s;
cursor: pointer;
}
nav a:hover { color: #e8eaf0; background: #1a2340; }
nav a.active { color: #0066ff; border-left-color: #0066ff; background: #0d1a33; }
main {
flex: 1;
overflow: auto;
padding: 32px 48px;
}
.graph-section {
display: none;
animation: fadeIn 0.2s ease;
}
.graph-section:target,
.graph-section.active { display: block; }
@keyframes fadeIn {
from { opacity: 0; }
to { opacity: 1; }
}
.graph-section h2 {
font-family: 'JetBrains Mono', monospace;
font-size: 15px;
font-weight: 500;
color: #8892a8;
margin-bottom: 8px;
letter-spacing: 1px;
}
.graph-section > p {
font-size: 13px;
color: #4a5568;
margin-bottom: 24px;
max-width: 800px;
}
.graph-container {
background: #0a0e17;
border: 1px solid #1e2a4a;
padding: 24px;
overflow: auto;
}
.graph-container a { display: block; }
.graph-container img { max-width: 100%; height: auto; }
.legend {
display: flex;
flex-wrap: wrap;
gap: 24px;
margin-top: 16px;
font-size: 11px;
font-family: 'JetBrains Mono', monospace;
color: #4a5568;
}
.legend span::before {
content: '';
display: inline-block;
width: 8px;
height: 8px;
margin-right: 6px;
border-radius: 50%;
}
.legend .browser::before { background: #e8eaf0; }
.legend .cluster::before { background: #0066ff; }
.legend .data::before { background: #b4bccf; }
.legend .gpu::before { background: #00c853; }
.legend .cloud::before { background: #ffc107; }
/* Prose */
.graph-section h3 {
font-family: 'JetBrains Mono', monospace;
font-size: 13px;
font-weight: 500;
color: #e8eaf0;
letter-spacing: 1px;
margin: 32px 0 10px;
text-transform: uppercase;
}
.prose { max-width: 820px; }
.prose p {
font-size: 14px;
color: #b4bccf;
margin-bottom: 14px;
line-height: 1.7;
}
.prose p b { color: #e8eaf0; font-weight: 600; }
.prose code {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: #7ab0ff;
background: #121829;
padding: 1px 5px;
border-radius: 3px;
}
.prose a { color: #0066ff; text-decoration: none; cursor: pointer; }
.prose a:hover { text-decoration: underline; }
.prose ul {
margin: 8px 0 16px 20px;
font-size: 14px;
color: #b4bccf;
line-height: 1.7;
}
.prose ul li { margin-bottom: 8px; }
pre.codeblock {
background: #121829;
border: 1px solid #1e2a4a;
color: #b4bccf;
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
padding: 14px 18px;
margin: 8px 0 20px;
overflow-x: auto;
line-height: 1.55;
}
pre.codeblock .c { color: #4a5568; }
pre.codeblock .k { color: #0066ff; }
/* Mobile menu toggle */
.menu-toggle {
display: none;
background: transparent;
border: 1px solid #1e2a4a;
color: #e8eaf0;
padding: 6px 10px;
font-family: 'JetBrains Mono', monospace;
font-size: 14px;
cursor: pointer;
line-height: 1;
margin-left: auto;
}
.menu-toggle:hover { background: #1a2340; }
.nav-backdrop {
display: none;
position: absolute;
inset: 0;
background: rgba(0, 0, 0, 0.5);
z-index: 10;
}
.layout.nav-open .nav-backdrop { display: block; }
@media (max-width: 720px) {
header { padding: 10px 12px; gap: 8px; }
header h1 { font-size: 16px; letter-spacing: 1px; }
header .subtitle { display: none; }
.menu-toggle { display: inline-block; }
.layout { position: relative; }
nav {
position: absolute;
left: 0; top: 0; bottom: 0;
width: 220px;
z-index: 20;
transform: translateX(-100%);
transition: transform 0.2s ease;
box-shadow: 2px 0 8px rgba(0, 0, 0, 0.5);
}
.layout.nav-open nav { transform: translateX(0); }
main { padding: 16px; }
.graph-section h2 { font-size: 13px; }
.prose p, .prose ul { font-size: 13px; }
}
</style>
</head> </head>
<body> <body>
<nav class="sidebar">
<h2>MPR</h2> <header>
<ul> <h1>MPR</h1>
<li><a href="#overview">System Overview</a></li> <span class="subtitle">Media Processing &amp; Detection Pipeline — Architecture</span>
<li><a href="#data-model">Data Model</a></li> <button class="menu-toggle" aria-label="Toggle navigation">☰</button>
<li><a href="#job-flow">Job Flow</a></li> </header>
<li><a href="#media-storage">Media Storage</a></li>
<li><a href="#chunker-pipeline">Chunker Pipeline</a></li> <div class="layout">
<li><a href="#api">API (GraphQL)</a></li> <div class="nav-backdrop"></div>
<li><a href="#access-points">Access Points</a></li>
<li><a href="#quick-reference">Quick Reference</a></li> <nav>
</ul> <a class="active" href="#overview">Overview</a>
<a href="#system">System</a>
<a href="#pipeline">Pipeline</a>
<a href="#profiles">Profiles</a>
<a href="#topology">Inference</a>
<a href="#data">Data Model</a>
<a href="#api">API</a>
<a href="#storage">Storage</a>
<a href="#modelgen">Codegen</a>
<a href="#dev">Dev Env</a>
<a href="#reference">Reference</a>
</nav> </nav>
<main class="content"> <main>
<h1>MPR - Media Processor</h1>
<p>
Media transcoding platform with three execution modes: local (Celery
+ MinIO), AWS (Step Functions + Lambda + S3), and GCP (Cloud Run
Jobs + GCS). Storage is S3-compatible across all environments.
</p>
<h2 id="overview">System Overview</h2> <section id="overview" class="graph-section active">
<div class="diagram-container"> <h2>OVERVIEW</h2>
<div class="diagram"> <p>A guided tour of the platform — start here for narrative context before the diagrams.</p>
<h3>Local Architecture (Development)</h3> <div class="prose">
<object
type="image/svg+xml"
data="architecture/01a-local-architecture.svg"
>
<img
src="architecture/01a-local-architecture.svg"
alt="Local Architecture"
/>
</object>
<a
href="architecture/01a-local-architecture.svg"
target="_blank"
>Open full size</a
>
</div>
<div class="diagram">
<h3>AWS Architecture (Production)</h3>
<object
type="image/svg+xml"
data="architecture/01b-aws-architecture.svg"
>
<img
src="architecture/01b-aws-architecture.svg"
alt="AWS Architecture"
/>
</object>
<a href="architecture/01b-aws-architecture.svg" target="_blank"
>Open full size</a
>
</div>
<div class="diagram">
<h3>GCP Architecture (Production)</h3>
<object
type="image/svg+xml"
data="architecture/01c-gcp-architecture.svg"
>
<img
src="architecture/01c-gcp-architecture.svg"
alt="GCP Architecture"
/>
</object>
<a href="architecture/01c-gcp-architecture.svg" target="_blank"
>Open full size</a
>
</div>
</div>
<h3>What MPR is</h3>
<p>MPR is a brand / logo / text detection pipeline for video. A user picks chunks of source material into a <b>Timeline</b>, then runs a <b>Profile</b> (pipeline topology + per-stage config) against it. The pipeline extracts frames, filters scenes, runs CV (field segmentation, edge detection) and detection (YOLO, OCR), resolves text to a session brand list, and escalates anything still unresolved to a local VLM and then to cloud VLM providers. Output is a brand timeline and per-brand stats.</p>
<h3>Where things run</h3>
<p>The architecture spans four boxes: the <b>browser</b> (Vue 3 detection-app + OpenCV WASM worker for fast CV iteration), the <b>K8s cluster</b> (Envoy Gateway, FastAPI, detection-ui, Postgres, Redis, MinIO — Kind in dev via Tilt), a separate <b>GPU host</b> on the LAN running the inference server (YOLO, OCR, local VLM), and <b>cloud VLM providers</b> (Anthropic, Gemini, OpenAI, Groq) for last-resort escalation. See <a href="#system">System</a>.</p>
<h3>Replay loop</h3>
<p>The system is built around iteration. <b>Checkpoint</b> rows form a tree of "what configs did we try at this stage" (no blobs); <b>StageOutput</b> is a flat upsert table holding each stage's output dict. A single stage can be re-run in place using upstream <code>StageOutput</code> rows, so the UI loop is "tweak config → replay one stage → look at the overlay" without rerunning the whole pipeline. Frame caches keyed by <code>timeline_id</code> are reused across replays.</p>
<h3>Profiles, not overrides</h3>
<p>Profiles live in Postgres as two JSONB blobs — <code>pipeline</code> (stages + edges + routing) and <code>configs</code> (per-stage parameters). The convention is to <i>duplicate a profile and tweak it</i>, not to layer overrides at the call site. Job-level <code>config_overrides</code> exist but are merged on top of the resolved profile in <code>core/detect/graph/nodes.py</code>.</p>
<h3>Inference indirection</h3>
<p>Every CV/ML stage takes an <code>INFERENCE_URL</code> argument. Empty (the dev default) runs CV in-process; set, the stage POSTs to <code>core/gpu/server.py</code> on the GPU host. Heavy ML deps (<code>torch</code>, <code>transformers</code>, <code>paddleocr</code>) live only in <code>core/gpu/pyproject.toml</code> — the API host doesn't need them.</p>
<h3>API and SSE</h3>
<p>FastAPI under <code>/detect/*</code> (<code>core/api/detect/</code>): sources, run/stop/pause/resume/step, status, replay, checkpoints, overlays, config. Pipeline events fan out through Redis to <code>GET /detect/stream/{job_id}</code> as SSE. Envoy keeps the SSE connection open for up to 3600s.</p>
<h3>Codegen</h3>
<p>Source-of-truth dataclasses live in <code>core/schema/models/</code>. The standalone <code>modelgen</code> tool emits SQLModel ORM (<code>core/db/models.py</code>), Pydantic schemas, TypeScript types, and Protobuf definitions. Regenerate everything with <code>bash ctrl/generate.sh</code>.</p>
</div>
</section>
<section id="system" class="graph-section">
<h2>SYSTEM ARCHITECTURE</h2>
<p>Browser ↔ Envoy Gateway ↔ FastAPI / detection-ui ↔ data plane (Postgres / Redis / MinIO) ↔ LAN GPU host ↔ cloud VLM providers.</p>
<div class="graph-container">
<a href="viewer.html?src=architecture/01-architecture.svg"><img src="architecture/01-architecture.svg" alt="System Architecture"></a>
</div>
<div class="legend"> <div class="legend">
<h3>Components</h3> <span class="browser">Browser</span>
<span class="cluster">K8s cluster</span>
<span class="gpu">GPU host (LAN)</span>
<span class="cloud">Cloud VLM</span>
</div>
</section>
<section id="pipeline" class="graph-section">
<h2>DETECTION PIPELINE</h2>
<p>11 named stages from <code>core/detect/graph/nodes.py</code>. The runner flattens the profile's <code>PipelineConfig</code> graph into a linear sequence and runs each stage with cancel / pause / resume / step control.</p>
<div class="graph-container">
<a href="viewer.html?src=architecture/03-detection-pipeline.svg"><img src="architecture/03-detection-pipeline.svg" alt="Detection Pipeline"></a>
</div>
<div class="legend">
<span class="cluster">Browser / WASM-eligible</span>
<span class="gpu">GPU inference</span>
<span class="cloud">Cloud VLM</span>
</div>
<div class="prose" style="margin-top: 24px;">
<p><b>Control flow.</b> Each stage runs inside <code>trace_node()</code>, emits <code>running</code> → <code>done</code>/<code>skipped</code> via <code>core/detect/emit.py</code>, and writes its result to a <code>StageOutput</code> row keyed by <code>(job_id, stage_name)</code>. Between stages the runner checks three job-keyed flags: cancel (<code>set_cancel_check</code>), pause/resume (<code>threading.Event</code>), and pause-after-stage / step.</p>
<p><b>Skip flags.</b> <code>SKIP_VLM=1</code> emits <code>skipped</code> for <code>escalate_vlm</code>; <code>SKIP_CLOUD=1</code> for <code>escalate_cloud</code>. Useful in CI and dev when you don't want to burn provider credits.</p>
<p><a href="architecture/05-detection-pipeline.md" target="_blank" rel="noopener">Full pipeline reference →</a></p>
</div>
</section>
<section id="profiles" class="graph-section">
<h2>PROFILES &amp; CHECKPOINTS</h2>
<p>Profiles are the config mechanism; checkpoints + StageOutput power the replay loop.</p>
<div class="prose">
<h3>Profile shape</h3>
<p>One <code>Profile</code> row per content type (e.g. <code>soccer_broadcast</code>) holds two JSONB blobs:</p>
<ul> <ul>
<li> <li><code>pipeline</code> — a <code>PipelineConfig</code>: stages + edges + routing rules. The runner topologically sorts the edges, falling back to stage order when no edges are defined.</li>
<span class="color-box" style="background: #e8f4f8"></span> <li><code>configs</code> — <code>{stage_name: {...}}</code> per-stage parameters: fps, thresholds, prompts, etc. Each stage parses its slice into a typed config (<code>FrameExtractionConfig</code>, <code>OCRConfig</code>, ...).</li>
Reverse Proxy (nginx) </ul>
</li> <p>Convention: <b>duplicate a profile and tweak it</b> rather than patching defaults at the call site. Job-level <code>config_overrides</code> exist for one-off experiments but the resolved profile is the durable artifact.</p>
<li>
<span class="color-box" style="background: #f0f8e8"></span> <h3>Checkpoint tree</h3>
Application Layer (Django Admin, GraphQL API, Timeline UI) <p>A <code>Checkpoint</code> row is a tree node: <code>(parent_id, stage_name, config_overrides, stats)</code>. <b>No blobs.</b> Lets the UI show a branching history of "what configs did we try at this stage" without dragging frame data around.</p>
</li>
<li> <h3>StageOutput (flat upsert)</h3>
<span class="color-box" style="background: #fff8e8"></span> <p>One row per <code>(job_id, stage_name)</code> holding the stage's output dict. Single-stage replay reads upstream outputs from here, so re-running <code>match_brands</code> with a tweaked threshold doesn't redo OCR. <code>POST /replay-stage</code> is the entry point.</p>
Worker Layer (Celery local mode)
</li> <h3>Replay loop</h3>
<li> <p>The detection-app UI is the test surface: change a config, replay one stage, see the overlay rendered from the cached frame plus the new <code>StageOutput</code>. Frame caches keyed by <code>timeline_id</code> survive across replays — <code>extract_frames</code> only fires on the first run for a timeline.</p>
<span class="color-box" style="background: #fde8d0"></span>
AWS (Step Functions, Lambda) </div>
</li> </section>
<li>
<span class="color-box" style="background: #e8f0fd"></span> <section id="topology" class="graph-section">
GCP (Cloud Run Jobs + GCS) <h2>INFERENCE TOPOLOGY</h2>
</li> <p>Stages can run in three places. The split is what keeps the dev box light and lets one GPU host serve the whole team.</p>
<li> <div class="prose">
<span class="color-box" style="background: #f8e8f0"></span>
Data Layer (PostgreSQL, Redis) <h3>Browser (OpenCV WASM)</h3>
</li> <p>Field and edge stages can run in a Web Worker via <code>ui/detection-app/src/cv/wasmBridge.ts</code> using OpenCV WASM directly — no TypeScript ports of the algorithms. This is the fast-iteration path for the replay loop: tweak a kernel size, rerun the stage on the cached frames, see the overlay update without touching a server.</p>
<li>
<span class="color-box" style="background: #f0f0f0"></span> <h3>API host (in-process)</h3>
S3-compatible Storage (MinIO / AWS S3 / GCS) <p>With <code>INFERENCE_URL=""</code> (the dev default in <code>ctrl/k8s/base/configmap.yaml</code>) every CV/ML stage calls its routine in-process. Useful when there's no GPU host wired up; works for everything except heavy YOLO/VLM workloads.</p>
</li>
<h3>GPU host (LAN)</h3>
<p>Set <code>INFERENCE_URL=http://gpu-host:8000</code> and the same stages POST to <code>core/gpu/server.py</code>. The GPU server exposes <code>/detect</code>, <code>/ocr</code>, <code>/preprocess</code>, <code>/vlm</code>, <code>/detect_edges</code>, <code>/segment_field</code> — each with a <code>/debug</code> variant that returns intermediate masks for the overlay viewer. Heavy ML deps live only in <code>core/gpu/pyproject.toml</code>; the API host doesn't import torch.</p>
<h3>Cloud VLM providers</h3>
<p>Last-resort escalation for unresolved candidates. <code>core/detect/providers/</code> wraps Anthropic, Gemini, OpenAI, and Groq. Selection is per-profile config; <code>SKIP_CLOUD=1</code> bypasses the stage entirely.</p>
</div>
</section>
<section id="data" class="graph-section">
<h2>DATA MODEL</h2>
<p>Tables generated by modelgen from <code>core/schema/models/</code> into <code>core/db/models.py</code> (SQLModel).</p>
<div class="graph-container">
<a href="viewer.html?src=architecture/02-data-model.svg"><img src="architecture/02-data-model.svg" alt="Data Model"></a>
</div>
<div class="prose" style="margin-top: 24px;">
<ul>
<li><code>MediaAsset</code> — source video file with probe metadata (duration, fps, codec).</li>
<li><code>Profile</code> — pipeline topology + per-stage config (JSONB).</li>
<li><code>Timeline</code> — user-created selection of chunks from a source asset.</li>
<li><code>Job</code> — one pipeline run on a timeline; <code>parent_id</code> chains replays into a tree.</li>
<li><code>Checkpoint</code> — tree node of stage state, no blobs.</li>
<li><code>StageOutput</code> — flat upsert per <code>(job, stage)</code>, holds output JSONB and an optional <code>checkpoint_id</code>.</li>
<li><code>Brand</code> — canonical name, aliases, source (ocr/local_vlm/cloud_llm/manual), airing history.</li>
</ul> </ul>
</div> </div>
</section>
<h2 id="data-model">Data Model</h2> <section id="api" class="graph-section">
<div class="diagram-container"> <h2>API</h2>
<div class="diagram"> <p>FastAPI under <code>/detect/*</code> (mounted from <code>core/api/detect/</code>). Through Envoy Gateway in dev the public path is <code>/api/detect/...</code>; <code>/api/detect/stream/*</code> gets an extended idle timeout for SSE.</p>
<h3>Entity Relationships</h3> <pre class="codeblock"><span class="c"># Sources / timelines</span>
<object <span class="k">GET</span> /sources
type="image/svg+xml" <span class="k">GET</span> /sources/{job_id}/chunks
data="architecture/02-data-model.svg" <span class="k">POST</span> /timeline
> <span class="k">GET</span> /timeline
<img <span class="k">GET</span> /timeline/{id}
src="architecture/02-data-model.svg" <span class="k">DELETE</span> /timeline/{id}/cache
alt="Data Model"
/>
</object>
<a href="architecture/02-data-model.svg" target="_blank"
>Open full size</a
>
</div>
</div>
<div class="legend"> <span class="c"># Run control</span>
<h3>Entities</h3> <span class="k">POST</span> /run
<span class="k">POST</span> /stop/{job_id}
<span class="k">POST</span> /pause/{job_id}
<span class="k">POST</span> /resume/{job_id}
<span class="k">POST</span> /step/{job_id}
<span class="k">POST</span> /pause-after-stage/{job_id}
<span class="k">GET</span> /status/{job_id}
<span class="k">POST</span> /clear/{job_id}
<span class="c"># Live events</span>
<span class="k">GET</span> /stream/{job_id} <span class="c"># SSE</span>
<span class="c"># Replay / checkpoints / overlays</span>
<span class="k">GET</span> /checkpoints/{timeline_id}
<span class="k">GET</span> /checkpoints/{timeline_id}/{stage}
<span class="k">GET</span> /scenarios
<span class="k">POST</span> /replay
<span class="k">POST</span> /replay-stage
<span class="k">POST</span> /overlays
<span class="k">GET</span> /overlays/{timeline_id}/{job_id}/{stage}/{seq}
<span class="c"># Config</span>
<span class="k">GET</span> /config
<span class="k">PUT</span> /config
<span class="k">GET</span> /config/profiles
<span class="k">GET</span> /config/profiles/{name}/pipeline
<span class="k">PUT</span> /config/edge-transform
<span class="k">GET</span> /config/stages
<span class="k">GET</span> /config/stages/{stage_name}
<span class="c"># Jobs</span>
<span class="k">GET</span> /jobs
<span class="k">GET</span> /jobs/{id}</pre>
</section>
<section id="storage" class="graph-section">
<h2>STORAGE</h2>
<p>S3-compatible everywhere — MinIO locally, real S3 / GCS / R2 in cloud targets. The same boto3 code path serves both; only <code>S3_ENDPOINT_URL</code> and credentials change.</p>
<div class="prose">
<ul> <ul>
<li> <li><code>mpr-media-in</code> — source video files (chunks).</li>
<span class="color-box" style="background: #4a90d9"></span> <li><code>mpr-media-out</code> — per-job artifacts: extracted frame caches, debug overlays.</li>
MediaAsset - Video/audio files with metadata </ul>
</li> <p>Heavy artifacts (frames, masks, overlays) live in object storage. <code>Checkpoint</code> and <code>StageOutput</code> rows in Postgres hold structured outputs and references to S3 keys, never blobs.</p>
<li> <p><a href="architecture/04-media-storage.md" target="_blank" rel="noopener">Full storage reference →</a></p>
<span class="color-box" style="background: #50b050"></span> </div>
TranscodePreset - Encoding configurations </section>
</li>
<li> <section id="modelgen" class="graph-section">
<span class="color-box" style="background: #d9534f"></span> <h2>CODE GENERATION</h2>
TranscodeJob - Processing queue items <p>Source-of-truth dataclasses in <code>core/schema/models/</code> → typed code in four targets.</p>
</li> <div class="prose">
<ul>
<li>SQLModel ORM tables → <code>core/db/models.py</code></li>
<li>Pydantic schemas (API request / response models)</li>
<li>TypeScript types (UI)</li>
<li>Protobuf definitions (gRPC stubs in <code>core/rpc/</code>)</li>
</ul> </ul>
</div> </div>
<pre class="codeblock"><span class="c"># regenerate everything</span>
bash ctrl/generate.sh</pre>
</section>
<h2 id="job-flow">Job Flow</h2> <section id="dev" class="graph-section">
<div class="diagram-container"> <h2>DEV ENVIRONMENT</h2>
<div class="diagram"> <p>Tilt + Kind for local dev. Routing via Envoy Gateway on port 8080 — no nginx-ingress.</p>
<h3>Job Lifecycle</h3> <div class="prose">
<object <p>The Tiltfile lives at <code>ctrl/Tiltfile</code> and applies the kustomize overlay <code>ctrl/k8s/overlays/dev/</code>. Cluster name: <code>kind-mpr</code>. Tilt port-forwards Envoy (8080) and MinIO (9000 API, 9001 console).</p>
type="image/svg+xml"
data="architecture/03-job-flow.svg"
>
<img src="architecture/03-job-flow.svg" alt="Job Flow" />
</object>
<a href="architecture/03-job-flow.svg" target="_blank"
>Open full size</a
>
</div>
</div>
<div class="legend">
<h3>Job States</h3>
<ul> <ul>
<li> <li><code>/api/detect/stream/*</code> → FastAPI SSE (3600s idle timeout)</li>
<span class="color-box" style="background: #ffc107"></span> <li><code>/api/*</code> → FastAPI</li>
PENDING - Waiting in queue <li><code>/</code>, <code>/detection/*</code> → detection-ui (with WS upgrade for Vite HMR)</li>
</li>
<li>
<span class="color-box" style="background: #17a2b8"></span>
PROCESSING - Worker executing
</li>
<li>
<span class="color-box" style="background: #28a745"></span>
COMPLETED - Success
</li>
<li>
<span class="color-box" style="background: #dc3545"></span>
FAILED - Error occurred
</li>
<li>
<span class="color-box" style="background: #6c757d"></span>
CANCELLED - User cancelled
</li>
</ul>
<h3>Execution Modes</h3>
<ul>
<li>
<span class="color-box" style="background: #e8f4e8"></span>
Local: Celery + MinIO (S3 API) + FFmpeg
</li>
<li>
<span class="color-box" style="background: #fde8d0"></span>
Lambda: Step Functions + Lambda + AWS S3
</li>
<li>
<span class="color-box" style="background: #e8f0fd"></span>
GCP: Cloud Run Jobs + GCS (S3 compat)
</li>
</ul> </ul>
</div> </div>
<pre class="codeblock"><span class="c"># Add to /etc/hosts</span>
127.0.0.1 mpr.local.ar k8s.mpr.local.ar
<h2 id="media-storage">Media Storage</h2> <span class="c"># Bring the cluster up</span>
<div class="diagram-container"> cd ctrl
<p> ./kind-create.sh <span class="c"># one-time</span>
MPR separates media into <strong>input</strong> and tilt up <span class="c"># builds + applies + port-forwards</span>
<strong>output</strong> paths, each independently configurable.
File paths are stored
<strong>relative to their respective root</strong> to ensure
portability between local development and cloud deployments.
</p>
</div>
<div class="legend"> <span class="c"># UI: http://k8s.mpr.local.ar:8080/</span>
<h3>Input / Output Separation</h3> <span class="c"># API: http://k8s.mpr.local.ar:8080/api/</span>
<ul> <span class="c"># MinIO: http://localhost:9001 (console; admin / minioadmin)</span>
<li>
<span class="color-box" style="background: #4a90d9"></span>
<code>MEDIA_IN</code> - Source media files to process
</li>
<li>
<span class="color-box" style="background: #50b050"></span>
<code>MEDIA_OUT</code> - Transcoded/trimmed output files
</li>
</ul>
<p><strong>Why Relative Paths?</strong></p>
<ul>
<li>Portability: Same database works locally and in cloud</li>
<li>Flexibility: Easy to switch between storage backends</li>
<li>Simplicity: No need to update paths when migrating</li>
</ul>
</div>
<div class="legend"> <span class="c"># Force a UI rebuild</span>
<h3>Local Development</h3> tilt trigger detection-ui</pre>
<pre><code>MEDIA_IN=/app/media/in </section>
MEDIA_OUT=/app/media/out
/app/media/ <section id="reference" class="graph-section">
├── in/ # Source files <h2>QUICK REFERENCE</h2>
│ ├── video1.mp4 <p>Common commands and switches for working in MPR.</p>
│ └── subfolder/video3.mp4 <pre class="codeblock"><span class="c"># Render SVGs from DOT files</span>
└── out/ # Transcoded output
└── video1_h264.mp4</code></pre>
</div>
<div class="legend">
<h3>AWS/Cloud Deployment</h3>
<pre><code>MEDIA_IN=s3://source-bucket/media/
MEDIA_OUT=s3://output-bucket/transcoded/
MEDIA_BASE_URL=https://source-bucket.s3.amazonaws.com/media/</code></pre>
<p>
Database paths remain unchanged (already relative). Just upload
files to S3 and update environment variables.
</p>
</div>
<p>
<a href="architecture/04-media-storage.md" target="_blank"
>Full Media Storage Documentation &rarr;</a
>
</p>
<h2 id="chunker-pipeline">Chunker Pipeline</h2>
<div class="diagram-container">
<p>
The chunker pipeline splits media into time-based segments,
streaming real-time events from worker threads through Redis
and gRPC-Web to the browser UI. 7 hops from worker thread to pixel.
</p>
</div>
<div class="legend">
<h3>Event Path</h3>
<pre><code>Worker thread → Pipeline._emit() → event_bridge() → Redis RPUSH
→ [50ms poll] gRPC server LRANGE → yield protobuf
→ HTTP/2 frame → Envoy (grpc-web filter)
→ HTTP/1.1 chunk → nginx (proxy_buffering off)
→ fetch ReadableStream → protobuf-ts decode
→ setEvents([...prev, evt]) → React re-render</code></pre>
</div>
<div class="legend">
<h3>Thread Model (inside Celery worker)</h3>
<pre><code>Celery worker process
└─ run_job task thread
└─ Pipeline.run()
├─ Producer thread — enqueues chunks
├─ Monitor thread — emits progress every 500ms
├─ Worker thread 0 — pulls from queue, processes
├─ Worker thread 1 — pulls from queue, processes
├─ Worker thread 2 — pulls from queue, processes
└─ Worker thread 3 — pulls from queue, processes</code></pre>
</div>
<div class="legend">
<h3>Infrastructure</h3>
<ul>
<li><code>nginx :80</code> - Reverse proxy, static file serving</li>
<li><code>fastapi :8702</code> - GraphQL API (Strawberry)</li>
<li><code>celery</code> - Task worker (runs pipeline)</li>
<li><code>redis :6379</code> - Event bus + Celery broker</li>
<li><code>grpc :50051</code> - gRPC server (StreamChunkPipeline)</li>
<li><code>envoy :8090</code> - gRPC-Web &harr; native gRPC translation</li>
<li><code>minio :9000</code> - S3-compatible source media storage</li>
<li><code>postgres :5432</code> - Job/asset metadata</li>
</ul>
</div>
<p>
<a href="architecture/05-chunker-pipeline.md" target="_blank"
>Full Chunker Pipeline Documentation &rarr;</a
>
</p>
<h2 id="api">API (GraphQL)</h2>
<div class="legend">
<p>
All client interactions go through GraphQL at
<code>/graphql</code>.
</p>
<pre><code># GraphiQL IDE
http://mpr.local.ar/graphql
# Queries
query { assets(status: "ready") { id filename duration } }
query { jobs(status: "processing") { id status progress } }
query { presets { id name container videoCodec } }
query { systemStatus { status version } }
# Mutations
mutation { scanMediaFolder { found registered skipped } }
mutation { createJob(input: { sourceAssetId: "...", presetId: "..." }) { id status } }
mutation { cancelJob(id: "...") { id status } }
mutation { retryJob(id: "...") { id status } }
mutation { updateAsset(id: "...", input: { comments: "..." }) { id comments } }
mutation { deleteAsset(id: "...") { ok } }
# Lambda callback (REST)
POST /api/jobs/{id}/callback - Lambda completion webhook</code></pre>
<p><strong>Supported File Types:</strong></p>
<p>
Video: mp4, mkv, avi, mov, webm, flv, wmv, m4v<br />
Audio: mp3, wav, flac, aac, ogg, m4a
</p>
</div>
<h2 id="access-points">Access Points</h2>
<pre><code># Add to /etc/hosts
127.0.0.1 mpr.local.ar
# URLs
http://mpr.local.ar/admin - Django Admin
http://mpr.local.ar/graphql - GraphiQL IDE
http://mpr.local.ar/ - Timeline UI
http://mpr.local.ar/chunker/ - Chunker UI
http://localhost:9001 - MinIO Console
# AWS deployment
https://mpr.mcrn.ar/ - Production</code></pre>
<h2 id="quick-reference">Quick Reference</h2>
<pre><code># Render SVGs from DOT files
for f in docs/architecture/*.dot; do dot -Tsvg "$f" -o "${f%.dot}.svg"; done for f in docs/architecture/*.dot; do dot -Tsvg "$f" -o "${f%.dot}.svg"; done
# Switch executor mode <span class="c"># Regenerate models from core/schema/models/</span>
MPR_EXECUTOR=local # Celery + MinIO bash ctrl/generate.sh
MPR_EXECUTOR=lambda # Step Functions + Lambda + S3
MPR_EXECUTOR=gcp # Cloud Run Jobs + GCS</code></pre> <span class="c"># Switch inference between local and GPU host</span>
INFERENCE_URL= <span class="c"># local (CV runs in API process)</span>
INFERENCE_URL=http://gpu-host:8000 <span class="c"># remote (core/gpu/server.py)</span>
<span class="c"># Skip VLM escalation paths</span>
SKIP_VLM=1
SKIP_CLOUD=1
<span class="c"># Tilt</span>
cd ctrl &amp;&amp; tilt up
tilt trigger detection-ui</pre>
<div class="prose">
<p>Reference docs:</p>
<ul>
<li><a href="architecture/05-detection-pipeline.md" target="_blank" rel="noopener">Detection pipeline reference</a></li>
<li><a href="architecture/04-media-storage.md" target="_blank" rel="noopener">Media &amp; artifact storage</a></li>
</ul>
</div>
</section>
</main> </main>
</div>
<script>
(function() {
var layout = document.querySelector('.layout');
var main = document.querySelector('main');
function syncActive() {
var hash = location.hash.slice(1) || 'overview';
document.querySelectorAll('.graph-section').forEach(function(s) { s.classList.remove('active'); });
document.querySelectorAll('nav a').forEach(function(a) { a.classList.remove('active'); });
var section = document.getElementById(hash);
if (section) section.classList.add('active');
var link = document.querySelector('nav a[href="#' + hash + '"]');
if (link) link.classList.add('active');
if (main) main.scrollTop = 0;
layout.classList.remove('nav-open');
}
window.addEventListener('hashchange', syncActive);
window.addEventListener('DOMContentLoaded', syncActive);
syncActive();
document.addEventListener('click', function(e) {
if (e.target.closest('.menu-toggle') || e.target.closest('.nav-backdrop')) {
layout.classList.toggle('nav-open');
}
});
})();
</script>
</body> </body>
</html> </html>

125
docs/media-storage.html
View File

@@ -1,125 +0,0 @@
<h1>Media Storage Architecture</h1>
<h2>Overview</h2>
<p>MPR separates media into <strong>input</strong> and <strong>output</strong> paths, each independently configurable. File paths are stored <strong>relative to their respective root</strong> to ensure portability between local development and cloud deployments (AWS S3, etc.).</p>
<h2>Storage Strategy</h2>
<h3>Input / Output Separation</h3>
<p>| Path | Env Var | Purpose |
|------|---------|---------|
| <code>MEDIA_IN</code> | <code>/app/media/in</code> | Source media files to process |
| <code>MEDIA_OUT</code> | <code>/app/media/out</code> | Transcoded/trimmed output files |</p>
<p>These can point to different locations or even different servers/buckets in production.</p>
<h3>File Path Storage</h3>
<ul>
<li><strong>Database</strong>: Stores only the relative path (e.g., <code>videos/sample.mp4</code>)</li>
<li><strong>Input Root</strong>: Configurable via <code>MEDIA_IN</code> env var</li>
<li><strong>Output Root</strong>: Configurable via <code>MEDIA_OUT</code> env var</li>
<li><strong>Serving</strong>: Base URL configurable via <code>MEDIA_BASE_URL</code> env var</li>
</ul>
<h3>Why Relative Paths?</h3>
<ol>
<li><strong>Portability</strong>: Same database works locally and in cloud</li>
<li><strong>Flexibility</strong>: Easy to switch between storage backends</li>
<li><strong>Simplicity</strong>: No need to update paths when migrating</li>
</ol>
<h2>Local Development</h2>
<h3>Configuration</h3>
<p><code>bash
MEDIA_IN=/app/media/in
MEDIA_OUT=/app/media/out</code></p>
<h3>File Structure</h3>
<p><code>/app/media/
├── in/ # Source files
│ ├── video1.mp4
│ ├── video2.mp4
│ └── subfolder/
│ └── video3.mp4
└── out/ # Transcoded output
├── video1_h264.mp4
└── video2_trimmed.mp4</code></p>
<h3>Database Storage</h3>
<p>```</p>
<h1>Source assets (scanned from media/in)</h1>
<p>filename: video1.mp4
file_path: video1.mp4</p>
<p>filename: video3.mp4
file_path: subfolder/video3.mp4
```</p>
<h3>URL Serving</h3>
<ul>
<li>Nginx serves input via <code>location /media/in { alias /app/media/in; }</code></li>
<li>Nginx serves output via <code>location /media/out { alias /app/media/out; }</code></li>
<li>Frontend accesses: <code>http://mpr.local.ar/media/in/video1.mp4</code></li>
<li>Video player: <code>&lt;video src="/media/in/video1.mp4" /&gt;</code></li>
</ul>
<h2>AWS/Cloud Deployment</h2>
<h3>S3 Configuration</h3>
<p>```bash</p>
<h1>Input and output can be different buckets/paths</h1>
<p>MEDIA_IN=s3://source-bucket/media/
MEDIA_OUT=s3://output-bucket/transcoded/
MEDIA_BASE_URL=https://source-bucket.s3.amazonaws.com/media/
```</p>
<h3>S3 Structure</h3>
<p>```
s3://source-bucket/media/
├── video1.mp4
└── subfolder/
└── video3.mp4</p>
<p>s3://output-bucket/transcoded/
├── video1_h264.mp4
└── video2_trimmed.mp4
```</p>
<h3>Database Storage (Same!)</h3>
<p>```
filename: video1.mp4
file_path: video1.mp4</p>
<p>filename: video3.mp4
file_path: subfolder/video3.mp4
```</p>
<h2>API Endpoints</h2>
<h3>Scan Media Folder</h3>
<p><code>http
POST /api/assets/scan</code></p>
<p><strong>Behavior:</strong>
1. Recursively scans <code>MEDIA_IN</code> directory
2. Finds all video/audio files (mp4, mkv, avi, mov, mp3, wav, etc.)
3. Stores paths <strong>relative to MEDIA_IN</strong>
4. Skips already-registered files (by filename)
5. Returns summary: <code>{ found, registered, skipped, files }</code></p>
<h3>Create Job</h3>
<p>```http
POST /api/jobs/
Content-Type: application/json</p>
<p>{
"source_asset_id": "uuid",
"preset_id": "uuid",
"trim_start": 10.0,
"trim_end": 30.0
}
```</p>
<p><strong>Behavior:</strong>
- Server sets <code>output_path</code> using <code>MEDIA_OUT</code> + generated filename
- Output goes to the output directory, not alongside source files</p>
<h2>Migration Guide</h2>
<h3>Moving from Local to S3</h3>
<ol>
<li>
<p><strong>Upload source files to S3:</strong>
<code>bash
aws s3 sync /app/media/in/ s3://source-bucket/media/
aws s3 sync /app/media/out/ s3://output-bucket/transcoded/</code></p>
</li>
<li>
<p><strong>Update environment variables:</strong>
<code>bash
MEDIA_IN=s3://source-bucket/media/
MEDIA_OUT=s3://output-bucket/transcoded/
MEDIA_BASE_URL=https://source-bucket.s3.amazonaws.com/media/</code></p>
</li>
<li>
<p><strong>Database paths remain unchanged</strong> (already relative)</p>
</li>
</ol>
<h2>Supported File Types</h2>
<p><strong>Video:</strong> <code>.mp4</code>, <code>.mkv</code>, <code>.avi</code>, <code>.mov</code>, <code>.webm</code>, <code>.flv</code>, <code>.wmv</code>, <code>.m4v</code>
<strong>Audio:</strong> <code>.mp3</code>, <code>.wav</code>, <code>.flac</code>, <code>.aac</code>, <code>.ogg</code>, <code>.m4a</code></p>

97
docs/viewer.html Normal file
View File

@@ -0,0 +1,97 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Graph Viewer</title>
<style>
* { margin: 0; padding: 0; }
body {
background: #0a0e17;
overflow: hidden;
width: 100vw;
height: 100vh;
}
#container {
width: 100vw;
height: 100vh;
overflow: hidden;
cursor: grab;
}
#container.dragging { cursor: grabbing; }
img {
transform-origin: 0 0;
user-select: none;
-webkit-user-drag: none;
}
</style>
</head>
<body>
<div id="container">
<img id="img" />
</div>
<script>
var src = new URLSearchParams(location.search).get('src');
var img = document.getElementById('img');
var container = document.getElementById('container');
img.src = src;
var scale = 1;
var x = 0, y = 0;
var dragging = false;
var startX, startY, startPanX, startPanY;
function apply() {
img.style.transform = 'translate(' + x + 'px,' + y + 'px) scale(' + scale + ')';
}
img.onload = function() {
var sw = window.innerWidth / img.naturalWidth;
var sh = window.innerHeight / img.naturalHeight;
scale = Math.min(sw, sh) * 0.95;
x = (window.innerWidth - img.naturalWidth * scale) / 2;
y = (window.innerHeight - img.naturalHeight * scale) / 2;
apply();
};
container.addEventListener('wheel', function(e) {
e.preventDefault();
var factor = e.deltaY < 0 ? 1.12 : 0.89;
var rect = container.getBoundingClientRect();
var mx = e.clientX - rect.left;
var my = e.clientY - rect.top;
x = mx - (mx - x) * factor;
y = my - (my - y) * factor;
scale *= factor;
apply();
}, { passive: false });
container.addEventListener('mousedown', function(e) {
if (e.button !== 0) return;
dragging = true;
startX = e.clientX;
startY = e.clientY;
startPanX = x;
startPanY = y;
container.classList.add('dragging');
e.preventDefault();
});
window.addEventListener('mousemove', function(e) {
if (!dragging) return;
x = startPanX + (e.clientX - startX);
y = startPanY + (e.clientY - startY);
apply();
});
window.addEventListener('mouseup', function() {
dragging = false;
container.classList.remove('dragging');
});
container.addEventListener('dblclick', function() {
img.onload();
});
</script>
</body>
</html>