rig major updates

rig/examples/data/README.md

@@ -0,0 +1,98 @@
+# `examples/data` — an overlay with a database, a scheduler and a worked pipeline
+
+postgres, redis and airflow, each an upstream image run unmodified, installed as
+this overlay's own addons. rig ships the mechanism that finds and runs them; which
+services a workload needs is the workload's business, so they live here rather
+than in rig's `ctrl/addons/`. Copy what you need into your own overlay's `addons/`.
+
+```bash
+OVERLAY=examples/data make cluster up     # cluster `data`: postgres, then airflow
+OVERLAY=examples/data make tilt           # the items-api simulator the DAG reads
+```
+
+```
+rig.env                    ADDONS (metallb is rig's; the rest are here), namespace, identities, image pins
+addons/postgres.sh         one replica on a PVC; the password generated once and kept
+addons/airflow.sh          one `standalone` pod on its own `airflow` database; needs postgres
+addons/redis.sh            only for switching airflow to CeleryExecutor (not in ADDONS)
+k8s/                       the items-api simulator
+dags/items_to_postgres.py  the worked example: API client → adapter → postgres
+Tiltfile                   names the simulator's resource
+```
+
+Everything lands in the `data` namespace (`DATA_NAMESPACE`, and the kustomization
+names it too), so resetting an app's namespace leaves the databases alone. Costs
+roughly 2 GB with airflow, under 1 without. Airflow's first boot runs the whole
+metadata migration, so expect a few minutes before it is ready.
+
+## The worked example: three links kept apart
+
+- **Metadata DB (infra).** `addons/airflow.sh` creates an `airflow` database on the
+  same postgres and points `SQL_ALCHEMY_CONN` there — airflow's own tables never land
+  in the app's database.
+- **DAG delivery.** The addon turns this overlay's `dags/` into the `airflow-dags`
+  ConfigMap, mounted at `/opt/airflow/dags`; re-run `make cluster up` after editing a
+  DAG. The faster path later: a kind `extraMount` of `dags/` plus a Tilt `sync` — noted,
+  not built.
+- **Data connection (operational logic).** `AIRFLOW_CONN_APP_DB`, composed each run
+  from the postgres secret, gives DAGs the app's database as the `app_db` connection.
+  One password reaches both URLs, with nowhere to drift.
+
+The DAG itself calls the simulator with its own HTTP client (the wire, as the API
+returns it), renames the wire's fields into the app's names in `to_app_row` — **the
+adapter, which belongs to whoever owns the app's model and so lives in the overlay** —
+and upserts into `items`. Hourly, no backfill, one retry, idempotent on `item_id`.
+
+```bash
+kubectl --context kind-data -n data exec deploy/airflow -- airflow dags unpause items_to_postgres
+kubectl --context kind-data -n data exec deploy/airflow -- airflow dags trigger items_to_postgres
+kubectl --context kind-data -n data exec deploy/postgres -- psql -U app -d app -c 'select * from items'
+```
+
+## Addons in an overlay
+
+Each one runs from rig's `ctrl/` (rig's `addons.sh` exports `RIG_CTRL`), so it
+starts with `cd "${RIG_CTRL:?...}"`, sources `./lib/config.sh` and calls
+`load_config` — and sees every key this overlay's `rig.env` sets. Run them through
+rig (`bash ctrl/addons.sh install`, or `make cluster up`), not directly.
+
+## postgres — plain manifests, one replica
+
+Plain manifests rather than a helm chart: a chart repo is a network dependency,
+and an offline machine needs a path with none. The image is pinned in `rig.env`
+and can be preloaded into a local registry like every other image.
+
+One replica on a PVC. This models a dependency for local work, not a
+highly-available database, and pretending otherwise on a kind node would be a
+more elaborate lie rather than a more useful one.
+
+The password is not in `rig.env`: `addons/postgres.sh` generates one on first
+install and keeps it across re-runs, so re-running the addon never rotates the
+credential out from under whatever is already connected.
+
+## redis
+
+Cache, and the broker anything queue-shaped runs on. No persistence: a broker
+that loses its queue on restart is the honest local model, and a PVC here buys
+nothing but a volume to clean up.
+
+## airflow
+
+Airflow needs a metadata database before it will start at all, so the script
+refuses rather than rolls a pod that will CrashLoopBackOff while the real problem
+(postgres missing from `ADDONS`) stays invisible in the logs.
+
+One pod on `standalone`: migration, admin user, scheduler and webserver in a
+single container, on LocalExecutor, which needs no broker. The official chart's
+five deployments model an installation; switching this on means wanting pipelines.
+redis is here for the day it moves to CeleryExecutor, and not before.
+
+## Reaching them
+
+Reach the databases with port-forward rather than binding more host ports:
+
+```bash
+kubectl --context kind-data -n data port-forward svc/postgres 5432:5432
+kubectl --context kind-data -n data port-forward svc/airflow 8080:8080
+kubectl --context kind-data -n data get secret postgres -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
+```