# `examples/data` — an overlay with a database, a scheduler and a worked pipeline

postgres, redis and airflow, each an upstream image run unmodified, installed as
this overlay's own addons. rig ships the mechanism that finds and runs them; which
services a workload needs is the workload's business, so they live here rather
than in rig's `ctrl/addons/`. Copy what you need into your own overlay's `addons/`.

```bash
OVERLAY=examples/data make cluster up     # cluster `data`: postgres, then airflow
OVERLAY=examples/data make tilt           # the items-api simulator the DAG reads
```

```
rig.env                    ADDONS (metallb is rig's; the rest are here), namespace, identities, image pins
addons/postgres.sh         one replica on a PVC; the password generated once and kept
addons/airflow.sh          one `standalone` pod on its own `airflow` database; needs postgres
addons/redis.sh            only for switching airflow to CeleryExecutor (not in ADDONS)
k8s/                       the items-api simulator
dags/items_to_postgres.py  the worked example: API client → adapter → postgres
Tiltfile                   names the simulator's resource
```

Everything lands in the `data` namespace (`DATA_NAMESPACE`, and the kustomization
names it too), so resetting an app's namespace leaves the databases alone. Costs
roughly 2 GB with airflow, under 1 without. Airflow's first boot runs the whole
metadata migration, so expect a few minutes before it is ready.

## The worked example: three links kept apart

- **Metadata DB (infra).** `addons/airflow.sh` creates an `airflow` database on the
  same postgres and points `SQL_ALCHEMY_CONN` there — airflow's own tables never land
  in the app's database.
- **DAG delivery.** The addon turns this overlay's `dags/` into the `airflow-dags`
  ConfigMap, mounted at `/opt/airflow/dags`; re-run `make cluster up` after editing a
  DAG. The faster path later: a kind `extraMount` of `dags/` plus a Tilt `sync` — noted,
  not built.
- **Data connection (operational logic).** `AIRFLOW_CONN_APP_DB`, composed each run
  from the postgres secret, gives DAGs the app's database as the `app_db` connection.
  One password reaches both URLs, with nowhere to drift.

The DAG itself calls the simulator with its own HTTP client (the wire, as the API
returns it), renames the wire's fields into the app's names in `to_app_row` — **the
adapter, which belongs to whoever owns the app's model and so lives in the overlay** —
and upserts into `items`. Hourly, no backfill, one retry, idempotent on `item_id`.

```bash
kubectl --context kind-data -n data exec deploy/airflow -- airflow dags unpause items_to_postgres
kubectl --context kind-data -n data exec deploy/airflow -- airflow dags trigger items_to_postgres
kubectl --context kind-data -n data exec deploy/postgres -- psql -U app -d app -c 'select * from items'
```

## Addons in an overlay

Each one runs from rig's `ctrl/` (rig's `addons.sh` exports `RIG_CTRL`), so it
starts with `cd "${RIG_CTRL:?...}"`, sources `./lib/config.sh` and calls
`load_config` — and sees every key this overlay's `rig.env` sets. Run them through
rig (`bash ctrl/addons.sh install`, or `make cluster up`), not directly.

## postgres — plain manifests, one replica

Plain manifests rather than a helm chart: a chart repo is a network dependency,
and an offline machine needs a path with none. The image is pinned in `rig.env`
and can be preloaded into a local registry like every other image.

One replica on a PVC. This models a dependency for local work, not a
highly-available database, and pretending otherwise on a kind node would be a
more elaborate lie rather than a more useful one.

The password is not in `rig.env`: `addons/postgres.sh` generates one on first
install and keeps it across re-runs, so re-running the addon never rotates the
credential out from under whatever is already connected.

## redis

Cache, and the broker anything queue-shaped runs on. No persistence: a broker
that loses its queue on restart is the honest local model, and a PVC here buys
nothing but a volume to clean up.

## airflow

Airflow needs a metadata database before it will start at all, so the script
refuses rather than rolls a pod that will CrashLoopBackOff while the real problem
(postgres missing from `ADDONS`) stays invisible in the logs.

One pod on `standalone`: migration, admin user, scheduler and webserver in a
single container, on LocalExecutor, which needs no broker. The official chart's
five deployments model an installation; switching this on means wanting pipelines.
redis is here for the day it moves to CeleryExecutor, and not before.

## Reaching them

Reach the databases with port-forward rather than binding more host ports:

```bash
kubectl --context kind-data -n data port-forward svc/postgres 5432:5432
kubectl --context kind-data -n data port-forward svc/airflow 8080:8080
kubectl --context kind-data -n data get secret postgres -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
```