99 lines
4.9 KiB
Markdown
99 lines
4.9 KiB
Markdown
# `examples/data` — an overlay with a database, a scheduler and a worked pipeline
|
|
|
|
postgres, redis and airflow, each an upstream image run unmodified, installed as
|
|
this overlay's own addons. rig ships the mechanism that finds and runs them; which
|
|
services a workload needs is the workload's business, so they live here rather
|
|
than in rig's `ctrl/addons/`. Copy what you need into your own overlay's `addons/`.
|
|
|
|
```bash
|
|
OVERLAY=examples/data make cluster up # cluster `data`: postgres, then airflow
|
|
OVERLAY=examples/data make tilt # the items-api simulator the DAG reads
|
|
```
|
|
|
|
```
|
|
rig.env ADDONS (metallb is rig's; the rest are here), namespace, identities, image pins
|
|
addons/postgres.sh one replica on a PVC; the password generated once and kept
|
|
addons/airflow.sh one `standalone` pod on its own `airflow` database; needs postgres
|
|
addons/redis.sh only for switching airflow to CeleryExecutor (not in ADDONS)
|
|
k8s/ the items-api simulator
|
|
dags/items_to_postgres.py the worked example: API client → adapter → postgres
|
|
Tiltfile names the simulator's resource
|
|
```
|
|
|
|
Everything lands in the `data` namespace (`DATA_NAMESPACE`, and the kustomization
|
|
names it too), so resetting an app's namespace leaves the databases alone. Costs
|
|
roughly 2 GB with airflow, under 1 without. Airflow's first boot runs the whole
|
|
metadata migration, so expect a few minutes before it is ready.
|
|
|
|
## The worked example: three links kept apart
|
|
|
|
- **Metadata DB (infra).** `addons/airflow.sh` creates an `airflow` database on the
|
|
same postgres and points `SQL_ALCHEMY_CONN` there — airflow's own tables never land
|
|
in the app's database.
|
|
- **DAG delivery.** The addon turns this overlay's `dags/` into the `airflow-dags`
|
|
ConfigMap, mounted at `/opt/airflow/dags`; re-run `make cluster up` after editing a
|
|
DAG. The faster path later: a kind `extraMount` of `dags/` plus a Tilt `sync` — noted,
|
|
not built.
|
|
- **Data connection (operational logic).** `AIRFLOW_CONN_APP_DB`, composed each run
|
|
from the postgres secret, gives DAGs the app's database as the `app_db` connection.
|
|
One password reaches both URLs, with nowhere to drift.
|
|
|
|
The DAG itself calls the simulator with its own HTTP client (the wire, as the API
|
|
returns it), renames the wire's fields into the app's names in `to_app_row` — **the
|
|
adapter, which belongs to whoever owns the app's model and so lives in the overlay** —
|
|
and upserts into `items`. Hourly, no backfill, one retry, idempotent on `item_id`.
|
|
|
|
```bash
|
|
kubectl --context kind-data -n data exec deploy/airflow -- airflow dags unpause items_to_postgres
|
|
kubectl --context kind-data -n data exec deploy/airflow -- airflow dags trigger items_to_postgres
|
|
kubectl --context kind-data -n data exec deploy/postgres -- psql -U app -d app -c 'select * from items'
|
|
```
|
|
|
|
## Addons in an overlay
|
|
|
|
Each one runs from rig's `ctrl/` (rig's `addons.sh` exports `RIG_CTRL`), so it
|
|
starts with `cd "${RIG_CTRL:?...}"`, sources `./lib/config.sh` and calls
|
|
`load_config` — and sees every key this overlay's `rig.env` sets. Run them through
|
|
rig (`bash ctrl/addons.sh install`, or `make cluster up`), not directly.
|
|
|
|
## postgres — plain manifests, one replica
|
|
|
|
Plain manifests rather than a helm chart: a chart repo is a network dependency,
|
|
and an offline machine needs a path with none. The image is pinned in `rig.env`
|
|
and can be preloaded into a local registry like every other image.
|
|
|
|
One replica on a PVC. This models a dependency for local work, not a
|
|
highly-available database, and pretending otherwise on a kind node would be a
|
|
more elaborate lie rather than a more useful one.
|
|
|
|
The password is not in `rig.env`: `addons/postgres.sh` generates one on first
|
|
install and keeps it across re-runs, so re-running the addon never rotates the
|
|
credential out from under whatever is already connected.
|
|
|
|
## redis
|
|
|
|
Cache, and the broker anything queue-shaped runs on. No persistence: a broker
|
|
that loses its queue on restart is the honest local model, and a PVC here buys
|
|
nothing but a volume to clean up.
|
|
|
|
## airflow
|
|
|
|
Airflow needs a metadata database before it will start at all, so the script
|
|
refuses rather than rolls a pod that will CrashLoopBackOff while the real problem
|
|
(postgres missing from `ADDONS`) stays invisible in the logs.
|
|
|
|
One pod on `standalone`: migration, admin user, scheduler and webserver in a
|
|
single container, on LocalExecutor, which needs no broker. The official chart's
|
|
five deployments model an installation; switching this on means wanting pipelines.
|
|
redis is here for the day it moves to CeleryExecutor, and not before.
|
|
|
|
## Reaching them
|
|
|
|
Reach the databases with port-forward rather than binding more host ports:
|
|
|
|
```bash
|
|
kubectl --context kind-data -n data port-forward svc/postgres 5432:5432
|
|
kubectl --context kind-data -n data port-forward svc/airflow 8080:8080
|
|
kubectl --context kind-data -n data get secret postgres -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
|
|
```
|