172 lines
6.7 KiB
Markdown
172 lines
6.7 KiB
Markdown
# Overlays: what runs lives outside rig
|
|
|
|
## Why
|
|
|
|
rig is the machine: the toolchain, the cluster, the registry, the port block, the
|
|
dev loop's plumbing. What runs on it — the services, their manifests, their
|
|
images, their settings — belongs to whoever owns that work, changes at a
|
|
different rate, and often cannot be shared at all. Keeping both in one tree meant
|
|
editing rig's own files to use it, and then carrying those edits into every copy.
|
|
|
|
So the use case is one folder, the **overlay**, kept outside rig's version
|
|
control. rig reads it; rig never writes into it and never knows what is in it.
|
|
The dependency points one way: an overlay knows about rig, rig knows about
|
|
overlays in general and about none in particular.
|
|
|
|
## Where an overlay lives
|
|
|
|
```
|
|
rig/local/<name>/ gitignored by rig: an overlay with no version control of its own,
|
|
or a clone of its own repo
|
|
anywhere/<name>/ a repo of its own, named by path
|
|
<project>/ a project folder that carries rig at <project>/rig/ (the vendored
|
|
layout, below)
|
|
```
|
|
|
|
Name it with `OVERLAY` — in `ctrl/.env` for this machine, or per call:
|
|
|
|
```bash
|
|
OVERLAY=local/myenv make cluster up
|
|
```
|
|
|
|
Relative paths are relative to rig's folder. With none named, rig uses its own
|
|
`examples/starter`, which sets nothing, so a plain rig behaves as it always did.
|
|
An overlay that is named and missing is an error; it never falls back.
|
|
|
|
## What rig reads from it
|
|
|
|
Every piece is optional.
|
|
|
|
| in the overlay | what rig does with it |
|
|
| --- | --- |
|
|
| `rig.env` | a config layer (below). Any key a profile could set. |
|
|
| `k8s/overlays/dev/` | the default `MANIFESTS_DIR`: rig's Tiltfile applies it with kustomize |
|
|
| `kind-config.yaml.tpl` | the default `KIND_CONFIG`: the cluster's shape, rendered like rig's own |
|
|
| `addons/<name>.sh` | an addon, found before rig's `ctrl/addons/<name>.sh` of the same name |
|
|
| `Tiltfile` | the workload's half of the dev loop, included by rig's `ctrl/Tiltfile` |
|
|
|
|
Anything else in the folder is the overlay's own business: Dockerfiles, DAGs,
|
|
folders of repos or data it mounts, its `.gitignore`, the secrets its kustomize
|
|
generators read. rig does not look.
|
|
|
|
## Layers
|
|
|
|
```
|
|
built-in defaults < ctrl/versions.env < ctrl/env.d/<profile>.env < <overlay>/rig.env < ctrl/.env < the caller
|
|
```
|
|
|
|
A profile says how this machine reaches the world (a registry mirror, an
|
|
air-gapped install); an overlay says what runs. `ctrl/.env` is still this
|
|
machine's, and the caller still wins over everything.
|
|
|
|
`rig.env` may not set `PROFILE` or `OVERLAY`: both are chosen before it loads.
|
|
The paths it sets (`MANIFESTS_DIR`, `KIND_CONFIG`) are relative to the overlay.
|
|
`MANIFESTS_DIR=none` means rig applies no manifests and the overlay's Tiltfile
|
|
does, e.g. when kustomize needs flags.
|
|
|
|
## Identity
|
|
|
|
With an overlay named, the cluster, the kubectl context and the port block
|
|
follow the overlay folder's name, sanitised the same way a rig folder's name is.
|
|
One rig can therefore serve several overlays, each in its own cluster, and a
|
|
project that carries rig at `./rig` does not name every cluster `rig`.
|
|
|
|
`ports.sh persist` refuses while an overlay is set: it writes to rig's
|
|
`ctrl/.env`, and a pin there would follow every overlay.
|
|
|
|
## The Tiltfile handoff
|
|
|
|
rig's `ctrl/Tiltfile` does rig's part — the context guard, `default_registry`,
|
|
the manifests, the namespaces they use — then publishes the facts and includes
|
|
the overlay's `Tiltfile`:
|
|
|
|
```
|
|
RIG_CLUSTER RIG_CONTEXT RIG_HTTP_PORT RIG_HTTPS_PORT RIG_TILT_PORT RIG_REGISTRY RIG_OVERLAY_DIR
|
|
```
|
|
|
|
Read them with `os.getenv`. An included Tiltfile runs from its own folder, so
|
|
every relative path in it (`docker_build` contexts, `sync`, `deps`, `local`) is
|
|
relative to the overlay — it never needs a path back into rig.
|
|
|
|
## Addons
|
|
|
|
An addon is a bash script run by `ctrl/addons.sh` from rig's `ctrl/`, with
|
|
`RIG_CTRL` exported. It starts like this, sources the config and does its work:
|
|
|
|
```bash
|
|
cd "${RIG_CTRL:?run it through rig: bash ctrl/addons.sh install}"
|
|
source ./lib/config.sh
|
|
load_config
|
|
```
|
|
|
|
`OVERLAY_DIR` is set, so an addon can find files beside it
|
|
(`$(_from_ctrl "$OVERLAY_DIR")/...`). rig's own `ctrl/addons/` holds only what
|
|
makes a cluster work (metallb, cert-manager, metrics-server);
|
|
`examples/data/addons/` shows workload ones.
|
|
|
|
## The kind config
|
|
|
|
An overlay's `kind-config.yaml.tpl` replaces rig's whole file, so start from a
|
|
copy of `ctrl/k8s/kind-config.yaml.tpl` and keep its containerd `config_path`
|
|
patch: `registry.sh` needs it, and `make check` says so when it is missing.
|
|
`${OVERLAY_DIR}` renders to the overlay's absolute path, for mounts:
|
|
|
|
```yaml
|
|
extraMounts:
|
|
- hostPath: ${OVERLAY_DIR}/datadir
|
|
containerPath: /rig/datadir
|
|
```
|
|
|
|
A kind config is fixed when the cluster is created: after changing it,
|
|
`make cluster reset`.
|
|
|
|
## The vendored layout
|
|
|
|
A project folder can carry rig inside it and be the overlay itself:
|
|
|
|
```
|
|
<project>/
|
|
Makefile the forwarder below
|
|
rig.env k8s/ Tiltfile kind-config.yaml.tpl addons/ ...
|
|
rig/ rig, placed as it is; tracked or ignored by the project, its call
|
|
```
|
|
|
|
The forwarder runs rig with `OVERLAY` set to this folder. It passes `OVERLAY` in
|
|
the environment, not as a make argument, so rig's own `$(shell ...)` sees it
|
|
under make 4.3 as well:
|
|
|
|
```make
|
|
HERE := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
|
|
ifeq ($(wildcard $(HERE)/rig/Makefile),)
|
|
$(error rig/ is missing — this folder is an overlay; put rig in ./rig)
|
|
endif
|
|
GOALS := $(or $(MAKECMDGOALS),help)
|
|
.PHONY: $(GOALS)
|
|
$(firstword $(GOALS)):
|
|
@OVERLAY='$(HERE)' $(MAKE) --no-print-directory -C '$(HERE)/rig' $(GOALS)
|
|
$(wordlist 2,$(words $(GOALS)),$(GOALS)):
|
|
@:
|
|
```
|
|
|
|
The cluster is then named after `<project>`, exactly as a copied rig named
|
|
`<project>` was, so moving a copied rig to this layout keeps its cluster and ports.
|
|
|
|
`rig.env` holds no secrets, by this contract. A repository whose `.gitignore` has a
|
|
broad `*.env` (a common secrets rule) would still hide it, so an overlay living in
|
|
such a repo re-includes it in its own `.gitignore`: `!rig.env`.
|
|
|
|
## Moving a copied rig to an overlay
|
|
|
|
A rig copied into a project and edited there splits cleanly:
|
|
|
|
| was, in the copy | goes to |
|
|
| --- | --- |
|
|
| `ctrl/k8s/base`, `ctrl/k8s/overlays` | `k8s/` |
|
|
| the workload parts of `ctrl/Tiltfile` | `Tiltfile` (paths now relative to the overlay) |
|
|
| `ctrl/env.d/<name>.env` | `rig.env` |
|
|
| edits to `ctrl/k8s/kind-config.yaml.tpl` | `kind-config.yaml.tpl` (`${HOST_WORKDIR}` → `${OVERLAY_DIR}`) |
|
|
| workload addons | `addons/` |
|
|
| Dockerfiles for the workload | beside the Tiltfile |
|
|
| `ctrl/.env` | `rig/ctrl/.env` (this machine's; drop a `MANIFESTS_DIR=ctrl/k8s/overlays/dev` line) |
|
|
| everything else of rig's | replaced by rig as it is |
|