6.7 KiB
Overlays: what runs lives outside rig
Why
rig is the machine: the toolchain, the cluster, the registry, the port block, the dev loop's plumbing. What runs on it — the services, their manifests, their images, their settings — belongs to whoever owns that work, changes at a different rate, and often cannot be shared at all. Keeping both in one tree meant editing rig's own files to use it, and then carrying those edits into every copy.
So the use case is one folder, the overlay, kept outside rig's version control. rig reads it; rig never writes into it and never knows what is in it. The dependency points one way: an overlay knows about rig, rig knows about overlays in general and about none in particular.
Where an overlay lives
rig/local/<name>/ gitignored by rig: an overlay with no version control of its own,
or a clone of its own repo
anywhere/<name>/ a repo of its own, named by path
<project>/ a project folder that carries rig at <project>/rig/ (the vendored
layout, below)
Name it with OVERLAY — in ctrl/.env for this machine, or per call:
OVERLAY=local/myenv make cluster up
Relative paths are relative to rig's folder. With none named, rig uses its own
examples/starter, which sets nothing, so a plain rig behaves as it always did.
An overlay that is named and missing is an error; it never falls back.
What rig reads from it
Every piece is optional.
| in the overlay | what rig does with it |
|---|---|
rig.env |
a config layer (below). Any key a profile could set. |
k8s/overlays/dev/ |
the default MANIFESTS_DIR: rig's Tiltfile applies it with kustomize |
kind-config.yaml.tpl |
the default KIND_CONFIG: the cluster's shape, rendered like rig's own |
addons/<name>.sh |
an addon, found before rig's ctrl/addons/<name>.sh of the same name |
Tiltfile |
the workload's half of the dev loop, included by rig's ctrl/Tiltfile |
Anything else in the folder is the overlay's own business: Dockerfiles, DAGs,
folders of repos or data it mounts, its .gitignore, the secrets its kustomize
generators read. rig does not look.
Layers
built-in defaults < ctrl/versions.env < ctrl/env.d/<profile>.env < <overlay>/rig.env < ctrl/.env < the caller
A profile says how this machine reaches the world (a registry mirror, an
air-gapped install); an overlay says what runs. ctrl/.env is still this
machine's, and the caller still wins over everything.
rig.env may not set PROFILE or OVERLAY: both are chosen before it loads.
The paths it sets (MANIFESTS_DIR, KIND_CONFIG) are relative to the overlay.
MANIFESTS_DIR=none means rig applies no manifests and the overlay's Tiltfile
does, e.g. when kustomize needs flags.
Identity
With an overlay named, the cluster, the kubectl context and the port block
follow the overlay folder's name, sanitised the same way a rig folder's name is.
One rig can therefore serve several overlays, each in its own cluster, and a
project that carries rig at ./rig does not name every cluster rig.
ports.sh persist refuses while an overlay is set: it writes to rig's
ctrl/.env, and a pin there would follow every overlay.
The Tiltfile handoff
rig's ctrl/Tiltfile does rig's part — the context guard, default_registry,
the manifests, the namespaces they use — then publishes the facts and includes
the overlay's Tiltfile:
RIG_CLUSTER RIG_CONTEXT RIG_HTTP_PORT RIG_HTTPS_PORT RIG_TILT_PORT RIG_REGISTRY RIG_OVERLAY_DIR
Read them with os.getenv. An included Tiltfile runs from its own folder, so
every relative path in it (docker_build contexts, sync, deps, local) is
relative to the overlay — it never needs a path back into rig.
Addons
An addon is a bash script run by ctrl/addons.sh from rig's ctrl/, with
RIG_CTRL exported. It starts like this, sources the config and does its work:
cd "${RIG_CTRL:?run it through rig: bash ctrl/addons.sh install}"
source ./lib/config.sh
load_config
OVERLAY_DIR is set, so an addon can find files beside it
($(_from_ctrl "$OVERLAY_DIR")/...). rig's own ctrl/addons/ holds only what
makes a cluster work (metallb, cert-manager, metrics-server);
examples/data/addons/ shows workload ones.
The kind config
An overlay's kind-config.yaml.tpl replaces rig's whole file, so start from a
copy of ctrl/k8s/kind-config.yaml.tpl and keep its containerd config_path
patch: registry.sh needs it, and make check says so when it is missing.
${OVERLAY_DIR} renders to the overlay's absolute path, for mounts:
extraMounts:
- hostPath: ${OVERLAY_DIR}/datadir
containerPath: /rig/datadir
A kind config is fixed when the cluster is created: after changing it,
make cluster reset.
The vendored layout
A project folder can carry rig inside it and be the overlay itself:
<project>/
Makefile the forwarder below
rig.env k8s/ Tiltfile kind-config.yaml.tpl addons/ ...
rig/ rig, placed as it is; tracked or ignored by the project, its call
The forwarder runs rig with OVERLAY set to this folder. It passes OVERLAY in
the environment, not as a make argument, so rig's own $(shell ...) sees it
under make 4.3 as well:
HERE := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
ifeq ($(wildcard $(HERE)/rig/Makefile),)
$(error rig/ is missing — this folder is an overlay; put rig in ./rig)
endif
GOALS := $(or $(MAKECMDGOALS),help)
.PHONY: $(GOALS)
$(firstword $(GOALS)):
@OVERLAY='$(HERE)' $(MAKE) --no-print-directory -C '$(HERE)/rig' $(GOALS)
$(wordlist 2,$(words $(GOALS)),$(GOALS)):
@:
The cluster is then named after <project>, exactly as a copied rig named
<project> was, so moving a copied rig to this layout keeps its cluster and ports.
rig.env holds no secrets, by this contract. A repository whose .gitignore has a
broad *.env (a common secrets rule) would still hide it, so an overlay living in
such a repo re-includes it in its own .gitignore: !rig.env.
Moving a copied rig to an overlay
A rig copied into a project and edited there splits cleanly:
| was, in the copy | goes to |
|---|---|
ctrl/k8s/base, ctrl/k8s/overlays |
k8s/ |
the workload parts of ctrl/Tiltfile |
Tiltfile (paths now relative to the overlay) |
ctrl/env.d/<name>.env |
rig.env |
edits to ctrl/k8s/kind-config.yaml.tpl |
kind-config.yaml.tpl (${HOST_WORKDIR} → ${OVERLAY_DIR}) |
| workload addons | addons/ |
| Dockerfiles for the workload | beside the Tiltfile |
ctrl/.env |
rig/ctrl/.env (this machine's; drop a MANIFESTS_DIR=ctrl/k8s/overlays/dev line) |
| everything else of rig's | replaced by rig as it is |