soleprint/rig/docs/notes/overlay.md

Overlays: what runs lives outside rig

Why

rig is the machine: the toolchain, the cluster, the registry, the port block, the dev loop's plumbing. What runs on it — the services, their manifests, their images, their settings — belongs to whoever owns that work, changes at a different rate, and often cannot be shared at all. Keeping both in one tree meant editing rig's own files to use it, and then carrying those edits into every copy.

So the use case is one folder, the overlay, kept outside rig's version control. rig reads it; rig never writes into it and never knows what is in it. The dependency points one way: an overlay knows about rig, rig knows about overlays in general and about none in particular.

Where an overlay lives

rig/local/<name>/          gitignored by rig: an overlay with no version control of its own,
                           or a clone of its own repo
anywhere/<name>/           a repo of its own, named by path
<project>/                 a project folder that carries rig at <project>/rig/ (the vendored
                           layout, below)

Name it with OVERLAY — in ctrl/.env for this machine, or per call:

OVERLAY=local/myenv make cluster up

Relative paths are relative to rig's folder. With none named, rig uses its own examples/starter, which sets nothing, so a plain rig behaves as it always did. An overlay that is named and missing is an error; it never falls back.

What rig reads from it

Every piece is optional.

in the overlay	what rig does with it
rig.env	a config layer (below). Any key a profile could set.
k8s/overlays/dev/	the default MANIFESTS_DIR: rig's Tiltfile applies it with kustomize
kind-config.yaml.tpl	the default KIND_CONFIG: the cluster's shape, rendered like rig's own
addons/<name>.sh	an addon, found before rig's ctrl/addons/<name>.sh of the same name
Tiltfile	the workload's half of the dev loop, included by rig's ctrl/Tiltfile

Anything else in the folder is the overlay's own business: Dockerfiles, DAGs, folders of repos or data it mounts, its .gitignore, the secrets its kustomize generators read. rig does not look.

Layers

built-in defaults < ctrl/versions.env < ctrl/env.d/<profile>.env < <overlay>/rig.env < ctrl/.env < the caller

A profile says how this machine reaches the world (a registry mirror, an air-gapped install); an overlay says what runs. ctrl/.env is still this machine's, and the caller still wins over everything.

rig.env may not set PROFILE or OVERLAY: both are chosen before it loads.

MEMORY_BUDGET (and optionally MEMORY_BUDGET_TIP) is how an overlay tells make check what its workload costs: rig only knows the node. See check.md. The paths it sets (MANIFESTS_DIR, KIND_CONFIG) are relative to the overlay. MANIFESTS_DIR=none means rig applies no manifests and the overlay's Tiltfile does, e.g. when kustomize needs flags.

Identity

With an overlay named, the cluster, the kubectl context and the port block follow the overlay folder's name, sanitised the same way a rig folder's name is. One rig can therefore serve several overlays, each in its own cluster, and a project that carries rig at ./rig does not name every cluster rig.

ports.sh persist refuses while an overlay is set: it writes to rig's ctrl/.env, and a pin there would follow every overlay.

The Tiltfile handoff

rig's ctrl/Tiltfile does rig's part — the context guard, default_registry, the manifests, the namespaces they use — then publishes the facts and includes the overlay's Tiltfile:

RIG_CLUSTER RIG_CONTEXT RIG_HTTP_PORT RIG_HTTPS_PORT RIG_TILT_PORT RIG_REGISTRY RIG_OVERLAY_DIR

Read them with os.getenv. An included Tiltfile runs from its own folder, so every relative path in it (docker_build contexts, sync, deps, local) is relative to the overlay — it never needs a path back into rig.

Addons

An addon is a bash script run by ctrl/addons.sh from rig's ctrl/, with RIG_CTRL exported. It starts like this, sources the config and does its work:

cd "${RIG_CTRL:?run it through rig: bash ctrl/addons.sh install}"
source ./lib/config.sh
load_config

OVERLAY_DIR is set, so an addon can find files beside it ($(_from_ctrl "$OVERLAY_DIR")/...). Its manifest is a <name>.yaml beside the script, rendered with render_manifest (see addons.md). rig's own ctrl/addons/ holds only what makes a cluster work (metallb, cert-manager, metrics-server); examples/data/addons/ shows workload ones.

The kind config

An overlay's kind-config.yaml.tpl replaces rig's whole file, so start from a copy of ctrl/k8s/kind-config.yaml.tpl and keep its containerd config_path patch: registry.sh needs it, and make check says so when it is missing. ${OVERLAY_DIR} renders to the overlay's absolute path, for mounts:

    extraMounts:
      - hostPath: ${OVERLAY_DIR}/datadir
        containerPath: /rig/datadir

A kind config is fixed when the cluster is created: after changing it, make cluster reset.

The vendored layout

A project folder can carry rig inside it and be the overlay itself:

<project>/
  Makefile        the forwarder below
  rig.env  k8s/  Tiltfile  kind-config.yaml.tpl  addons/  ...
  rig/            rig, placed as it is; tracked or ignored by the project, its call

The forwarder runs rig with OVERLAY set to this folder. It passes OVERLAY in the environment, not as a make argument, so rig's own $(shell ...) sees it under make 4.3 as well:

HERE := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
ifeq ($(wildcard $(HERE)/rig/Makefile),)
$(error rig/ is missing — this folder is an overlay; put rig in ./rig)
endif
GOALS := $(or $(MAKECMDGOALS),help)
.PHONY: $(GOALS)
$(firstword $(GOALS)):
	@OVERLAY='$(HERE)' $(MAKE) --no-print-directory -C '$(HERE)/rig' $(GOALS)
$(wordlist 2,$(words $(GOALS)),$(GOALS)):
	@:

The cluster is then named after <project>, exactly as a copied rig named <project> was, so moving a copied rig to this layout keeps its cluster and ports.

rig can also live outside the project, which then holds only the overlay. The forwarder names it with one variable, a machine path that is never committed (the shell, or an ignored .env.local), defaulting to the vendored layout:

RIG_DIR ?= $(HERE)/rig
override RIG_DIR := $(abspath $(patsubst ~/%,$(HOME)/%,$(RIG_DIR)))

No search path: an unset RIG_DIR means ./rig, and nothing else is tried. Moving rig changes nothing in the cluster's name, context or ports; moving or renaming the overlay folder does.

.version, at rig's top, names the rig. It travels with the folder, vendored, a worktree, a plain copy or a git archive release, and needs no git. Patch and minor steps are edits to that file, with no tag; a rig-<version> tag marks a milestone only. A project records the version it was last run against and warns on another; it does not refuse. Releases before 0.1.1 carry VERSION instead, so the project reads that as a fallback:

RIG_REF := 0.1.1
RIG_AT  := $(strip $(shell cat '$(RIG_DIR)/.version' 2>/dev/null || cat '$(RIG_DIR)/VERSION' 2>/dev/null))
ifneq ($(RIG_AT),$(RIG_REF))
$(warning rig at $(RIG_DIR) is $(or $(RIG_AT),unversioned); this overlay was last run against rig $(RIG_REF))
endif

rig.env holds no secrets, by this contract. A repository whose .gitignore has a broad *.env (a common secrets rule) would still hide it, so an overlay living in such a repo re-includes it in its own .gitignore: !rig.env.