soleprint/rig/docs/notes/overlay.md

Overlays: what runs lives outside rig

Why

rig is the machine: the toolchain, the cluster, the registry, the port block, the dev loop's plumbing. What runs on it — the services, their manifests, their images, their settings — belongs to whoever owns that work, changes at a different rate, and often cannot be shared at all. Keeping both in one tree meant editing rig's own files to use it, and then carrying those edits into every copy.

So the use case is one folder, the overlay, kept outside rig's version control. rig reads it; rig never writes into it and never knows what is in it. The dependency points one way: an overlay knows about rig, rig knows about overlays in general and about none in particular.

Where an overlay lives

rig/local/<name>/          gitignored by rig: an overlay with no version control of its own,
                           or a clone of its own repo
anywhere/<name>/           a repo of its own, named by path
<project>/                 a project folder that carries rig at <project>/rig/ (the vendored
                           layout, below)

Name it with OVERLAY — in ctrl/.env for this machine, or per call:

OVERLAY=local/myenv make cluster up

Relative paths are relative to rig's folder. With none named, rig uses its own examples/starter, which sets nothing, so a plain rig behaves as it always did. An overlay that is named and missing is an error; it never falls back.

What rig reads from it

Every piece is optional.

in the overlay	what rig does with it
rig.env	a config layer (below). Any key a profile could set.
k8s/overlays/dev/	the default MANIFESTS_DIR: rig's Tiltfile applies it with kustomize
kind-config.yaml.tpl	the default KIND_CONFIG: the cluster's shape, rendered like rig's own
addons/<name>.sh	an addon, found before rig's ctrl/addons/<name>.sh of the same name
Tiltfile	the workload's half of the dev loop, included by rig's ctrl/Tiltfile

Anything else in the folder is the overlay's own business: Dockerfiles, DAGs, folders of repos or data it mounts, its .gitignore, the secrets its kustomize generators read. rig does not look.

Layers

built-in defaults < ctrl/versions.env < ctrl/env.d/<profile>.env < <overlay>/rig.env < ctrl/.env < the caller

A profile says how this machine reaches the world (a registry mirror, an air-gapped install); an overlay says what runs. ctrl/.env is still this machine's, and the caller still wins over everything.

rig.env may not set PROFILE or OVERLAY: both are chosen before it loads.

MEMORY_BUDGET (and optionally MEMORY_BUDGET_TIP) is how an overlay tells make check what its workload costs: rig only knows the node. See check.md. The paths it sets (MANIFESTS_DIR, KIND_CONFIG) are relative to the overlay. MANIFESTS_DIR=none means rig applies no manifests and the overlay's Tiltfile does, e.g. when kustomize needs flags.

Identity

With an overlay named, the cluster, the kubectl context and the port block follow the overlay folder's name, sanitised the same way a rig folder's name is. One rig can therefore serve several overlays, each in its own cluster, and a project that carries rig at ./rig does not name every cluster rig.

That is the default. The convention is to pin the name in the overlay's rig.env:

CLUSTER=<name>

The name only has to be unique on the machine: the port block derives from it, so two overlays with the same name share one cluster and one set of ports. Pinned, it no longer follows the folder, so moving or renaming folders changes nothing. A second copy of the same project (another worktree, say) runs beside it by setting its own CLUSTER= in the project's ignored .env.local: the caller beats rig.env. Ports the workload forwards itself (a Tiltfile's port_forwards) are the overlay's, not the block's, and can still collide.

A repository that runs on rig keeps it in a devenv/ folder, with rig as the project's sibling:

devenv/
  rig/              rig
  <project>/        the forwarder (RIG_DIR ?= $(HERE)/../rig), its knobs, its data
    overlay/        what rig reads

The overlay folder is then called overlay/ in every project, so its name says nothing and CLUSTER= in its rig.env is required, not optional: left to the default, every such project would be the cluster overlay, on the same ports. The project folder, the cluster and the workload's namespace are three names: a folder may hold _, the other two are DNS labels.

ports.sh persist refuses while an overlay is set: it writes to rig's ctrl/.env, and a pin there would follow every overlay.

The Tiltfile handoff

rig's ctrl/Tiltfile does rig's part — the context guard, default_registry, the manifests, the namespaces they use — then publishes the facts and includes the overlay's Tiltfile:

RIG_CLUSTER RIG_CONTEXT RIG_HTTP_PORT RIG_HTTPS_PORT RIG_TILT_PORT RIG_REGISTRY RIG_OVERLAY_DIR

Read them with os.getenv. An included Tiltfile runs from its own folder, so every relative path in it (docker_build contexts, sync, deps, local) is relative to the overlay — it never needs a path back into rig.

Addons

An addon is a bash script run by ctrl/addons.sh from rig's ctrl/, with RIG_CTRL exported. It starts like this, sources the config and does its work:

cd "${RIG_CTRL:?run it through rig: bash ctrl/addons.sh install}"
source ./lib/config.sh
load_config

OVERLAY_DIR is set, so an addon can find files beside it ($(_from_ctrl "$OVERLAY_DIR")/...). Its manifest is a <name>.yaml beside the script, rendered with render_manifest (see addons.md). rig's own ctrl/addons/ holds only what makes a cluster work (metallb, cert-manager, metrics-server); examples/data/addons/ shows workload ones.

The kind config

An overlay's kind-config.yaml.tpl replaces rig's whole file, so start from a copy of ctrl/k8s/kind-config.yaml.tpl and keep its containerd config_path patch: registry.sh needs it, and make check says so when it is missing. ${OVERLAY_DIR} renders to the overlay's absolute path, for mounts:

    extraMounts:
      - hostPath: ${OVERLAY_DIR}/datadir
        containerPath: /rig/datadir

A kind config is fixed when the cluster is created: after changing it, make cluster reset.

The vendored layout

A project folder can carry rig inside it and be the overlay itself:

<project>/
  Makefile        the forwarder below
  rig.env  k8s/  Tiltfile  kind-config.yaml.tpl  addons/  ...
  rig/            rig, placed as it is; tracked or ignored by the project, its call

The forwarder runs rig with OVERLAY set to this folder. It passes OVERLAY in the environment, not as a make argument, so rig's own $(shell ...) sees it under make 4.3 as well:

HERE := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
ifeq ($(wildcard $(HERE)/rig/Makefile),)
$(error rig/ is missing — this folder is an overlay; put rig in ./rig)
endif
GOALS := $(or $(MAKECMDGOALS),help)
.PHONY: $(GOALS)
$(firstword $(GOALS)):
	@OVERLAY='$(HERE)' $(MAKE) --no-print-directory -C '$(HERE)/rig' $(GOALS)
$(wordlist 2,$(words $(GOALS)),$(GOALS)):
	@:

The cluster is then named after <project>, exactly as a copied rig named <project> was, so moving a copied rig to this layout keeps its cluster and ports.

rig can also live outside the project, which then holds only the overlay. The forwarder names it with one variable, a machine path that is never committed (the shell, or an ignored .env.local), defaulting to the vendored layout:

RIG_DIR ?= $(HERE)/rig
override RIG_DIR := $(abspath $(patsubst ~/%,$(HOME)/%,$(RIG_DIR)))

make docs should serve the project's pages, not rig's: the forwarder names them with DOCS_DIR := $(HERE)/docs (see docs.md). rig's own stay make -C <rig> docs.

No search path: an unset RIG_DIR means ./rig, and nothing else is tried. Moving rig changes nothing in the cluster's name, context or ports; moving or renaming the overlay folder does.

.version, at rig's top, names the rig. It travels with the folder, vendored, a worktree, a plain copy or a git archive release, and needs no git. Patch and minor steps are edits to that file, with no tag; a rig-<version> tag marks a milestone only. A project records the version it was last run against and warns on another; it does not refuse. Releases before 0.1.1 carry VERSION instead, so the project reads that as a fallback:

RIG_REF := 0.1.1
RIG_AT  := $(strip $(shell cat '$(RIG_DIR)/.version' 2>/dev/null || cat '$(RIG_DIR)/VERSION' 2>/dev/null))
ifneq ($(RIG_AT),$(RIG_REF))
$(warning rig at $(RIG_DIR) is $(or $(RIG_AT),unversioned); this overlay was last run against rig $(RIG_REF))
endif

rig.env holds no secrets, by this contract. A repository whose .gitignore has a broad *.env (a common secrets rule) would still hide it, so an overlay living in such a repo re-includes it in its own .gitignore: !rig.env.