init rig

rig/ctrl/k8s/README.md

@@ -0,0 +1,71 @@
+# `ctrl/k8s` — cluster shape, and what runs on it
+
+Same layout as every other project here (`unt`, `nvi`, `eth`, `mpr`, and
+soleprint's generated rooms): a kind config, a kustomize `base/`, and an
+`overlays/dev/` that patches it. See ALL `projects/templates/conventions.md`.
+
+```
+kind-config*.yaml.tpl   the cluster itself — nodes, ports, audit
+base/                   the components, as plain manifests
+overlays/dev/           how this rig differs from the base
+audit-policy.yaml       mounted into the apiserver by the audit shapes
+```
+
+## Why the cluster config is a template
+
+Every other project checks in a literal `kind-config.yaml`, because there is
+exactly one `unt` and one `nvi`. A rig is copied and renamed to make a second
+environment, and both the cluster name and the host port block follow the
+directory name — so a literal would make every copy collide on both.
+
+`ctrl/cluster.sh` renders it with `sed`, substituting `${CLUSTER}`,
+`${NODE_IMAGE}`, `${HTTP_PORT}` and `${HOST_WORKDIR}`. Not `envsubst`: that is
+`gettext-base`, which a minimal Debian does not have, and Docker being the only
+prerequisite is the one promise rig makes.
+
+**The chosen file is the source of truth for node count and audit.**
+`lib/config.sh` reads both back out of it, so a profile names a shape and does
+not restate what the YAML already says.
+
+| file | nodes | audit | profiles |
+| --- | --- | --- | --- |
+| `kind-config.yaml.tpl` | 1 | off | `minimal`, `data` |
+| `kind-config.audit.yaml.tpl` | 1 | on | `offline` |
+| `kind-config.client.yaml.tpl` | 3 | on | `client` |
+
+A profile picks one with `KIND_CONFIG` in `ctrl/env.d/<profile>.env`. Adding a
+shape is adding a file — there is no dispatcher to edit.
+
+Audit is an apiserver flag and therefore fixed at creation: changing it is
+`make cluster reset`, not a re-apply.
+
+## `base/` — replace these
+
+**The two components in `base/` are examples, not the system.** They exist so
+the real manifests have a shape to be written against.
+
+The real ones are expected to be versioned **separately from the installer** —
+they change on a different cadence, by different people, under different review.
+Point `MANIFESTS_DIR` in `ctrl/.env` at their overlay and rig stops owning them:
+
+```
+MANIFESTS_DIR=../platform-manifests/overlays/dev
+```
+
+Until then it defaults to `ctrl/k8s/overlays/dev`.
+
+### The three states a component can be in
+
+Switching between them should be a one-line change, never a rewrite. The DNS
+name stays the same in every case, so callers never know the difference:
+
+| state | what exists | when |
+| --- | --- | --- |
+| **real** | an image built from source, hot-reloaded | the one thing you are working on |
+| **mock** | a stub returning canned responses (`example-mock.yaml`) | everything else — most of the estate |
+| **remote** | no pod at all, just a Service (`example-remote.yaml`) | when the real system is reachable and you want it |
+
+Most components should be **mock**. What has to be faithful is the topology —
+names, ports, dependency order, who can reach whom, how it fails. The workloads
+are noise, and mocking them is what makes several copies of a large estate fit
+on one laptop.

rig/ctrl/k8s/audit-policy.yaml

@@ -0,0 +1,44 @@
+# Apiserver audit policy. Mounted into the control plane at creation when a
+# profile sets AUDIT=on — an apiserver flag, so it cannot be added to a running
+# cluster without recreating it.
+#
+# Deliberately modest: enough to make "who changed what, and when" answerable
+# during onboarding without filling the disk. Read the log with:
+#   docker exec <cluster>-control-plane cat /var/log/kubernetes/audit.log
+apiVersion: audit.k8s.io/v1
+kind: Policy
+
+# Never log the request body for these — they contain credentials.
+omitStages:
+  - RequestReceived
+
+rules:
+  # Secrets/configmaps: record that access happened, never the contents.
+  - level: Metadata
+    resources:
+      - group: ""
+        resources: ["secrets", "configmaps"]
+
+  # Authn/authz decisions — the part an auditor actually asks about.
+  - level: Metadata
+    nonResourceURLs:
+      - /apis*
+      - /api*
+
+  # Mutations to workloads and policy: full request, so a diff is reconstructable.
+  - level: Request
+    verbs: ["create", "update", "patch", "delete"]
+    resources:
+      - group: ""
+        resources: ["pods", "services", "serviceaccounts", "namespaces"]
+      - group: "apps"
+      - group: "networking.k8s.io"
+      - group: "rbac.authorization.k8s.io"
+
+  # Everything else that changes state: metadata only.
+  - level: Metadata
+    verbs: ["create", "update", "patch", "delete"]
+
+  # Reads are dropped entirely — otherwise controller polling drowns the log.
+  - level: None
+    verbs: ["get", "list", "watch"]

rig/ctrl/k8s/base/example-mock.yaml

@@ -0,0 +1,104 @@
+# EXAMPLE — a mocked component. Copy, rename, replace.
+#
+# A stub that answers on the right name and port with canned responses. No image
+# to build: the script is mounted from the ConfigMap, so changing the behaviour
+# is a kubectl apply, not a rebuild.
+#
+# Deliberately boring and readable. This is onboarding material — someone should
+# be able to read the generated object and recognise what it is.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: example-service-stub
+data:
+  # Canned responses by path. Add entries as the contract becomes clear;
+  # anything unmatched returns 404 so a missing route is visible, not silent.
+  routes.json: |
+    {
+      "/health":     {"status": 200, "body": {"status": "ok"}},
+      "/v1/example": {"status": 200, "body": {"items": [], "mocked": true}}
+    }
+  serve.py: |
+    import json, os
+    from http.server import BaseHTTPRequestHandler, HTTPServer
+
+    ROUTES = json.load(open("/etc/stub/routes.json"))
+    NAME = os.environ.get("STUB_NAME", "stub")
+
+    class H(BaseHTTPRequestHandler):
+        def do_GET(self):
+            r = ROUTES.get(self.path)
+            if r is None:
+                self.send_response(404)
+                self.end_headers()
+                # Say which stub rejected it — with everything mocked, "404"
+                # alone tells you nothing about where the call actually landed.
+                self.wfile.write(json.dumps(
+                    {"error": "no canned route", "stub": NAME, "path": self.path}
+                ).encode())
+                return
+            body = json.dumps(r["body"]).encode()
+            self.send_response(r["status"])
+            self.send_header("Content-Type", "application/json")
+            self.send_header("X-Mocked-By", NAME)
+            self.end_headers()
+            self.wfile.write(body)
+
+        def log_message(self, fmt, *args):
+            print("%s %s" % (NAME, fmt % args), flush=True)
+
+    HTTPServer(("0.0.0.0", 8080), H).serve_forever()
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: example-service
+  labels:
+    app: example-service
+    rig.component/impl: mock      # so `kubectl get deploy -L rig.component/impl`
+                                  # shows at a glance what is real and what is not
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: example-service
+  template:
+    metadata:
+      labels:
+        app: example-service
+    spec:
+      containers:
+        - name: stub
+          image: python:3.12-slim
+          command: ["python3", "/etc/stub/serve.py"]
+          env:
+            - name: STUB_NAME
+              value: example-service
+          ports:
+            - containerPort: 8080
+          volumeMounts:
+            - name: stub
+              mountPath: /etc/stub
+          readinessProbe:
+            httpGet: { path: /health, port: 8080 }
+            initialDelaySeconds: 2
+          # Small enough that a whole estate of these fits alongside the real
+          # thing you are working on.
+          resources:
+            requests: { memory: 32Mi, cpu: 10m }
+            limits:   { memory: 64Mi }
+      volumes:
+        - name: stub
+          configMap:
+            name: example-service-stub
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: example-service
+spec:
+  selector:
+    app: example-service
+  ports:
+    - port: 80
+      targetPort: 8080

rig/ctrl/k8s/base/example-remote.yaml

@@ -0,0 +1,46 @@
+# EXAMPLE — a component that is NOT simulated, pointed at the real system.
+#
+# This is the payoff of keeping the topology honest: there is no pod here at
+# all, yet `example-remote.<namespace>.svc.cluster.local` resolves exactly as it
+# does when the same component is mocked. Callers are identical in both cases,
+# so moving a dependency from mocked to real is a one-line change and nothing
+# downstream is touched.
+#
+# Use this when the real system is reachable and you want it in the loop.
+# Note that reachability depends on where you are running: systems restricted to
+# a managed workspace will not resolve from a laptop at all, which is the whole
+# reason most components should stay mocked.
+apiVersion: v1
+kind: Service
+metadata:
+  name: example-remote
+  labels:
+    rig.component/impl: remote
+spec:
+  type: ExternalName
+  externalName: real-system.internal.example.com
+---
+# If the real system has no DNS name — only an IP, which is common for legacy
+# hosts — ExternalName cannot express it. Use a bare Service plus manual
+# Endpoints instead, and delete the block above.
+#
+# apiVersion: v1
+# kind: Service
+# metadata:
+#   name: example-remote
+#   labels:
+#     rig.component/impl: remote
+# spec:
+#   ports:
+#     - port: 80
+#       targetPort: 8080
+# ---
+# apiVersion: v1
+# kind: Endpoints
+# metadata:
+#   name: example-remote      # must match the Service name exactly
+# subsets:
+#   - addresses:
+#       - ip: 10.0.0.42
+#     ports:
+#       - port: 8080

rig/ctrl/k8s/base/kustomization.yaml

@@ -0,0 +1,11 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+# The namespace every component lands in. The overlay overrides it, so a rig
+# modelling two estates can apply the same base twice under different names.
+namespace: rig
+
+resources:
+  - namespace.yaml
+  - example-mock.yaml
+  - example-remote.yaml

rig/ctrl/k8s/base/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: rig

rig/ctrl/k8s/kind-config.audit.yaml.tpl

@@ -0,0 +1,57 @@
+# Cluster shape: one node, apiserver audit ON. Used by the `offline` profile.
+#
+# Audit is an apiserver flag, so it is fixed when the cluster is created —
+# changing it means `make cluster reset`, not a re-apply. That is why it is a
+# property of the cluster file rather than something switched at runtime.
+#
+# k8s >= 1.31 uses kubeadm v1beta4, where extraArgs is a LIST of name/value
+# pairs. The older map form is silently ignored — it does not error, audit
+# simply never turns on.
+#
+# Substituted by ctrl/cluster.sh: CLUSTER, NODE_IMAGE, HTTP_PORT, HOST_WORKDIR
+# (named without the ${...} braces so this line survives the substitution)
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+name: ${CLUSTER}
+
+containerdConfigPatches:
+  - |-
+    [plugins."io.containerd.grpc.v1.cri".registry]
+      config_path = "/etc/containerd/certs.d"
+
+kubeadmConfigPatches:
+  - |
+    kind: ClusterConfiguration
+    apiServer:
+      extraArgs:
+        - name: audit-policy-file
+          value: /etc/kubernetes/audit/policy.yaml
+        - name: audit-log-path
+          value: /var/log/kubernetes/audit.log
+        - name: audit-log-maxage
+          value: "7"
+      extraVolumes:
+        - name: audit-policy
+          hostPath: /etc/kubernetes/audit
+          mountPath: /etc/kubernetes/audit
+          readOnly: true
+        - name: audit-log
+          hostPath: /var/log/kubernetes
+          mountPath: /var/log/kubernetes
+          readOnly: false
+
+nodes:
+  - role: control-plane
+    image: ${NODE_IMAGE}
+    # hostPath is resolved by the HOST dockerd, so this must be a host path even
+    # when cluster.sh runs inside the wizard container. HOST_WORKDIR says where
+    # this rig lives on the host; bare on a host it is just the repo root.
+    extraMounts:
+      - hostPath: ${HOST_WORKDIR}/ctrl/k8s/audit-policy.yaml
+        containerPath: /etc/kubernetes/audit/policy.yaml
+        readOnly: true
+    extraPortMappings:
+      - containerPort: 30080
+        hostPort: ${HTTP_PORT}
+        listenAddress: "0.0.0.0"
+        protocol: TCP

rig/ctrl/k8s/kind-config.client.yaml.tpl

@@ -0,0 +1,55 @@
+# Cluster shape: three nodes, apiserver audit ON. Used by the `client` profile —
+# the regulated-estate shape.
+#
+# Multi-node so taints, affinity and topology spread are real rather than
+# vacuously satisfied by a single node. It costs roughly 4-6 GB; run
+# `make cluster list` before starting this alongside other work.
+#
+# Substituted by ctrl/cluster.sh: CLUSTER, NODE_IMAGE, HTTP_PORT, HOST_WORKDIR
+# (named without the ${...} braces so this line survives the substitution)
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+name: ${CLUSTER}
+
+containerdConfigPatches:
+  - |-
+    [plugins."io.containerd.grpc.v1.cri".registry]
+      config_path = "/etc/containerd/certs.d"
+
+kubeadmConfigPatches:
+  - |
+    kind: ClusterConfiguration
+    apiServer:
+      extraArgs:
+        - name: audit-policy-file
+          value: /etc/kubernetes/audit/policy.yaml
+        - name: audit-log-path
+          value: /var/log/kubernetes/audit.log
+        - name: audit-log-maxage
+          value: "7"
+      extraVolumes:
+        - name: audit-policy
+          hostPath: /etc/kubernetes/audit
+          mountPath: /etc/kubernetes/audit
+          readOnly: true
+        - name: audit-log
+          hostPath: /var/log/kubernetes
+          mountPath: /var/log/kubernetes
+          readOnly: false
+
+nodes:
+  - role: control-plane
+    image: ${NODE_IMAGE}
+    extraMounts:
+      - hostPath: ${HOST_WORKDIR}/ctrl/k8s/audit-policy.yaml
+        containerPath: /etc/kubernetes/audit/policy.yaml
+        readOnly: true
+    extraPortMappings:
+      - containerPort: 30080
+        hostPort: ${HTTP_PORT}
+        listenAddress: "0.0.0.0"
+        protocol: TCP
+  - role: worker
+    image: ${NODE_IMAGE}
+  - role: worker
+    image: ${NODE_IMAGE}

rig/ctrl/k8s/kind-config.yaml.tpl

@@ -0,0 +1,36 @@
+# Cluster shape: one node, no audit. Used by the `minimal` and `data` profiles.
+#
+# A TEMPLATE rather than a plain kind-config.yaml because a rig is copied and
+# renamed to make a second environment, and both the cluster name and the host
+# port follow the directory. A checked-in literal would make every copy collide
+# on both. ctrl/cluster.sh renders it with sed — not envsubst, which is
+# gettext-base and absent from a minimal Debian, and rig's whole premise is that
+# Docker is the only prerequisite.
+#
+# Substituted by ctrl/cluster.sh: CLUSTER, NODE_IMAGE, HTTP_PORT, HOST_WORKDIR
+# (named without the ${...} braces so this line survives the substitution)
+# Node count and audit are READ BACK from this file by lib/config.sh, so this
+# YAML is the source of truth for both — there is no second place to update.
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+name: ${CLUSTER}
+
+# Point containerd at a certs.d directory. registry.sh drops per-host hosts.toml
+# files in there afterwards, so switching registry mode never requires
+# recreating the cluster.
+containerdConfigPatches:
+  - |-
+    [plugins."io.containerd.grpc.v1.cri".registry]
+      config_path = "/etc/containerd/certs.d"
+
+nodes:
+  - role: control-plane
+    image: ${NODE_IMAGE}
+    # One NodePort bridged to the host; an in-cluster gateway owns it. There is
+    # deliberately no ingress controller — they pin a narrow window of k8s
+    # versions, and running a trailing-edge control plane is the point.
+    extraPortMappings:
+      - containerPort: 30080
+        hostPort: ${HTTP_PORT}
+        listenAddress: "0.0.0.0"
+        protocol: TCP

rig/ctrl/k8s/overlays/dev/kustomization.yaml

@@ -0,0 +1,22 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - ../../base
+
+# The dev overlay is where a rig says how its estate differs from the base —
+# which components are real, which are mocked, which point at a live system.
+# Kept empty on purpose: the base already boots, and an overlay full of examples
+# is harder to read than one that starts blank.
+#
+# The shape a patch takes, for when the first one is needed:
+#
+# patches:
+#   - target: {kind: Service, name: example-service}
+#     patch: |
+#       - op: replace
+#         path: /spec/type
+#         value: NodePort
+#       - op: add
+#         path: /spec/ports/0/nodePort
+#         value: 30080