attemp to develop rig in spr without an actual use case

This commit is contained in:
2026-08-26 07:27:12 -03:00
parent 83b6cbebe3
commit 966f8fc821
65 changed files with 5887 additions and 2853 deletions

4
.gitignore vendored
View File

@@ -40,7 +40,5 @@ cfg/dlt/
# not land here. They are versioned in their own repo.
#
# Anchored at the ROOT on purpose: a copy is a SIBLING of rig/, so a rule inside
# rig/.gitignore cannot see it. The negation must name the full path for the same
# reason — `*-rig/` is unanchored and matches at any depth, including rig/sample-rig.
# rig/.gitignore cannot see it.
*-rig/
!rig/sample-rig/

7
rig/.gitignore vendored
View File

@@ -11,10 +11,9 @@ ctrl/.env
arch/*.dot
ctrl/Tiltfile.gen
# binaries pulled by `make deps-bundle` for the air-gapped wizard image
# binaries pulled by `make deps-bundle` for the air-gapped installer image
vendor
# Client rigs are NOT ignored here. A copy is a SIBLING of this directory
# (spr/acme-rig), so a rule in this file cannot see it — the rules live in
# spr/.gitignore, anchored at spr's root, where `*-rig/` matches the siblings and
# `!rig/sample-rig/` keeps the committed stand-in.
# (../acme-rig), so a rule in this file cannot see it — the rules live in the
# parent repo's .gitignore, anchored at its root, where `*-rig/` matches them.

View File

@@ -4,9 +4,8 @@ The README says the prerequisite is Docker and nothing else. This is what that
actually looks like end to end: a bare Linux box, and a new project running under
Tilt at the end of it.
rig lives inside soleprint, at `spr/rig` — it is soleprint's cluster half, and
a client copy is a sibling (`spr/acme-rig`). Paths below are relative to
soleprint's checkout.
A copy of this directory is a sibling of it, named after the environment it
models (`acme-rig`). Paths below are relative to the parent checkout.
It spans three repos because the work does. **rig** prepares the machine — the
pinned toolchain, the cluster, the port arithmetic. **all** owns the shape a
@@ -50,7 +49,7 @@ isn't.
## Read the docs before installing anything
```bash
cd spr/rig
cd rig
make docs
```
@@ -67,11 +66,11 @@ persists; ctrl-c ends it.
## Ask what is wrong with this machine
```bash
make station
make check
cp ctrl/.env.example ctrl/.env
```
`station.sh` reports and instructs, and fixes nothing. It runs bare rather than
`check.sh` reports and instructs, and fixes nothing. It runs bare rather than
in a container because host detection only ever reads `/proc` and `/etc` — no
dependency beyond coreutils.
@@ -80,7 +79,7 @@ port rig binds derives from this directory's name, so the answer is specific to
this copy, and a clash here surfaces as an opaque `failed to bind host port` in
the middle of cluster creation if you skip it.
Copy the `.env` even though station only warns about it. It is gitignored, it is
Copy the `.env` even though the check only warns about it. It is gitignored, it is
where a machine-local override goes, and `ports.sh persist` expects it to exist.
@@ -88,33 +87,33 @@ where a machine-local override goes, and `ports.sh persist` expects it to exist.
This is the step where "nothing installed" stops being rhetorical.
`make deps` runs `ctrl/wizard.sh install` directly on the host, and the wizard
`make deps` runs `ctrl/deps.sh install` directly on the host, and the installer
fetches with `curl`. A stock `debian:trixie-slim` has no curl — detection runs
fine, then the first download dies with `curl: command not found` and an exit
code of 127. That is the bootstrap paradox `ctrl/Dockerfile.wizard` exists
to kill — the wizard carries its own toolchain so the host needs only Docker —
code of 127. That is the bootstrap paradox `ctrl/Dockerfile.deps` exists
to kill — the installer carries its own toolchain so the host needs only Docker —
but building the image and running it are two different things, and only the
build has a Makefile target today. **On a genuinely bare machine, run it by
hand:**
```bash
make wizard # builds rig-wizard:wizard
make deps-image # builds rig-deps:deps
mkdir -p ~/.local/bin
docker run --rm \
-v /:/host:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v "$HOME/.local/bin:/out/bin" \
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
rig-wizard:wizard install dev
rig-deps:deps install dev
```
The image name follows the directory, like everything else here: in `spr/rig`
it is `rig-wizard`, in a copy called `spr/acme-rig` it is `acme-rig-wizard`. The
tag is `wizard` (or `full`, below), not `latest`.
The image name follows the directory, like everything else here: in `rig`
it is `rig-deps`, in a copy called `acme-rig` it is `acme-rig-deps`. The
tag is `deps` (or `full`, below), not `latest`.
None of the four arguments are guessable, so:
- **`/:/host:ro`** — the wizard reads the *host's* `/etc/os-release` and
- **`/:/host:ro`** — the installer reads the *host's* `/etc/os-release` and
`/etc/wsl.conf`, not the container's. `HOST_ROOT=/host` is already baked into
the image; this is what it points at. Read-only, and it is the only reason
detection inside a container tells you anything about the machine.
@@ -122,7 +121,7 @@ None of the four arguments are guessable, so:
and how it counts kind clusters already running.
- **`/out/bin`** — the image's `OUT_BIN`. Whatever you mount here is where the
four binaries land.
- **`HOST_UID` / `HOST_GID`** — the wizard runs as root so it can reach that
- **`HOST_UID` / `HOST_GID`** — the installer runs as root so it can reach that
socket, which means everything it writes into a mounted volume is root-owned
and useless to you. These drive the `chown` back. Omit them and the install
looks like it worked.
@@ -131,18 +130,18 @@ None of the four arguments are guessable, so:
tooling — which is the right answer on a managed or corporate-issued machine and
is why the split exists.
Then put them on PATH, which the wizard will remind you about because it cannot
Then put them on PATH, which the installer will remind you about because it cannot
edit your shell for you:
```bash
export PATH="$HOME/.local/bin:$PATH" # and add the same line to ~/.bashrc
```
If something else on this machine already provides `kubectl`, the wizard says so
If something else on this machine already provides `kubectl`, the installer says so
by name rather than shadowing it quietly. `OUT_BIN=$PWD/def/bin` installs
somewhere private instead.
**Two variants worth knowing before you need them.** `make wizard full` bakes
**Two variants worth knowing before you need them.** `make deps-image full` bakes
every pinned binary into the image at build time (`DEPS_SOURCE=baked`), so
`docker save` gives you the entire installer as one file to carry into an
air-gapped network. And `DEPS_SOURCE=artifactory` with `DEPS_ARTIFACTORY_URL`

View File

@@ -20,7 +20,7 @@ SLUG := $(shell echo '$(notdir $(CURDIR))' | tr '[:upper:]' '[:lower:]' | t
CLUSTER := $(or $(shell sed -n 's/^CLUSTER=//p' ctrl/.env 2>/dev/null),$(SLUG))
KCTX := --context kind-$(CLUSTER)
TILT_PORT := $(shell sed -n 's/^TILT_PORT=//p' ctrl/.env 2>/dev/null)
WIZARD := $(SLUG)-wizard
DEPSIMG := $(SLUG)-deps
# Words after the target become the script's subcommand. Make would otherwise
# treat them as goals of their own, so each gets a no-op rule.
@@ -35,7 +35,7 @@ $(eval $(ARGS):;@:)
.PHONY: $(ARGS)
endif
.PHONY: help setup station deps wizard cluster registry addons ports \
.PHONY: help setup check mem deps deps-image cluster registry addons ports \
newbox dockerhost docs tilt \
kind-up kind-down kind-reset tilt-up tilt-down
@@ -47,16 +47,19 @@ help: ## list targets
setup: ## prepare this machine [core] [--share-docker] [--cluster]
bash ctrl/setup.sh $(ARGS)
station: ## is this workstation ready? reports, never fixes
bash ctrl/station.sh
check: ## is this machine ready? reports, never fixes
bash ctrl/check.sh
mem: ## memory, and any cap holding it [status|backup|restore]
bash ctrl/mem.sh $(or $(ARGS),status)
deps: ## install the toolchain [core|dev] (default dev)
bash ctrl/wizard.sh install $(or $(ARGS),dev)
bash ctrl/deps.sh install $(or $(ARGS),dev)
wizard: ## build the installer image [full]
docker build -f ctrl/Dockerfile.wizard \
--target $(if $(filter full,$(ARGS)),wizard-full,wizard) \
-t $(WIZARD):$(if $(filter full,$(ARGS)),full,wizard) .
deps-image: ## build the installer image [full]
docker build -f ctrl/Dockerfile.deps \
--target $(if $(filter full,$(ARGS)),deps-full,deps) \
-t $(DEPSIMG):$(if $(filter full,$(ARGS)),full,deps) .
# ── cluster ────────────────────────────────────────────────────────────────

View File

@@ -9,6 +9,46 @@ topology, not the workloads.
**Docker.** Nothing else — no curl, no jq, no python, no apt repositories.
### Starting from plain Windows
Everything here is bash and runs *inside* a Linux shell, so on a Windows machine
that means WSL. Nothing in rig installs WSL, and nothing will: `wsl --install`
enables Windows features and requires a reboot, which is not something a script
should do to a machine on your behalf — and there is no tested undo for it.
From an elevated PowerShell or Command Prompt, once:
```powershell
wsl --install
```
Then reboot and open the Linux shell it installed.
**If you cloned this on the Windows side, copy it into WSL before carrying on.**
WSL can reach the Windows drives at `/mnt/c`, and working from there mostly
functions — slowly — but file watching does not: that filesystem raises no
inotify events, so anything watching for edits silently stops seeing them.
```bash
cp -r /mnt/c/Users/<you>/rig ~/rig
cd ~/rig
```
`make deps` reports it if you are running from `/mnt/...`. Then carry on below.
If it fails, the usual causes give unhelpful messages:
| symptom | cause |
| --- | --- |
| "the virtual machine could not be started" | virtualization disabled in BIOS/UEFI |
| the command is not recognised | Windows build too old — needs 2004 or later |
| the install starts, then nothing works | a reboot is still pending |
Running the scripts from **Git Bash, MSYS or Cygwin does not work** — those look
close enough to a Linux shell to get started and then fail without `/proc` or a
docker socket. `ctrl/deps.sh` detects that and says so rather than letting you
find out the slow way.
## Read the docs first
```bash
@@ -21,16 +61,43 @@ instructions for everything else. No cluster and no toolchain required.
## Then
```bash
make station # report host and config problems; changes nothing
make check # report host and config problems; changes nothing
make deps # install the toolchain (add `core` on a managed machine)
make cluster up # build the cluster for the active profile
```
`make cluster up` also starts this environment's local registry and wires it
into the node, so an image built locally is pullable by the cluster without
going near docker.io:
```bash
make registry status # prints: endpoint localhost:<port>
docker build -t localhost:<port>/app:1 .
docker push localhost:<port>/app:1
kubectl --context kind-$(basename $PWD) run app --image=localhost:<port>/app:1
```
The port block is derived from the directory name, so two copies of rig never
collide:
```bash
make ports show # HTTP / HTTPS / TILT / REGISTRY
make cluster list # every cluster on this machine, with memory
make cluster free # stop the others if memory is tight
make cluster down # remove this cluster and its registry
```
**`make tilt` has nothing to run yet.** The target and its `tilt-up` / `tilt-down`
aliases exist so rig answers to the same spelling as every other project here,
but rig ships no `Tiltfile` — it builds the estate, it is not itself a service
with a dev loop. Add a `ctrl/Tiltfile` and the target works; until then it fails
on the missing file, not on anything rig did.
`make help` lists every target.
On a machine where Docker really is the only thing installed, `make deps` has
nothing to download with — see [BOOTSTRAP.md](BOOTSTRAP.md), which runs the
toolchain through the wizard container and carries on to scaffolding and running
toolchain through the installer container and carries on to scaffolding and running
a new project.
## One directory is one environment
@@ -39,10 +106,10 @@ Copy this directory, rename it, run it. Cluster name, kubectl context, image
tags and the host port block all derive from the directory name, so copies never
collide and neither one's teardown can touch the other.
rig lives inside soleprint, at `spr/rig` — it is soleprint's cluster half, and a
copy is a **sibling**: `spr/acme-rig`. That is why the ignore rules for client
rigs sit in `spr/.gitignore` rather than here; a rule in this directory cannot
see a directory beside it.
A copy of this directory is a **sibling** of it, named after the environment it
models (`acme-rig`). That is why the ignore rules for copies sit in the *parent*
repo's `.gitignore` rather than here: a rule in this directory cannot see a
directory beside it.
## Profiles
@@ -54,7 +121,7 @@ apiserver audits. They live in `ctrl/env.d/`, and the active one is `PROFILE`.
| `minimal` | the default. One node, no addons, boots fast. |
| `client` | the regulated-estate shape — multi-node, audit on, registry mirror. |
| `offline` | air-gapped: everything from a preloaded local registry. |
| `data` | the dependency containers a soleprint room asks for. |
| `data` | the cabinets an environment asks for. |
```bash
PROFILE=data make cluster up
@@ -98,10 +165,10 @@ cluster does.
| `redis` | cache and broker |
| `airflow` | scheduled pipelines; needs postgres and redis |
The last three are the cluster half of **soleprint's cabinets**. A room declares
what it needs once, in `cfg/<room>/data/cabinets.json`; soleprint's `build.py`
composes those services into `docker-compose.yml` for a laptop, and these
install the same ones here. The names match on purpose each cabinet carries a
The last three are **cabinets**: a public service dropped in as-is, the upstream
image unmodified, reachable at a known address. A cabinet is declared once and
installs on either target — a `service.yml` composes it for a laptop, and these
install the same one here. The names match on purpose: each cabinet carries a
`rig_addon` field pointing at `ctrl/addons/<name>.sh`.
Plain manifests rather than helm charts, like every other addon: a chart repo is

View File

@@ -26,10 +26,10 @@ PROFILE=minimal
# MANIFESTS_DIR=../platform-manifests/overlays/dev
MANIFESTS_DIR=ctrl/k8s/overlays/dev
# Where the wizard fetches the pinned binaries from.
# Where the installer fetches the pinned binaries from.
# upstream GitHub releases / dl.k8s.io (needs internet)
# artifactory a generic repo — what a locked-down client usually allows
# baked already inside the wizard image; no network at all
# baked already inside the installer image; no network at all
DEPS_SOURCE=upstream
DEPS_ARTIFACTORY_URL=
@@ -43,7 +43,7 @@ REGISTRY_PASSWORD=
# Corporate root CA, if Artifactory is fronted by an internal CA (it usually is).
# Trust has to reach THREE places and nothing does it for you: the host docker
# daemon, every kind node's containerd, and any in-cluster client. registry.sh
# handles the first two; station.sh reports when it's configured but not trusted.
# handles the first two; check.sh reports when it's configured but not trusted.
# Symptom when missing: x509: certificate signed by unknown authority
REGISTRY_CA_FILE=

View File

@@ -1,18 +1,18 @@
# The installation wizard. It does NOT run the cluster — it installs a toolchain
# The toolchain installer image. It does NOT run the cluster — it installs a toolchain
# onto the host and gets out of the way.
#
# This exists to kill a bootstrap paradox: a plain bash installer needs curl, jq
# and sha256sum to already be present, and a minimal Debian has none of them.
# The wizard carries its own toolchain, so the only host prerequisite is Docker.
# It carries its own toolchain, so the only host prerequisite is Docker.
#
# Two variants from one file:
# docker build -f ctrl/Dockerfile.wizard --target wizard -t <slug>-wizard .
# docker build -f ctrl/Dockerfile.wizard --target wizard-full -t <slug>-wizard:full .
# docker build -f ctrl/Dockerfile.deps --target deps -t <slug>-deps .
# docker build -f ctrl/Dockerfile.deps --target deps-full -t <slug>-deps:full .
#
# wizard-full bakes every pinned binary in at build time. `docker save` it and
# deps-full bakes every pinned binary in at build time. `docker save` it and
# you have the whole installer as one file to carry into an air-gapped network.
FROM debian:trixie-slim AS wizard
FROM debian:trixie-slim AS deps
# ca-certificates + curl: fetch and verify. graphviz + python3: render diagrams
# and validate the arch model, so the host never needs an apt package.
@@ -26,21 +26,21 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
WORKDIR /work
COPY ctrl/versions.env /work/ctrl/versions.env
COPY ctrl/wizard.sh /work/ctrl/wizard.sh
RUN chmod +x /work/ctrl/wizard.sh
COPY ctrl/deps.sh /work/ctrl/deps.sh
RUN chmod +x /work/ctrl/deps.sh
# Defaults; every one is overridable with -e at run time.
ENV DEPS_SOURCE=upstream \
OUT_BIN=/out/bin \
HOST_ROOT=/host
ENTRYPOINT ["/work/ctrl/wizard.sh"]
ENTRYPOINT ["/work/ctrl/deps.sh"]
CMD ["install"]
# ---------------------------------------------------------------------------
# wizard-full — same wizard, binaries baked in, works with no network at all.
FROM wizard AS wizard-full
RUN /work/ctrl/wizard.sh fetch --to /opt/rig/bin
# deps-full — same image, binaries baked in, works with no network at all.
FROM deps AS deps-full
RUN /work/ctrl/deps.sh fetch --to /opt/rig/bin
ENV DEPS_SOURCE=baked \
BAKED_BIN=/opt/rig/bin

View File

@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Apache Airflow — the cluster half of soleprint's airflow cabinet.
# Apache Airflow — the cluster half of the airflow cabinet.
#
# Airflow needs a metadata database before it will start at all, so this refuses
# rather than rolls a pod that will CrashLoopBackOff while the real problem
@@ -7,7 +7,7 @@
#
# One pod on `standalone`, matching the compose cabinet: migration, admin user,
# scheduler and webserver in a single container. The official chart's five
# deployments model an installation; a room switching this on wants pipelines.
# deployments model an installation; switching this on means wanting pipelines.
set -euo pipefail
cd "$(dirname "$0")/.."

View File

@@ -1,9 +1,10 @@
#!/usr/bin/env bash
# PostgreSQL — the cluster half of soleprint's postgres cabinet.
# PostgreSQL — the cluster half of the postgres cabinet.
#
# A room declares the dependency once, in cfg/<room>/data/cabinets.json. On a
# laptop `build.py` composes it into docker-compose.yml; here it becomes a pod,
# so the same declaration works either way and nothing has to be remembered
# A cabinet is a public service dropped into the environment as-is — the
# upstream image, unmodified, reachable at a known address. This is the cluster
# half of it; the compose half is a `service.yml` beside a `cabinet.json`. The
# declaration is made once and both paths read it, so nothing is remembered
# twice.
#
# Plain manifests rather than a helm chart, matching the other addons: a chart
@@ -32,8 +33,8 @@ if $K get secret -n "$NS" postgres >/dev/null 2>&1; then
else
password=$(head -c 18 /dev/urandom | base64 | tr -d '/+=' | head -c 24)
$K create secret generic postgres -n "$NS" \
--from-literal=POSTGRES_DB="${POSTGRES_DB:-soleprint}" \
--from-literal=POSTGRES_USER="${POSTGRES_USER:-soleprint}" \
--from-literal=POSTGRES_DB="${POSTGRES_DB:-postgres}" \
--from-literal=POSTGRES_USER="${POSTGRES_USER:-postgres}" \
--from-literal=POSTGRES_PASSWORD="$password" >/dev/null
echo " generated a password (read it back with the command printed below)"
fi

View File

@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Redis — the cluster half of soleprint's redis cabinet.
# Redis — the cluster half of the redis cabinet.
#
# Cache, and the broker anything queue-shaped runs on. No persistence: a broker
# that loses its queue on restart is the honest local model, and a PVC here buys

View File

@@ -1,20 +1,20 @@
#!/usr/bin/env bash
# Station check: is this workstation ready to run rig?
# Readiness check: is this machine ready to run rig?
#
# Reports and instructs; never silently fixes anything. Everything it finds is
# either already fine, or something a human has to decide on.
#
# Runs the wizard's host detection in a container when Docker is the only thing
# Runs ctrl/deps.sh host detection in a container when Docker is the only thing
# installed, or directly when the toolchain is already present. Then adds the
# checks that need this repo's config: profile sanity, CA trust, port clashes.
set -euo pipefail
cd "$(dirname "$0")"
WIZARD_IMAGE="${WIZARD_IMAGE:-$(basename "$(cd .. && pwd)")-wizard}"
DEPS_IMAGE="${DEPS_IMAGE:-$(basename "$(cd .. && pwd)")-deps}"
# Host detection. Prefer running it bare — it needs no dependencies beyond
# coreutils — and fall back to the container only if this shell can't.
bash ./wizard.sh detect
bash ./deps.sh detect
# ── repo-level checks ──────────────────────────────────────────────────────

View File

@@ -1,17 +1,22 @@
#!/usr/bin/env bash
# The installation wizard: detect the host, install a pinned toolchain onto it,
# then report what it could not do. It never runs the cluster and never mutates
# the host outside the directories mounted into it.
# Toolchain installer: detect the host, install a pinned toolchain onto it, then
# report what it could not do.
#
# Usage (normally via `make station` / `make deps`, or directly):
# wizard.sh detect # report host facts only, change nothing
# wizard.sh fetch [core|dev] [--to DIR] # download + verify into DIR
# wizard.sh install [core|dev] # detect, fetch, install, report
# It never runs the cluster, never uses sudo or apt, and writes only into
# $OUT_BIN (default ~/.local/bin). Everything that would touch the host proper —
# systemd, inotify limits, .wslconfig, docker group — is REPORTED for a human to
# decide on, never performed. That is what makes it safe to run on a machine that
# already has a working setup.
#
# Usage (normally via `make deps`, or directly):
# deps.sh detect # report host facts only, change nothing
# deps.sh fetch [core|dev] [--to DIR] # download + verify into DIR
# deps.sh install [core|dev] # detect, fetch, install, report
#
# Tiers: 'core' is kubectl + jq (talk to a cluster); 'dev' adds kind and tilt
# Default is dev.
#
# Runs both inside the wizard container and bare on a host. Inside the
# Runs both inside the installer container and bare on a host. Inside the
# container, host files are read through $HOST_ROOT (mount / as :ro); bare, it
# falls back to /.
@@ -55,6 +60,29 @@ host_file() {
# ── detect ─────────────────────────────────────────────────────────────────
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
# the tooling. Detectable, so name it instead.
require_linux() {
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
cat >&2 <<'EOF'
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
wsl --install
That enables Windows features and needs a reboot, so it is not something this
script will do for you. Afterwards, open the Linux shell it installs and run
this from there.
See "Starting from plain Windows" in README.md.
EOF
exit 1 ;;
esac
}
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
detect() {
@@ -76,6 +104,7 @@ detect() {
fi
detect_wsl
detect_filesystem
detect_docker
detect_inotify
}
@@ -115,18 +144,46 @@ detect_wsl() {
if [ -n "$wcfg" ] && grep -qE '^\s*memory\s*=' "$wcfg"; then
echo " wslconfig memory set: $(grep -E '^\s*memory\s*=' "$wcfg" | tr -d ' ')"
else
MANUAL+=("Cap/raise the WSL VM memory — in %USERPROFILE%\\.wslconfig on Windows:
[wsl2]
memory=8GB
then from a WINDOWS terminal: wsl --shutdown")
MANUAL+=("Cap/raise the WSL VM memory — see what is set versus what booted:
make mem status
It prints the edit to make and the command to apply it.")
fi
}
# Not a path check: /mnt is an ordinary mount point and an ext4 disk mounted
# there is perfectly fine. What matters is the filesystem. The Windows drives
# arrive as 9p (WSL2) or drvfs (WSL1); network and fuse mounts behave the same
# way. None of them deliver inotify events, so anything watching files goes
# quiet without saying why.
watch_hostile_fs() {
local dir="$1" fstype
fstype=$(findmnt -no FSTYPE --target "$dir" 2>/dev/null || true)
[ -n "$fstype" ] || fstype=$(stat -f -c %T "$dir" 2>/dev/null || true)
case "$fstype" in
9p|v9fs|drvfs|cifs|smb3|nfs|nfs4|fuse.sshfs|fuseblk) echo "$fstype" ;;
*) echo "" ;;
esac
}
detect_filesystem() {
local root fstype
root=$(cd .. && pwd -P)
fstype=$(watch_hostile_fs "$root")
if [ -n "$fstype" ]; then
echo " ! this directory is on $fstype — file watching will not work"
MANUAL+=("Move this onto the local disk. Nothing watching files sees changes
on a $fstype mount, and everything else is slower:
cp -r \"$root\" ~/ && cd ~/$(basename "$root")")
else
echo " filesystem $root ($(findmnt -no FSTYPE --target "$root" 2>/dev/null || echo local))"
fi
}
detect_docker() {
# Reachability of the daemon is the real question, and the CLI is only how
# we ask it. Note that when this runs inside the wizard container, Docker
# we ask it. Note that when this runs inside the installer container, Docker
# necessarily exists on the host — otherwise nothing would be executing —
# so a missing CLI in here is a wizard packaging bug, not a host problem.
# so a missing CLI in here is an installer packaging bug, not a host problem.
if ! command -v docker >/dev/null 2>&1; then
if [ -S /var/run/docker.sock ]; then
echo " docker socket present (no cli in this context)"
@@ -230,7 +287,7 @@ fetch_tgz() {
chmod +x "$dest/$name"
}
# The wizard runs as root so it can reach the docker socket, which means
# The installer runs as root so it can reach the docker socket, which means
# everything it writes into a mounted volume lands root-owned and unusable from
# the host. Hand it back to whoever owns the mount point (the host user created
# that directory before mounting it).
@@ -257,7 +314,13 @@ fix_ownership() {
CORE_TOOLS="kubectl jq"
# No helm: every addon installs with `kubectl apply -f <url>`, so nothing here
# has ever invoked it. Add it back the day something actually needs a chart.
DEV_TOOLS="kind tilt"
#
# ctlptl is 'dev' rather than 'core' for the same reason kind is: core is "talk
# to a cluster someone else runs", and ctlptl builds them. It earns its place
# because it is what wires a cluster to a local registry — without one, an
# unqualified image name resolves to docker.io/library/<name> and there is
# nothing structural stopping a push there.
DEV_TOOLS="kind tilt ctlptl"
fetch() {
local dest="$OUT_BIN" tier="${TIER:-dev}"
@@ -284,7 +347,8 @@ fetch() {
fetch_bin jq "$JQ_URL" "$JQ_SHA256" "$dest"
if [ "$tier" = "dev" ]; then
fetch_bin kind "$KIND_URL" "$KIND_SHA256" "$dest"
fetch_tgz tilt "$TILT_URL" "$TILT_SHA256" "$dest" tilt 0
fetch_tgz tilt "$TILT_URL" "$TILT_SHA256" "$dest" tilt 0
fetch_tgz ctlptl "$CTLPTL_URL" "$CTLPTL_SHA256" "$dest" ctlptl 0
fi
fix_ownership "$dest"
@@ -301,7 +365,7 @@ report_manual() {
echo "nothing left to do by hand."
return
fi
echo "host actions the wizard cannot perform (${#MANUAL[@]}):"
echo "host actions this cannot perform (${#MANUAL[@]}):"
echo
local n=1
for m in "${MANUAL[@]}"; do
@@ -373,6 +437,8 @@ install() {
# ── main ───────────────────────────────────────────────────────────────────
require_linux
case "${1:-install}" in
detect) detect; report_manual ;;
fetch) shift; fetch "$@" ;;

View File

@@ -19,7 +19,7 @@ DNS_MODE=hosts
# Opt in to the real ports below only when this is the ONLY environment and
# nothing else owns :80. They fail to bind otherwise, and docker reports it as an
# opaque "failed to bind host port 0.0.0.0:80/tcp: address already in use"
# halfway through cluster creation. `make station` checks before you spend the
# halfway through cluster creation. `make check` checks before you spend the
# time. Uncommenting also means only one environment can exist at a time.
# HTTP_PORT=80
# HTTPS_PORT=443

View File

@@ -1,11 +1,10 @@
# data — a cluster with the dependency containers a soleprint room asks for.
# data — a cluster with the cabinets an environment asks for.
#
# The point of this profile is that a room declares what it needs once, in
# cfg/<room>/data/cabinets.json, and gets it on either target: `build.py`
# composes those services into docker-compose.yml for a laptop, and the addons
# below install the same ones here. The names match deliberately —
# soleprint/station/cabinets/<name>/cabinet.json carries a `rig_addon` field
# pointing at ctrl/addons/<name>.sh.
# A cabinet is a public service dropped in as-is — the upstream image,
# unmodified, reachable at a known address. It is declared once and installs on
# either target: a `service.yml` composes it for a laptop, and the addons below
# install the same one here. The names match deliberately — each cabinet.json
# carries a `rig_addon` field pointing at ctrl/addons/<name>.sh.
#
# Everything lands in the `data` namespace (DATA_NAMESPACE to move it), so
# `make cluster reset` on the app namespace leaves the databases alone.
@@ -30,8 +29,8 @@ DATA_NAMESPACE=data
# Postgres identity. The password is not here: postgres.sh generates one on
# first install and keeps it across re-runs, so re-running the addon never
# rotates the credential out from under whatever is already connected.
POSTGRES_DB=soleprint
POSTGRES_USER=soleprint
POSTGRES_DB=app
POSTGRES_USER=app
POSTGRES_STORAGE=2Gi
AIRFLOW_ADMIN_USER=admin

View File

@@ -1,5 +1,5 @@
# offline — air-gapped. Everything comes from a local registry that was loaded
# ahead of time; nothing reaches the internet. Pair with the wizard-full image
# ahead of time; nothing reaches the internet. Pair with the deps-full image
# (DEPS_SOURCE=baked) so the toolchain install is offline too.
#
# The heavier addons are left out to keep first boot viable.

View File

@@ -1,8 +1,7 @@
# `ctrl/k8s` — cluster shape, and what runs on it
Same layout as every other project here (`unt`, `nvi`, `eth`, `mpr`, and
soleprint's generated rooms): a kind config, a kustomize `base/`, and an
`overlays/dev/` that patches it. See ALL `projects/templates/conventions.md`.
Same layout as every other project here: a kind config, a kustomize `base/`,
and an `overlays/dev/` that patches it.
```
kind-config*.yaml.tpl the cluster itself — nodes, ports, audit

View File

@@ -44,7 +44,7 @@ nodes:
- role: control-plane
image: ${NODE_IMAGE}
# hostPath is resolved by the HOST dockerd, so this must be a host path even
# when cluster.sh runs inside the wizard container. HOST_WORKDIR says where
# when cluster.sh runs inside the installer container. HOST_WORKDIR says where
# this rig lives on the host; bare on a host it is just the repo root.
extraMounts:
- hostPath: ${HOST_WORKDIR}/ctrl/k8s/audit-policy.yaml

View File

@@ -112,7 +112,7 @@ load_config() {
fi
# Read the shape back out of the YAML rather than trusting a profile to
# restate it. station.sh sizes the memory warning on NODES, and cluster.sh
# restate it. check.sh sizes the memory warning on NODES, and cluster.sh
# prints AUDIT before spending minutes building something that cannot be
# changed afterwards — both would mislead if the numbers drifted.
NODES=$(grep -c '^ - role:' "$KIND_CONFIG_PATH")
@@ -125,7 +125,7 @@ load_config() {
# depending on something the caller does not set.
#
# hostPath entries are resolved by the HOST dockerd, so HOST_WORKDIR must stay a
# host path even when this runs inside the wizard container.
# host path even when this runs inside the installer container.
render_kind_config() {
local host_workdir="${HOST_WORKDIR:-$(cd .. && pwd)}"
sed -e "s|\${CLUSTER}|${CLUSTER}|g" \

233
rig/ctrl/mem.sh Executable file
View File

@@ -0,0 +1,233 @@
#!/usr/bin/env bash
# What memory this machine has, what is left, and — where there is one — what
# cap is holding it there.
#
# Runs on native Linux and under WSL, because rig is developed on one and used
# on the other. The difference is not cosmetic: on WSL the memory you see is a
# VM allocation that can be raised, and the commonest failure is raising it
# without restarting, so the number on disk and the number in /proc disagree.
# On native Linux there is no such cap and pretending otherwise sends you to a
# file that does not exist.
#
# This reports and instructs. It never writes a .wslconfig — applying one costs
# a full VM restart that takes every shell, mount and container with it, and
# choosing that moment is yours.
#
# `backup` exists so `restore` has something to read: back up, hand-edit
# following the printed instruction, restore if it goes wrong. Both are
# WSL-only, because .wslconfig is the only thing here worth backing up.
#
# Usage: mem.sh status | backup | restore
set -euo pipefail
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
# the tooling. Detectable, so name it instead.
require_linux() {
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
cat >&2 <<'EOF'
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
wsl --install
That enables Windows features and needs a reboot, so it is not something this
script will do for you. Afterwards, open the Linux shell it installs and run
this from there.
See "Starting from plain Windows" in README.md.
EOF
exit 1 ;;
esac
}
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
require_wsl() {
if ! is_wsl; then
echo "$1 acts on .wslconfig, which only exists under WSL." >&2
echo "This is native Linux — there is no VM allocation to save or roll back." >&2
echo "Use 'mem.sh status' to see what the machine actually has." >&2
exit 1
fi
}
mb() { echo $(( $(awk "/^$1:/{print \$2}" /proc/meminfo) / 1024 )); }
# /mnt/c/Users can hold several real accounts — a renamed login leaves the old
# directory behind — so picking the first alphabetically is a coin toss. Ask
# Windows, then fall back to whichever profile actually owns a config.
wslconfig_path() {
local profile winpath found
profile=$(cmd.exe /c "echo %USERPROFILE%" 2>/dev/null | tr -d "\r\n" || true)
case "$profile" in
""|*%*) ;;
*) winpath=$(wslpath -u "$profile" 2>/dev/null || true)
if [ -n "$winpath" ] && [ -d "$winpath" ]; then
echo "$winpath/.wslconfig"; return
fi ;;
esac
found=$(ls -d /mnt/c/Users/*/.wslconfig 2>/dev/null | head -1 || true)
if [ -n "$found" ]; then echo "$found"; return; fi
echo "cannot tell which Windows profile owns .wslconfig. Candidates:" >&2
ls -d /mnt/c/Users/*/ 2>/dev/null \
| grep -viE "/(All Users|Default|Default User|Public)/$" | sed "s/^/ /" >&2
exit 1
}
configured_memory() {
[ -r "$1" ] || { echo ""; return; }
sed -n 's/^[[:space:]]*memory[[:space:]]*=[[:space:]]*//p' "$1" | tail -1 | tr -d '[:space:]'
}
# "9GB" / "8192MB" / "9G" -> MB, so it can be compared with /proc/meminfo.
to_mb() {
local v="${1^^}" n
n=$(echo "$v" | tr -dc '0-9')
[ -n "$n" ] || { echo ""; return; }
case "$v" in
*GB|*G) echo $(( n * 1024 )) ;;
*MB|*M) echo "$n" ;;
*) echo $(( n / 1024 / 1024 )) ;;
esac
}
hogs() {
echo "holding the most:"
ps -eo rss,comm --sort=-rss 2>/dev/null | awk 'NR>1 && NR<=6 {printf " %6.0f MB %s\n", $1/1024, $2}'
}
status() {
local total avail swap_total swap_free
total=$(mb MemTotal); avail=$(mb MemAvailable)
swap_total=$(mb SwapTotal); swap_free=$(mb SwapFree)
if is_wsl; then
local cfg conf conf_mb
cfg=$(wslconfig_path)
conf=$(configured_memory "$cfg")
echo "platform WSL"
echo "config $cfg"
if [ -n "$conf" ]; then
conf_mb=$(to_mb "$conf")
echo "configured $conf (${conf_mb} MB)"
else
conf_mb=""
echo "configured (no memory= set — WSL defaults to 50% of host RAM, or 8GB, whichever is less)"
fi
echo "booted ${total} MB"
echo "available ${avail} MB"
echo "swap ${swap_total} MB ($(( swap_total - swap_free )) MB used)"
if [ -n "$conf_mb" ]; then
# The VM reports a little less than allocated; 15% covers the kernel
# without calling every healthy machine a mismatch.
if [ "$total" -lt $(( conf_mb * 85 / 100 )) ]; then
echo
echo "! configured ${conf_mb} MB but booted ${total} MB."
echo " The change has not been applied. From a WINDOWS terminal:"
echo
echo " wsl --shutdown"
echo
echo " then start the distro again."
fi
else
echo
echo "To raise it, add to $cfg on the Windows side:"
echo
echo " [wsl2]"
echo " memory=8GB"
echo
echo "then, from a WINDOWS terminal: wsl --shutdown"
fi
local n
n=$(ls "$cfg".*.bak 2>/dev/null | wc -l)
[ "$n" -gt 0 ] && echo "backups $n (newest: $(ls -t "$cfg".*.bak 2>/dev/null | head -1))"
else
echo "platform native linux"
echo "total ${total} MB"
echo "available ${avail} MB"
echo "swap ${swap_total} MB ($(( swap_total - swap_free )) MB used)"
echo
echo "No VM allocation to raise here — this is the machine's own memory."
echo "If it is tight the levers are freeing something or adding swap."
fi
# Under a fifth left is worth naming wherever you are running.
if [ "$avail" -lt $(( total / 5 )) ]; then
echo
hogs
fi
return 0
}
backup() {
require_wsl backup
local cfg dest
cfg=$(wslconfig_path)
[ -r "$cfg" ] || { echo "nothing to back up: $cfg does not exist" >&2; exit 1; }
# Timestamped and never overwritten: a backup that can destroy itself on a
# second run is not a backup.
dest="${cfg}.$(date +%Y%m%d-%H%M%S).bak"
cp "$cfg" "$dest"
echo "backed up $dest"
echo
echo "Edit $cfg by hand, then from a WINDOWS terminal: wsl --shutdown"
}
restore() {
require_wsl restore
local cfg newest count
cfg=$(wslconfig_path)
newest=$(ls -t "$cfg".*.bak 2>/dev/null | head -1 || true)
[ -n "$newest" ] || { echo "no backups found beside $cfg" >&2; exit 1; }
echo "restoring $newest"
echo " -> $cfg"
echo
# Newest is the right default — undo the last edit — but if you backed up
# *after* editing, the state you want is older. Show the rest so a no-op
# restore is obviously a no-op rather than a mystery.
count=$(ls "$cfg".*.bak 2>/dev/null | wc -l)
if [ "$count" -gt 1 ]; then
echo "$count backups exist, newest first:"
ls -t "$cfg".*.bak | sed 's/^/ /'
echo " (restoring the newest; copy another by hand to pick an older one)"
echo
fi
if [ -r "$cfg" ]; then
echo "what changes:"
if diff "$cfg" "$newest" > /tmp/mem.diff 2>&1 && [ ! -s /tmp/mem.diff ]; then
echo " nothing — that backup is identical to the current config"
else
sed 's/^/ /' /tmp/mem.diff
fi
rm -f /tmp/mem.diff
echo
fi
printf "proceed? [y/N] "
read -r reply
case "$reply" in
y|Y|yes|Yes) ;;
*) echo "left alone"; return 0 ;;
esac
cp "$newest" "$cfg"
echo "restored. From a WINDOWS terminal: wsl --shutdown"
}
require_linux
case "${1:-status}" in
status) status ;;
backup) backup ;;
restore) restore ;;
*) echo "usage: $0 [status|backup|restore]" >&2; exit 1 ;;
esac

View File

@@ -58,10 +58,13 @@ require_wsl() {
cat >&2 <<'EOF'
newbox is WSL-only for now.
If WSL is not installed, run `wsl --install` from an elevated Windows prompt
first — see "Starting from plain Windows" in README.md.
On native Linux you do not need it: rig already isolates environments by
directory (own cluster, context, images and port block), so a second copy in a
second directory is the clean slate. To validate the installer itself against a
bare system, run the wizard against a stock Debian container instead.
bare system, run ctrl/deps.sh against a stock Debian container instead.
EOF
exit 1
fi
@@ -250,7 +253,7 @@ create() {
echo
echo "next:"
echo " make newbox shell # a shell inside it"
echo " then: cd ~/rig && make station && make deps && make cluster up"
echo " then: cd ~/rig && make check && make deps && make cluster up"
echo
echo "For a browser on Windows to resolve the hostnames, paste this into"
echo "C:\\Windows\\System32\\drivers\\etc\\hosts (it has no wildcard support):"

View File

@@ -43,7 +43,7 @@ K="kubectl --context ${KUBECONTEXT}"
# 2. every kind node's containerd — nodes do NOT inherit host trust
# 3. anything doing HTTPS from inside the cluster, in its own trust store
#
# We handle (2) here because it's ours to handle. (1) is reported by station.sh
# We handle (2) here because it's ours to handle. (1) is reported by check.sh
# since it needs root. (3) belongs to the workload.
install_ca_into_nodes() {
[ -n "${REGISTRY_CA_FILE:-}" ] || return 0

View File

@@ -73,11 +73,11 @@ record() {
step_host() {
local out
if ! out=$(bash ./wizard.sh detect 2>&1); then
if ! out=$(bash ./deps.sh detect 2>&1); then
record host fail "detection failed"
return
fi
# Anything the wizard flagged with '!' needs a human; surface the count here
# Anything flagged with '!' needs a human; surface the count here
# and the detail below rather than burying it.
local warns; warns=$(echo "$out" | grep -c '^\s*!' || true)
HOST_DETAIL="$out"
@@ -102,7 +102,7 @@ step_toolchain() {
return
fi
if bash ./wizard.sh install "$TIER" >/tmp/rig-deps.$$ 2>&1; then
if bash ./deps.sh install "$TIER" >/tmp/rig-deps.$$ 2>&1; then
local still=""
for b in $want; do
[ -x "${OUT_BIN:-$HOME/.local/bin}/$b" ] || still="$still $b"

View File

@@ -1,4 +1,4 @@
# Pinned toolchain — the single manifest the wizard installs from.
# Pinned toolchain — the single manifest ctrl/deps.sh installs from.
# Every entry is a single binary; none of them needs an apt repo.
# kubectl fully static
# kind libc only
@@ -6,8 +6,19 @@
# jq upstream static build (Debian's is linked against libjq/libonig)
#
# Checksums are the upstream-published SHA256 of the linux/amd64 artifact.
# To bump: change the version, then re-run `bash ctrl/versions-refresh.sh` and
# commit the result — never hand-edit a checksum.
#
# To bump: change the version, then take the checksum from the release's own
# published list — never hand-edit or hand-copy one from a download you did.
# For anything hosted on GitHub releases that is:
#
# curl -sSL https://github.com/<org>/<repo>/releases/download/<tag>/checksums.txt \
# | grep linux.x86_64
#
# (kubectl publishes its own instead: <KUBECTL_URL>.sha256.)
#
# There was a `ctrl/versions-refresh.sh` named here that has never existed. If
# bumping stops being rare enough to do by hand, write it — but a comment
# pointing at a missing script is worse than no comment.
KIND_VERSION=v0.32.0
KIND_SHA256=50030de23cf40a18505f20426f6a8506bedf13c6e509244bd1fa9463721b0f54
@@ -21,6 +32,14 @@ TILT_VERSION=0.37.6
TILT_SHA256=e9672b8a18d43501f35dcfe98465969a7db0e436b36cf0c50c7e6f8d40de5fe6
TILT_URL=https://github.com/tilt-dev/tilt/releases/download/v${TILT_VERSION}/tilt.${TILT_VERSION}.linux.x86_64.tar.gz
# ctlptl — creates a kind cluster WITH a local registry wired in, which is what
# keeps images off docker.io (an unqualified name means docker.io/library/<name>).
# Same publisher and same archive shape as tilt: binary at the archive root, so
# fetch_tgz handles it with strip=0 and no special case.
CTLPTL_VERSION=0.9.4
CTLPTL_SHA256=c63a1ec28e60bc3faf6becb76f53355c5cf5e0143dafdd27ad85db5584fa6b1e
CTLPTL_URL=https://github.com/tilt-dev/ctlptl/releases/download/v${CTLPTL_VERSION}/ctlptl.${CTLPTL_VERSION}.linux.x86_64.tar.gz
JQ_VERSION=1.8.2
JQ_SHA256=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f
JQ_URL=https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-amd64
@@ -44,9 +63,9 @@ CERT_MANAGER_VERSION=v1.21.1
METRICS_SERVER_VERSION=v0.9.0
METALLB_VERSION=v0.16.0
# Dependency containers. These mirror soleprint's cabinets
# (soleprint/station/cabinets/), so a room that declares postgres gets the same
# thing whether it runs on compose or in the cluster. Pinned by tag rather than
# Cabinets — public services dropped in as-is, the upstream image unmodified.
# The same declaration installs on compose or in the cluster, so a dependency is
# named once and works either way. Pinned by tag rather than
# digest because they are ordinary upstream images with no supply chain claim
# attached — bump freely, and preload them for the offline profile.
POSTGRES_IMAGE=postgres:16-alpine

View File

@@ -20,13 +20,13 @@ digraph rig_install {
bin [label="~/.local/bin\nkind · kubectl · tilt\njq" fillcolor="#121829" shape=cylinder]
}
subgraph cluster_wizard {
subgraph cluster_installer {
label="Installer container (transient)"
style=dashed
color="#1e2a4a"
fontcolor="#8892a8"
wizard [label="wizard\ncurl · jq · python · graphviz" fillcolor="#121829"]
installer [label="deps installer\ncurl · jq · python · graphviz" fillcolor="#121829"]
detect [label="detect host\nWSL · memory · inotify · docker" fillcolor="#121829"]
fetch [label="fetch + verify\nSHA256, pinned versions" fillcolor="#121829"]
}
@@ -34,14 +34,14 @@ digraph rig_install {
upstream [label="upstream\nreleases / corporate mirror" fillcolor="#1a3a1a" fontcolor="#00c853" shape=octagon]
report [label="report what it\nCANNOT do" fillcolor="#3a1a1a" fontcolor="#ffc107"]
docker -> wizard [label="docker run"]
wizard -> detect
docker -> installer [label="docker run"]
installer -> detect
detect -> fetch
fetch -> upstream [label="pinned + checksummed" color="#00c853"]
fetch -> bin [label="install"]
detect -> report [style=dashed label="sudo / Windows-side steps" color="#ffc107"]
// The container is gone after this; nothing depends on it at run time.
wizard -> gone [style=dotted label="exits"]
installer -> gone [style=dotted label="exits"]
gone [label="(container discarded)" fillcolor="#0a0e17" fontcolor="#4a5568" color="#1e2a4a" style="filled,dashed"]
}

View File

@@ -16,7 +16,7 @@
<text xml:space="preserve" text-anchor="middle" x="1011.15" y="-170.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Your machine</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_wizard</title>
<title>cluster_installer</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="8,-95 8,-175 745.5,-175 745.5,-95 8,-95"/>
<text xml:space="preserve" text-anchor="middle" x="376.75" y="-155.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Installer container (transient)</text>
</g>
@@ -27,16 +27,16 @@
<text xml:space="preserve" text-anchor="middle" x="1011.15" y="-46.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">Docker</text>
<text xml:space="preserve" text-anchor="middle" x="1011.15" y="-32.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">(the one prerequisite)</text>
</g>
<!-- wizard -->
<!-- installer -->
<g id="node3" class="node">
<title>wizard</title>
<title>installer</title>
<polygon fill="#121829" stroke="#1e2a4a" points="181.25,-139 16,-139 16,-103 181.25,-103 181.25,-139"/>
<text xml:space="preserve" text-anchor="middle" x="98.62" y="-124.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">wizard</text>
<text xml:space="preserve" text-anchor="middle" x="98.62" y="-124.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">deps installer</text>
<text xml:space="preserve" text-anchor="middle" x="98.62" y="-110.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">curl · jq · python · graphviz</text>
</g>
<!-- docker&#45;&gt;wizard -->
<!-- docker&#45;&gt;installer -->
<g id="edge1" class="edge">
<title>docker&#45;&gt;wizard</title>
<title>docker&#45;&gt;installer</title>
<path fill="none" stroke="#4a5568" d="M906.12,-45.7C757.47,-50.51 475.98,-63.12 238.25,-94 223.38,-95.93 207.7,-98.48 192.45,-101.24"/>
<polygon fill="#4a5568" stroke="#4a5568" points="192.13,-97.74 182.93,-103.01 193.4,-104.63 192.13,-97.74"/>
<text xml:space="preserve" text-anchor="middle" x="506.62" y="-74.39" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">docker run</text>
@@ -57,9 +57,9 @@
<text xml:space="preserve" text-anchor="middle" x="333.62" y="-124.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">detect host</text>
<text xml:space="preserve" text-anchor="middle" x="333.62" y="-110.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">WSL · memory · inotify · docker</text>
</g>
<!-- wizard&#45;&gt;detect -->
<!-- installer&#45;&gt;detect -->
<g id="edge2" class="edge">
<title>wizard&#45;&gt;detect</title>
<title>installer&#45;&gt;detect</title>
<path fill="none" stroke="#4a5568" d="M181.44,-121C195.97,-121 211.28,-121 226.37,-121"/>
<polygon fill="#4a5568" stroke="#4a5568" points="226.33,-124.5 236.33,-121 226.33,-117.5 226.33,-124.5"/>
</g>
@@ -69,9 +69,9 @@
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="400.5,-47 266.75,-47 266.75,-11 400.5,-11 400.5,-47"/>
<text xml:space="preserve" text-anchor="middle" x="333.62" y="-25.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#4a5568">(container discarded)</text>
</g>
<!-- wizard&#45;&gt;gone -->
<!-- installer&#45;&gt;gone -->
<g id="edge7" class="edge">
<title>wizard&#45;&gt;gone</title>
<title>installer&#45;&gt;gone</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="1,5" d="M119.29,-102.62C138.26,-86 168.54,-62.29 199.25,-49.75 216.76,-42.6 236.49,-37.9 255.29,-34.81"/>
<polygon fill="#4a5568" stroke="#4a5568" points="255.67,-38.29 265.04,-33.35 254.64,-31.37 255.67,-38.29"/>
<text xml:space="preserve" text-anchor="middle" x="209.75" y="-52.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">exits</text>

Before

Width:  |  Height:  |  Size: 9.0 KiB

After

Width:  |  Height:  |  Size: 9.0 KiB

View File

@@ -20,7 +20,7 @@ digraph rig_environment {
cname [label="cluster name\nacmebank" fillcolor="#121829"]
ctx [label="kubectl context\nkind-acmebank" fillcolor="#121829"]
img [label="image tag\nacmebank-wizard" fillcolor="#121829"]
img [label="image tag\nacmebank-deps" fillcolor="#121829"]
ports [label="port block\n2130021309" fillcolor="#121829"]
reg [label="registry container\nacmebank-registry" fillcolor="#121829"]
}

View File

@@ -4,28 +4,28 @@
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: rig_environment Pages: 1 -->
<svg width="971pt" height="481pt"
viewBox="0.00 0.00 971.00 481.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<svg width="962pt" height="481pt"
viewBox="0.00 0.00 962.00 481.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 476.83)">
<title>rig_environment</title>
<polygon fill="#0a0e17" stroke="none" points="-4,4 -4,-476.83 967,-476.83 967,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="481.5" y="-453.63" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">One environment per directory — copies never collide</text>
<polygon fill="#0a0e17" stroke="none" points="-4,4 -4,-476.83 958,-476.83 958,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="477" y="-453.63" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">One environment per directory — copies never collide</text>
<g id="clust1" class="cluster">
<title>cluster_derived</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="8,-65 8,-144.5 605,-144.5 605,-65 8,-65"/>
<text xml:space="preserve" text-anchor="middle" x="306.5" y="-125.3" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Everything below is derived from it</text>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="8,-65 8,-144.5 596,-144.5 596,-65 8,-65"/>
<text xml:space="preserve" text-anchor="middle" x="302" y="-125.3" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Everything below is derived from it</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_config</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="613,-65 613,-437.33 955,-437.33 955,-65 613,-65"/>
<text xml:space="preserve" text-anchor="middle" x="784" y="-418.13" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Configuration — weakest first, later wins</text>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="604,-65 604,-437.33 946,-437.33 946,-65 604,-65"/>
<text xml:space="preserve" text-anchor="middle" x="775" y="-418.13" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Configuration — weakest first, later wins</text>
</g>
<!-- dirname -->
<g id="node1" class="node">
<title>dirname</title>
<polygon fill="#1f6feb" stroke="#1e2a4a" points="375.11,-197.44 375.11,-219.63 329.94,-235.33 266.06,-235.33 220.89,-219.63 220.89,-197.44 266.06,-181.75 329.94,-181.75 375.11,-197.44"/>
<text xml:space="preserve" text-anchor="middle" x="298" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">directory name</text>
<text xml:space="preserve" text-anchor="middle" x="298" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">e.g. acmebank/</text>
<polygon fill="#1f6feb" stroke="#1e2a4a" points="370.11,-197.44 370.11,-219.63 324.94,-235.33 261.06,-235.33 215.89,-219.63 215.89,-197.44 261.06,-181.75 324.94,-181.75 370.11,-197.44"/>
<text xml:space="preserve" text-anchor="middle" x="293" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">directory name</text>
<text xml:space="preserve" text-anchor="middle" x="293" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">e.g. acmebank/</text>
</g>
<!-- cname -->
<g id="node2" class="node">
@@ -37,8 +37,8 @@
<!-- dirname&#45;&gt;cname -->
<g id="edge1" class="edge">
<title>dirname&#45;&gt;cname</title>
<path fill="none" stroke="#4a5568" d="M232.76,-193.05C195.81,-183.01 149.78,-167.28 113,-144.5 101.37,-137.29 90.31,-127.09 81.33,-117.6"/>
<polygon fill="#4a5568" stroke="#4a5568" points="84.04,-115.37 74.73,-110.3 78.85,-120.07 84.04,-115.37"/>
<path fill="none" stroke="#4a5568" d="M228.68,-192.51C192.88,-182.36 148.52,-166.7 113,-144.5 101.4,-137.25 90.34,-127.04 81.36,-117.55"/>
<polygon fill="#4a5568" stroke="#4a5568" points="84.07,-115.32 74.76,-110.26 78.88,-120.02 84.07,-115.32"/>
</g>
<!-- ctx -->
<g id="node3" class="node">
@@ -50,120 +50,120 @@
<!-- dirname&#45;&gt;ctx -->
<g id="edge2" class="edge">
<title>dirname&#45;&gt;ctx</title>
<path fill="none" stroke="#4a5568" d="M269.64,-181.32C248.71,-161.98 220.42,-135.83 199.86,-116.83"/>
<polygon fill="#4a5568" stroke="#4a5568" points="202.4,-114.41 192.68,-110.19 197.65,-119.55 202.4,-114.41"/>
<path fill="none" stroke="#4a5568" d="M265.77,-181.32C245.77,-162.07 218.77,-136.06 199.05,-117.08"/>
<polygon fill="#4a5568" stroke="#4a5568" points="201.55,-114.63 191.91,-110.21 196.69,-119.67 201.55,-114.63"/>
</g>
<!-- img -->
<g id="node4" class="node">
<title>img</title>
<polygon fill="#121829" stroke="#1e2a4a" points="354,-109 242,-109 242,-73 354,-73 354,-109"/>
<text xml:space="preserve" text-anchor="middle" x="298" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">image tag</text>
<text xml:space="preserve" text-anchor="middle" x="298" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;wizard</text>
<polygon fill="#121829" stroke="#1e2a4a" points="344.12,-109 241.88,-109 241.88,-73 344.12,-73 344.12,-109"/>
<text xml:space="preserve" text-anchor="middle" x="293" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">image tag</text>
<text xml:space="preserve" text-anchor="middle" x="293" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;deps</text>
</g>
<!-- dirname&#45;&gt;img -->
<g id="edge3" class="edge">
<title>dirname&#45;&gt;img</title>
<path fill="none" stroke="#4a5568" d="M298,-181.32C298,-163.19 298,-139.07 298,-120.47"/>
<polygon fill="#4a5568" stroke="#4a5568" points="301.5,-120.67 298,-110.67 294.5,-120.67 301.5,-120.67"/>
<path fill="none" stroke="#4a5568" d="M293,-181.32C293,-163.19 293,-139.07 293,-120.47"/>
<polygon fill="#4a5568" stroke="#4a5568" points="296.5,-120.67 293,-110.67 289.5,-120.67 296.5,-120.67"/>
</g>
<!-- ports -->
<g id="node5" class="node">
<title>ports</title>
<polygon fill="#121829" stroke="#1e2a4a" points="460.38,-109 371.62,-109 371.62,-73 460.38,-73 460.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="416" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">port block</text>
<text xml:space="preserve" text-anchor="middle" x="416" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">2130021309</text>
<polygon fill="#121829" stroke="#1e2a4a" points="451.38,-109 362.62,-109 362.62,-73 451.38,-73 451.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="407" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">port block</text>
<text xml:space="preserve" text-anchor="middle" x="407" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">2130021309</text>
</g>
<!-- dirname&#45;&gt;ports -->
<g id="edge4" class="edge">
<title>dirname&#45;&gt;ports</title>
<path fill="none" stroke="#4a5568" d="M324.94,-181.53C336.66,-170.19 350.54,-156.71 363,-144.5 372.11,-135.57 382.06,-125.74 390.85,-117.02"/>
<polygon fill="#4a5568" stroke="#4a5568" points="393.15,-119.67 397.79,-110.14 388.22,-114.7 393.15,-119.67"/>
<path fill="none" stroke="#4a5568" d="M318.87,-181.32C337.78,-162.15 363.29,-136.3 382,-117.34"/>
<polygon fill="#4a5568" stroke="#4a5568" points="384.47,-119.81 389,-110.24 379.49,-114.9 384.47,-119.81"/>
</g>
<!-- reg -->
<g id="node6" class="node">
<title>reg</title>
<polygon fill="#121829" stroke="#1e2a4a" points="597.38,-109 478.62,-109 478.62,-73 597.38,-73 597.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="538" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">registry container</text>
<text xml:space="preserve" text-anchor="middle" x="538" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;registry</text>
<polygon fill="#121829" stroke="#1e2a4a" points="588.38,-109 469.62,-109 469.62,-73 588.38,-73 588.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="529" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">registry container</text>
<text xml:space="preserve" text-anchor="middle" x="529" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;registry</text>
</g>
<!-- dirname&#45;&gt;reg -->
<g id="edge5" class="edge">
<title>dirname&#45;&gt;reg</title>
<path fill="none" stroke="#4a5568" d="M356.87,-190.63C390.74,-179.74 433.49,-163.98 469,-144.5 483.3,-136.65 497.86,-126.04 509.9,-116.41"/>
<polygon fill="#4a5568" stroke="#4a5568" points="511.88,-119.31 517.39,-110.26 507.44,-113.9 511.88,-119.31"/>
<path fill="none" stroke="#4a5568" d="M350.86,-190.3C383.87,-179.35 425.43,-163.64 460,-144.5 474.27,-136.6 488.83,-125.98 500.87,-116.36"/>
<polygon fill="#4a5568" stroke="#4a5568" points="502.85,-119.26 508.36,-110.21 498.41,-113.84 502.85,-119.26"/>
</g>
<!-- cluster -->
<g id="node11" class="node">
<title>cluster</title>
<polygon fill="#1a1a3a" stroke="#1e2a4a" points="469.81,-10.54 469.81,-25.46 438.29,-36 393.71,-36 362.19,-25.46 362.19,-10.54 393.71,0 438.29,0 469.81,-10.54"/>
<text xml:space="preserve" text-anchor="middle" x="416" y="-14.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">kind cluster</text>
<polygon fill="#1a1a3a" stroke="#1e2a4a" points="460.81,-10.54 460.81,-25.46 429.29,-36 384.71,-36 353.19,-25.46 353.19,-10.54 384.71,0 429.29,0 460.81,-10.54"/>
<text xml:space="preserve" text-anchor="middle" x="407" y="-14.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">kind cluster</text>
</g>
<!-- cname&#45;&gt;cluster -->
<g id="edge9" class="edge">
<title>cname&#45;&gt;cluster</title>
<path fill="none" stroke="#4a5568" d="M93.06,-72.62C99.54,-69.72 106.38,-67.02 113,-65 192.78,-40.7 288.45,-28.91 350.65,-23.42"/>
<polygon fill="#4a5568" stroke="#4a5568" points="350.69,-26.93 360.36,-22.6 350.1,-19.96 350.69,-26.93"/>
<path fill="none" stroke="#4a5568" d="M93.35,-72.51C99.75,-69.67 106.48,-67 113,-65 189.55,-41.49 281.19,-29.58 341.58,-23.85"/>
<polygon fill="#4a5568" stroke="#4a5568" points="341.71,-27.35 351.35,-22.96 341.07,-20.38 341.71,-27.35"/>
</g>
<!-- ports&#45;&gt;cluster -->
<g id="edge10" class="edge">
<title>ports&#45;&gt;cluster</title>
<path fill="none" stroke="#4a5568" d="M416,-72.81C416,-65.23 416,-56.1 416,-47.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="419.5,-47.54 416,-37.54 412.5,-47.54 419.5,-47.54"/>
<path fill="none" stroke="#4a5568" d="M407,-72.81C407,-65.23 407,-56.1 407,-47.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="410.5,-47.54 407,-37.54 403.5,-47.54 410.5,-47.54"/>
</g>
<!-- versions -->
<g id="node7" class="node">
<title>versions</title>
<polygon fill="#121829" stroke="#1e2a4a" points="759.38,-401.83 652.62,-401.83 652.62,-365.83 759.38,-365.83 759.38,-401.83"/>
<text xml:space="preserve" text-anchor="middle" x="706" y="-386.88" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">versions.env</text>
<text xml:space="preserve" text-anchor="middle" x="706" y="-373.38" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">pinned toolchain</text>
<polygon fill="#121829" stroke="#1e2a4a" points="750.38,-401.83 643.62,-401.83 643.62,-365.83 750.38,-365.83 750.38,-401.83"/>
<text xml:space="preserve" text-anchor="middle" x="697" y="-386.88" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">versions.env</text>
<text xml:space="preserve" text-anchor="middle" x="697" y="-373.38" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">pinned toolchain</text>
</g>
<!-- profile -->
<g id="node8" class="node">
<title>profile</title>
<polygon fill="#121829" stroke="#1e2a4a" points="790.5,-318.58 621.5,-318.58 621.5,-282.58 790.5,-282.58 790.5,-318.58"/>
<text xml:space="preserve" text-anchor="middle" x="706" y="-303.63" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">env.d/&lt;profile&gt;.env</text>
<text xml:space="preserve" text-anchor="middle" x="706" y="-290.13" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">nodes · CNI · audit · addons</text>
<polygon fill="#121829" stroke="#1e2a4a" points="781.5,-318.58 612.5,-318.58 612.5,-282.58 781.5,-282.58 781.5,-318.58"/>
<text xml:space="preserve" text-anchor="middle" x="697" y="-303.63" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">env.d/&lt;profile&gt;.env</text>
<text xml:space="preserve" text-anchor="middle" x="697" y="-290.13" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">nodes · CNI · audit · addons</text>
</g>
<!-- versions&#45;&gt;profile -->
<g id="edge6" class="edge">
<title>versions&#45;&gt;profile</title>
<path fill="none" stroke="#4a5568" d="M706,-365.59C706,-355.32 706,-342.03 706,-330.21"/>
<polygon fill="#4a5568" stroke="#4a5568" points="709.5,-330.58 706,-320.58 702.5,-330.58 709.5,-330.58"/>
<text xml:space="preserve" text-anchor="middle" x="737.5" y="-339.28" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
<path fill="none" stroke="#4a5568" d="M697,-365.59C697,-355.32 697,-342.03 697,-330.21"/>
<polygon fill="#4a5568" stroke="#4a5568" points="700.5,-330.58 697,-320.58 693.5,-330.58 700.5,-330.58"/>
<text xml:space="preserve" text-anchor="middle" x="728.5" y="-339.28" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
</g>
<!-- localenv -->
<g id="node9" class="node">
<title>localenv</title>
<polygon fill="#121829" stroke="#1e2a4a" points="761.88,-226.54 646.12,-226.54 646.12,-190.54 761.88,-190.54 761.88,-226.54"/>
<text xml:space="preserve" text-anchor="middle" x="704" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">ctrl/.env</text>
<text xml:space="preserve" text-anchor="middle" x="704" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">secrets, overrides</text>
<polygon fill="#121829" stroke="#1e2a4a" points="752.88,-226.54 637.12,-226.54 637.12,-190.54 752.88,-190.54 752.88,-226.54"/>
<text xml:space="preserve" text-anchor="middle" x="695" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">ctrl/.env</text>
<text xml:space="preserve" text-anchor="middle" x="695" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">secrets, overrides</text>
</g>
<!-- profile&#45;&gt;localenv -->
<g id="edge7" class="edge">
<title>profile&#45;&gt;localenv</title>
<path fill="none" stroke="#4a5568" d="M705.61,-282.22C705.34,-269.76 704.96,-252.69 704.64,-238.23"/>
<polygon fill="#4a5568" stroke="#4a5568" points="708.14,-238.28 704.42,-228.36 701.14,-238.43 708.14,-238.28"/>
<text xml:space="preserve" text-anchor="middle" x="736.68" y="-256.03" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
<path fill="none" stroke="#4a5568" d="M696.61,-282.22C696.34,-269.76 695.96,-252.69 695.64,-238.23"/>
<polygon fill="#4a5568" stroke="#4a5568" points="699.14,-238.28 695.42,-228.36 692.14,-238.43 699.14,-238.28"/>
<text xml:space="preserve" text-anchor="middle" x="727.68" y="-256.03" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
</g>
<!-- shell -->
<g id="node10" class="node">
<title>shell</title>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="772.38,-109 623.62,-109 623.62,-73 772.38,-73 772.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="698" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">the environment</text>
<text xml:space="preserve" text-anchor="middle" x="698" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">PROFILE=client make …</text>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="763.38,-109 614.62,-109 614.62,-73 763.38,-73 763.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="689" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">the environment</text>
<text xml:space="preserve" text-anchor="middle" x="689" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">PROFILE=client make …</text>
</g>
<!-- localenv&#45;&gt;shell -->
<g id="edge8" class="edge">
<title>localenv&#45;&gt;shell</title>
<path fill="none" stroke="#00c853" d="M703.11,-190.49C702.16,-172.16 700.63,-142.72 699.49,-120.79"/>
<polygon fill="#00c853" stroke="#00c853" points="703,-120.81 698.99,-111.01 696.01,-121.18 703,-120.81"/>
<text xml:space="preserve" text-anchor="middle" x="733.21" y="-155.2" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
<path fill="none" stroke="#00c853" d="M694.11,-190.49C693.16,-172.16 691.63,-142.72 690.49,-120.79"/>
<polygon fill="#00c853" stroke="#00c853" points="694,-120.81 689.99,-111.01 687.01,-121.18 694,-120.81"/>
<text xml:space="preserve" text-anchor="middle" x="724.21" y="-155.2" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
</g>
<!-- shell&#45;&gt;cluster -->
<g id="edge11" class="edge">
<title>shell&#45;&gt;cluster</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M637.29,-72.6C627.83,-69.99 618.17,-67.38 609,-65 562.72,-52.99 509.89,-40.48 471.22,-31.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="472.2,-28.18 461.67,-29.35 470.63,-35 472.2,-28.18"/>
<path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M628.29,-72.6C618.83,-69.99 609.17,-67.38 600,-65 553.72,-52.99 500.89,-40.48 462.22,-31.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="463.2,-28.18 452.67,-29.35 461.63,-35 463.2,-28.18"/>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 11 KiB

After

Width:  |  Height:  |  Size: 11 KiB

View File

@@ -265,11 +265,11 @@
<h3>The only prerequisite</h3>
<p><b>Docker.</b> No curl, no jq, no python, no apt repositories to configure.</p>
<pre><code><span class="c"># then, in the environment directory:</span>
make station <span class="c"># is this workstation ready? reports, never fixes</span>
make check <span class="c"># is this machine ready? reports, never fixes</span>
make deps <span class="c"># install the pinned toolchain</span>
make cluster up <span class="c"># build the cluster for the active profile</span>
</code></pre>
<p>Read <code>make station</code> before <code>make deps</code>. It never changes
<p>Read <code>make check</code> before <code>make deps</code>. It never changes
anything — it prints what it found and, at the end, the steps it cannot perform
for you.</p>
</div>
@@ -280,13 +280,13 @@ make cluster up <span class="c"># build the cluster for the active profile</spa
<p class="lede">Start to finish, in order, with what each one actually does.</p>
<div class="prose">
<h3>1 &middot; make station</h3>
<p>Asks whether this workstation is ready. It <b>changes nothing</b> — it
<h3>1 &middot; make check</h3>
<p>Asks whether this machine is ready. It <b>changes nothing</b> — it
reports what it found and, at the end, the things only a human can do
(anything needing <code>sudo</code>, or a Windows-side restart). Read it
before installing anything; it is faster than discovering the same problems
one failure at a time.</p>
<pre><code>make station</code></pre>
<pre><code>make check</code></pre>
<h3>2 &middot; make setup</h3>
<p>Does the preparation that can be automated: installs the pinned
@@ -381,8 +381,8 @@ make setup core <span class="c"># same distinction, via setup</span>
wants only Docker.</p>
<h3>Air-gapped</h3>
<pre><code>make wizard full <span class="c"># bakes every binary into the image</span>
docker save …-wizard:full | gzip &gt; rig.tgz
<pre><code>make deps-image full <span class="c"># bakes every binary into the image</span>
docker save …-deps:full | gzip &gt; rig.tgz
<span class="c"># carry that one file in, then:</span>
docker load &lt; rig.tgz &amp;&amp; make cluster up PROFILE=offline
</code></pre>
@@ -485,7 +485,7 @@ docker load &lt; rig.tgz &amp;&amp; make cluster up PROFILE=offline
registry is usually behind an internal CA, and trust has to reach
<b>three</b> places: the host Docker daemon, every cluster node's containerd
(nodes do <i>not</i> inherit host trust), and any in-cluster client. Set
<code>REGISTRY_CA_FILE</code> and <code>make station</code> reports which is
<code>REGISTRY_CA_FILE</code> and <code>make check</code> reports which is
still missing. The symptom otherwise is an opaque
<code>x509: certificate signed by unknown authority</code>.</p></div>
@@ -528,11 +528,11 @@ docker load &lt; rig.tgz &amp;&amp; make cluster up PROFILE=offline
<h3>Tilt stops noticing file changes</h3>
<p>Almost always <code>inotify</code> limits, and it fails <i>silently</i>
nothing errors, changes just stop being picked up. Defaults on WSL are far too
low. <code>make station</code> reports it and prints the fix.</p>
low. <code>make check</code> reports it and prints the fix.</p>
<h3>Cluster creation dies halfway with a port error</h3>
<p>Docker reports <code>failed to bind host port … address already in use</code>
partway through creating the cluster. Run <code>make station</code> first — it
partway through creating the cluster. Run <code>make check</code> first — it
checks every port in this environment's block before anything is built.</p>
<h3>Every node stays NotReady</h3>

View File

@@ -1,9 +0,0 @@
# NOTE: generated/ is deliberately NOT ignored. The artifact IS the deliverable —
# the whole point is a folder you copy, apply and boot without generating
# anything first. Regenerate it with `make manifest` after editing bundle.json
# or app/serve.py, and commit the result.
#
# (This differs from rig's ctrl/k8s/generated, which is a local build artifact.)
__pycache__/
*.pyc

View File

@@ -1,50 +0,0 @@
# Thin control Makefile — one target per ctrl/ script, subcommand as an
# argument. Same shape as rig's, for the same reason: the logic lives in the
# script, never here.
#
# make up -> ctrl/bundle.sh up
# make bundle down
#
# This directory is a BUNDLE, not an installer. It needs a cluster, which rig
# owns:
#
# cd .. && make cluster up # kind cluster for this environment
# make up # then deploy this bundle into it
#
# Start with: make up
ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
ifneq ($(ARGS),)
$(eval $(ARGS):;@:)
endif
.DEFAULT_GOAL := help
.PHONY: help bundle manifest up down status url list dev
help: ## list targets
@grep -hE '^[a-z]+:.*?##' $(MAKEFILE_LIST) | sed 's/:.*##/\t/' | expand -t16
bundle: ## the bundle [manifest|up|down|status|url|list|dev]
bash ctrl/bundle.sh $(or $(ARGS),status)
# Shorthands for the ones used constantly.
manifest: ## regenerate generated/<slug>.yaml — no cluster needed
bash ctrl/bundle.sh manifest
up: ## deploy this rig (installs MetalLB if absent)
bash ctrl/bundle.sh up
down: ## remove this rig (leaves cluster, MetalLB, siblings)
bash ctrl/bundle.sh down
status: ## what is deployed for this rig
bash ctrl/bundle.sh status
url: ## the address MetalLB assigned
bash ctrl/bundle.sh url
list: ## every rig in this cluster, with addresses
bash ctrl/bundle.sh list
dev: ## run the UI locally with vite — no cluster needed
bash ctrl/bundle.sh dev

View File

@@ -1,140 +0,0 @@
# sample-rig
A minimal, non-sensitive bundle that proves an installation works and shows what
shipped. Copy it, rename it, and you have another rig.
```bash
make manifest # generate the artifact — no cluster, no kubectl needed
make up # deploy it into the local cluster
make list # every rig in this cluster, with addresses
make dev # run the UI locally with vite, no cluster at all
```
`make up` prints an address. Open it and the page says **IT WORKS**, then lists
the tools and rigs in the bundle.
## What it is for
Three jobs, in the order you hit them:
1. **Prove the install.** kind is there, a cluster exists, MetalLB hands out an
address, a `type: LoadBalancer` Service actually resolves, and a pod serves.
If all of that works, the environment is sound.
2. **Say what shipped.** The page renders [`bundle.json`](bundle.json) —
standalone tools and rigs, flat, with none of soleprint's internal hierarchy.
Editing that file is the only step needed to change the listing.
3. **Stand in for the real thing.** Nothing here is sensitive. The real
architecture connects separately, against a setup already known to work.
## The UI is a complement, not the product
`rig-ui/` is just a vite app. It complements a rig; a rig is complete and useful
without it, and nothing depends on it being there. It is deliberately **not**
generated by kind or tilt — you copy the folder into a rig after that rig is
pulled, and apply one manifest:
```bash
kubectl apply -n <namespace> -f rig-ui/k8s.yaml
```
That file is the whole integration: one Pod running `npm run dev` on
`node:22-alpine`, one Service. A bare Pod rather than a Deployment because this
is a dev-loop convenience, not a workload to keep alive.
The app and `bundle.json` arrive as a ConfigMap, so nothing is baked into an
image and editing the bundle is the entire update cycle. The container runs
`npm install` at start, which needs egress to a registry — on a locked-down
cluster point npm at the internal one, or bake an image instead. Nothing else
changes if you do.
## One artifact, two destinations
`ctrl/manifest.py` emits `generated/<slug>.yaml` — namespace, the app and
bundle embedded in a ConfigMap, Pod, Service. It is self-contained and applies
unmodified anywhere:
```bash
kubectl apply -f generated/sample-rig.yaml # local kind, or an external cluster
```
`make up` applies **that same file**. There is no separate local path, so what
works here cannot quietly differ from what is applied elsewhere.
This is what `type: LoadBalancer` buys. MetalLB answers it on kind; the AWS load
balancer controller answers it on EKS. NodePort would not survive the trip — it
is a single cluster-wide port range, so two rigs would have to negotiate numbers.
**VPC-agnostic on purpose.** The target is EKS, but the Service carries no
annotations — no `aws-load-balancer-subnets`, no security groups, no `-scheme`,
no `-type: nlb`. Each of those encodes a specific network layout, and one of them
appearing here would pin the artifact to the account and VPC it was written
against, which is precisely what stops it also working on kind. Subnet discovery
is the cluster's business: EKS resolves it from the tags its own subnets carry.
That leaves one thing genuinely environment-specific — internal versus
internet-facing. A bare `LoadBalancer` provisions internet-facing, which a
regulated account will usually refuse, and should. That belongs in a
per-environment overlay applied on top, never inlined into this artifact.
**MetalLB only — no ingress-nginx.** Its controller supports a narrow window of
Kubernetes versions, so depending on it constrains which k8s a rig can be built
with. That undercuts running trailing-edge control planes to model a legacy
estate, which is the reason `versions.env` pins v1_33..v1_36. MetalLB carries no
such constraint, so reachability costs nothing in version coverage.
## Several rigs, one cluster
Identity follows the **folder name**, the same rule rig uses for cluster
identity. The namespace is the folder; resource names are generic, and names only
have to be unique within a namespace.
```bash
cp -r sample-rig corporate-rig
cd corporate-rig && make up # its own namespace, its own address
```
No edits, no collisions, both in the same local cluster. `make list` shows them
together. `make down` removes only this one — siblings, MetalLB and the cluster
are left alone.
Client rigs are gitignored (`*-rig/`, with `sample-rig/` the deliberate
exception): a rig's k8s files spell out a real architecture, and that is exactly
what must not land in this repo.
## Staging workstations
`ctrl/manifest.py` is stdlib-only on purpose: it runs on a bare machine before
anything is installed. The toolchain itself is rig's job — `make deps` installs
the pinned kind and tilt binaries, which is what makes a staging AWS workspace
reachable from the same commands as a laptop.
## Layout
```
sample-rig/
├── Makefile # thin — one target per ctrl/ script
├── bundle.json # what shipped; the UI renders THIS
├── rig-ui/ # the vite app — optional, copied into a rig to enable it
│ ├── k8s.yaml # how to plug it in: one Pod, one Service
│ ├── index.html
│ ├── package.json
│ ├── vite.config.js
│ └── src/{main.js,style.css}
├── ctrl/
│ ├── manifest.py # emits the artifact
│ └── bundle.sh # generate / deploy / inspect
└── generated/ # the artifact — committed, this is the deliverable
```
Editing `bundle.json` or anything in `rig-ui/` means re-running `make manifest`.
The ConfigMap carries a checksum of everything embedded, so a stale deployment is
visible rather than silent.
## Not built, but not foreclosed
Everything derives from `bundle.json` plus a target namespace. A Pulumi or
Terraform emitter would sit beside `ctrl/manifest.py` consuming the same inputs;
nothing above it assumes the artifact is YAML.
Licence terms for the compiled UI component belong in the soleprint-generated
bundle, not here — this sample carries no proprietary component.

View File

@@ -1,51 +0,0 @@
{
"_comment": "What this bundle contains. Single source of truth — the landing page renders THIS file, so adding an entry here is the only edit needed. Deliberately FLAT: standalone tools and rigs, with none of soleprint's internal hierarchy (no artery/atlas/station layering). Nothing here is sensitive; the real architecture connects separately.",
"bundle": {
"name": "sample-rig",
"description": "Non-sensitive sample bundle. Proves the kind install works and shows what ships.",
"sensitive": false
},
"tools": [
{
"name": "modelgen",
"summary": "Generate models from config",
"standalone": true
},
{
"name": "datagen",
"summary": "Generate test data from rig-owned generators",
"standalone": true
},
{
"name": "graphgen",
"summary": "Generate navigable model graphs",
"standalone": true
},
{
"name": "tester",
"summary": "HTTP contract test runner — one suite, any environment",
"standalone": true
},
{
"name": "databrowse",
"summary": "SQL data browser",
"standalone": true
},
{
"name": "sbwrapper",
"summary": "Sandbox wrapper",
"standalone": true
}
],
"rigs": [
{
"name": "sample-rig",
"summary": "This bundle — a minimal, copyable environment",
"active": true
}
],
"next": [
"Point MANIFESTS_DIR at the real manifests to connect the actual architecture.",
"Real k8s files are versioned separately and are not part of this bundle."
]
}

View File

@@ -1,40 +0,0 @@
{
"_comment": "MOCKED cluster state. Nothing here is read from a live cluster — it exists so the UI can be shown when there is no cluster at all (a locked-down machine, a laptop with no memory to spare, a demo where kind will not start). The page labels it as mocked; a demo that looks live but is not is worse than one that says so. When a real cluster is present the same shapes come from kubectl.",
"mocked": true,
"cluster": {
"name": "sample-rig",
"context": "kind-sample-rig",
"provider": "kind",
"profile": "minimal",
"k8s": "v1.36.1",
"nodes": 1
},
"workloads": [
{
"name": "rig-ui",
"summary": "Pod · node:22-alpine · vite on :5173",
"state": "Running"
},
{
"name": "metallb-system/controller",
"summary": "Deployment · assigns LoadBalancer addresses",
"state": "Running"
},
{
"name": "metallb-system/speaker",
"summary": "DaemonSet · answers ARP in layer 2 mode",
"state": "Running"
}
],
"services": [
{
"name": "rig-ui",
"summary": "LoadBalancer · 80 -> 5173 · no annotations, so it resolves on kind and on EKS alike",
"state": "172.18.255.200"
}
]
}

View File

@@ -1,223 +0,0 @@
#!/usr/bin/env bash
# The rig bundle: generate it, deploy it, tear it down, find it.
#
# Usage: bundle.sh manifest | up | down | status | url | list | dev
#
# What `up` proves, in order: kind installed and a cluster exists, MetalLB can
# hand out an address, a Service of type LoadBalancer actually resolves, and a
# pod serves the bundle listing. If all of that works the installation is sound,
# and the only thing missing is the real architecture.
#
# ONE ARTIFACT
# `up` applies generated/<slug>.yaml — the same self-contained file you would
# hand to an external cluster. There is no separate local path, so what works
# here cannot quietly differ from the master deployment applied elsewhere.
#
# ONE CLUSTER, SEVERAL RIGS
# Identity follows the FOLDER NAME, exactly as rig's cluster identity does. This
# directory deploys into a namespace named after itself, so copying it to
# corporate-rig/ yields a second rig in the SAME local cluster with no edits and
# no collisions — different namespace, its own MetalLB address. `list` shows all
# of them. The cluster itself is rig's business; this only ever owns a namespace.
#
# MetalLB is installed by calling rig's own addon script rather than
# reimplementing it — deriving the pool from the kind Docker network is the
# fiddly part and there should be exactly one copy of it.
set -euo pipefail
cd "$(dirname "$0")/.."
BUNDLE_ROOT="$(pwd)"
RIG_CTRL="$(cd .. && pwd)/ctrl"
# The containing folder's name, reduced to a DNS label (same rule as rig's
# default_cluster_name and ctrl/manifest.py, so all three agree on the slug).
slug() {
local n
n=$(basename "$BUNDLE_ROOT")
n=$(echo "$n" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9-' '-')
n=$(echo "$n" | sed 's/^-*//; s/-*$//')
echo "${n:-rig-bundle}"
}
NS="$(slug)"
ARTIFACT="generated/${NS}.yaml"
# Resolved lazily, not at load time: `manifest` and `dev` deliberately work
# with no cluster and no kubectl at all, and a top-level check would break that.
#
# Follows whatever context rig's cluster.sh selected, so this bundle works in a
# copied-and-renamed environment without being told which cluster it is in.
init_kube() {
KUBECONTEXT="${KUBECONTEXT:-$(kubectl config current-context 2>/dev/null || true)}"
if [ -z "$KUBECONTEXT" ]; then
echo "no kubectl context — bring a cluster up first: (cd .. && make cluster up)" >&2
exit 1
fi
KCTX="kubectl --context ${KUBECONTEXT}"
K="kubectl --context ${KUBECONTEXT} --namespace ${NS}"
}
require_cluster() {
if ! $KCTX cluster-info >/dev/null 2>&1; then
echo "context '$KUBECONTEXT' does not reach a cluster" >&2
echo "bring one up: (cd .. && make cluster up)" >&2
exit 1
fi
}
ensure_metallb() {
if $KCTX get deployment -n metallb-system controller >/dev/null 2>&1; then
echo "metallb: present"
return 0
fi
# Only kind needs it. On a real cluster the cloud load balancer answers a
# `type: LoadBalancer` Service, and installing MetalLB there would be wrong.
case "$KUBECONTEXT" in
kind-*) ;;
*)
echo "metallb: skipped — '$KUBECONTEXT' is not a kind context"
echo " (a cloud load balancer answers LoadBalancer services there)"
return 0
;;
esac
if [ ! -f "$RIG_CTRL/addons/metallb.sh" ]; then
echo "metallb is not installed and rig's addon script was not found at" >&2
echo " $RIG_CTRL/addons/metallb.sh" >&2
echo "a Service of type LoadBalancer will sit at <pending> without it." >&2
exit 1
fi
# rig's addons derive their target cluster from RIG'S OWN folder name via
# load_config, so left alone this bundle would install into `kind-rig` —
# a cluster that need not exist — while deploying everything else into the
# context actually selected. CLUSTER is in load_config's overridable set,
# so passing it here points the addon at the same cluster we are using.
local target="${KUBECONTEXT#kind-}"
echo "metallb: installing via rig's addon into '$target'"
CLUSTER="$target" bash "$RIG_CTRL/addons/metallb.sh"
}
# Regenerate the artifact. No cluster and no kubectl required — this is the step
# a staging workstation runs before anything is installed.
manifest() {
mkdir -p generated
python3 ctrl/manifest.py "$NS" > "$ARTIFACT"
echo "wrote $ARTIFACT ($(wc -l < "$ARTIFACT") lines)"
echo " applies as-is anywhere: kubectl apply -f ${BUNDLE_ROOT}/${ARTIFACT}"
}
up() {
manifest
init_kube
require_cluster
ensure_metallb
echo
echo "applying '${NS}' to context '${KUBECONTEXT}'"
$KCTX apply -f "$ARTIFACT"
# `rollout status` does not work on a bare Pod — it only understands
# Deployments, StatefulSets and DaemonSets. Wait on the condition instead.
# This is the slow step: the container npm-installs before vite serves.
echo "waiting for the pod to be ready (npm install runs first)..."
$K wait --for=condition=Ready pod/rig-ui --timeout=300s
echo
url
}
down() {
init_kube
# Delete the namespace and everything in it goes with it. Scoped to THIS
# rig — a sibling rig in the same cluster is untouched.
$KCTX delete namespace "$NS" --ignore-not-found
echo "'${NS}' removed (cluster, metallb and any sibling rig are left alone)"
}
status() {
init_kube
require_cluster
if ! $KCTX get namespace "$NS" >/dev/null 2>&1; then
echo "'${NS}' is not deployed — run: make up"
return 0
fi
$K get pod,svc,configmap -o wide
}
# Every rig in this cluster, not just this one — the point of the namespace
# split is that several coexist, so there has to be a way to see them together.
list() {
init_kube
require_cluster
local names
names=$($KCTX get namespace -l rig.bundle/name \
-o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true)
if [ -z "$names" ]; then
echo "no rigs deployed in context '${KUBECONTEXT}'"
return 0
fi
printf "%-20s %-16s %s\n" RIG ADDRESS ""
local n ip
for n in $names; do
ip=$($KCTX -n "$n" get svc rig-ui \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true)
printf "%-20s %-16s %s\n" "$n" "${ip:-<pending>}" \
"$([ "$n" = "$NS" ] && echo '<- this one')"
done
}
# The address MetalLB (or a cloud load balancer) assigned. <pending> here is the
# classic silent failure: everything reports healthy and nothing is reachable.
url() {
init_kube
local ip
ip=$($K get svc rig-ui \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true)
if [ -z "$ip" ]; then
ip=$($K get svc rig-ui \
-o jsonpath='{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null || true)
fi
if [ -z "$ip" ]; then
echo "no external address yet — nothing has assigned one."
echo "on kind: kubectl --context $KUBECONTEXT -n metallb-system get pods"
return 1
fi
echo "IT WORKS -> http://${ip}/"
echo " bundle http://${ip}/bundle.json"
}
# Run the UI locally with no cluster at all — the fast way to iterate on
# bundle.json. Same vite command the pod runs, so what you see here is what
# gets served there.
dev() {
if ! command -v npm >/dev/null 2>&1; then
echo "npm not found — the UI needs node locally for this." >&2
echo "(in-cluster it runs on the node:22-alpine image instead)" >&2
exit 1
fi
# bundle.json lives one level up so it stays the rig's data rather than the
# app's; vite serves public/ at the root, which is where the app fetches it.
mkdir -p rig-ui/public
cp bundle.json rig-ui/public/bundle.json
# The mocked cluster is a DEMO asset and is deliberately not embedded in the
# deployed artifact — on a real rig the UI would then show canned values
# beside a live cluster, which is precisely the lie its banner warns about.
# It is served here, and in the static build for the public UI-only page.
cp cluster.mock.json rig-ui/public/cluster.mock.json
cd rig-ui
[ -d node_modules ] || npm install --no-audit --no-fund
VITE_RIG_NAME="$NS" npm run dev
}
case "${1:-status}" in
manifest) manifest ;;
up) up ;;
down) down ;;
status) status ;;
url) url ;;
list) list ;;
dev) dev ;;
*) echo "usage: $0 [manifest|up|down|status|url|list|dev]" >&2; exit 1 ;;
esac

View File

@@ -1,141 +0,0 @@
#!/usr/bin/env python3
"""Emit the complete, self-contained deployment for this rig.
python3 ctrl/manifest.py [namespace] > generated/<slug>.yaml
The output is the ARTIFACT. It carries everything — namespace, the vite app and
bundle.json embedded in a ConfigMap, the Pod and the Service — so it applies
unmodified to any cluster:
kubectl apply -f generated/sample-rig.yaml
On kind, MetalLB answers the `type: LoadBalancer` Service. On a real external
cluster the cloud load balancer does. Same file, no edits, no branch — which is
the point: what runs locally is byte-identical to the deployment applied
elsewhere, so local success actually means something.
`ctrl/bundle.sh up` applies this same generated output rather than a separate
code path, so the local convenience wrapper can never drift from the artifact.
Stdlib only, deliberately: this must run on a bare staging workstation before
anything is installed, so it cannot depend on PyYAML or a template engine.
Open seam — not built: everything here derives from bundle.json plus a target
namespace. A Pulumi or Terraform emitter would sit beside this file consuming the
same inputs; nothing above it assumes the artifact is YAML.
"""
import json
import re
import sys
import zlib
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
UI = ROOT / "rig-ui"
# Files embedded into the ConfigMap, mounted read-only at /src in the pod and
# copied into vite's layout at start (see rig-ui/k8s.yaml). Flat on purpose:
# ConfigMap keys cannot contain '/'.
EMBEDDED = {
"bundle.json": ROOT / "bundle.json",
"package.json": UI / "package.json",
"vite.config.js": UI / "vite.config.js",
"index.html": UI / "index.html",
"main.js": UI / "src" / "main.js",
"style.css": UI / "src" / "style.css",
}
def slug(name: str) -> str:
"""Reduce a folder name to a DNS label, matching ctrl/bundle.sh's rule."""
out = re.sub(r"[^a-z0-9-]", "-", name.lower()).strip("-")
return out or "rig-bundle"
def block(text: str, indent: int) -> str:
"""Indent a file's contents for a YAML literal block scalar.
Blank lines are emitted truly empty rather than as whitespace: trailing
spaces on an otherwise blank line are legal YAML but show up as diff noise
in a committed artifact.
"""
pad = " " * indent
return "\n".join(pad + line if line.strip() else "" for line in text.splitlines())
def checksum(parts: list[str]) -> str:
"""Stable content hash of everything embedded, stamped as a label.
A mounted ConfigMap updates in place without restarting anything, so without
a visible change nothing signals that the pod is serving stale content.
"""
return str(zlib.crc32("".join(parts).encode()) & 0xFFFFFFFF)
def build(namespace: str) -> str:
contents = {}
for key, path in EMBEDDED.items():
if not path.exists():
sys.exit(f"missing input: {path}")
contents[key] = path.read_text()
# Fail loudly here rather than shipping an artifact that renders an error.
try:
json.loads(contents["bundle.json"])
except json.JSONDecodeError as exc:
sys.exit(f"bundle.json is not valid JSON: {exc}")
app = (UI / "k8s.yaml").read_text()
app = app.replace("__RIG_NAME__", namespace)
data = "\n".join(
f" {key}: |\n{block(text, 4)}" for key, text in sorted(contents.items())
)
return f"""# GENERATED by ctrl/manifest.py — do not edit.
# Regenerate with: make manifest
#
# Self-contained: applies as-is to any cluster, local kind or external.
# kubectl apply -f this-file.yaml
#
# Namespace carries the identity, so several rigs coexist in one cluster.
apiVersion: v1
kind: Namespace
metadata:
name: {namespace}
labels:
rig.bundle/name: {namespace}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: rig-ui
namespace: {namespace}
labels:
rig.bundle/checksum: "{checksum(list(contents.values()))}"
data:
{data}
---
{_namespaced(app, namespace)}
"""
def _namespaced(doc: str, namespace: str) -> str:
"""Add `namespace:` to each resource so the artifact applies without -n.
rig-ui/k8s.yaml omits it on purpose — applied by hand it should land in
whatever namespace you choose. Pinning it belongs to the generated artifact,
which has to be self-contained.
"""
return re.sub(
r"^(metadata:\n(?:[ \t]+.*\n)*?)([ \t]+)(name: rig-ui)$",
lambda m: f"{m.group(1)}{m.group(2)}{m.group(3)}\n{m.group(2)}namespace: {namespace}",
doc.strip(),
flags=re.MULTILINE,
)
if __name__ == "__main__":
target = sys.argv[1] if len(sys.argv) > 1 else slug(ROOT.name)
sys.stdout.write(build(target))

View File

@@ -1,406 +0,0 @@
# GENERATED by ctrl/manifest.py — do not edit.
# Regenerate with: make manifest
#
# Self-contained: applies as-is to any cluster, local kind or external.
# kubectl apply -f this-file.yaml
#
# Namespace carries the identity, so several rigs coexist in one cluster.
apiVersion: v1
kind: Namespace
metadata:
name: sample-rig
labels:
rig.bundle/name: sample-rig
---
apiVersion: v1
kind: ConfigMap
metadata:
name: rig-ui
namespace: sample-rig
labels:
rig.bundle/checksum: "2074194964"
data:
bundle.json: |
{
"_comment": "What this bundle contains. Single source of truth — the landing page renders THIS file, so adding an entry here is the only edit needed. Deliberately FLAT: standalone tools and rigs, with none of soleprint's internal hierarchy (no artery/atlas/station layering). Nothing here is sensitive; the real architecture connects separately.",
"bundle": {
"name": "sample-rig",
"description": "Non-sensitive sample bundle. Proves the kind install works and shows what ships.",
"sensitive": false
},
"tools": [
{
"name": "modelgen",
"summary": "Generate models from config",
"standalone": true
},
{
"name": "datagen",
"summary": "Generate test data from rig-owned generators",
"standalone": true
},
{
"name": "graphgen",
"summary": "Generate navigable model graphs",
"standalone": true
},
{
"name": "tester",
"summary": "HTTP contract test runner — one suite, any environment",
"standalone": true
},
{
"name": "databrowse",
"summary": "SQL data browser",
"standalone": true
},
{
"name": "sbwrapper",
"summary": "Sandbox wrapper",
"standalone": true
}
],
"rigs": [
{
"name": "sample-rig",
"summary": "This bundle — a minimal, copyable environment",
"active": true
}
],
"next": [
"Point MANIFESTS_DIR at the real manifests to connect the actual architecture.",
"Real k8s files are versioned separately and are not part of this bundle."
]
}
index.html: |
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>IT WORKS</title>
</head>
<body>
<main id="app"></main>
<script type="module" src="/src/main.js"></script>
</body>
</html>
main.js: |
import "./style.css";
/* The IT WORKS page: renders bundle.json as the list of what shipped.
*
* Plain vite, no framework — this is a complement to the rig, not part of it,
* and it should stay small enough that nobody has to adopt a stack to read it.
*
* bundle.json is fetched at runtime rather than imported, so the same built app
* serves whatever rig it was copied into. Editing the ConfigMap changes the page
* without rebuilding.
*
* Styling is a handful of rules on purpose. The real visual identity lives in
* the soleprint UI package; nothing here should grow into a theme.
*/
const esc = (s) =>
String(s).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
const tag = (text, on = false) =>
`<span class="tag${on ? " on" : ""}">${esc(text)}</span>`;
function items(list, activeKey) {
if (!list?.length) return `<li><span class="summary">nothing listed</span></li>`;
return list
.map((it) => {
const tags = [
it.standalone ? tag("standalone") : "",
it.state ? tag(it.state) : "",
activeKey && it[activeKey] ? tag("active", true) : "",
].join("");
return `<li><span class="name">${esc(it.name ?? "?")}</span>
<span class="summary">${esc(it.summary ?? "")}</span>${tags}</li>`;
})
.join("");
}
/* Cluster state, when there is any to show.
*
* Fetched separately and allowed to fail: the bundle listing is the point, and a
* rig with no cluster reachable is a normal state, not an error. Renders nothing
* at all when absent.
*
* When the payload says `mocked`, say so loudly. This exists to demo the UI on a
* machine where kind will not run — and a demo that looks live but is not is
* worse than one that admits it. */
function clusterSection(c) {
if (!c) return "";
const m = c.cluster ?? {};
const banner = c.mocked
? `<p class="mock">mocked — no cluster was queried; these are canned values</p>`
: "";
const meta = [m.context, m.k8s, m.profile ? `profile ${m.profile}` : "",
m.nodes ? `${m.nodes} node${m.nodes > 1 ? "s" : ""}` : ""]
.filter(Boolean).join(" · ");
return `
<h2>Cluster${c.mocked ? " (mocked)" : ""}</h2>
${banner}
${meta ? `<p class="sub">${esc(meta)}</p>` : ""}
<ul>${items(c.workloads)}</ul>
<h2>Services (${c.services?.length ?? 0})</h2>
<ul>${items(c.services)}</ul>`;
}
function render(b, name, cluster) {
const meta = b.bundle ?? {};
const next = (b.next ?? []).map((n) => `<li>${esc(n)}</li>`).join("");
return `
<h1><span class="ok">IT WORKS</span> — ${esc(name || meta.name || "rig")}</h1>
<p class="sub">${esc(meta.description ?? "")}</p>
<h2>Tools (${b.tools?.length ?? 0})</h2>
<ul>${items(b.tools)}</ul>
<h2>Rigs (${b.rigs?.length ?? 0})</h2>
<ul>${items(b.rigs, "active")}</ul>
${clusterSection(cluster)}
${next ? `<div class="next"><ul>${next}</ul></div>` : ""}`;
}
const app = document.getElementById("app");
const json = (path, required) =>
fetch(path).then((r) => {
if (r.ok) return r.json();
if (required) throw new Error(`${path} -> HTTP ${r.status}`);
return null; // optional: absent is a normal state, not an error
}).catch((err) => {
if (required) throw err;
return null;
});
Promise.all([json("/bundle.json", true), json("/cluster.mock.json", false)])
// RIG_NAME is injected by vite from the pod env, so two rigs sharing a
// cluster are distinguishable even if a copied bundle.json kept its old name.
.then(([b, cluster]) => {
app.innerHTML = render(b, import.meta.env.VITE_RIG_NAME, cluster);
})
.catch((err) => {
app.innerHTML = `<h1 class="err">bundle unavailable</h1>
<p class="sub">${esc(err.message)}</p>`;
});
package.json: |
{
"name": "rig-ui",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite --host 0.0.0.0 --port 5173",
"build": "vite build",
"preview": "vite preview --host 0.0.0.0 --port 5173"
},
"devDependencies": {
"vite": "^6"
}
}
style.css: |
/* Minimal, self-contained. The real visual identity ships with the soleprint UI
package, which is a separate artifact — nothing here should grow into a theme. */
body {
margin: 0;
padding: 2.5rem 1.5rem;
background: #0d0d0f;
color: #e8e8f0;
font: 14px/1.6 ui-monospace, "JetBrains Mono", Menlo, monospace;
}
main { max-width: 52rem; margin: 0 auto; }
h1 { margin: 0; font-size: 1.6rem; letter-spacing: 0.02em; }
h1 .ok { color: #3ecf8e; }
h1.err { color: #f06565; }
.sub { color: #8888a0; margin: 0.35rem 0 2.25rem; }
h2 {
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.12em;
color: #8888a0;
margin: 2rem 0 0.75rem;
font-weight: 600;
}
ul { list-style: none; margin: 0; padding: 0; }
li {
display: flex;
gap: 0.75rem;
align-items: baseline;
padding: 0.5rem 0.75rem;
border: 1px solid #2e2e38;
border-radius: 6px;
margin-bottom: 0.4rem;
background: #16161a;
}
.name { font-weight: 600; min-width: 9rem; }
.summary { color: #8888a0; flex: 1; }
.tag {
font-size: 0.7rem;
padding: 0.1rem 0.45rem;
border-radius: 3px;
background: #26262f;
color: #8888a0;
white-space: nowrap;
}
.tag.on { background: #3ecf8e; color: #0d0d0f; }
.next {
color: #555568;
font-size: 0.8rem;
margin-top: 2.5rem;
border-top: 1px solid #2e2e38;
padding-top: 1rem;
}
.next li {
display: list-item;
border: 0;
background: none;
padding: 0.15rem 0;
margin: 0 0 0 1.1rem;
list-style: disc;
}
/* Mocked-data banner. Deliberately loud: this only appears when the cluster
payload is canned, and a demo that looks live but is not is worse than one
that says so. */
.mock {
margin: 0 0 0.75rem;
padding: 0.4rem 0.75rem;
border: 1px dashed #f5a623;
border-radius: 6px;
color: #f5a623;
font-size: 0.8rem;
}
vite.config.js: |
import { defineConfig } from "vite";
/* Serves on 0.0.0.0 so the pod is reachable through the Service, and allows any
* Host header because the address is assigned at runtime (MetalLB locally, a
* cloud load balancer on EKS) and is never known at build time. */
export default defineConfig({
server: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
preview: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
});
---
# How to plug the UI into whatever k8s you generated. THIS IS THE WHOLE THING:
# one Pod running the vite app, one Service to reach it.
#
# Optional by design. The UI complements a rig; it is not part of the end
# product, and a rig is complete and useful without it. Apply this only when you
# want the listing:
#
# kubectl apply -n <your-namespace> -f rig-ui/k8s.yaml
#
# A bare Pod, not a Deployment — this is a dev-loop convenience, not a workload
# to keep alive. If it dies you re-apply it; nothing depends on it staying up.
#
# The app and bundle.json arrive as a ConfigMap named `rig-ui`, which
# ctrl/manifest.py generates from the folder. Nothing is baked into an image, so
# editing bundle.json and re-applying is the whole update cycle.
apiVersion: v1
kind: Pod
metadata:
name: rig-ui
namespace: sample-rig
labels:
app: rig-ui
spec:
containers:
- name: vite
image: node:22-alpine
workingDir: /app
# npm install at start: no image to build and no registry to publish to,
# which is the point of a minimal plug-in. It needs egress to a registry —
# on a locked-down cluster point npm at the internal one, or bake an image
# instead. Nothing else here changes if you do.
command: ["sh", "-c"]
# A ConfigMap mounts flat (keys cannot contain '/'), so the files are
# placed into vite's expected layout here. bundle.json goes to public/
# because that is what vite serves at /bundle.json, which is where the
# app fetches it.
args:
- |
mkdir -p /app/src /app/public &&
cp /src/package.json /src/vite.config.js /src/index.html /app/ &&
cp /src/main.js /src/style.css /app/src/ &&
cp /src/bundle.json /app/public/ &&
npm install --no-audit --no-fund &&
npm run dev
env:
# Rendered in the heading so two rigs sharing a cluster stay
# distinguishable. Set from the namespace by ctrl/manifest.py.
- name: VITE_RIG_NAME
value: sample-rig
ports:
- name: http
containerPort: 5173
volumeMounts:
# /src is read-only from the ConfigMap; the app is copied to a writable
# /app because npm install has to create node_modules.
- name: rig-ui
mountPath: /src
- name: app
mountPath: /app
readinessProbe:
httpGet: { path: /, port: 5173 }
# npm install decides how long this takes, and it is the slow part.
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 30
resources:
requests: { memory: 128Mi, cpu: 50m }
limits: { memory: 512Mi }
volumes:
- name: rig-ui
configMap:
name: rig-ui
- name: app
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: rig-ui
namespace: sample-rig
labels:
app: rig-ui
# No annotations, deliberately — see k8s/app.yaml. The target is EKS but this
# stays VPC-agnostic: no subnets, no security groups, no -scheme, no -type.
# A bare LoadBalancer is what lets one manifest work on kind and on EKS.
spec:
type: LoadBalancer
selector:
app: rig-ui
ports:
- name: http
port: 80
targetPort: 5173
protocol: TCP
# Pinned, because a LoadBalancer Service also allocates a NodePort and
# this is the only address that works everywhere.
#
# On WSL the MetalLB address is on a docker bridge INSIDE the Linux VM,
# and Windows has no route to it — the page looks broken while the
# cluster is perfectly healthy. 30080 is what rig's `hostport` ingress
# mode publishes to the host, so this is reachable at
# localhost:$HTTP_PORT from a Windows browser with nothing configured.
#
# Costs nothing elsewhere: MetalLB still assigns an external IP on Linux,
# and on EKS the load balancer targets this NodePort anyway. One Service,
# no per-environment branch.
#
# A pinned NodePort is cluster-unique, so two rigs must live in separate
# clusters — which is how they are run anyway.
nodePort: 30080

View File

@@ -1,3 +0,0 @@
node_modules/
public/
dist/

View File

@@ -1,12 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>IT WORKS</title>
</head>
<body>
<main id="app"></main>
<script type="module" src="/src/main.js"></script>
</body>
</html>

View File

@@ -1,108 +0,0 @@
# How to plug the UI into whatever k8s you generated. THIS IS THE WHOLE THING:
# one Pod running the vite app, one Service to reach it.
#
# Optional by design. The UI complements a rig; it is not part of the end
# product, and a rig is complete and useful without it. Apply this only when you
# want the listing:
#
# kubectl apply -n <your-namespace> -f rig-ui/k8s.yaml
#
# A bare Pod, not a Deployment — this is a dev-loop convenience, not a workload
# to keep alive. If it dies you re-apply it; nothing depends on it staying up.
#
# The app and bundle.json arrive as a ConfigMap named `rig-ui`, which
# ctrl/manifest.py generates from the folder. Nothing is baked into an image, so
# editing bundle.json and re-applying is the whole update cycle.
apiVersion: v1
kind: Pod
metadata:
name: rig-ui
labels:
app: rig-ui
spec:
containers:
- name: vite
image: node:22-alpine
workingDir: /app
# npm install at start: no image to build and no registry to publish to,
# which is the point of a minimal plug-in. It needs egress to a registry —
# on a locked-down cluster point npm at the internal one, or bake an image
# instead. Nothing else here changes if you do.
command: ["sh", "-c"]
# A ConfigMap mounts flat (keys cannot contain '/'), so the files are
# placed into vite's expected layout here. bundle.json goes to public/
# because that is what vite serves at /bundle.json, which is where the
# app fetches it.
args:
- |
mkdir -p /app/src /app/public &&
cp /src/package.json /src/vite.config.js /src/index.html /app/ &&
cp /src/main.js /src/style.css /app/src/ &&
cp /src/bundle.json /app/public/ &&
npm install --no-audit --no-fund &&
npm run dev
env:
# Rendered in the heading so two rigs sharing a cluster stay
# distinguishable. Set from the namespace by ctrl/manifest.py.
- name: VITE_RIG_NAME
value: __RIG_NAME__
ports:
- name: http
containerPort: 5173
volumeMounts:
# /src is read-only from the ConfigMap; the app is copied to a writable
# /app because npm install has to create node_modules.
- name: rig-ui
mountPath: /src
- name: app
mountPath: /app
readinessProbe:
httpGet: { path: /, port: 5173 }
# npm install decides how long this takes, and it is the slow part.
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 30
resources:
requests: { memory: 128Mi, cpu: 50m }
limits: { memory: 512Mi }
volumes:
- name: rig-ui
configMap:
name: rig-ui
- name: app
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: rig-ui
labels:
app: rig-ui
# No annotations, deliberately — see k8s/app.yaml. The target is EKS but this
# stays VPC-agnostic: no subnets, no security groups, no -scheme, no -type.
# A bare LoadBalancer is what lets one manifest work on kind and on EKS.
spec:
type: LoadBalancer
selector:
app: rig-ui
ports:
- name: http
port: 80
targetPort: 5173
protocol: TCP
# Pinned, because a LoadBalancer Service also allocates a NodePort and
# this is the only address that works everywhere.
#
# On WSL the MetalLB address is on a docker bridge INSIDE the Linux VM,
# and Windows has no route to it — the page looks broken while the
# cluster is perfectly healthy. 30080 is what rig's `hostport` ingress
# mode publishes to the host, so this is reachable at
# localhost:$HTTP_PORT from a Windows browser with nothing configured.
#
# Costs nothing elsewhere: MetalLB still assigns an external IP on Linux,
# and on EKS the load balancer targets this NodePort anyway. One Service,
# no per-environment branch.
#
# A pinned NodePort is cluster-unique, so two rigs must live in separate
# clusters — which is how they are run anyway.
nodePort: 30080

File diff suppressed because it is too large Load Diff

View File

@@ -1,14 +0,0 @@
{
"name": "rig-ui",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite --host 0.0.0.0 --port 5173",
"build": "vite build",
"preview": "vite preview --host 0.0.0.0 --port 5173"
},
"devDependencies": {
"vite": "^6"
}
}

View File

@@ -1,136 +0,0 @@
import "./style.css";
/* The IT WORKS page: renders bundle.json as the list of what shipped.
*
* Plain vite, no framework — this is a complement to the rig, not part of it,
* and it should stay small enough that nobody has to adopt a stack to read it.
*
* Laid out like soleprint's templated vein pages, because it does the same job:
* name each component, list what it exposes, show what comes back. Tool chrome
* and output are styled apart on purpose (see style.css) — that separation is
* what tells you whether you are reading the tool or its result.
*
* bundle.json is fetched at runtime rather than imported, so the same built app
* serves whatever rig it was copied into. Editing the ConfigMap changes the page
* without rebuilding.
*/
const esc = (s) =>
String(s).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
const tag = (text, on = false) =>
`<span class="tag${on ? " on" : ""}">${esc(text)}</span>`;
/* Tool chrome: one bordered card per component. */
function components(list, activeKey) {
if (!list?.length)
return `<div class="component"><p>nothing listed</p></div>`;
return list
.map((it) => {
const tags = [
it.standalone ? tag("standalone") : "",
it.state ? tag(it.state) : "",
activeKey && it[activeKey] ? tag("active", true) : "",
].join("");
return `<div class="component">
<h4>${esc(it.name ?? "?")} ${tags}</h4>
<p>${esc(it.summary ?? "")}</p>
</div>`;
})
.join("");
}
/* Endpoint rows: path on the left, what it returns on the right. */
function endpoints(list) {
return list
.map(
(e) => `<li><code>${esc(e.path)}</code>
<span class="desc">${esc(e.desc)}</span></li>`
)
.join("");
}
/* Output: what the endpoint above actually returns, so the page demonstrates
itself rather than describing what a demonstration would look like. */
function example(bundle) {
const sample = {
bundle: bundle.bundle?.name,
tools: (bundle.tools ?? []).map((t) => t.name),
rigs: (bundle.rigs ?? []).map((r) => r.name),
};
return `<pre class="output">${esc(JSON.stringify(sample, null, 2))}</pre>`;
}
/* Cluster state, when there is any to show.
*
* Fetched separately and allowed to fail: the bundle listing is the point, and a
* rig with no cluster reachable is a normal state, not an error. Renders nothing
* at all when absent. When the payload says `mocked`, say so loudly. */
function clusterSection(c) {
if (!c) return "";
const m = c.cluster ?? {};
const meta = [m.context, m.k8s, m.profile && `profile ${m.profile}`,
m.nodes && `${m.nodes} node${m.nodes > 1 ? "s" : ""}`]
.filter(Boolean).join(" · ");
return `
<h2>Cluster${c.mocked ? " (mocked)" : ""}</h2>
${c.mocked ? `<p class="mock">mocked — no cluster was queried; these are canned values</p>` : ""}
${meta ? `<p class="tagline">${esc(meta)}</p>` : ""}
<div class="components">${components(c.workloads)}</div>
<h2>Services</h2>
<div class="components">${components(c.services)}</div>`;
}
function render(b, name, cluster) {
const meta = b.bundle ?? {};
const next = (b.next ?? []).map((n) => `<li>${esc(n)}</li>`).join("");
return `
<h1><span class="ok">IT WORKS</span> — ${esc(name || meta.name || "rig")}</h1>
<p class="tagline">${esc(meta.description ?? "")}</p>
<h2>Tools (${b.tools?.length ?? 0})</h2>
<div class="components">${components(b.tools)}</div>
<h2>Rigs (${b.rigs?.length ?? 0})</h2>
<div class="components">${components(b.rigs, "active")}</div>
<h2>Endpoints</h2>
<ul class="endpoints">${endpoints([
{ path: "/", desc: "this page" },
{ path: "/bundle.json", desc: "the manifest it renders" },
])}</ul>
<h2>Example — GET /bundle.json</h2>
${example(b)}
${clusterSection(cluster)}
${next ? `<div class="next"><ul>${next}</ul></div>` : ""}`;
}
const app = document.getElementById("app");
const json = (path, required) =>
fetch(path)
.then((r) => {
if (r.ok) return r.json();
if (required) throw new Error(`${path} -> HTTP ${r.status}`);
return null; // optional: absent is a normal state, not an error
})
.catch((err) => {
if (required) throw err;
return null;
});
Promise.all([json("/bundle.json", true), json("/cluster.mock.json", false)])
// RIG_NAME is injected by vite from the pod env, so two rigs sharing a
// cluster are distinguishable even if a copied bundle.json kept its old name.
.then(([b, cluster]) => {
app.innerHTML = render(b, import.meta.env.VITE_RIG_NAME, cluster);
})
.catch((err) => {
app.innerHTML = `<h1 class="err">bundle unavailable</h1>
<p class="tagline">${esc(err.message)}</p>`;
});

View File

@@ -1,139 +0,0 @@
/* Minimal and self-contained — no framework dependency.
*
* The visual language follows soleprint's templated vein pages, because this
* page does the same job: say what a component is, list what it exposes, and
* show what comes back. Two treatments, deliberately distinct:
*
* TOOL CHROME bordered cards on the darker background, accent-coloured
* titles, endpoint rows separated by rules.
* OUTPUT a lighter raised block, monospace, pre-wrap and selectable —
* it is data, not furniture, and should read as a payload.
*
* Keeping them apart matters more than either looks: it is what tells you at a
* glance whether you are reading the tool or the thing it produced. */
:root {
--bg: #0d0d0f;
--surface: #16161a;
--surface-raised: #1e1e24;
--border: #2e2e38;
--border-strong: #3d3d4a;
--text: #e8e8f0;
--muted: #8888a0;
--accent: #3ecf8e;
--accent-dim: #f5a623;
--mono: "JetBrains Mono", "Cascadia Mono", Consolas, ui-monospace, monospace;
}
body {
margin: 0;
padding: 2.5rem 1.5rem;
background: var(--bg);
color: var(--text);
font: 14px/1.6 var(--mono);
}
main { max-width: 56rem; margin: 0 auto; }
h1 { margin: 0; font-size: 1.6rem; letter-spacing: 0.02em; }
h1 .ok { color: var(--accent); }
h1.err { color: #f06565; }
.tagline { color: var(--muted); margin: 0.35rem 0 2.25rem; }
h2 {
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.12em;
color: var(--muted);
margin: 2.25rem 0 0.75rem;
font-weight: 600;
}
/* ── tool chrome ────────────────────────────────────────────────────────── */
.components {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 0.75rem;
}
.component {
background: var(--bg);
border: 1px solid var(--border-strong);
border-radius: 8px;
padding: 0.75rem;
}
.component h4 {
margin: 0 0 0.25rem;
font-size: 0.95rem;
color: var(--accent);
}
.component p {
margin: 0;
font-size: 0.85rem;
color: var(--muted);
}
.endpoints { list-style: none; margin: 0; padding: 0; }
.endpoints li {
display: flex;
justify-content: space-between;
flex-wrap: wrap;
gap: 0.5rem;
padding: 0.6rem 0;
border-bottom: 1px solid var(--border-strong);
}
.endpoints li:last-child { border-bottom: none; }
.endpoints code {
background: var(--surface);
color: var(--accent);
padding: 0.25rem 0.5rem;
border-radius: 4px;
}
.endpoints .desc { color: var(--muted); font-size: 0.9rem; }
.tag {
font-size: 0.7rem;
padding: 0.1rem 0.45rem;
border-radius: 3px;
background: var(--border);
color: var(--muted);
white-space: nowrap;
}
.tag.on { background: var(--accent); color: var(--bg); }
/* ── output ─────────────────────────────────────────────────────────────── */
.output {
background: var(--surface);
border: 1px solid var(--border);
border-radius: 8px;
padding: 1rem;
font-size: 0.85rem;
white-space: pre-wrap;
word-break: break-word;
user-select: text;
color: var(--text);
margin: 0;
}
.output .k { color: var(--accent); }
/* Mocked-data banner. Loud on purpose: it only appears when the payload is
canned, and a demo that looks live but is not is worse than one that says so. */
.mock {
margin: 0 0 0.75rem;
padding: 0.4rem 0.75rem;
border: 1px dashed var(--accent-dim);
border-radius: 6px;
color: var(--accent-dim);
font-size: 0.8rem;
}
.next {
color: #555568;
font-size: 0.8rem;
margin-top: 2.5rem;
border-top: 1px solid var(--border);
padding-top: 1rem;
}
.next ul { margin: 0; padding: 0 0 0 1.1rem; }
.next li { list-style: disc; padding: 0.15rem 0; }

View File

@@ -1,9 +0,0 @@
import { defineConfig } from "vite";
/* Serves on 0.0.0.0 so the pod is reachable through the Service, and allows any
* Host header because the address is assigned at runtime (MetalLB locally, a
* cloud load balancer on EKS) and is never known at build time. */
export default defineConfig({
server: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
preview: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
});

View File

@@ -577,7 +577,7 @@ def station_index(request: Request):
# not the server. Route registration raises more than ImportError — FastAPI
# turns a missing optional dependency into a RuntimeError at decoration time —
# and catching only ImportError meant one such tool took the whole app down.
for _tool in ("tester", "graphgen", "datagen", "shuntgen"):
for _tool in ("tester", "graphgen", "datagen", "shuntgen", "histgen"):
try:
_module = importlib.import_module(f"station.tools.{_tool}.api")
app.include_router(_module.router, prefix="/station")

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,5 @@
```bash
./ctrl/explode.sh --list bundle.txt # what is in there, write nothing
./ctrl/explode.sh -o ./restored bundle.txt # write the tree
./ctrl/explode.sh -o ./restored --force x.md # overwrite what is already there
```

View File

@@ -0,0 +1,346 @@
#!/usr/bin/env bash
# Explode one file back into the tree of files it describes.
#
# The inverse of ctrl/distill.sh's digest: something hands you a single text
# file with many files inside it, each introduced by its path, and you want the
# directory back.
#
# Three layouts are understood, picked automatically. Prefer the first if you
# control what writes the file:
#
# === FILE: pkg/models/domain.py explicit open and close. Nothing has to be
# <the file> counted or inferred, and a block that is
# === END never closed is an error rather than a
# file quietly missing its tail.
#
# === ./pkg/models/domain.py a marker line, then the file, until the
# <the file> next marker or the end
#
# ## pkg/models/domain.py distill.sh's own digest: a heading, then
# ```python a fenced block. The fence may be longer
# <the file> than three backticks, and the closing one
# ``` has to match it exactly.
#
# Usage:
# explode.sh [opts] FILE
#
# Options:
# -o DEST where to write the tree (default: the current directory)
# --list print what the file contains and write nothing
# -n same as --list
# --force overwrite files that already exist
# --format F fenced | marker | digest | auto (default: auto)
# --selftest check this copy of the script against known input and exit
#
# Examples:
# explode.sh --list bundle.txt
# explode.sh -o ./restored bundle.txt
# explode.sh -o ./restored --force repo.md
#
# Why the explicit form is worth asking for: a writer that emits plain three-
# backtick fences truncates any file that itself contains a fence — every README
# with a shell example — and does it silently, because the nested fence looks
# exactly like the closing one. distill.sh avoids that by making its fences
# longer than anything inside the file, but nothing else will bother.
#
# Two limits worth knowing. A file whose last line has no trailing newline comes
# back with one: the digest has to put a newline before the closing fence, so the
# distinction is not in the input to recover. And in marker layout a line
# starting with "=== " inside a file's own content is indistinguishable from a
# real marker — there are no fences to say otherwise. The digest layout has no
# such ambiguity, which is the reason to prefer it when something else is
# generating the file.
#
# Paths come out of a text file, so they are treated as untrusted: anything
# absolute, or reaching upward with .., is refused and nothing is written. A
# file that describes /etc/cron.d/x is not a file you want to expand blindly.
set -euo pipefail
SELF="$(basename "$0")"
usage() { awk 'NR==1{next} /^#/{sub(/^# ?/,""); print; next} {exit}' "$0"; }
die() { echo "$SELF: $*" >&2; exit 1; }
DEST="."
LIST=""
FORCE=""
FORMAT="auto"
SRC=""
SELFTEST=""
while [ $# -gt 0 ]; do
case "$1" in
-o) shift; DEST="${1:-}" ;;
--list|-n) LIST=1 ;;
--force) FORCE=1 ;;
--format) shift; FORMAT="${1:-}" ;;
--selftest) SELFTEST=1 ;;
-h|--help) usage; exit 0 ;;
-*) die "unknown option: $1" ;;
*) [ -z "$SRC" ] && SRC="$1" || die "one input file at a time (got '$SRC' and '$1')" ;;
esac
shift
done
# ── self-test ──────────────────────────────────────────────────────────────
# So a copy of this script on another machine can be checked without any real
# input, and without asking whether it is the version that knows a given format.
# Every case here is one that has actually gone wrong.
selftest() {
local t rc=0 got want
t="$(mktemp -d)"; trap 'rm -rf "$t"' RETURN
check() { # name, expected, actual
if [ "$2" = "$3" ]; then printf ' ok %s\n' "$1"
else printf ' FAIL %s\n expected: %s\n got: %s\n' "$1" "$2" "$3"; rc=1
fi
}
# A file containing a fence and a stray === line: the two things that break
# naive parsers.
cat > "$t/a.txt" <<'FIXTURE'
Here is the code.
=== FILE: pkg/core/client.py
class Client:
pass
=== END
=== FILE: README.md
# proj
```bash
pip install proj
```
A line that says === not a marker
=== END
FIXTURE
"$0" -o "$t/a" "$t/a.txt" >/dev/null 2>&1 || true
check "fenced: file count" "2" "$(find "$t/a" -type f 2>/dev/null | wc -l)"
check "fenced: nested fences" "2" "$(grep -c '```' "$t/a/README.md" 2>/dev/null || echo 0)"
check "fenced: === in content" "1" "$(grep -c 'not a marker' "$t/a/README.md" 2>/dev/null || echo 0)"
check "fenced: subdirectory" "class Client:" "$(head -1 "$t/a/pkg/core/client.py" 2>/dev/null)"
check "fenced: prose skipped" "0" "$(find "$t/a" -name 'Here*' 2>/dev/null | wc -l)"
# An unclosed block must be refused, not written short.
printf '=== FILE: a.py\nx = 1\n=== END\n\n=== FILE: b.py\ny = 2\n' > "$t/b.txt"
"$0" -o "$t/b" "$t/b.txt" >/dev/null 2>&1 || true
check "unterminated: refused" "1" "$([ -e "$t/b" ] && echo 0 || echo 1)"
# Paths out of a text file are untrusted.
printf '=== FILE: ../escape.py\nx\n=== END\n' > "$t/c.txt"
"$0" -o "$t/c" "$t/c.txt" >/dev/null 2>&1 || true
check "traversal: refused" "1" "$([ -e "$t/c" ] && echo 0 || echo 1)"
# The stale-copy failure: explicit format read by the marker parser.
"$0" --format marker -o "$t/d" "$t/a.txt" >/dev/null 2>&1 || true
check "wrong parser: refused" "1" "$([ -e "$t/d" ] && echo 0 || echo 1)"
# The other two layouts still work.
printf '=== ./x/y.py\nz = 1\n' > "$t/e.txt"
"$0" -o "$t/e" "$t/e.txt" >/dev/null 2>&1 || true
check "marker layout" "z = 1" "$(cat "$t/e/x/y.py" 2>/dev/null)"
printf '# d\n\n## x/y.py\n\n```python\nz = 1\n```\n' > "$t/f.txt"
"$0" -o "$t/f" "$t/f.txt" >/dev/null 2>&1 || true
check "digest layout" "z = 1" "$(cat "$t/f/x/y.py" 2>/dev/null)"
echo
if [ "$rc" -eq 0 ]; then echo "all checks passed — this copy is current"
else echo "SOME CHECKS FAILED — this copy is out of date or broken" >&2
fi
return "$rc"
}
if [ -n "$SELFTEST" ]; then selftest; exit $?; fi
[ -n "$SRC" ] || { usage >&2; exit 1; }
[ -f "$SRC" ] || die "no such file: $SRC"
case "$FORMAT" in fenced|marker|digest|auto) ;; *) die "--format must be fenced, marker, digest or auto" ;; esac
# Which layout is it? Count the two shapes and take the commoner one, rather
# than trusting the first line that happens to match: a digest of a repo full of
# markdown will contain plenty of '=== ' inside its own fenced content, and a
# marker file can quote a '## ' heading just as easily.
if [ "$FORMAT" = auto ]; then
n_fenced=$(grep -cE '^=== +FILE: +[^ ]' "$SRC" || true)
n_marker=$(grep -cE '^=== +\.?/?[^ ]' "$SRC" || true)
n_marker=$((n_marker - n_fenced - $(grep -cE '^=== +END[ \t]*$' "$SRC" || true)))
[ "$n_marker" -lt 0 ] && n_marker=0
n_digest=$(grep -cE '^## +[^ ]' "$SRC" || true)
if [ "$n_fenced" -gt 0 ]; then
FORMAT=fenced
elif [ "$n_marker" -eq 0 ] && [ "$n_digest" -eq 0 ]; then
die "found no '=== FILE:' blocks, no '=== path' markers and no '## path' headings in $SRC"
elif [ "$n_marker" -ge "$n_digest" ]; then FORMAT=marker
else FORMAT=digest
fi
echo "format: $FORMAT"
fi
# ── the parser ─────────────────────────────────────────────────────────────
# One awk, two modes. In list mode it prints "path<TAB>lines"; otherwise it
# writes each file under DEST. Reading the whole thing in awk rather than a
# bash read-loop matters once the input is a few megabytes.
#
# In digest mode a heading only opens a file if a fence follows it. distill.sh
# writes '## Tree' and '## Binary files ...' sections that are prose, and
# treating those as files would scatter junk through the output.
parse() {
awk -v dest="$DEST" -v mode="$1" -v fmt="$FORMAT" '
function flush() {
if (path != "") {
if (mode == "list") { printf "%s\t%d\n", path, n }
path = ""
}
n = 0
}
function clean(p) {
sub(/^\.\//, "", p)
sub(/[ \t\r]+$/, "", p)
return p
}
function unsafe(p) {
return (p == "" || p ~ /^\// || p ~ /^[A-Za-z]:/ || p ~ /(^|\/)\.\.(\/|$)/)
}
function open_file(p) {
path = p
n = 0
if (mode == "write") {
out = dest "/" path
d = out; sub(/\/[^\/]*$/, "", d)
system("mkdir -p \"" d "\"")
printf "" > out
}
}
function emit(line) {
n++
if (mode == "write") print line >> (dest "/" path)
}
# Explicit open/close. The whole point is that nothing is inferred:
# content is content until the END line, whatever it looks like.
fmt == "fenced" && path == "" && /^=== +FILE: +/ {
p = substr($0, index($0, "FILE:") + 5)
sub(/^[ \t]+/, "", p)
p = clean(p)
if (unsafe(p)) { print "UNSAFE\t" p; bad = 1; next }
open_file(p)
next
}
fmt == "fenced" && path != "" && /^=== +END[ \t]*$/ { flush(); next }
fmt == "fenced" && path == "" { next } # anything between blocks is prose
fmt == "marker" && /^=== +/ {
flush()
p = clean(substr($0, index($0, " ") + 1))
# "FILE: ./x.py" and "END" are not paths, they are the explicit
# format being read by the wrong parser. Left alone this writes a
# directory literally called "FILE: ." and a file called "END",
# which is what an out-of-date copy of this script did once.
if (p ~ /^FILE:/ || p == "END") { print "WRONGFMT\t" p; bad = 1; next }
if (unsafe(p)) { print "UNSAFE\t" p; bad = 1; next }
open_file(p)
next
}
# Only when NOT already inside a file. The content of any markdown file
# in the input is full of "## " headings, and the fence is the only
# thing that says which ones are structure and which are text. Once a
# file is open, the matching close fence is the sole way out.
fmt == "digest" && path == "" && /^## +/ {
flush()
pending = clean(substr($0, 4))
expect = 1
next
}
fmt == "digest" && expect == 1 {
if ($0 ~ /^[ \t]*$/) next # blank line between the two
if ($0 ~ /^`{3,}/) { # a fence: this is a file
match($0, /^`+/)
fence = substr($0, 1, RLENGTH)
expect = 0
if (unsafe(pending)) { print "UNSAFE\t" pending; bad = 1; next }
open_file(pending)
next
}
expect = 0 # prose section, not a file
pending = ""
next
}
fmt == "digest" && path != "" && $0 == fence { flush(); next }
{ if (path != "") emit($0) }
END {
if (fmt == "fenced" && path != "") {
print "UNTERMINATED\t" path
bad = 1
}
flush()
exit (bad ? 3 : 0)
}
' "$SRC"
}
# Validate before writing anything: a refusal after half the tree is on disk is
# not a refusal.
# awk exits non-zero when it found something wrong; that is the signal, not a
# crash, so let it through and report it properly below.
scan="$(parse list || true)"
refused="$(printf '%s\n' "$scan" | grep '^UNSAFE' || true)"
if [ -n "$refused" ]; then
echo "$SELF: refusing — these paths escape the destination:" >&2
printf '%s\n' "$refused" | sed 's/^UNSAFE\t/ /' >&2
exit 1
fi
# A block that never closed means the input is malformed, or a file contained
# the END line. Either way the tail is missing, and a truncated source file that
# looks complete is the failure this format exists to prevent.
wrongfmt="$(printf '%s\n' "$scan" | grep '^WRONGFMT' || true)"
if [ -n "$wrongfmt" ]; then
echo "$SELF: this file uses '=== FILE: path' / '=== END', but it was read as" >&2
echo "the plain marker format, which would create a directory called 'FILE: .'" >&2
echo "and files called 'END'. Re-run with --format fenced, or update this script." >&2
exit 1
fi
unterminated="$(printf '%s\n' "$scan" | grep '^UNTERMINATED' || true)"
if [ -n "$unterminated" ]; then
echo "$SELF: refusing — this block was never closed with '=== END':" >&2
printf '%s\n' "$unterminated" | sed 's/^UNTERMINATED\t/ /' >&2
echo "the file it describes would be silently truncated" >&2
exit 1
fi
listing="$(printf '%s\n' "$scan" | grep -vE '^(UNSAFE|UNTERMINATED|WRONGFMT)' || true)"
[ -n "$listing" ] || die "no files found in $SRC (format: $FORMAT)"
count=$(printf '%s\n' "$listing" | grep -c . )
if [ -n "$LIST" ]; then
printf '%s\n' "$listing" | awk -F'\t' '{ printf " %-60s %5d lines\n", $1, $2 }'
echo "$count files"
exit 0
fi
# Existing files are someone's work until proven otherwise.
if [ -z "$FORCE" ]; then
clashes=""
while IFS=$'\t' read -r p _; do
[ -e "$DEST/$p" ] && clashes="$clashes $p"$'\n'
done <<< "$listing"
if [ -n "$clashes" ]; then
echo "$SELF: these already exist under $DEST:" >&2
printf '%s' "$clashes" >&2
echo "re-run with --force to overwrite" >&2
exit 1
fi
fi
mkdir -p "$DEST"
parse write >/dev/null
printf '%s\n' "$listing" | awk -F'\t' '{ printf " %s\n", $1 }'
echo "wrote $count files to $DEST"

View File

@@ -0,0 +1,5 @@
# Local settings: whose repo this machine points at is not a fact about the
# tool. Copy the folder, run `make init-config`, and the answer stays here.
histgen.json
def
__pycache__/

View File

@@ -0,0 +1,137 @@
# histgen — one target per verb.
#
# The folder is meant to be copied out of soleprint and used on its own, so
# everything here is derived from where this file sits rather than written down:
# copy the directory anywhere, `cd` into it, and `make` works. Renaming it works
# too, since the package name comes from the directory.
#
# Two directories, and the whole tool hangs off the difference:
#
# SOURCE the tree to read. Read-only, always. Nothing is written into it.
# OUT the plan, the briefs, and OUT/<name>/ — a copy of the source with
# the designed history committed into it.
#
# make check prove it works, on its own fixture
# make copy SOURCE=~/work/x OUT=~/out just the files, no repo, no keys
# make run SOURCE=~/work/x OUT=~/out scan + plan + brief
# make list the commits, to confirm
# make commands copy the files, hand back git commands
# make export ...or have it commit them for you
#
# Set them once and the verbs take no arguments:
#
# make init-config SOURCE=... OUT=...
# make config what everything resolves to
# make status what is in OUT, and what is left
#
# The logic lives in the Python, never here. Each target is one invocation.
HERE := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
PKG := $(notdir $(HERE))
PARENT := $(patsubst %/,%,$(dir $(HERE)))
PY ?= python3
PREFIX ?= $(HOME)/.local
# Run the package from its parent, which is what `python -m` needs and what
# lets this work without installing anything.
HISTGEN := PYTHONPATH=$(PARENT) $(PY) -m $(PKG)
# Extra flags for the verb being run: make plan REPO=x ARGS=--max-files=12
ARGS ?=
# Left empty, these say nothing and the config file decides. Passing REPO= or
# OUT= on the command line overrides it, which is the precedence the tool
# already applies — the Makefile just has to not invent a default of its own.
SOURCE ?=
OUT ?=
CONFIG ?=
WHERE := $(if $(SOURCE),--source $(SOURCE)) $(if $(OUT),--out $(OUT)) \
$(if $(CONFIG),--config $(CONFIG))
.PHONY: help check run copy scan plan list brief export keep commands dry-run verify status \
against-history clean install uninstall doctor config init-config
help: ## List every target
@echo "histgen — seed a clean, logical history into a repo"
@echo
@grep -E '^[a-z-]+:.*?## .*$$' $(MAKEFILE_LIST) \
| awk 'BEGIN{FS=":.*?## "}{printf " \033[1m%-16s\033[0m %s\n", $$1, $$2}'
@echo
@echo " SOURCE=/path/to/tree what to read (read-only, never written to)"
@echo " OUT=/path/to/out what to write (plan, briefs, and OUT/<name>/)"
@echo " ARGS=... extra flags, e.g. ARGS=\"--max-files 25\""
@echo
@echo " Both can live in histgen.json instead: make init-config SOURCE=.. OUT=.."
check: ## Prove the whole pipeline works, needing no repo and nothing installed
@$(PY) $(HERE)/selftest.py
config: ## Show what repo, out and max-files resolve to
@$(HISTGEN) config $(WHERE)
init-config: ## Write a starter histgen.json beside the tool
@$(HISTGEN) config --init $(WHERE)
doctor: ## Report whether this machine can run it
@printf 'python : '; $(PY) --version 2>&1 || echo MISSING
@printf 'git : '; git --version 2>&1 || echo MISSING
@printf 'package: %s (from %s)\n' '$(PKG)' '$(PARENT)'
@$(HISTGEN) --help >/dev/null 2>&1 \
&& echo 'import : ok' || echo 'import : FAILED — is the folder intact?'
run: ## scan + plan + brief, everything before the messages are needed
@$(HISTGEN) run $(WHERE) $(ARGS)
list: ## Print the commits, to confirm before exporting
@$(HISTGEN) list $(WHERE) $(ARGS)
status: ## What is in OUT, what state it is in, and what is left to do
@$(HISTGEN) status $(WHERE) $(ARGS)
copy: ## Copy the files out: no .git, nothing ignored, no keys, no build output
@$(HISTGEN) copy $(WHERE) $(ARGS)
scan: ## Read the source and cache what was read
@$(HISTGEN) scan $(WHERE) $(ARGS)
plan: ## Order the files and cut them into commits
@$(HISTGEN) plan $(WHERE) $(ARGS)
brief: ## Write one brief per commit, for the messages
@$(HISTGEN) brief $(WHERE) $(ARGS)
against-history: ## Report how an existing history compares. Reads only
@$(HISTGEN) plan $(WHERE) --against-history $(ARGS)
dry-run: ## Write the export as a reviewable regen.sh instead of running it
@$(HISTGEN) export $(WHERE) --dry-run $(ARGS)
export: ## Copy the source into OUT and commit the history. Resumes if interrupted
@$(HISTGEN) export $(WHERE) $(ARGS)
keep: ## Export, carrying an existing history over onto its own branch
@$(HISTGEN) export $(WHERE) --keep-history $(ARGS)
commands: ## Copy the files, create no repo, print the git commands to run yourself
@$(HISTGEN) export $(WHERE) --commands $(ARGS)
verify: ## Nothing left untracked, and the exported tree matches the source
@$(HISTGEN) verify $(WHERE) $(ARGS)
clean: ## Delete the whole OUT directory. The source is not touched
@d=$$($(HISTGEN) config $(WHERE) | awk '/^out /{print $$2}'); \
test -n "$$d" -a "$$d" != "(unset)" || { echo "Error: no OUT set." >&2; exit 1; }; \
rm -rf "$$d" && echo "Removed $$d. The source was never written to."
install: ## Put a `histgen` command on PATH, pointing back at this folder
@mkdir -p $(PREFIX)/bin
@printf '#!/bin/sh\n# Generated by histgen'"'"'s Makefile; points at the folder it was run from.\nPYTHONPATH="%s" exec "%s" -m %s "$$@"\n' \
'$(PARENT)' '$(shell command -v $(PY))' '$(PKG)' > $(PREFIX)/bin/histgen
@chmod +x $(PREFIX)/bin/histgen
@echo "Installed $(PREFIX)/bin/histgen -> $(HERE)"
@case ":$$PATH:" in *":$(PREFIX)/bin:"*) ;; \
*) echo "Note: $(PREFIX)/bin is not on PATH." ;; esac
uninstall: ## Remove that command
@rm -f $(PREFIX)/bin/histgen && echo "Removed $(PREFIX)/bin/histgen"

View File

@@ -0,0 +1,545 @@
# histgen
Seeds a clean, logical history into a repo — reading one directory and writing
another, so the tree it reads is never touched.
```bash
histgen copy # just the files: no .git, nothing ignored, no keys
histgen run # scan the source, plan the commits, write the briefs
histgen list # the commits, to confirm
histgen commands # copy the files, hand back the git add/commit commands
histgen export # ...or have it do the committing itself
```
Two directories, and the whole tool hangs off the difference:
| | |
|---|---|
| **source** | the tree to read. Opened read-only, always. Not a commit, not a `.git`, not a state file is written into it — it can be a checkout you do not own or a read-only mount. |
| **out** | everything produced. The index, the plan, the briefs, and `out/<name>/` — a copy of the source with the designed history committed into it. |
That separation is what makes the history safe to argue with. It is an argument
you will have more than once, and every attempt is a directory you can delete
rather than a repo you have to put back.
Stdlib only, no network, no API key. Also readable in the browser at
`/station/tools/histgen/`, which shows a plan and never writes one.
## Copying it out
Copy the folder anywhere, `cd` into it, and use the Makefile. It derives the
package name and path from where it sits, so the directory can be renamed and
still work, and nothing has to be installed.
```bash
cp -r histgen ~/tools/ && cd ~/tools/histgen
make check # prove it works, on its own fixture
make doctor # what this machine has
make init-config SOURCE=~/work/x OUT=~/out # set both once
make run # scan + plan + brief
make list # the commits, to confirm
make export # write them into ~/out/x
make help # every target
```
One target per verb, plus `dry-run`, `against-history`, `verify` and `clean`.
Extra flags go in `ARGS`:
```bash
make plan REPO=/path/to/repo ARGS="--max-files 25"
```
`make install` drops a `histgen` command in `~/.local/bin` pointing back at the
folder, if you would rather not `cd` into it.
## Just the files, without the repo
```bash
make copy SOURCE=~/code/myproject OUT=~/clean
```
Gives you `~/clean/myproject` holding what the project actually is — no `.git`,
nothing gitignored, nothing a build regenerates, and nothing that looks like a
key. No history is planned and nothing is committed; this is the plain utility
underneath the rest.
```
14 files tracked, 7 to copy, 7 left behind.
secret — looks like a key or a credential (2):
.env
certs/server.key
ignored — tracked, but the ignore rules say they should not be (1):
data/raw/dump.sql
derived — a build regenerates these (3):
dist/bundle.js.map
dist/bundle.min.js
package-lock.json
oversize — larger than --max-bytes (1):
data/big.csv
Copied to /home/you/clean/myproject
no .git — the source has one and it was not copied
what was left behind: /home/you/clean/copied.md
```
**Every drop is named**, on screen and in `copied.md`. A file quietly missing
from a copy is the same class of failure as a file quietly missing from a
history, one directory earlier.
The same filter runs on the history path. `scan` drops secrets and
ignored-but-tracked files before they ever reach a plan, and says so:
```
Scanned 5 files: 5 read, 0 reused from cache.
3 left out of the history:
.env (looks like a key or a credential)
certs/tls.key (looks like a key or a credential)
data/dump.sql (tracked, but the ignore rules say they should not be)
```
so `make commands` and `make export` cannot commit a key that `make copy` would
have left behind. `make status` keeps saying it afterwards, and
`--keep-secrets` turns it off.
**One deliberate difference between the two.** `copy` also drops what a build
regenerates — lockfiles, maps, minified output — because a snapshot is for
reading. `scan` keeps them, because a lockfile is content in a repo somebody is
going to use. `copy --all` keeps them too.
### What gets left behind, and why
| | |
|---|---|
| **secret** | `.env`, `*.pem`, `*.key`, `id_rsa`, `.netrc`, `credentials.json`, `service-account*.json`, `.ssh/` |
| **ignored** | tracked *despite* the repo's own ignore rules — someone ran `git add -f` once |
| **derived** | lockfiles, `*.map`, `*.min.js`, `*.pyc`, `*.so`, `__pycache__/`, `node_modules/` |
| **oversize** | whatever `--max-bytes` says |
The derived list is ported from `ppl/ctrl/distill.sh`, including the lesson in
its comments: **the line is derived-vs-content, not text-vs-binary.** Images,
fonts, spreadsheets and PDFs are content and are kept — none of them can be
regenerated from what is left, which is the only thing that makes a file safe to
drop. That distinction was wrong in distill once and cost real files.
**Lookalikes are kept on purpose.** `.env.example` is the documented way to say
what the real one needs, and dropping it takes the documentation with the
secret. Same for `server.key.pub` — a public key is not a private one.
The **ignored** case is the one worth reading. `git add -f` is not always a
mistake, so these are named rather than assumed either way; but a dump or a
credentials file that went in once and was never noticed since looks exactly
like this, and the repo is already contradicting itself about them.
### Knobs
`--keep-secrets`, `--exclude` and `--include` work on `copy`, `scan` and `run`
alike, and can live in `histgen.json`. The rest are `copy`'s own.
```bash
make copy ARGS="--dry-run" # report only, write nothing
make copy ARGS="--max-bytes 100000" # leave anything bigger
make copy ARGS="--exclude '*.csv' --exclude data/"
make copy ARGS="--include package-lock.json" # keep it, whatever the filters say
make copy ARGS="--all --keep-secrets" # turn the two filters off
```
`--exclude` follows distill's rule: a pattern with no `/` matches basenames at
any depth. `--include` is checked first and wins outright, so one file can be
rescued without turning a whole filter off.
To then plan a history from the cleaned tree, point a fresh run at it:
```bash
make run SOURCE=~/clean/myproject OUT=~/history
```
## Recipe: a copy with a clean history
The common case. You have a repo, you want the same files somewhere new with a
history that reads like the thing was built on purpose, and you do not care what
the old history said.
Nothing is written to the original. The old history is simply not carried over —
that is the default, and `--keep-history` is the opt-in for when you do want it.
**`OUT` is the parent directory, not the repo.** The copy keeps the source's own
name underneath it. `OUT` does not have to exist yet; it is created on the first
command.
```bash
cd ~/tools/histgen # wherever you copied the folder
make init-config SOURCE=~/code/myproject OUT=~/clean
make config # check both paths before anything runs
```
```
source /home/you/code/myproject read-only
out /home/you/clean
exported to /home/you/clean/myproject <- the copy ends up here
```
### 1. Plan it
```bash
make run # reads the source, groups the files, writes a brief per commit
make list # the proposed commits, in order
```
`make list` is the thing to look at. Every commit is marked `*` until it has a
message. If a commit holds two unrelated ideas, move a path between groups in
`~/clean/plan.json` and run `make plan && make list` again — the grouping is a
proposal, and re-planning keeps every message whose group still holds the same
files.
### 2. Write the messages
`~/clean/briefs/` has one markdown file per commit, holding each file's opening
comment. Read them and write a `title` and `body` into each group in
`~/clean/plan.json`.
This is the part worth doing properly: the briefs carry the reasoning already in
the code, which is what makes a message worth reading. A message reconstructed
from the diff just restates the diff.
```bash
make list # again — the titles you wrote now show instead of the * marks
```
To see the shape end to end before writing any of them, use
`ARGS=--allow-untitled` in the next step; the commits get their group name as a
subject, which is fine for a throwaway pass and not fine for anything you keep.
### 3. Get the commands, and run them yourself
```bash
make commands
```
This copies the planned files into `~/clean/myproject` and **creates no repo**
no `git init`, no `.git`. What comes back is the list, also saved to
`~/clean/commands.sh`:
```
cd /home/you/clean/myproject
git init
# 01 Repo skeleton: ignore rules and line-endings policy
git add -- .gitattributes .gitignore
git commit -F /home/you/clean/messages/01-skeleton.txt
# 02 Pin the toolchain in one manifest
git add -- versions.env
git commit -F /home/you/clean/messages/02-versions.txt
...
# Worth running afterwards. The first says no file was silently
# missed; the second says the result is byte-identical to the source.
git status --porcelain
git rev-parse HEAD^{tree} # expect 3132703a817922f9f83bafa0e86e6bdf002ce8cb
```
`git init` is the first line of the list rather than something already done: a
repo that appeared without you asking is exactly what someone reaching for this
mode does not want. Read the list, edit it, reorder it, run it a line at a time.
Messages go in files rather than `-m` because bodies are multi-line, and the
body is where the *why* lives. Edit the message files directly if you want to
reword something — nothing has been committed yet.
The last two commands are worth running when you are done. `git status
--porcelain` printing nothing means no file was silently missed, which is the
failure this whole exercise exists to prevent. The tree hash matching means the
result is byte-identical to the source.
Only the files in the plan are copied: not the source's `.git`, not anything
gitignored. If `~/clean/myproject` already contains a repo, this refuses rather
than handing you commands that would commit into it.
### Or let it do the committing
```bash
make dry-run # optional: writes ~/clean/regen.sh, a script that does everything
make export # copy and commit, checking both guards itself
```
```
18 commits on main in /home/you/clean/myproject. Checking:
nothing left untracked: ok
tree matches source (3132703a8179): ok
```
Three modes, and the difference is who does what:
| | copies the files | makes the repo | commits |
|---|---|---|---|
| `make commands` | yes | no — you run `git init` | you |
| `make dry-run` | no — writes a script that would | in the script | in the script |
| `make export` | yes | yes | yes, and checks both guards |
```bash
cd ~/clean/myproject
git log --oneline
```
### If something goes wrong
```bash
make status # says which of the four states out is in, and what to do next
```
- Interrupted partway? Run `make export` again — it continues from the commit
after the last one recorded, rather than starting over or refusing. (This
applies to `make export`; with `make commands` the repo is yours, so a
half-finished run is yours to continue from the list.)
- Changed the plan after exporting? `make status` says so; `make export
ARGS=--force` discards the copy and redoes it.
- Want to start completely fresh? `make clean` deletes the whole `OUT`
directory. The source is not touched, so there is nothing to put back.
### Handing this to someone else
Everything above needs the folder, `python3` and `git` — nothing installed, no
network, no API key. Copy the directory, then:
```bash
cd histgen && make check # proves the whole pipeline on a fixture it builds
make help # every target
```
## What is already in out
`export` writes, so it starts by working out what it is writing into. Four
states, and they are genuinely different — `histgen status` prints which one:
| | |
|---|---|
| **absent** | nothing there yet. Copy the tree, init, commit. |
| **unfinished** | commits this tool made, and a record of where it stopped. Something interrupted the run. **Continue from the group after the last one recorded.** |
| **foreign** | commits this tool did *not* make. That history is someone's, so nothing is rewritten, moved or deleted: the designed account goes on its own orphan branch and the existing branch is left exactly where it was. |
| **stale** | the plan changed, or the copy moved underneath us. Refuse, and say which of the two it was. `--force` discards and starts over. |
Telling **unfinished** from **foreign** is the whole reason `progress.json`
exists. Without it both read as "there are commits here", and the tool either
destroys work it should have kept or refuses to finish work it started — which
is exactly what it used to do.
```
$ histgen status
source /home/mariano/wdir/rdir/adapter
out /home/mariano/histories/adapter
census 91 files
plan 24 commits, 6 without a message
export unfinished — 18 group(s) committed by this tool, 6 to go
committed: 1-18
remaining: 19-24
Run `export` again; it continues from where it stopped.
```
The record is written after **each** commit, not at the end — the point is to
survive the run not reaching the end. It stores the plan's fingerprint (the
groups and their paths, never the messages, so rewording commit 20 does not
invalidate the nineteen already made) and each commit's sha, which must still
be where the branch tip is or the copy has moved and it says so.
## Keeping a history that already exists
```bash
histgen export --keep-history
```
Carries the source's `.git` into the copy and commits the designed account to an
orphan branch, leaving the original branch pointing exactly where it did. Two
tiers, which is what `all/ctrl/handover.sh` has been saying all along:
```
main o-o-o-o "updates 33.1 139" (untouched)
designed-history o-o-o-o-o-o-o-o the designed account (no shared parent)
```
Both are present in `out/<name>/`; which one to publish is a decision for later
and by hand. Nothing is rewritten and the source is not touched either way.
## Settings
```bash
make init-config SOURCE=~/work/thing OUT=~/histories/thing
make config # what everything resolves to, and where it came from
make run # no arguments
```
Writes `histgen.json` beside the tool — the arrangement `ppl/ctrl/distill.sh`
already uses, where the JSON next to the script is picked up when nothing else
says otherwise.
```json
{
"source": "~/work/thing",
"out": "~/histories/thing",
"max_files": null,
"keep_history": false,
"branch": null
}
```
Precedence, most specific first:
```
the command line -> --config FILE -> histgen.json beside the tool -> defaults
```
so a config sets a starting point and never wins an argument with a flag typed
deliberately. A mistyped key is refused rather than ignored, because a setting
plainly written in the file and silently not applied is a bad thing to debug.
`out` inside `source` is refused too — the source is read-only by design, and
`out` would end up in its own census.
`histgen.json` is gitignored by the folder's own `.gitignore`: which tree this
machine points at is not a fact about the tool.
## The verbs
Each writes one file under the repo's `.histgen/`, so the step before it is
never repeated.
| | | |
|---|---|---|
| `scan` | `out/index.json` | what is in the source, cached by content hash |
| `plan` | `out/plan.json` | the order, cut into commits |
| `brief` | `out/briefs/*.md` | one pack per commit, for the messages |
| `copy` | `out/<name>/` | the files alone, no repo — needs no plan |
| `list` | — | the commits, printed to confirm |
| `export` | `out/<name>/` | the copy and its history, with both guards |
| `status` | — | which of the four states `out` is in |
| `verify` | — | the guards, on their own |
`plan.json` is the seam. Everything above it is analysis that can be recomputed
from the tree; everything below is git commands. That split is the whole design:
the expensive half is a model reading code, and it should run once.
## Where the messages come from
`brief` writes a markdown pack per commit holding each file's **opening
comment** — not the file. An agent reads the packs and writes `title` and
`body` back into `plan.json`.
That is deliberate. A commit message reconstructed from a diff restates the
diff, and the thing worth recording was never in the diff: it was in the comment
explaining why the ignore rules exist before the code they exclude, or why a
port offset has to mean the same thing in two different projects. Handing over
the reasoning that is already written down produces a message worth reading;
handing over the diff produces `Update files`.
Keeping the model outside the tool is also what keeps the tool offline, keeps
every message editable before a single commit exists, and keeps the cost of a
500-file repo to the comments rather than the code.
## The order
Role first, references second.
```
skeleton (.gitignore) -> README -> version pins -> config layer -> profiles
-> templates -> the things that source them -> front door (Makefile) LATE
-> the bootstrap account LAST
```
The front door is late because it only dispatches; the bootstrap account is last
because it narrates everything above it. References refine within that, so a
config lands before the script that sources it.
**References never override roles.** A reference in code is a dependency; a
reference in a comment is a footnote. `.gitignore` names `ctrl/wizard.sh` to say
the opposite of "I need this", and a README names every file in the repo.
Counting those as edges commits the ignore rules after the code they exclude —
consistent, and unreadable. So refs are taken from non-comment lines only, and
narrative files (`.gitignore`, README, docs, BOOTSTRAP) contribute no outgoing
edges at all.
## The grouping is a proposal
One coherent idea per commit, not one directory per commit. What holds a group
together is that its files name each other; a hub and the directory named after
it (`addons.sh` and `addons/`) always travel together, because committing a
loader without the things it loads produces a commit that cannot run.
Where it cannot know — five scripts in one directory that never mention each
other are five ideas or one, and nothing in the text says which — it guesses and
says so. **Moving a path from one group to another in `plan.json` is the
expected way to use this**, and `plan` re-run afterwards keeps every message
whose group still holds the same files.
`--max-files` sets the cap. The default is about a twentieth of the tree with a
floor of eight, which lands near how these repos were actually built — rig plans
18 against a real 18, spr 77 against a real 78. Raising it gives fewer, larger
commits; lowering it gives more.
## The two guards
`export` refuses a plan whose paths are missing, duplicated, or do not cover the
source — before it writes anything. After the last commit it checks both:
1. **nothing left untracked**, with nothing exempt. The state lives in `out`
and the copy lives inside it, so there is genuinely nothing of ours in the
tree being checked. A file silently missed is the failure this whole tool
exists to prevent. It is quiet at the time and surfaces much later, when
something does not build on a fresh clone and the history offers no clue
which commit should have carried it.
2. **the exported tree still matches the source**, by tree hash. Every path
committed is not the same claim as the same tree: a stale index, a path in
two groups, or a file edited mid-plan all pass the first check and fail this
one.
`--dry-run` writes `out/regen.sh` and `out/messages/` instead — ordinary git
commands that copy the files and make the commits, both guards included,
reviewable before anything runs.
## Reporting on a history that already exists
```bash
python -m station.tools.histgen plan /path/to/repo --against-history
```
Maps each of the source's existing commits onto the group holding most of the files it touched,
then reports what agrees and what does not:
```
= 01 skeleton a127b1d matched one commit
~ 03 ctrl split across 2 one idea, committed piecemeal
+ 04 ctrl-lib no commit never landed as its own change
! 4 commit(s) land earlier in the proposed order than work already done
? 31 commit(s) carry no usable account of the change ("updates 33.1 139")
```
It reads and prints. It never rewrites: published history is someone else's
clone.
## Verified against
`rig`'s 18-commit history, which was built by hand and is what this reproduces.
Run over the same 64 files, histgen plans 18 commits; the profiles, the cluster
templates, the addons hub, the k8s manifests, the Makefile, `sample-rig` and
`BOOTSTRAP.md` all land as their own commits in the same places. Replaying it
produces a tree hash identical to the one rig ships.
Where it differs is where the difference is semantic: rig splits its wizard,
host checks, cluster lifecycle and registry into four commits, and nothing in
those four files' text says they are four things.
## The CLI shape
`cli.py` is a shared scaffold — subcommand registration, one spelling for
`--source/-o/-n/--force/--dry-run`, `Error: … -> stderr -> exit 1` as the only
exit path, deferred heavy imports so `--help` stays instant, and
`refuse_to_clobber`. It exists because every tool here grew its own slightly
different copy of the same three things.
histgen is its first user. Nothing else was rewritten to use it: a scaffold
earns adoption by being there when the next tool is written.

View File

@@ -0,0 +1,27 @@
"""
Histgen — seed a clean, logical history into a repo.
The general case is a tree with no git at all: read the code, then commit it in
parts, ordered so each commit stands on what came before. Done by hand it means
uploading everything and asking a model; done twice it stops being worth the
time.
The expensive half (reading the code) and the mechanical half (applying a plan)
are split on purpose, and `plan.json` is the seam. Everything before it is
analysis and can be cached; everything after it is git commands that fail fast.
python -m station.tools.histgen scan /path/to/repo
python -m station.tools.histgen plan /path/to/repo
python -m station.tools.histgen brief /path/to/repo
# an agent reads briefs/ and writes title+body back into plan.json
python -m station.tools.histgen apply /path/to/repo --dry-run
python -m station.tools.histgen verify /path/to/repo
Stdlib only, no network. The directory can be copied out of soleprint and run
on its own — a repo that needs a history seeded is, by definition, not one that
already has this framework on its path.
"""
__version__ = "0.1.0"
__all__ = ["census", "order", "brief", "snapshot", "export"]

View File

@@ -0,0 +1,318 @@
"""
Histgen CLI — seed a clean, logical history into a repo.
python -m station.tools.histgen run --source ~/work/thing --out ~/out
python -m station.tools.histgen list # the commits, to confirm
python -m station.tools.histgen export # write them into out/thing
Two directories and one rule: the source is read-only, everything is written
under out. Set both once with `config --init` and the verbs take no arguments.
Run from the soleprint/ directory so `station.tools...` resolves, or copy the
folder out and use its Makefile.
"""
import sys
from pathlib import Path
from .cli import Tool, fail
VERBS = (
("copy", "Copy the files out, with no .git and nothing private."),
("scan", "Read the source and cache what was read."),
("plan", "Order the files and cut them into commits."),
("list", "Print the commits, to confirm before exporting."),
("brief", "Write one brief per commit, for the messages."),
("export", "Copy the source into out and commit the history."),
("verify", "Nothing left untracked, and the tree still matches."),
("status", "What is in the out directory, and what is left to do."),
("run", "scan + plan + brief."),
("config", "Show the resolved settings, or write a starter file."),
)
def _settings(args, need_out=True):
"""
Where to read and where to write, with the command line on top.
Resolved once per invocation and passed down, rather than each module
working it out again — two places deciding where things live is how `scan`
and `plan` end up disagreeing about it.
"""
from . import config
s = config.resolve(args, getattr(args, "config", None))
if not s["source"]:
fail("No source given.",
f"Pass --source, or set it in {config.default_path()} "
"(see `histgen config --init`).")
if not s["source"].is_dir():
fail(f"Not a directory: {s['source']}")
if need_out and not s["out"]:
fail("No out directory given.",
"Pass --out, or set it in the config. It is where the plan and "
"the exported repo go; the source is never written to.")
if s["out"]:
# The source must stay clean, and an out inside it would be scanned as
# part of the tree it describes on the very next run.
try:
if s["out"] == s["source"] or s["out"].is_relative_to(s["source"]):
fail("out is inside source.",
"The source is read-only by design, and out would end up "
"in its own census. Put out somewhere else.")
except (OSError, ValueError):
pass
return s
def _index(out):
"""The census, which must already exist — scanning is its own verb."""
from . import census
index = census.load_index(out)
if not index.get("files"):
fail(f"No census at {census.state_path(out)}.", "Run `scan` first.")
return index
def cmd_copy(args):
"""The plain utility: files out, repo and private things left behind."""
from . import snapshot
s = _settings(args)
snapshot.take(s["source"], s["out"],
keep_noise=args.all, keep_secrets=s["keep_secrets"],
max_bytes=args.max_bytes, exclude=s["exclude"],
include=s["include"], force=args.force, dry_run=args.dry_run)
def cmd_scan(args):
from . import census
s = _settings(args)
census.scan(s["source"], s["out"], keep_secrets=s["keep_secrets"],
exclude=s["exclude"], include=s["include"])
def cmd_plan(args):
from . import order
s = _settings(args)
plan = order.build_plan(_index(s["out"]), s["out"], max_files=s["max_files"])
if args.against_history:
from . import history
print()
history.compare(s["source"], plan)
def cmd_list(args):
"""
Print the commits as a numbered list, which is the thing to confirm.
Deliberately the plainest output here: a number, a title, and the files
under it. Deciding whether a commit is one idea is done by reading it, and
anything else on the line is in the way.
"""
from . import export as exporter
s = _settings(args)
plan = exporter.load_plan(s["out"])
files = _index(s["out"])["files"] if args.roles else {}
for g in plan["groups"]:
title = (g.get("title") or "").strip()
mark = " " if title else "*"
print(f"{mark}{g['n']:3}. {title or g['slug'] + ' (no message yet)'}")
for p in g["paths"]:
role = f" [{files[p]['role']}]" if args.roles and p in files else ""
print(f" {p}{role}")
untitled = [g["n"] for g in plan["groups"] if not (g.get("title") or "").strip()]
print(f"\n{len(plan['groups'])} commits, "
f"{sum(len(g['paths']) for g in plan['groups'])} files.")
if untitled:
print(f"* {len(untitled)} still without a message — read "
f"{s['out']}/briefs/ and write them into {s['out']}/plan.json.")
def cmd_brief(args):
from . import brief, order
s = _settings(args)
if not order.plan_path(s["out"]).exists():
fail(f"No plan at {order.plan_path(s['out'])}.", "Run `plan` first.")
brief.write_briefs(s["out"])
def cmd_export(args):
from . import export as exporter
s = _settings(args)
exporter.export(s["source"], s["out"], dry_run=args.dry_run,
commands=args.commands, allow_untitled=args.allow_untitled,
keep_history=s["keep_history"], branch=s["branch"],
force=args.force)
def cmd_verify(args):
from . import export as exporter
s = _settings(args)
plan = exporter.load_plan(s["out"])
planned = [p for g in plan["groups"] for p in g["paths"]]
copy = exporter.repo_dir(s["source"], s["out"])
if not copy.is_dir():
fail(f"Nothing exported at {copy}.", "Run `export` first.")
print("Checking:", flush=True)
if not exporter.verify(copy, exporter.source_tree_hash(s["source"], planned)):
sys.exit(1)
def cmd_status(args):
"""What is in out, what state it is in, and what is left."""
from . import census, export as exporter, order
s = _settings(args)
index = census.load_index(s["out"])
plan = None
if order.plan_path(s["out"]).exists():
plan = exporter.load_plan(s["out"])
print(f"source {s['source']}")
print(f"out {s['out']}")
if index.get("files"):
left = index.get("left_out", [])
print(f"census {len(index['files'])} files"
+ (f", {len(left)} left out" if left else ""))
for item in left:
print(f" left out: {item['path']} ({item['why']})")
else:
print("census none — run `scan`")
if plan:
untitled = [g["n"] for g in plan["groups"] if not (g.get("title") or "").strip()]
print(f"plan {len(plan['groups'])} commits"
+ (f", {len(untitled)} without a message" if untitled else
", all messages written"))
else:
print("plan none — run `plan`")
report = exporter.inspect(s["source"], s["out"], plan)
print(f"export {report['state']}{report['detail']}")
if report["done"]:
print(f" committed: {_ranges(report['done'])}")
if report["remaining"]:
print(f" remaining: {_ranges(report['remaining'])}")
advice = {
"absent": "Run `export`.",
"unfinished": "Run `export` again; it continues from where it stopped.",
"foreign": "Run `export`; the designed history goes on its own branch "
"and nothing existing is touched.",
"stale": "Run `export --force` to discard and redo, or point --out elsewhere.",
"complete": "Nothing to do.",
}
print(f"\n{advice.get(report['state'], '')}")
def _ranges(numbers):
"""[1,2,3,7,8] -> '1-3, 7-8'. A list of eighteen numbers is unreadable."""
if not numbers:
return "none"
out, start, previous = [], numbers[0], numbers[0]
for n in numbers[1:] + [None]:
if n == previous + 1:
previous = n
continue
out.append(str(start) if start == previous else f"{start}-{previous}")
start = previous = n
return ", ".join(out)
def cmd_run(args):
from . import brief, census, order
s = _settings(args)
index = census.scan(s["source"], s["out"], keep_secrets=s["keep_secrets"],
exclude=s["exclude"], include=s["include"])
order.build_plan(index, s["out"], max_files=s["max_files"])
brief.write_briefs(s["out"])
def cmd_config(args):
from . import config
if args.init:
path = config.write_template(
Path(args.config).expanduser() if args.config else config.default_path(),
source=args.source, out=args.out)
print(f"Wrote {path}. Edit \"source\" and \"out\".")
return
s = config.resolve(args, args.config)
print(f"config {s['config_path'] or '(none; using defaults)'}")
print(f"source {s['source'] or '(unset)'} read-only")
print(f"out {s['out'] or '(unset)'}")
if s["source"] and s["out"]:
from .export import repo_dir
print(f"exported to {repo_dir(s['source'], s['out'])}")
print(f"max-files {s['max_files'] or '(scales with the source)'}")
print(f"keep-history {s['keep_history']}")
print(f"keep-secrets {s['keep_secrets']}"
+ ("" if s["keep_secrets"] else " keys and credentials are left out"))
if s["exclude"]:
print(f"exclude {', '.join(s['exclude'])}")
if s["include"]:
print(f"include {', '.join(s['include'])}")
def main(argv=None):
tool = Tool("histgen", __doc__, package=__package__)
handlers = {name: globals()[f"cmd_{name}"] for name, _ in VERBS}
for verb, help_text in VERBS:
tool.command(verb, handlers[verb], help_text)
tool.argument(verb, "--source", "-s", default=None, metavar="DIR",
help="The tree to read. Never written to.")
tool.argument(verb, "--out", "-o", default=None, metavar="DIR",
help="Where the plan and the exported repo go.")
tool.argument(verb, "--config", "-c", default=None, metavar="FILE",
help="Settings file, instead of histgen.json beside the tool.")
for verb in ("plan", "run"):
tool.argument(verb, "--max-files", type=int, default=None, metavar="N",
help="Files per commit before a group is cut. Default "
"scales with the source (about N/20, floor 8).")
tool.argument("plan", "--against-history", action="store_true",
help="Also report how the source's existing history compares.")
tool.argument("list", "--roles", action="store_true",
help="Show each file's detected role.")
# Whatever reads the source can filter it, so the same three answers hold
# for a snapshot and for a history. Keeping them on `copy` alone was how a
# tracked key stayed out of one and went straight into the other.
for verb in ("copy", "scan", "run"):
tool.argument(verb, "--keep-secrets", action="store_true",
help="Keep files that look like keys or credentials.")
tool.argument(verb, "--exclude", action="append", default=[], metavar="GLOB",
help="Leave these out. Repeatable; a pattern with no / "
"matches basenames at any depth.")
tool.argument(verb, "--include", action="append", default=[], metavar="GLOB",
help="Keep these whatever the filters say. Repeatable.")
tool.common("copy", "dry_run")
tool.argument("copy", "--all", action="store_true",
help="Keep what a build regenerates too: lockfiles, maps, "
"minified and compiled output.")
tool.argument("copy", "--max-bytes", type=int, default=None, metavar="N",
help="Leave behind anything larger.")
tool.argument("copy", "--force", action="store_true",
help="Write into a destination that is not empty.")
tool.common("export", "dry_run")
tool.argument("export", "--commands", action="store_true",
help="Copy the files, create no repo, and print the git add "
"and git commit commands to run yourself.")
tool.argument("export", "--allow-untitled", action="store_true",
help="Commit groups whose message was never written.")
tool.argument("export", "--keep-history", action="store_true",
help="Carry the source's existing history into the copy, and "
"put the designed one on its own branch.")
tool.argument("export", "--branch", default=None, metavar="NAME",
help="Branch for the designed history when one is kept.")
tool.argument("export", "--force", action="store_true",
help="Discard an out directory that no longer matches the plan.")
tool.argument("config", "--init", action="store_true",
help="Write a starter config file rather than reading one.")
tool.run(argv)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,98 @@
"""
Histgen's HTTP surface.
Read-only on purpose. The CLI seeds histories; this shows what a plan looks
like before anyone runs it, because the interesting failure — a group that
holds two unrelated ideas — is one you see by reading, not by testing.
Nothing here writes commits. A browser tab is the wrong place to decide that a
repository's history is about to be rebuilt.
"""
import logging
from pathlib import Path
from fastapi import APIRouter, HTTPException
from fastapi.responses import HTMLResponse
log = logging.getLogger(__name__)
router = APIRouter(prefix="/tools/histgen", tags=["histgen"])
SPR_ROOT = Path(__file__).parents[3]
HERE = Path(__file__).parent
# ─────────────────────────────────────────────────────────────────────────────
# Pages
# ─────────────────────────────────────────────────────────────────────────────
@router.get("/", response_class=HTMLResponse)
def index():
template = HERE / "templates" / "index.html"
if template.exists():
return template.read_text()
return "<h1>histgen</h1>"
@router.get("/health")
def health():
return {"status": "ok", "tool": "histgen"}
# ─────────────────────────────────────────────────────────────────────────────
# API
# ─────────────────────────────────────────────────────────────────────────────
def _resolve(out: str) -> Path:
"""
An out directory, kept inside the tree this instance was built from.
The parameter is a filesystem path from a query string, so it is the one
input here worth distrusting: without the containment check, `../..` reads
any directory the server can.
"""
target = Path(out).resolve() if Path(out).is_absolute() else (SPR_ROOT.parent / out).resolve()
root = SPR_ROOT.parent.resolve()
if root not in target.parents and target != root:
raise HTTPException(400, f"Outside the tree: {out}")
if not target.is_dir():
raise HTTPException(404, f"Not a directory: {out}")
return target
@router.get("/api/plan")
def get_plan(out: str):
"""The plan as it stands, with each group's files and message."""
import json
from .order import plan_path
path = plan_path(_resolve(out))
if not path.exists():
raise HTTPException(404, "No plan yet. Run `histgen run`.")
plan = json.loads(path.read_text())
return {
"groups": plan["groups"],
"commits": len(plan["groups"]),
"files": sum(len(g["paths"]) for g in plan["groups"]),
"untitled": [g["n"] for g in plan["groups"] if not (g.get("title") or "").strip()],
}
@router.get("/api/census")
def get_census(out: str):
"""What the scan found, without the per-file detail."""
from collections import Counter
from .census import load_index
index = load_index(_resolve(out))
if not index.get("files"):
raise HTTPException(404, "No census yet. Run `scan` first.")
files = index["files"]
return {
"files": len(files),
"roles": dict(Counter(f["role"] for f in files.values())),
"edges": sum(len(f["refs"]) for f in files.values()),
}

View File

@@ -0,0 +1,111 @@
"""
One pack per commit, for whoever writes the message.
This is the interface to the expensive reader, and it is a directory of
markdown rather than a network call. The tool does not know how to reach a
model and does not want to: an agent already in this repo reads the briefs and
writes titles and bodies back into plan.json, which keeps the messages editable
before a single commit exists and keeps an API key out of a tool that otherwise
runs offline.
What travels is the reasoning already in the code — each file's opening
comment — and not the file. That is both what makes the pack cheap and what
makes the message right: a commit message reconstructed from a diff restates
the diff, and the thing worth recording was never in the diff. It was in the
comment explaining why the port offsets have to match another project's, or why
the ignore rules exist before the code they exclude.
"""
import json
from .census import INDEX_FILE, state_dir
from .order import plan_path
BRIEF_DIR = "briefs"
WHY_CHARS = 700 # an opening comment past this is an essay; the head carries it
MAX_LISTED = 40
HEADER = """# {n:02d}{slug}
**{count} file(s), commit {n} of {total}.**
Write a title and a body for this commit, then put them in
`{plan}` under group {n} as `"title"` and `"body"`.
- The title says what this commit establishes, in the repo's own words.
- The body carries the **why** — take it from the reasoning already in the
comments below. Do not restate the diff; the diff is already in the commit.
- If a file below does not belong in this commit, move its path to another
group in plan.json. The grouping is a proposal.
"""
def _fmt_why(text):
text = (text or "").strip()
if not text:
return "_(no opening comment)_"
if len(text) > WHY_CHARS:
text = text[:WHY_CHARS].rsplit("\n", 1)[0] + "\n"
return "\n".join("> " + line if line.strip() else ">" for line in text.split("\n"))
def write_briefs(out, quiet=False):
state = state_dir(out)
plan = json.loads(plan_path(out).read_text())
index = json.loads((state / INDEX_FILE).read_text())
files = index["files"]
groups = plan["groups"]
# Which commit each path lands in, so a dependency can be named by the
# commit that introduced it rather than by a bare path. "stands on 04" is
# the sentence the ordering exists to make true.
landed = {p: g["n"] for g in groups for p in g["paths"]}
out_dir = state / BRIEF_DIR
out_dir.mkdir(parents=True, exist_ok=True)
for stale in out_dir.glob("*.md"):
stale.unlink()
for g in groups:
lines = [HEADER.format(n=g["n"], slug=g["slug"], count=len(g["paths"]),
total=len(groups), plan=plan_path(out))]
earlier, later = {}, set()
for p in g["paths"]:
for dep in files.get(p, {}).get("refs", []):
n = landed.get(dep)
if n is None or dep in g["paths"]:
continue
(earlier.setdefault(n, set()).add(dep) if n < g["n"] else later.add(dep))
if earlier:
lines.append("\n## Stands on\n")
for n in sorted(earlier):
names = ", ".join(f"`{d}`" for d in sorted(earlier[n])[:MAX_LISTED])
lines.append(f"- commit {n:02d}: {names}")
if later:
# Worth stating plainly rather than hiding: it is the one thing a
# reader of the finished history would notice and the tool cannot
# fix, because the fix is a judgement about which comes first.
names = ", ".join(f"`{d}`" for d in sorted(later)[:MAX_LISTED])
lines.append("\n## Forward references (this commit names things not yet committed)\n")
lines.append(f"- {names}")
lines.append("\n## Files\n")
for p in g["paths"]:
e = files.get(p, {})
meta = f"{e.get('role', '?')}, {e.get('lines', 0)} lines"
if e.get("binary"):
meta += ", binary"
lines.append(f"\n### `{p}`\n\n_{meta}_\n")
lines.append(_fmt_why(e.get("why")))
path = out_dir / f"{g['n']:02d}-{g['slug']}.md"
path.write_text("\n".join(lines) + "\n")
if not quiet:
print(f"Wrote {len(groups)} briefs -> {out_dir}")
print(f"Read them, then write title and body into {plan_path(out)}.")
return out_dir

View File

@@ -0,0 +1,456 @@
"""
What is in the tree, and what each file says about itself.
This is the expensive half. Walking is cheap; reading is not, so what gets read
is cached by content hash and a second run costs only the files that changed.
The cache lives beside the output, in the repo's own .histgen/, because a
destination that carries its own state cannot be confused with another one's.
Two things are extracted from every file, and both are used twice:
the opening comment why the file exists, in the author's words. `order`
does not read it; `brief` hands it to whoever writes
the commit message, because that reasoning is the
message. A commit that restates its own diff is noise.
path references which other files this one names. `order` turns them
into edges, so a config lands before the script that
sources it. `brief` reports them as the seam between
one commit and the last.
"""
import hashlib
import json
import os
import re
import subprocess
import tempfile
from pathlib import Path
STATE_DIR = ".histgen"
INDEX_FILE = "index.json"
# Read caps. A file's opening comment is at the top by definition, and nothing
# below the cap has ever been the reason a file exists. The byte cap is what
# keeps a vendored 2 MB bundle from being tokenised for no reason.
HEAD_LINES = 60
MAX_BYTES = 400_000
# ── the file set ───────────────────────────────────────────────────────────
#
# Never parse .gitignore. It has negations, directory semantics, precedence
# across nested files and a global excludes file, and a hand-rolled parser that
# gets 95% of that right is worse than none: it is wrong silently, on exactly
# the files someone took care to exclude. Ask git, which is always installed
# here because the output of this tool is a git repository.
def _git(args, **kw):
return subprocess.run(["git", *args], capture_output=True, text=True, **kw)
def is_git(path: Path) -> bool:
r = _git(["-C", str(path), "rev-parse", "--git-dir"])
return r.returncode == 0
def file_set(source: Path):
"""
The paths a history would contain, relative to the source, sorted.
A source with git is asked what it tracks — the same question handover.sh
asks, and for the same reason: a hand-maintained list drifts, and the drift
shows up as a file that silently never got committed.
A source with NO git is the general case, and the interesting one. Rather
than reimplementing the ignore rules, git is pointed at the tree with its
own directory kept in a temporary path: `ls-files -o --exclude-standard`
then means untracked-and-not-ignored, which is exactly the candidate set.
Nothing is written inside the tree, so a dry run leaves no .git behind to
explain later.
"""
if is_git(source):
listed = _git(["-C", str(source), "ls-files", "-z"]).stdout
paths = [p for p in listed.split("\0") if p]
# A tracked file that has been deleted but not committed is still in
# ls-files. It would abort `apply` partway through with a missing path,
# so drop it here and let `verify` be the thing that complains.
return sorted(p for p in paths if (source / p).is_file())
with tempfile.TemporaryDirectory(prefix="histgen-git-") as tmp:
env = dict(os.environ, GIT_DIR=str(Path(tmp) / "git"), GIT_WORK_TREE=str(source))
subprocess.run(["git", "init", "-q"], env=env, capture_output=True)
listed = subprocess.run(
["git", "ls-files", "-o", "--exclude-standard", "-z"],
env=env, capture_output=True, text=True,
).stdout
return sorted(p for p in listed.split("\0")
if p and (source / p).is_file())
# ── roles ──────────────────────────────────────────────────────────────────
#
# A file's role is what decides where it lands when nothing references it, and
# most files reference nothing. The ranks are the ordering heuristic itself,
# read bottom-up off rig's log: ignore rules and README first, then the config
# layer, then the things that source it, the front door late because it only
# dispatches, and the bootstrap account last because it narrates the rest.
ROLE_RANK = {
"skeleton": 0, # .gitignore, .gitattributes — the rules before the files
"readme": 10, # what this is, and the one prerequisite
"pin": 20, # versions, dependency manifests
"config": 30, # the layer everything else reads
"profile": 40, # named variants of that config
"template": 50, # shapes rendered later
"source": 60, # the work
"test": 70,
"asset": 76,
"lock": 78, # generated from a pin; never interesting, never first
"doc": 80,
"frontdoor": 90, # Makefile, Tiltfile — a dispatcher, so it comes after
"bootstrap": 100, # BOOTSTRAP/INSTALL — the account of everything above
}
_SKELETON = {".gitignore", ".gitattributes", ".editorconfig", ".dockerignore",
"license", "license.md", "license.txt", "copying", "notice"}
_FRONTDOOR = {"makefile", "gnumakefile", "tiltfile", "justfile", "taskfile.yml",
"dockerfile", "docker-compose.yml", "docker-compose.yaml"}
_LOCK = {"package-lock.json", "poetry.lock", "pnpm-lock.yaml", "yarn.lock",
"cargo.lock", "go.sum", "composer.lock", "gemfile.lock", "uv.lock"}
_PIN = {"requirements.txt", "pyproject.toml", "package.json", "go.mod",
"cargo.toml", "gemfile", "versions.env", "setup.py", "setup.cfg"}
_ASSET_EXT = {".png", ".jpg", ".jpeg", ".gif", ".ico", ".svg", ".webp", ".pdf",
".woff", ".woff2", ".ttf", ".eot", ".mp4", ".zip", ".ods", ".xlsx",
".bundle", ".so", ".dylib", ".dll", ".wasm"}
_DOC_EXT = {".md", ".rst", ".txt", ".adoc"}
def ignored_but_tracked(source: Path, paths):
"""
Files git tracks that the ignore rules say it should not.
`git add -f` is how they get there, and it is not always a mistake — a
built artifact committed on purpose looks exactly like this. But so does a
dump, a credentials file or a data directory that someone forced in once
and nobody noticed since, and those are the ones worth catching.
`--no-index` is the whole trick: without it check-ignore stays quiet about
anything already tracked, which is precisely the set being asked about.
"""
if not paths or not is_git(source):
return set()
r = subprocess.run(
["git", "-C", str(source), "check-ignore", "--no-index", "--stdin", "-z"],
input="\0".join(paths) + "\0", text=True, capture_output=True)
# 0 = some matched, 1 = none matched, anything else is a real failure and
# not a reason to refuse to copy.
if r.returncode not in (0, 1):
return set()
return {p for p in r.stdout.split("\0") if p}
def role_of(path: str) -> str:
p = Path(path)
name, low = p.name, p.name.lower()
parts = [s.lower() for s in p.parts]
stem = p.stem.lower()
if low in _SKELETON:
return "skeleton"
if low in _LOCK:
return "lock"
if low in _FRONTDOOR or low.startswith("dockerfile"):
return "frontdoor"
if stem in ("bootstrap", "install", "installing", "getting-started", "quickstart"):
return "bootstrap"
if stem == "readme":
# Only the repo's own README opens the history. A README inside a
# subdirectory documents that subdirectory and travels with it.
return "readme" if len(p.parts) == 1 else "doc"
if low in _PIN or low.endswith(".lock"):
return "lock" if low.endswith(".lock") else "pin"
if p.suffix.lower() in _ASSET_EXT:
return "asset"
if "test" in parts or "tests" in parts or stem.startswith("test_") or stem.endswith("_test"):
return "test"
if p.suffix in (".tpl", ".tmpl", ".j2", ".mustache") or low.endswith((".yaml.tpl", ".tmpl")):
return "template"
if "templates" in parts:
return "template"
# A profile is a named variant sitting in a directory of siblings: env.d/,
# profiles/, overlays/. The directory is the signal, not the extension.
if any(d in parts for d in ("env.d", "profiles", "environments")):
return "profile"
if stem in ("config", "settings", "conf", "defaults") or low in (".env.example", "env.example"):
return "config"
if low.endswith(".env") or low.endswith(".env.example"):
return "profile"
if p.suffix.lower() in _DOC_EXT or "docs" in parts or "doc" in parts:
return "doc"
return "source"
# ── what a file says about itself ──────────────────────────────────────────
_COMMENT = {
"#": (".sh", ".bash", ".py", ".yaml", ".yml", ".toml", ".env", ".cfg", ".conf", ".tf", ""),
"//": (".js", ".ts", ".jsx", ".tsx", ".go", ".java", ".c", ".h", ".cpp", ".rs", ".scala"),
}
def opening_comment(text: str, path: str) -> str:
"""
The comment block at the top of the file, or the module docstring.
The shebang and any editor modeline are skipped — they are not prose. The
block ends at the first line that is not a comment, which is what makes it
the file's own statement of purpose rather than a running commentary.
"""
lines = text.split("\n")[:HEAD_LINES]
i = 0
while i < len(lines) and (
lines[i].startswith("#!") or not lines[i].strip()
or lines[i].lstrip().startswith(("# -*-", "# vim:", "# shellcheck"))
):
i += 1
# A docstring: take it whole, it is the same statement in another syntax.
rest = "\n".join(lines[i:]).lstrip()
for quote in ('"""', "'''"):
if rest.startswith(quote):
end = rest.find(quote, len(quote))
if end != -1:
return rest[len(quote):end].strip()
suffix = Path(path).suffix.lower()
markers = [m for m, exts in _COMMENT.items() if suffix in exts] or ["#"]
block = []
for line in lines[i:]:
stripped = line.strip()
if not any(stripped.startswith(m) for m in markers):
break
for m in markers:
if stripped.startswith(m):
block.append(stripped[len(m):].strip())
break
return "\n".join(block).strip()
_TOKEN = re.compile(r"[A-Za-z0-9_./+-]{4,}")
# What opens a comment, by language. Used to tell a reference apart from a
# mention, which is the difference between an edge and a footnote.
_COMMENT_PREFIX = ("#", "//", "--", ";", "*", "/*")
def _strip_comment(line: str) -> str:
stripped = line.strip()
if stripped.startswith(_COMMENT_PREFIX):
return ""
# A trailing comment on a real line: keep the code, drop the aside.
for marker in (" #", " //"):
cut = line.find(marker)
if cut != -1:
line = line[:cut]
return line
def referenced_paths(text: str, path: str, by_path, by_base):
"""
Which other files in this repo this one names, split by how it names them.
Deliberately textual rather than per-language. A shell `source
"$DIR/lib/config.sh"`, a Makefile's `ctrl/cluster.sh`, a kustomization's
`- namespace.yaml` and a Dockerfile's `COPY run.py .` are all the same fact
— this file needs that one — and four parsers would find it four ways and
disagree at the edges.
The split matters more than the matching does. A reference in code is a
dependency: config.sh has to exist before the script that sources it. A
reference in a comment is a footnote — .gitignore names `ctrl/wizard.sh`
to say the opposite of "I need this", and README names every file in the
repo. Counting those as edges puts the ignore rules after the code they
exclude, which is exactly backwards. So prose informs the brief and never
the order.
Bare filenames only count when they are unique in the repo, and only with
their extension. Without both, every `config.py` in a tree of them becomes
an edge to all the others and the ordering collapses into one cycle.
"""
def hits(blob):
found = set()
for token in set(_TOKEN.findall(blob)):
token = token.strip("./")
if not token:
continue
if token in by_path:
found.update(by_path[token])
continue
# A bare name is only a reference if it carries an extension and
# names exactly one file. "cluster" is a word; "cluster.sh" is not.
if "." in token:
candidates = by_base.get(token)
if candidates and len(candidates) == 1:
found.update(candidates)
found.discard(path)
return found
lines = text.split("\n")
code = "\n".join(_strip_comment(l) for l in lines)
strong = hits(code)
mentions = hits(text) - strong
return sorted(strong), sorted(mentions)
def read_text(full: Path):
"""Text, or None if this is not text. Size is checked before reading."""
try:
if full.stat().st_size > MAX_BYTES:
return None
raw = full.read_bytes()
except OSError:
return None
if b"\0" in raw[:8000]:
return None
return raw.decode("utf-8", errors="replace")
# ── the index ──────────────────────────────────────────────────────────────
def _digest(full: Path) -> str:
h = hashlib.sha256()
with full.open("rb") as fh:
for chunk in iter(lambda: fh.read(65536), b""):
h.update(chunk)
return h.hexdigest()[:16]
def state_dir(out) -> Path:
"""
Where this run's index, plan, briefs and messages live.
Always `out`, never the source. The source is opened read-only and nothing
is written into it — which is what removed a whole class of special cases
that used to be here: skipping the state directory during its own census,
exempting it from `git status`, and writing a .git/info/exclude entry to
keep it quiet. None of that has anywhere to happen now.
"""
return Path(out)
def state_path(out) -> Path:
return state_dir(out) / INDEX_FILE
def current_file_set(source: Path, index) -> set:
"""
What the source holds right now, filtered the way the census was.
Recomputed rather than read back, because the guard it feeds exists to
catch a file added after the scan. Reading the stored list would answer the
easy question — "did the plan cover what we saw?" — instead of the one
worth asking, which is "does the plan cover what is there?".
"""
from .sift import sift
settings = (index or {}).get("filter", {})
tracked = file_set(source)
kept, _ = sift(source, tracked,
keep_noise=True,
keep_secrets=settings.get("keep_secrets", False),
exclude=settings.get("exclude", ()),
include=settings.get("include", ()),
ignored=ignored_but_tracked(source, tracked))
return set(kept)
def load_index(out) -> dict:
p = state_path(out)
if not p.exists():
return {}
try:
return json.loads(p.read_text())
except (OSError, json.JSONDecodeError):
# A half-written cache is a cache miss, not a crash. Rebuilding costs
# one run; refusing to start costs an explanation.
return {}
def scan(source: Path, out, keep_secrets=False, exclude=(), include=(),
quiet=False) -> dict:
"""
Census the tree, reusing everything whose content has not changed.
The reuse is per file and keyed on content, not on mtime: a checkout, a
branch switch or a `touch` all move mtimes without changing a byte, and
re-reading the whole tree because git rewrote it is the cost this exists to
avoid.
"""
from .sift import REASONS, sift
# The filter runs here, not at export time, so a key never reaches a plan
# in the first place. Secrets and files the repo's own ignore rules
# contradict are dropped; what a build regenerates is NOT — a lockfile is
# content in a repo somebody is going to use, however little it says.
# `copy` is the one that drops those, because a snapshot is for reading.
tracked = file_set(source)
paths, left_out = sift(source, tracked, keep_noise=True,
keep_secrets=keep_secrets, exclude=exclude,
include=include,
ignored=ignored_but_tracked(source, tracked))
previous = load_index(out).get("files", {})
by_path, by_base = {}, {}
for p in paths:
by_path.setdefault(p, []).append(p)
by_base.setdefault(Path(p).name, []).append(p)
files, reused = {}, 0
for rel in paths:
full = source / rel
try:
digest = _digest(full)
except OSError:
continue
old = previous.get(rel)
if old and old.get("hash") == digest and "mentions" in old:
files[rel] = old
reused += 1
continue
text = read_text(full)
refs, mentions = ([], []) if text is None else referenced_paths(text, rel, by_path, by_base)
entry = {
"hash": digest,
"size": full.stat().st_size,
"role": role_of(rel),
"binary": text is None,
"why": "" if text is None else opening_comment(text, rel),
"refs": refs, # in code: a dependency, and an edge
"mentions": mentions, # in prose: context for the brief, never an edge
}
entry["lines"] = 0 if text is None else text.count("\n") + 1
files[rel] = entry
# The filter settings travel with the index so the guards can re-derive
# the same set later. Without them `check_plan` has to choose between
# trusting a stale list and flagging every filtered file as missing.
index = {"version": 1, "source": str(source), "files": files,
"filter": {"keep_secrets": bool(keep_secrets),
"exclude": list(exclude), "include": list(include)},
"left_out": [{"path": p, "why": w} for p, w in left_out]}
destination = state_path(out)
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_text(json.dumps(index, indent=2, sort_keys=True))
if not quiet:
fresh = len(files) - reused
print(f"Scanned {len(files)} files: {fresh} read, {reused} reused from cache.")
if left_out:
# Never silent. A key that was tracked is a thing to know about,
# and it stays true after the file stops travelling.
print(f" {len(left_out)} left out of the history:")
for rel, why in left_out:
print(f" {rel} ({REASONS[why]})")
print(f" -> {destination}")
return index

View File

@@ -0,0 +1,132 @@
"""
The shape a station tool's command line has.
Every tool here grew its own copy of the same three things: argparse subcommands
wired to `cmd_*` functions, a flag vocabulary that is nearly but not quite the
same between tools, and an error convention. shuntgen spells `-s` as --spec in
one subcommand and --source in another; modelgen calls the same idea --source
everywhere; tester has neither. The differences are accidents, not decisions.
This is that shared shape, factored out. histgen is the first user. Nothing else
is rewritten to use it — a scaffold earns adoption by being there when the next
tool is written, not by a flag-day.
from .cli import Tool, fail
tool = Tool("histgen", __doc__)
tool.command("scan", cmd_scan, "Read the repo and cache what was read.")
tool.argument("scan", "repo", help="The tree to read.")
tool.run()
Conventions it encodes, so they stop being re-decided:
--source/-s what to read --output/-o where to write
--name/-n what to call it --force/-f write anyway
--dry-run print, do not do
Progress goes to stdout as plain print(). Errors go to stderr prefixed
'Error: ' and exit 1 — never a traceback, which tells a user nothing they
can act on. `fail()` is the only exit path.
Heavy imports live inside the cmd_* function, never at module top, so
`--help` stays instant and an optional dependency only costs the one
subcommand that needs it.
"""
import argparse
import sys
# The flags worth having exactly one spelling of. A tool adds its own on top;
# it does not redefine these.
COMMON = {
"source": dict(flags=("--source", "-s"), help="What to read."),
"output": dict(flags=("--output", "-o"), help="Where to write."),
"name": dict(flags=("--name", "-n"), help="What to call it."),
"force": dict(flags=("--force", "-f"), action="store_true",
help="Write even if the destination is occupied."),
"dry_run": dict(flags=("--dry-run",), action="store_true",
help="Print what would happen; change nothing."),
}
def fail(message, hint=None):
"""The only way out on error: a line a user can act on, never a traceback."""
print(f"Error: {message}", file=sys.stderr)
if hint:
print(f" {hint}", file=sys.stderr)
sys.exit(1)
def refuse_to_clobber(path, force, marker, what):
"""
Regenerating is fine; overwriting something we did not write is not.
Lifted from shuntgen, which refuses a non-empty output directory unless it
carries the file its own generator leaves behind. The check is cheap and the
failure it prevents — silently eating a directory someone hand-wrote — is
not recoverable from.
"""
if force or not path.exists():
return
if not any(path.iterdir()):
return
if (path / marker).exists():
return
fail(f"{path} already exists and was not written by {what}.",
"Pick another path, or pass --force to write into it anyway.")
def _prog(package, name):
"""
How this tool was actually invoked, for the usage line.
The folder is meant to be copied out and run on its own, so a usage line
hardcoding `python -m station.tools.histgen` is wrong the moment it is —
it names a path that does not exist on the machine reading it.
"""
return f"python -m {package or name}"
class Tool:
"""A tool's whole command line: subcommands, shared flags, one exit path."""
def __init__(self, name, description, package=None):
self.name = name
self.parser = argparse.ArgumentParser(
prog=_prog(package, name),
description=(description or "").strip().split("\n\n")[0],
formatter_class=argparse.RawDescriptionHelpFormatter,
)
self._subparsers = self.parser.add_subparsers(dest="command", required=True)
self._commands = {}
def command(self, verb, func, help_text):
"""Register a subcommand. `func` takes parsed args and returns None."""
sub = self._subparsers.add_parser(verb, help=help_text, description=help_text)
sub.set_defaults(func=func)
self._commands[verb] = sub
return sub
def argument(self, verb, *args, **kwargs):
"""Add a positional or flag to one subcommand."""
self._commands[verb].add_argument(*args, **kwargs)
def common(self, verb, *names, **overrides):
"""Add shared flags by name, so their spelling is decided in one place."""
for key in names:
spec = dict(COMMON[key])
flags = spec.pop("flags")
spec.update(overrides.get(key, {}))
self._commands[verb].add_argument(*flags, **spec)
def run(self, argv=None):
args = self.parser.parse_args(argv)
try:
args.func(args)
except KeyboardInterrupt:
# Ctrl-C is a decision, not a crash. Say so and leave quietly.
print("\nInterrupted.", file=sys.stderr)
sys.exit(130)
except BrokenPipeError:
# `... | head` closes the pipe early; that is the caller's business.
sys.exit(0)

View File

@@ -0,0 +1,153 @@
"""
Where to read from and where to write to.
Two directories, and the whole tool hangs off the difference between them:
source the tree to read. Opened read-only, always. Nothing is written
into it, ever — not a commit, not a .git, not a state file. It
can be a checkout you do not own or a read-only mount.
out everything this produces. The index, the plan, the briefs, and
`out/<name>/` — a copy of the source with the designed history
committed into it. Delete the directory and you have lost
nothing but time.
The same repo gets scanned, planned and re-planned a dozen times while its
grouping is argued with, and passing the pair of paths to every one of five
verbs gets old. So they can live in a file instead — the arrangement
`ppl/ctrl/distill.sh` already uses, where the JSON beside the script is picked
up when nothing else says otherwise.
{
"source": "~/work/some-project",
"out": "~/histories/some-project",
"max_files": null
}
That separation is what makes the thing safe to experiment with. The history
is an argument you will have more than once, and every attempt is a directory
you can throw away rather than a repo you have to put back.
Precedence is the usual one, most specific first:
the command line -> --config FILE -> histgen.json beside the tool
-> the defaults
so a config file sets a starting point and never wins an argument with a flag
that was typed deliberately.
"""
import json
from pathlib import Path
CONFIG_NAME = "histgen.json"
HERE = Path(__file__).resolve().parent
KEYS = ("source", "out", "max_files", "keep_history", "branch",
"keep_secrets", "exclude", "include")
# `repo` was what `source` used to be called, back when the tool committed
# into the tree it read. Accepted rather than rejected, because a config file
# written last week should not be an error message.
ALIASES = {"repo": "source"}
TEMPLATE = {
"source": None,
"out": None,
"max_files": None,
"keep_history": False,
"branch": None,
"keep_secrets": False,
"exclude": [],
"include": [],
}
def default_path() -> Path:
"""The config beside the tool, used when nothing else is named."""
return HERE / CONFIG_NAME
def find(explicit=None):
"""The config file to read, or None. An explicit one that is missing is an error."""
if explicit:
path = Path(explicit).expanduser()
if not path.is_file():
from .cli import fail
fail(f"No such config file: {path}")
return path
beside = default_path()
return beside if beside.is_file() else None
def load(explicit=None) -> dict:
"""Read the config, or return the defaults. Unknown keys are an error."""
settings = dict(TEMPLATE)
path = find(explicit)
if not path:
return settings
from .cli import fail
try:
raw = json.loads(path.read_text())
except json.JSONDecodeError as e:
fail(f"{path} is not valid JSON: {e}")
if not isinstance(raw, dict):
fail(f"{path} should hold an object, not a {type(raw).__name__}.")
# A typo in a key would otherwise be silent, and the symptom — the tool
# ignoring a setting that is plainly written in the file — is a bad one to
# debug. Naming the valid keys costs one line.
raw = {ALIASES.get(k, k): v for k, v in raw.items()}
unknown = sorted(set(raw) - set(KEYS))
if unknown:
fail(f"{path}: unknown key(s) {', '.join(unknown)}.",
f"Known keys: {', '.join(KEYS)}.")
settings.update({k: v for k, v in raw.items() if v is not None})
settings["_path"] = str(path)
return settings
def resolve(args, explicit=None):
"""
Fold the config under the command line and hand back what to actually use.
Paths are expanded and made absolute here rather than at each use, so
everything downstream compares like with like — a `~` that survived into a
path comparison is a bug that only shows up on someone else's machine.
"""
settings = load(explicit)
source = getattr(args, "source", None) or settings.get("source")
out = getattr(args, "out", None) or settings.get("out")
return {
"source": Path(source).expanduser().resolve() if source else None,
"out": Path(out).expanduser().resolve() if out else None,
"max_files": getattr(args, "max_files", None) or settings.get("max_files"),
"keep_history": (getattr(args, "keep_history", False)
or settings.get("keep_history", False)),
"branch": getattr(args, "branch", None) or settings.get("branch"),
"keep_secrets": (getattr(args, "keep_secrets", False)
or settings.get("keep_secrets", False)),
"exclude": list(getattr(args, "exclude", None) or [])
+ list(settings.get("exclude") or []),
"include": list(getattr(args, "include", None) or [])
+ list(settings.get("include") or []),
"config_path": settings.get("_path"),
}
def write_template(path: Path, source=None, out=None) -> Path:
"""Write a starter config, never over one that already exists."""
from .cli import fail
if path.exists():
fail(f"{path} already exists.", "Edit it, or name another path.")
body = dict(TEMPLATE)
if source:
body["source"] = str(source)
if out:
body["out"] = str(out)
path.write_text(json.dumps(body, indent=2) + "\n")
return path

View File

@@ -0,0 +1,613 @@
"""
Materialise the source into the out directory and commit the designed history.
The source is never touched. What gets committed is a copy, made here, and the
copy is the only thing that ends up with a history — so an attempt that goes
wrong costs a `rm -rf` rather than an afternoon putting a real checkout back.
Everything up to this point is analysis and can be recomputed. This part writes,
so it starts by working out what it is writing into. Four states, and they are
genuinely different:
absent nothing there yet. Copy the tree, init, commit.
unfinished a copy is there with commits this tool made and a record of
where it stopped. Something interrupted the run — a signal, a
full disk, a hook that refused. Continue from the group after
the last one recorded.
foreign a copy is there with commits this tool did not make. That is
history someone else is entitled to, so nothing is rewritten,
moved or deleted: the designed account is committed to its own
orphan branch and the existing branch is left exactly as it
was. Two tiers, which is what `all/ctrl/handover.sh` has been
saying all along.
stale a copy is there that does not match the plan any more. Refuse,
and say which of the two moved.
Telling the second apart from the third is the whole reason progress.json
exists. Without it both read as "there are commits here", and the tool either
destroys work it should have kept or refuses to finish work it started.
"""
import hashlib
import json
import os
import shlex
import shutil
import subprocess
import sys
from pathlib import Path
from .census import current_file_set, is_git, load_index, state_dir
from .cli import fail
from .order import plan_path
MESSAGE_DIR = "messages"
SCRIPT_FILE = "regen.sh"
PROGRESS_FILE = "progress.json"
# Where the designed account goes when the copy already carries a history that
# is not ours. A name, not a number, because it has to mean something in a
# branch list six months from now.
DEFAULT_BRANCH = "designed-history"
# ── where the copy lives ───────────────────────────────────────────────────
def repo_dir(source: Path, out: Path) -> Path:
"""
The copy, named after the source.
Named rather than called `repo/`, because this directory gets `cd`-ed into,
pushed from and looked at in a file manager, and "adapter" answers a
question there that "repo" does not.
"""
return Path(out) / source.name
def progress_path(out) -> Path:
return state_dir(out) / PROGRESS_FILE
def plan_fingerprint(plan) -> str:
"""
Identifies the plan a history was built from, by its groups and their paths.
Messages are deliberately not in it. Rewording a commit that has not been
made yet must not invalidate the twelve that have — that is the normal way
this tool gets used, one group's message at a time.
"""
shape = [[g["n"], sorted(g["paths"])] for g in plan["groups"]]
return hashlib.sha256(json.dumps(shape, sort_keys=True).encode()).hexdigest()[:16]
def load_progress(out) -> dict:
p = progress_path(out)
if not p.exists():
return {}
try:
return json.loads(p.read_text())
except (OSError, json.JSONDecodeError):
# Unreadable progress means we cannot prove which commits are ours, and
# guessing is exactly the thing this file exists to avoid.
return {}
def save_progress(out, data) -> None:
progress_path(out).parent.mkdir(parents=True, exist_ok=True)
progress_path(out).write_text(json.dumps(data, indent=2))
# ── git ────────────────────────────────────────────────────────────────────
def _git(repo: Path, *args, check=True):
r = subprocess.run(["git", "-C", str(repo), *args], capture_output=True, text=True)
if check and r.returncode != 0:
fail(f"git {' '.join(args[:2])} failed: {r.stderr.strip() or r.stdout.strip()}")
return r
def _head(repo: Path):
r = _git(repo, "rev-parse", "HEAD", check=False)
return r.stdout.strip() if r.returncode == 0 else None
def _commit_count(repo: Path) -> int:
r = _git(repo, "rev-list", "--count", "HEAD", check=False)
return int(r.stdout.strip()) if r.returncode == 0 and r.stdout.strip() else 0
# ── what state is the out directory in ─────────────────────────────────────
def inspect(source: Path, out: Path, plan=None):
"""
Read the out directory and say what is there. Writes nothing.
`status` prints this; `export` branches on it. One function so the two can
never disagree about what they are looking at, which they would within a
week of being written separately.
"""
copy = repo_dir(source, out)
progress = load_progress(out)
report = {
"copy": copy,
"exists": copy.is_dir(),
"git": copy.is_dir() and is_git(copy),
"commits": 0,
"state": "absent",
"done": [],
"remaining": [],
"branch": None,
"detail": "",
}
if not report["exists"]:
report["detail"] = "nothing exported yet"
if plan:
report["remaining"] = [g["n"] for g in plan["groups"]]
return report
if not report["git"]:
report["state"] = "stale"
report["detail"] = ("a directory is there but it is not a git repo — "
"an export that died before `git init`")
return report
report["commits"] = _commit_count(copy)
report["branch"] = _git(copy, "rev-parse", "--abbrev-ref", "HEAD",
check=False).stdout.strip() or None
if report["commits"] == 0:
report["state"] = "absent"
report["detail"] = "a repo with no commits"
if plan:
report["remaining"] = [g["n"] for g in plan["groups"]]
return report
ours = progress.get("commits", [])
head = _head(copy)
if not ours:
report["state"] = "foreign"
report["detail"] = (f"{report['commits']} commit(s) this tool did not make")
if plan:
report["remaining"] = [g["n"] for g in plan["groups"]]
return report
if plan and progress.get("plan") != plan_fingerprint(plan):
report["state"] = "stale"
report["detail"] = ("the plan changed after this history was started — "
"the groups are not the ones these commits were made from")
report["done"] = [c["n"] for c in ours]
return report
# The record has to still describe reality. If the branch moved underneath
# us — a rebase, a reset, an amend — continuing would build on something
# other than what was recorded, and quietly.
if head != ours[-1]["sha"]:
report["state"] = "stale"
report["detail"] = ("the copy has moved since this tool last wrote to it "
f"(expected {ours[-1]['sha'][:9]}, found "
f"{(head or '-')[:9]})")
report["done"] = [c["n"] for c in ours]
return report
report["done"] = [c["n"] for c in ours]
if plan:
report["remaining"] = [g["n"] for g in plan["groups"]
if g["n"] not in set(report["done"])]
report["state"] = "complete" if plan and not report["remaining"] else "unfinished"
report["detail"] = (f"{len(report['done'])} group(s) committed by this tool"
+ (f", {len(report['remaining'])} to go" if report["remaining"] else ""))
return report
# ── checks ─────────────────────────────────────────────────────────────────
def load_plan(out):
p = plan_path(out)
if not p.exists():
fail(f"No plan at {p}.", "Run `scan` then `plan` first.")
try:
return json.loads(p.read_text())
except json.JSONDecodeError as e:
# Hand-editing plan.json is the expected workflow, so a trailing comma
# is a normal event and deserves a line number rather than a traceback.
fail(f"{p} is not valid JSON: {e}")
def check_plan(source: Path, out, plan, require_messages=True):
"""
Refuse a plan that could not produce the tree it claims to.
What the plan is measured against is the source as it stands *now*, with
the same filter the census used. Not the stored list: a file added after
the scan is exactly what this is here to catch, and a stored list cannot
see it. Not the unfiltered source either, or every deliberately dropped key
comes back as a file no group covers.
"""
planned, dupes = [], []
for g in plan["groups"]:
for p in g["paths"]:
(dupes if p in planned else planned).append(p)
problems = []
if dupes:
problems.append(f"{len(dupes)} path(s) appear in more than one group: "
+ ", ".join(sorted(set(dupes))[:5]))
missing = [p for p in planned if not (source / p).is_file()]
if missing:
problems.append(f"{len(missing)} planned path(s) are not in the source: "
+ ", ".join(missing[:5]))
present = current_file_set(source, load_index(out))
unplanned = sorted(present - set(planned))
if unplanned:
problems.append(f"{len(unplanned)} file(s) are in the source but in no group: "
+ ", ".join(unplanned[:5])
+ "\n Re-run `scan` and `plan` if the source changed since.")
empty = [g["n"] for g in plan["groups"] if not g["paths"]]
if empty:
problems.append(f"group(s) {empty} have no paths")
if require_messages:
unwritten = [g["n"] for g in plan["groups"] if not (g.get("title") or "").strip()]
if unwritten:
problems.append(
f"{len(unwritten)} group(s) have no title: "
+ ", ".join(str(n) for n in unwritten[:8])
+ f"\n Read {state_dir(out) / 'briefs'}/ and fill them in, "
"or pass --allow-untitled.")
if problems:
for p in problems:
print(f"Error: {p}", file=sys.stderr)
sys.exit(1)
return planned
def _message(group):
title = (group.get("title") or "").strip() or f"{group['slug']} ({len(group['paths'])} files)"
body = (group.get("body") or "").strip()
return f"{title}\n\n{body}\n" if body else f"{title}\n"
def source_tree_hash(source: Path, paths):
"""
The tree hash the source files would produce, without committing anything.
Runs against a temporary index and, when the source has no git, a temporary
git directory too. The source's own index is never touched: someone running
this mid-edit must not lose their staging area to a verification step.
"""
import tempfile
with tempfile.TemporaryDirectory(prefix="histgen-idx-") as tmp:
env = dict(os.environ, GIT_INDEX_FILE=str(Path(tmp) / "index"))
if not (source / ".git").exists():
env["GIT_DIR"] = str(Path(tmp) / "git")
env["GIT_WORK_TREE"] = str(source)
subprocess.run(["git", "init", "-q"], env=env, capture_output=True)
proc = subprocess.run(
["git", "-C", str(source), "update-index", "--add", "--stdin"],
input="\n".join(paths) + "\n", text=True, capture_output=True, env=env)
if proc.returncode != 0:
return None
r = subprocess.run(["git", "-C", str(source), "write-tree"],
capture_output=True, text=True, env=env)
return r.stdout.strip() if r.returncode == 0 else None
# ── the guards ─────────────────────────────────────────────────────────────
def verify(copy: Path, expected_tree=None, quiet=False):
"""
Nothing left untracked, and the tree still matches the source.
Nothing is exempt from the first check. The state directory lives in `out`
and the copy lives inside it, so there is genuinely nothing of ours in the
tree being checked — which is stricter than the version that had to forgive
its own scaffolding.
"""
ok = True
status = _git(copy, "status", "--porcelain").stdout.strip()
if status:
ok = False
print("Error: the tree is not clean — these never made it into a commit:",
file=sys.stderr)
for line in status.split("\n")[:20]:
print(f" {line}", file=sys.stderr)
if len(status.split("\n")) > 20:
print(f" ... and {len(status.split(chr(10))) - 20} more", file=sys.stderr)
elif not quiet:
print(" nothing left untracked: ok")
if expected_tree:
head = _git(copy, "rev-parse", "HEAD^{tree}", check=False).stdout.strip()
if head != expected_tree:
ok = False
print(f"Error: the exported tree does not match the source.\n"
f" source {expected_tree}\n HEAD {head}", file=sys.stderr)
elif not quiet:
print(f" tree matches source ({head[:12]}): ok")
return ok
# ── materialising the copy ─────────────────────────────────────────────────
def materialise(source: Path, copy: Path, paths, keep_history=False, quiet=False):
"""
Put the planned files into the copy, and nothing else.
Copied file by file from the plan rather than with `cp -r`, because the
plan is the definition of what belongs in the history: anything gitignored,
anything untracked, and the source's own .git are all things the source has
and the export must not.
`keep_history` is the exception, and the only reason the source's .git ever
comes across: it is what lets an existing history be carried into the copy
so the designed account can sit beside it instead of replacing it.
"""
copy.mkdir(parents=True, exist_ok=True)
if keep_history and (source / ".git").is_dir() and not (copy / ".git").exists():
shutil.copytree(source / ".git", copy / ".git", symlinks=True)
# A copied .git still points its index at files that are about to be
# rewritten underneath it; reset so status reflects the copy, not the
# source's staging area at the moment it was cloned.
_git(copy, "reset", "-q", check=False)
for rel in paths:
target = copy / rel
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source / rel, target)
if not quiet:
print(f" copied {len(paths)} files -> {copy}")
def _start_branch(copy: Path, report, branch, quiet=False):
"""
Decide which ref the designed history goes on, and get there.
A foreign history is not ours to move, so the designed account starts from
an orphan — no parent, nothing shared — and the branch that was there keeps
pointing exactly where it did.
"""
if report["state"] != "foreign":
return None
name = branch or DEFAULT_BRANCH
if _git(copy, "rev-parse", "--verify", name, check=False).returncode == 0:
fail(f"Branch '{name}' already exists in {copy}.",
"Pick another with --branch, or delete it if it was a failed attempt.")
kept = report["branch"] or "the existing branch"
if not quiet:
print(f" {report['commits']} existing commit(s) on {kept}: kept, untouched")
print(f" the designed history goes on a new orphan branch '{name}'")
_git(copy, "checkout", "-q", "--orphan", name)
# --orphan keeps the index, which would make the first designed commit
# carry every file the old branch had staged.
_git(copy, "rm", "-rq", "--cached", ".", check=False)
return name
def export(source: Path, out, dry_run=False, commands=False, allow_untitled=False,
keep_history=False, branch=None, force=False, quiet=False):
plan = load_plan(out)
planned = check_plan(source, out, plan, require_messages=not allow_untitled)
expected = source_tree_hash(source, planned)
if not expected:
fail("Could not compute the source tree hash.",
"Without it the export cannot be checked, and an unchecked export "
"is the thing this refuses to produce.")
copy = repo_dir(source, out)
report = inspect(source, out, plan)
if dry_run:
return _emit_script(source, out, plan, planned, expected, report,
keep_history, branch, quiet)
if commands:
return _emit_commands(source, out, plan, planned, expected, force, quiet)
if report["state"] == "stale" and not force:
fail(f"{copy}: {report['detail']}.",
"Pass --force to discard what is there and export again, or point "
"--out somewhere else to keep it.")
if report["state"] == "complete":
print(f"Already exported: {len(report['done'])} groups committed in {copy}.")
print("Nothing to do. Re-plan, or use --force to start over.")
return True
if report["state"] == "stale" and force:
if not quiet:
print(f" discarding {copy}")
shutil.rmtree(copy)
save_progress(out, {})
report = inspect(source, out, plan)
progress = load_progress(out)
resuming = report["state"] == "unfinished"
if resuming:
done = set(report["done"])
if not quiet:
print(f"Resuming: {len(done)} of {len(plan['groups'])} groups already "
f"committed in {copy}.")
# The files are already there from the interrupted run, but a source
# edited since would otherwise be silently ignored.
materialise(source, copy, planned, quiet=quiet)
else:
done = set()
materialise(source, copy, planned, keep_history=keep_history, quiet=quiet)
if not is_git(copy):
_git(copy, "init", "-q")
# Re-read the copy. --keep-history has only just put a history into it,
# so the state worked out before the directory existed cannot have seen
# it — and acting on the stale answer commits the designed account on
# top of the history it was supposed to sit beside.
report = inspect(source, out, plan)
active = _start_branch(copy, report, branch, quiet)
progress = {"plan": plan_fingerprint(plan), "source": str(source),
"branch": active, "commits": []}
save_progress(out, progress)
for g in plan["groups"]:
if g["n"] in done:
continue
_git(copy, "add", "--", *g["paths"])
msg = copy / ".git" / "HISTGEN_MSG"
msg.write_text(_message(g))
_git(copy, "commit", "-q", "-F", str(msg))
msg.unlink(missing_ok=True)
# Recorded after each commit, not at the end. The whole point is to
# survive the run not reaching the end.
progress.setdefault("commits", []).append({"n": g["n"], "sha": _head(copy)})
progress["plan"] = plan_fingerprint(plan)
save_progress(out, progress)
if not quiet:
print(f" {g['n']:02d} {_message(g).splitlines()[0]}")
if not quiet:
# The repo's own count, not the plan's. With a kept history the two
# differ, and the number a reader wants is what is actually in there.
where = _git(copy, "rev-parse", "--abbrev-ref", "HEAD",
check=False).stdout.strip()
print(f"\n{_commit_count(copy)} commits on {where} in {copy}. "
"Checking:", flush=True)
if not verify(copy, expected, quiet=quiet):
sys.exit(1)
return True
def _emit_script(source, out, plan, planned, expected, report,
keep_history, branch, quiet):
"""
Write the export as a shell script instead of running it.
Reviewing plain git commands before they run is worth more here than
anywhere else: this is the one operation whose mistakes are baked into
every commit that follows.
"""
state = state_dir(out)
copy = repo_dir(source, out)
msg_dir = state / MESSAGE_DIR
msg_dir.mkdir(parents=True, exist_ok=True)
for stale in msg_dir.glob("*.txt"):
stale.unlink()
q = shlex.quote
lines = [
"#!/usr/bin/env bash",
"# Generated by histgen. Review, then run from anywhere.",
"set -euo pipefail",
"",
f"SOURCE={q(str(source))}",
f"COPY={q(str(copy))}",
"",
'mkdir -p "$COPY"',
]
if keep_history and (source / ".git").is_dir():
lines.append('test -d "$COPY/.git" || cp -a "$SOURCE/.git" "$COPY/.git"')
lines += [
"# Only the planned files: not the source's .git, not anything ignored.",
'while IFS= read -r f; do mkdir -p "$COPY/$(dirname "$f")"; '
'cp -p "$SOURCE/$f" "$COPY/$f"; done <<\'PATHS\'',
*planned,
"PATHS",
"",
'cd "$COPY"',
"test -d .git || git init -q",
]
if report["state"] == "foreign":
name = branch or DEFAULT_BRANCH
lines += [f"# {report['commits']} existing commit(s) stay where they are.",
f"git checkout -q --orphan {q(name)}",
"git rm -rq --cached . || true", ""]
for g in plan["groups"]:
name = f"{g['n']:02d}-{g['slug']}.txt"
(msg_dir / name).write_text(_message(g))
lines += [f"# {g['n']:02d} {_message(g).splitlines()[0]}",
"git add -- " + " ".join(q(p) for p in g["paths"]),
f"git commit -q -F {q(str(msg_dir / name))}",
""]
lines += [
"# The two guards. A history that fails these is not worth keeping.",
'test -z "$(git status --porcelain)" || '
'{ echo "FAILED: files left untracked" >&2; exit 1; }',
f'test "$(git rev-parse HEAD^{{tree}})" = "{expected}" || '
'{ echo "FAILED: exported tree does not match source" >&2; exit 1; }',
'echo "ok: $(git rev-list --count HEAD) commits, tree verified"',
"",
]
script = state / SCRIPT_FILE
script.write_text("\n".join(lines))
script.chmod(0o755)
if not quiet:
print(f"Wrote {len(plan['groups'])} commits as commands -> {script}")
print(f" messages -> {msg_dir}")
return True
def _emit_commands(source: Path, out, plan, planned, expected, force, quiet):
"""
Copy the files, create no repo, and print the commands to make the history.
The other two modes each decide something for you: `export` runs the whole
thing, `--dry-run` writes a script that would. This one does the half that
is tedious and gets the other half out of the way — the copy is made, and
what comes back is a list you read, edit and run yourself.
Nothing here creates a .git. `git init` is the first line of the list rather
than something already done, because a repo that appeared without you asking
is exactly what someone reaching for this mode does not want.
"""
copy = repo_dir(source, out)
# Asked for no repo, so an existing one is a contradiction worth stopping
# for: the commands below would commit into it rather than into a fresh one.
if (copy / ".git").exists() and not force:
fail(f"{copy} already contains a git repo.",
"This mode creates none and the commands assume none. Delete it, "
"point --out elsewhere, or pass --force to copy the files in anyway.")
materialise(source, copy, planned, quiet=quiet)
state = state_dir(out)
msg_dir = state / MESSAGE_DIR
msg_dir.mkdir(parents=True, exist_ok=True)
for stale in msg_dir.glob("*.txt"):
stale.unlink()
q = shlex.quote
lines = [f"cd {q(str(copy))}", "git init", ""]
for g in plan["groups"]:
name = f"{g['n']:02d}-{g['slug']}.txt"
(msg_dir / name).write_text(_message(g))
lines += [f"# {g['n']:02d} {_message(g).splitlines()[0]}",
"git add -- " + " ".join(q(p) for p in g["paths"]),
f"git commit -F {q(str(msg_dir / name))}",
""]
lines += [
"# Worth running afterwards. The first says no file was silently",
"# missed; the second says the result is byte-identical to the source.",
"git status --porcelain",
f"git rev-parse HEAD^{{tree}} # expect {expected}",
"",
]
listing = "\n".join(lines)
(state / "commands.sh").write_text(listing)
if not quiet:
print(f" messages -> {msg_dir}")
print(f" this list -> {state / 'commands.sh'}\n")
print(listing)
return True

View File

@@ -0,0 +1,116 @@
"""
What the history already there says, next to what the plan proposes.
Reads and reports. It never rewrites: published history is someone else's
clone, and the useful output here is an argument about ordering, not a
force-push.
Most repos this gets pointed at are in the checkpoint tier — "updates 33.1
139", "working state", "debugging" — where the honest finding is that the log
records when work was saved and nothing about how the thing is built. That is
worth printing plainly, because it is the case for keeping a second, designed
history rather than trying to repair this one.
"""
import subprocess
from collections import defaultdict
# Subjects that carry no information about what changed. Matching is on the
# whole subject, lowercased, with trailing numbers dropped — "updates 33.1 139"
# and "updates 33.1 84" are the same non-statement.
NOISE = {"update", "updates", "wip", "fix", "fixes", "changes", "some changes",
"working state", "debugging", "init commit", "initial commit", "misc",
"checkpoint", "save", "final", "for final test", "cleanup", "tmp"}
def _subject_is_noise(subject):
s = subject.strip().lower().rstrip("0123456789. ")
return s in NOISE or not s
def read_history(repo):
"""[(sha, subject, [paths])] oldest first, or [] if there is no history."""
r = subprocess.run(
["git", "-C", str(repo), "log", "--reverse", "--name-only",
"--format=%x00%h%x1f%s"],
capture_output=True, text=True)
if r.returncode != 0:
return []
commits = []
for chunk in r.stdout.split("\0"):
if not chunk.strip():
continue
head, _, rest = chunk.partition("\n")
sha, _, subject = head.partition("\x1f")
paths = [l for l in rest.split("\n") if l.strip()]
commits.append((sha, subject, paths))
return commits
def compare(source, plan, quiet=False):
"""Print how the existing history lines up with the proposed one."""
commits = read_history(source)
groups = plan["groups"]
if not commits:
print(f"{len(groups)} groups proposed; the source has no history to compare.")
return {"commits": 0, "groups": len(groups)}
where = {p: g["n"] for g in groups for p in g["paths"]}
# A commit maps to the group holding most of the files it touched. Files
# that no longer exist are dropped rather than counted against it — a
# commit that deleted something is not disagreeing about order.
mapped, noise, touches = {}, [], defaultdict(list)
for sha, subject, paths in commits:
hits = [where[p] for p in paths if p in where]
if not hits:
noise.append((sha, subject, "touches nothing that still exists"))
continue
best = max(set(hits), key=lambda n: (hits.count(n), -n))
spread = len(set(hits))
mapped[sha] = (best, subject, spread, len(hits))
touches[best].append(sha)
if _subject_is_noise(subject):
noise.append((sha, subject, f"says nothing; touches {spread} group(s)"))
print(f"{len(groups)} groups proposed, {len(commits)} existing commits.\n")
# Order disagreement: walking the real history, does the group number ever
# go backwards? That is the concrete "this was built in a different order".
seen_max, inversions = 0, []
for sha, subject, _ in commits:
if sha not in mapped:
continue
n = mapped[sha][0]
if n < seen_max:
inversions.append((sha, n, seen_max, subject))
seen_max = max(seen_max, n)
for g in groups:
shas = touches.get(g["n"], [])
title = g.get("title") or g["slug"]
if not shas:
mark, note = "+", "no existing commit builds this"
elif len(shas) == 1:
mark, note = "=", f"{shas[0]}"
else:
mark, note = "~", f"split across {len(shas)} commits ({', '.join(shas[:4])})"
print(f" {mark} {g['n']:02d} {title[:44]:46} {note}")
if inversions:
print(f"\n ! {len(inversions)} commit(s) land earlier in the proposed order "
f"than work already done:")
for sha, n, high, subject in inversions[:10]:
print(f" {sha} group {n:02d} after group {high:02d} {subject[:44]}")
if noise:
print(f"\n ? {len(noise)} commit(s) carry no usable account of the change:")
for sha, subject, why in noise[:10]:
print(f" {sha} {subject[:44]:46} {why}")
if len(noise) > 10:
print(f" ... and {len(noise) - 10} more")
print("\n = matched one commit ~ split + not in history "
"! out of order ? uninformative")
return {"commits": len(commits), "groups": len(groups),
"inversions": len(inversions), "noise": len(noise)}

View File

@@ -0,0 +1,450 @@
"""
The order the files go in, and where one commit stops and the next begins.
Two decisions, and they are not the same decision. Order answers "what can be
understood before what"; grouping answers "what is one idea". Getting the first
right and the second wrong gives you 64 correct commits nobody wants to read.
The order is role first, references second. Roles carry the heuristic — ignore
rules and README, then the config layer, then the things that source it, the
front door late because it only dispatches, the bootstrap account last because
it narrates everything above it. References refine within that, so a config
lands before the script that sources it.
References never override roles. A reference in code is a dependency, but roles
already encode dependencies that no reference states: nothing in the repo
*refers to* .gitignore, and the README is named by nothing while naming
everything. Letting edges win produces the ignore rules committed after the
code they exclude — technically consistent, and unreadable.
Nothing here is authoritative. plan.json is a file, and moving a path from one
group to another is the expected way to use it: this gets the shape right so
the argument is about two or three groups, not sixty-four paths.
"""
import json
from collections import defaultdict
from pathlib import Path
from .census import ROLE_RANK, state_dir
PLAN_FILE = "plan.json"
# Roles that name other files without depending on them. Their outgoing edges
# are dropped: a README mentioning every script in the tree is a table of
# contents, not a build order.
NARRATIVE = {"skeleton", "readme", "doc", "bootstrap", "asset", "lock"}
# A directory that carries its own README, Makefile or package manifest is a
# project in its own right. It is committed whole and late — it stands on the
# repo around it, so it cannot be read before it. rig's sample-rig is the case:
# sixteen files, one idea, and it calls rig's own addon script rather than
# reimplementing it.
# A README is NOT one of these. Any directory worth having explains itself, and
# ctrl/k8s/README.md documenting four manifests does not make them a project —
# it made them sort after the Makefile, which is where this rule came from. What
# marks a project is something that builds or resolves it.
SUBPROJECT_MARKERS = {"makefile", "package.json", "pyproject.toml",
"go.mod", "cargo.toml", "gemfile", "build.gradle"}
SUBPROJECT_RANK = 95 # after the front door, before the bootstrap account
# How many files one commit may hold before it stops being one idea. Soft: a
# subproject and a hub with its satellites are exempt, because splitting those
# produces a commit that does not build.
DEFAULT_MAX_FILES = 8
# Roles whose files earn their way into a commit by referring to each other,
# rather than by sitting in the same directory.
CODE_ROLES = {"source", "test", "frontdoor"}
def _subprojects(paths):
"""Directories that are their own project -> the files beneath them."""
marked = set()
for p in paths:
parent = str(Path(p).parent)
if parent not in (".", "") and Path(p).name.lower() in SUBPROJECT_MARKERS:
marked.add(parent)
# A subproject inside a subproject belongs to the outer one; one commit,
# not two nested ones.
roots = {d for d in marked
if not any(d != o and d.startswith(o + "/") for o in marked)}
owned = {}
for p in paths:
for root in roots:
if p == root or p.startswith(root + "/"):
owned[p] = root
break
return owned
# Roles that are one idea when they sit side by side. A diagram and the source
# it renders from belong in the same commit; so do a pin and the config that
# reads it. Grouping only — the ordering still keeps them apart.
GROUP_TIER = {"asset": "doc", "lock": "pin", "pin": "pin", "config": "pin"}
def _cluster(path, owner, dirs=()):
"""
The directory a file is grouped under: its subproject, else its own parent.
The parent, not the top-level directory. Under `ctrl` everything in a tree
this shape lands in one bucket — eleven scripts, four profiles and a k8s
tree — and the split has to be reconstructed afterwards from references
that were never going to describe it.
"""
if owner:
return owner
# `ctrl/addons.sh` belongs with `ctrl/addons/`, not with its own siblings.
# Clustering is what decides which files are even considered together, so a
# hub parted from its satellites here can never be rejoined later.
hub = _hub_of(path)
if hub and hub in dirs:
return hub
return str(Path(path).parent)
def adaptive_cap(count, requested=None):
"""
How many files one commit may hold, for a repo this size.
A fixed cap does not survive the range. rig is 64 files and wants commits
of three or four; spr is 507 and at a flat eight plans a hundred and
forty-three, which is the same failure as one commit from the other end.
A twentieth of the tree, floored at the default, lands close to what these
repos were actually built as: rig plans 18 against a real 18, spr 77
against a real 78. It is a starting point, not a claim — --max-files
overrides it and plan.json is editable either way.
"""
if requested:
return requested
return max(DEFAULT_MAX_FILES, -(-count // 20))
def order_files(index, max_files=None):
"""Return an ordered list of groups, each a list of repo-relative paths."""
files = index["files"]
paths = sorted(files)
max_files = adaptive_cap(len(paths), max_files)
owner = _subprojects(paths)
def rank(p):
return SUBPROJECT_RANK if p in owner else ROLE_RANK.get(files[p]["role"], 60)
# ── edges ──────────────────────────────────────────────────────────────
# An edge b -> a means "b must come after a". Only real dependencies count,
# and only between files whose roles do not already disagree: a reference
# pointing backwards up the role order is a mention the extractor could not
# tell from a dependency, and honouring it inverts the tier.
after = defaultdict(set)
for p in paths:
if files[p]["role"] in NARRATIVE or p in owner:
continue
for dep in files[p]["refs"]:
if dep in files and dep != p and rank(dep) <= rank(p):
after[p].add(dep)
# ── ordering ───────────────────────────────────────────────────────────
# Kahn's algorithm, taking the lowest (rank, path) that is ready. Ties are
# broken by path so two runs on the same tree produce the same history.
blockers = {p: set(after[p]) for p in paths}
dependents = defaultdict(set)
for p, deps in blockers.items():
for d in deps:
dependents[d].add(p)
ready = sorted((p for p in paths if not blockers[p]), key=lambda p: (rank(p), p))
ordered = []
while ready:
p = ready.pop(0)
ordered.append(p)
for d in sorted(dependents[p]):
blockers[d].discard(p)
if not blockers[d]:
ready.append(d)
ready.sort(key=lambda q: (rank(q), q))
# A cycle leaves files unplaced. Two shell scripts that source each other is
# a real thing and not an error here, so append them in role order rather
# than refusing to produce a plan at all.
if len(ordered) < len(paths):
ordered += sorted(set(paths) - set(ordered), key=lambda p: (rank(p), p))
return _coalesce(_group(ordered, files, owner, after, max_files), owner, max_files)
def _common_dir(paths):
"""The deepest directory every path in the group sits under."""
parts = list(Path(paths[0]).parent.parts)
for p in paths[1:]:
other = Path(p).parent.parts
keep = []
for a, b in zip(parts, other):
if a != b:
break
keep.append(a)
parts = keep
return tuple(parts)
def _coalesce(groups, owner, max_files):
"""
Merge neighbouring groups that are really one idea in one place.
Cutting on directory is right for a shallow tree and wrong for a deep one:
a directory holding a single file is not an idea, and a repo of five
hundred files has a lot of them. Left alone spr planned three hundred and
thirty-seven commits, which is the same failure as one commit, from the
other end.
Only neighbours already adjacent in the order merge, only when they share
a top-level directory, only under the cap, and only when BOTH are small.
Both, not either. Letting a lone file join whatever it happened to sit next
to put `dockerhost.sh` inside the addons commit — a coherent group of seven
with an eighth file that has nothing to do with it. A run of scattered
singletons is fragmentation and should close up; a group that already says
something should not absorb a stray because it had room.
What counts as small scales with the cap, so raising --max-files actually
buys fewer commits. Gated at a flat two it did not: on a 500-file repo the
cap could be raised from 8 to 40 and the count moved by four, because
nothing was ever allowed to merge. Asking for bigger commits should widen
what is considered fragmentation, not just what is allowed to survive.
"""
loose = max(2, max_files // 4)
out = []
for group in groups:
if not out:
out.append(group)
continue
previous = out[-1]
if (len(previous) + len(group) <= max_files
and max(len(previous), len(group)) <= loose
and not any(p in owner for p in previous + group)
and _shares_ancestor(_common_dir(previous), _common_dir(group))):
out[-1] = previous + group
else:
out.append(group)
return out
def _shares_ancestor(a, b):
"""Both at the root, or under a common top-level directory."""
if not a and not b:
return True
return bool(a) and bool(b) and a[0] == b[0]
def _group(ordered, files, owner, after, max_files):
"""
Cut the ordered list into commits.
One coherent idea per commit, not one directory per commit. What holds a
group together is that its files refer to each other — ports.sh and the
hosts template it renders, a kustomization and the manifests it lists. What
separates two groups in the same directory is that neither names the other.
A hub and the directory named after it (addons.sh and addons/) travel
together whatever their references say: committing the loader without the
things it loads produces a commit that cannot run.
"""
dirs = {str(Path(p).parent) for p in ordered}
groups, buf = [], []
seen_cluster = seen_role = None
def flush():
nonlocal buf
if buf:
groups.append(buf)
buf = []
for path in ordered:
own = owner.get(path)
cluster = _cluster(path, own, dirs)
role = "subproject" if own else GROUP_TIER.get(
files[path]["role"], files[path]["role"])
if cluster != seen_cluster or role != seen_role:
flush()
seen_cluster, seen_role = cluster, role
buf.append(path)
flush()
out = []
for group in groups:
# A subproject is one commit no matter how many files it holds.
if any(p in owner for p in group):
out.append(group)
continue
out.extend(_split(group, files, after, max_files))
return out
def _hub_of(path):
"""`ctrl/addons.sh` is the hub of `ctrl/addons/`; returns that directory."""
p = Path(path)
return str(p.parent / p.stem) if p.suffix else None
def _pure_hub(members, hubs):
"""True when the group is exactly one hub and things inside its directory."""
for hub, owner_file in hubs.items():
if owner_file in members and all(
m == owner_file or m.startswith(hub + "/") for m in members
):
return True
return False
def _split(group, files, after, max_files):
"""
Break one cluster into commits.
A run of siblings of the same kind in the same directory is left alone —
four profile files under env.d/ are one idea, and nothing in them refers to
anything, so splitting on references turns them into four commits saying
the same thing four times. References are only asked about a group that is
already too big or already spans directories.
Past that: connected components, because what holds a commit together is
that its files name each other. A hub and the directory named after it
(addons.sh and addons/) survive the cap, since committing the loader
without the things it loads produces a commit that cannot run. Anything
else over the cap gets its hub peeled off into its own commit and the
remainder reconsidered — a cut at a named seam rather than at a count.
"""
if len(group) <= 1:
return [group]
# ...but only for roles where sitting side by side IS the relationship.
# Four env.d profiles are one idea. Seven scripts that happen to share a
# directory are seven ideas, and `ctrl/` is full of them, so code always
# gets asked about its references.
parents = {str(Path(p).parent) for p in group}
kinds = {files[p]["role"] for p in group}
if len(parents) == 1 and not (kinds & CODE_ROLES):
if len(group) <= max_files:
return [group]
# Over the cap and still one kind in one directory: cut it into runs.
# Nothing here refers to anything, so asking references to find the
# seam yields one commit per file — thirteen commits each saying "a
# project note", which is worse than an admitted arbitrary cut.
return [group[i:i + max_files] for i in range(0, len(group), max_files)]
index = {p: i for i, p in enumerate(group)}
parent = list(range(len(group)))
def find(i):
while parent[i] != i:
parent[i] = parent[parent[i]]
i = parent[i]
return i
def union(a, b):
ra, rb = find(a), find(b)
if ra != rb:
parent[max(ra, rb)] = min(ra, rb)
hubs = {}
for p in group:
hub = _hub_of(p)
if hub:
hubs[hub] = p
for p in group:
for hub, owner_file in hubs.items():
if p.startswith(hub + "/"):
union(index[p], index[owner_file])
for dep in after.get(p, ()):
if dep in index:
union(index[p], index[dep])
components = defaultdict(list)
for p in group:
components[find(index[p])].append(p)
out = []
for key in sorted(components):
members = components[key]
while len(members) > max_files and not _pure_hub(members, hubs):
degree = {m: sum(1 for o in members if m in after.get(o, ())) for m in members}
hub = max(degree, key=lambda m: (degree[m], m))
if degree[hub] == 0:
# Nothing holds this together and nothing names anything: an
# arbitrary cut is the honest answer, so cut on the order we
# already have rather than inventing a reason.
out.extend([members[i:i + max_files]
for i in range(0, len(members), max_files)])
members = []
break
members.remove(hub)
out.append([hub])
if members:
out.append(members)
return [g for g in out if g]
# ── the plan ───────────────────────────────────────────────────────────────
def plan_path(out) -> Path:
return state_dir(out) / PLAN_FILE
def build_plan(index, out, max_files=None, quiet=False):
"""
Write plan.json: ordered groups of paths with empty message slots.
This is the seam. Everything above is analysis and can be recomputed from
the tree; everything below is git commands. An existing plan's messages are
carried over when the group's paths still match exactly, so re-planning
after editing three files does not throw away sixty written messages.
"""
groups = order_files(index, max_files)
written = {}
existing = plan_path(out)
if existing.exists():
try:
for g in json.loads(existing.read_text()).get("groups", []):
if g.get("title"):
written[tuple(sorted(g["paths"]))] = (g.get("title"), g.get("body", ""))
except (OSError, json.JSONDecodeError, KeyError, TypeError):
pass
document = {"version": 1, "groups": []}
for i, paths in enumerate(groups, 1):
title, body = written.get(tuple(sorted(paths)), ("", ""))
document["groups"].append({
"n": i,
"slug": _slug(paths, index),
"paths": paths,
"title": title,
"body": body,
})
destination = plan_path(out)
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_text(json.dumps(document, indent=2))
if not quiet:
kept = sum(1 for g in document["groups"] if g["title"])
print(f"Planned {len(groups)} commits over "
f"{sum(len(g) for g in groups)} files"
+ (f", {kept} messages carried over" if kept else "") + ".")
print(f" -> {destination}")
return document
def _slug(paths, index):
"""A stable handle for a group, for filenames and for talking about it."""
common = Path(paths[0]).parent
for p in paths[1:]:
parts = []
for a, b in zip(common.parts, Path(p).parent.parts):
if a != b:
break
parts.append(a)
common = Path(*parts) if parts else Path(".")
base = str(common).strip("./").replace("/", "-")
if not base:
base = Path(paths[0]).stem if len(paths) == 1 else index["files"][paths[0]]["role"]
return base.lower().replace("_", "-").replace(".", "")[:40] or "root"

View File

@@ -0,0 +1,413 @@
"""
Prove the whole pipeline on a tree this builds itself.
`make check` after copying the folder somewhere new, with no repo to point at
and nothing installed. It builds a small tree with the shapes that matter —
ignore rules, a README, a pin, a config that sources it, a directory of
profiles, a hub with satellites, a front door, an ignored directory — runs all
five verbs over it, and asserts what has to be true.
The guards get the same treatment as the happy path. A check that only ever
proves things work would have missed the one real bug found while writing this:
the tree-hash guard returned None on exactly the trees it was written for, and
reported success anyway.
python3 selftest.py # or: make check
"""
import json
import os
import subprocess
import sys
import tempfile
from pathlib import Path
TREE = {
".gitignore": "# Generated output; regenerate rather than commit.\nbuild/\n*.log\n",
".gitattributes": "# LF everywhere: a CRLF checkout breaks the shebang.\n* text=auto eol=lf\n",
"README.md": "# demo\n\nWhat this is, and the one prerequisite.\n",
"ctrl/versions.env": "# Pinned toolchain — one manifest, one answer.\nKUBECTL=1.31.0\n",
"ctrl/lib/config.sh": "# Shared config loading. Sourced, never executed.\n"
'. "$(dirname "$0")/../versions.env"\n',
"ctrl/env.d/minimal.env": "# The smallest profile that still starts.\nADDONS=\n",
"ctrl/env.d/full.env": "# Everything on, for a demo machine.\nADDONS=redis\n",
"ctrl/addons.sh": "# Install the addons the profile asked for.\n"
"# Adding one is adding a file, not editing a dispatcher.\n"
'for a in ctrl/addons/*.sh; do sh "$a"; done\n',
"ctrl/addons/redis.sh": "# Redis — the broker half, nothing else uses it.\necho redis\n",
"ctrl/addons/postgres.sh": "# Postgres — the metadata store.\necho postgres\n",
"Makefile": "# Thin front door: one target per ctrl/ script.\nup:\n\tsh ctrl/addons.sh\n",
"BOOTSTRAP.md": "# From a bare machine to something running.\n",
"build/generated.txt": "this is ignored and must never be committed",
"noisy.log": "also ignored",
}
IGNORED = {"build/generated.txt", "noisy.log"}
# Things a copy should leave behind, and the two lookalikes it must not. Added
# to the tree only for the `copy` checks, and force-added so the ignored-but-
# tracked case is real rather than described.
SIFTABLE = {
"package-lock.json": "lockfile contents",
"dist/app.min.js": "minified",
".env": "API_KEY=real-secret-value",
".env.example": "API_KEY=",
"certs/server.key": "-----BEGIN PRIVATE KEY-----",
"certs/server.key.pub": "ssh-rsa AAAA",
"assets/logo.png": "PNG",
"build/forced.bin": "ignored yet tracked",
}
DROPPED = {"package-lock.json", "dist/app.min.js", ".env",
"certs/server.key", "build/forced.bin"}
KEPT_LOOKALIKES = {".env.example", "certs/server.key.pub", "assets/logo.png"}
def run(pkg, parent, *args):
return subprocess.run([sys.executable, "-m", pkg, *args],
capture_output=True, text=True,
env={"PYTHONPATH": str(parent), "PATH": "/usr/bin:/bin",
"HOME": str(Path.home())})
def build(root: Path):
for rel, body in TREE.items():
path = root / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(body)
def main():
here = Path(__file__).resolve().parent
pkg, parent = here.name, here.parent
failures = []
def check(label, condition, detail=""):
print(f" {'ok ' if condition else 'FAIL'} {label}")
if not condition:
failures.append(f"{label}{': ' + detail if detail else ''}")
def hg(*args):
return run(pkg, parent, *args)
with tempfile.TemporaryDirectory(prefix="histgen-selftest-") as tmp:
source, out = Path(tmp) / "demo", Path(tmp) / "out"
build(source)
W = ["--source", str(source), "--out", str(out)]
copy = out / "demo"
r = hg("run", *W)
check("run: scan + plan + brief", r.returncode == 0, r.stderr.strip())
if r.returncode != 0:
print(r.stderr)
return 1
index = json.loads((out / "index.json").read_text())
plan = json.loads((out / "plan.json").read_text())
planned = [p for g in plan["groups"] for p in g["paths"]]
check("the ignore rules were honoured", not (set(planned) & IGNORED),
f"ignored files got planned: {sorted(set(planned) & IGNORED)}")
check("every other file is in exactly one group",
sorted(planned) == sorted(set(TREE) - IGNORED) and len(planned) == len(set(planned)))
check("the ignore rules are committed first",
plan["groups"][0]["paths"][0].startswith(".git"))
check("the front door is not", "Makefile" not in plan["groups"][0]["paths"])
check("the hub travels with its satellites",
any({"ctrl/addons.sh", "ctrl/addons/redis.sh", "ctrl/addons/postgres.sh"}
<= set(g["paths"]) for g in plan["groups"]))
check("a comment is not mistaken for a dependency",
index["files"][".gitignore"]["refs"] == [])
check("a real dependency is found",
"ctrl/versions.env" in index["files"]["ctrl/lib/config.sh"]["refs"])
check("the reasoning was extracted for the message",
"not editing a dispatcher" in index["files"]["ctrl/addons.sh"]["why"])
check("a brief exists per commit",
len(list((out / "briefs").glob("*.md"))) == len(plan["groups"]))
check("list prints one line per commit",
all(f"{g['n']:3}." in hg("list", *W).stdout for g in plan["groups"]))
# The guards refuse before they approve.
check("export refuses a plan with no messages",
"no title" in hg("export", *W).stderr)
(source / "appeared-late.sh").write_text("# added after planning\n")
r = hg("export", *W, "--allow-untitled")
check("export refuses a file no group covers",
r.returncode != 0 and "no group" in r.stderr)
(source / "appeared-late.sh").unlink()
# --- commands: copy the files, make no repo, hand back the list ---
# Its own out directory, so it needs its own plan in it: the plan is a
# fact about an out directory, not about the source.
hands = Path(str(out) + "-byhand")
H = ["--source", str(source), "--out", str(hands)]
hg("run", *H)
r = hg("export", *H, "--allow-untitled", "--commands")
check("commands: succeeds", r.returncode == 0, r.stderr.strip())
made = hands / "demo"
check("commands: the files were copied", (made / "README.md").is_file())
check("commands: NO repo was created", not (made / ".git").exists())
check("commands: gitignored files did not come across",
not any((made / i).exists() for i in IGNORED))
check("commands: git init is the first thing offered, not done for you",
"git init" in r.stdout)
check("commands: one add and one commit per group",
r.stdout.count("git add -- ") == len(plan["groups"])
and r.stdout.count("git commit -F ") == len(plan["groups"]))
check("commands: the list is saved too", (hands / "commands.sh").is_file())
# The list has to actually work, which is the only claim that matters.
subprocess.run(["bash", str(hands / "commands.sh")], capture_output=True,
env={**os.environ, "GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@t", "GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@t"})
check("commands: running the list builds the history",
subprocess.run(["git", "-C", str(made), "rev-list", "--count", "HEAD"],
capture_output=True, text=True).stdout.strip()
== str(len(plan["groups"])))
check("commands: it leaves nothing untracked",
not subprocess.run(["git", "-C", str(made), "status", "--porcelain"],
capture_output=True, text=True).stdout.strip())
expected = [l for l in r.stdout.split("\n") if "expect " in l][0].split("expect ")[1].strip()
check("commands: the tree matches the hash it told you to expect",
subprocess.run(["git", "-C", str(made), "rev-parse", "HEAD^{tree}"],
capture_output=True, text=True).stdout.strip() == expected)
check("commands: refuses when a repo is already there",
hg("export", *H, "--allow-untitled", "--commands").returncode != 0)
# --- absent -> exported ---
check("status says absent before anything is exported",
"absent" in hg("status", *W).stdout)
r = hg("export", *W, "--allow-untitled")
check("export writes the history", r.returncode == 0, r.stderr.strip())
check("both guards ran",
"nothing left untracked: ok" in r.stdout and "tree matches source" in r.stdout)
check("the copy is named after the source", copy.is_dir())
check("one commit per group",
subprocess.run(["git", "-C", str(copy), "rev-list", "--count", "HEAD"],
capture_output=True, text=True).stdout.strip()
== str(len(plan["groups"])))
check("the ignored files never entered the history",
not (set(subprocess.run(["git", "-C", str(copy), "ls-files"],
capture_output=True, text=True).stdout.split()) & IGNORED))
# The invariant the whole design rests on.
check("THE SOURCE WAS NEVER WRITTEN TO",
not (source / ".git").exists() and not (source / ".histgen").exists()
and sorted(p.relative_to(source).as_posix()
for p in source.rglob("*") if p.is_file()) == sorted(TREE))
check("a finished export says so and stops",
"Nothing to do" in hg("export", *W, "--allow-untitled").stdout)
check("verify passes on its own", hg("verify", *W).returncode == 0)
# --- unfinished: interrupt, then resume ---
progress = json.loads((out / "progress.json").read_text())
# Half of them, whatever the fixture happens to plan. Hardcoding four
# made this pass vacuously the moment the fixture planned exactly four:
# nothing was left to resume, so "complete" was the honest answer and
# the resume path was never entered.
keep = progress["commits"][:max(1, len(progress["commits"]) // 2)]
subprocess.run(["git", "-C", str(copy), "reset", "-q", "--hard", keep[-1]["sha"]])
progress["commits"] = keep
(out / "progress.json").write_text(json.dumps(progress))
check("status spots a half-finished history",
"unfinished" in hg("status", *W).stdout)
r = hg("export", *W, "--allow-untitled")
check("export resumes rather than restarting",
r.returncode == 0 and "Resuming" in r.stdout, r.stdout + r.stderr)
check("resuming did not redo the commits already made",
f"{len(keep)} of {len(plan['groups'])}" in r.stdout, r.stdout)
check("the resumed history is complete and verified",
"tree matches source" in r.stdout)
# --- stale: the plan moved ---
plan["groups"][1]["paths"].append(plan["groups"][2]["paths"].pop())
(out / "plan.json").write_text(json.dumps(plan))
check("status spots a plan that no longer matches",
"stale" in hg("status", *W).stdout)
check("export refuses a stale export",
hg("export", *W, "--allow-untitled").returncode != 0)
check("--force starts over",
hg("export", *W, "--allow-untitled", "--force").returncode == 0)
# --- foreign: a history that has to be kept ---
with tempfile.TemporaryDirectory(prefix="histgen-selftest-keep-") as tmp:
source, out = Path(tmp) / "demo", Path(tmp) / "out"
build(source)
for cmd in (["init", "-q"], ["add", "-A"], ["commit", "-qm", "init commit"]):
subprocess.run(["git", "-C", str(source), *cmd], capture_output=True,
env={**os.environ, "GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@t", "GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@t"})
before = subprocess.run(["git", "-C", str(source), "rev-parse", "HEAD"],
capture_output=True, text=True).stdout.strip()
W = ["--source", str(source), "--out", str(out)]
copy = out / "demo"
hg("run", *W)
r = hg("export", *W, "--allow-untitled", "--keep-history")
check("keep: export succeeds", r.returncode == 0, r.stderr.strip())
check("keep: the old history is reported as kept", "kept, untouched" in r.stdout)
branches = subprocess.run(["git", "-C", str(copy), "branch", "--format=%(refname:short)"],
capture_output=True, text=True).stdout.split()
check("keep: the designed history is on its own branch",
"designed-history" in branches and len(branches) >= 2, str(branches))
main = [b for b in branches if b != "designed-history"][0]
check("keep: the old branch still points where it did",
subprocess.run(["git", "-C", str(copy), "rev-parse", main],
capture_output=True, text=True).stdout.strip() == before)
check("keep: the two histories share no commit",
subprocess.run(["git", "-C", str(copy), "merge-base", main, "designed-history"],
capture_output=True, text=True).returncode != 0)
check("keep: the source's own history is untouched",
subprocess.run(["git", "-C", str(source), "rev-parse", "HEAD"],
capture_output=True, text=True).stdout.strip() == before)
# --- the filter has to hold on the HISTORY path, not just on copy ---
# This is the case that was wrong: `copy` left a tracked key behind while
# `scan` walked straight past it, so the key stayed out of the snapshot and
# went into the commits.
with tempfile.TemporaryDirectory(prefix="histgen-selftest-secret-") as tmp:
source, out = Path(tmp) / "demo", Path(tmp) / "out"
build(source)
for rel, body in SIFTABLE.items():
(source / rel).parent.mkdir(parents=True, exist_ok=True)
(source / rel).write_text(body)
env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
for cmd in (["init", "-q"], ["add", "-A", "-f", "."], ["commit", "-qm", "init"]):
subprocess.run(["git", "-C", str(source), *cmd], capture_output=True, env=env)
W = ["--source", str(source), "--out", str(out)]
r = hg("run", *W)
check("filter: scan says what it left out", "left out of the history" in r.stdout)
index = json.loads((out / "index.json").read_text())
plan = json.loads((out / "plan.json").read_text())
planned = {p for g in plan["groups"] for p in g["paths"]}
secrets = {".env", "certs/server.key"}
check("filter: no secret reached the census", not (set(index["files"]) & secrets))
check("filter: no secret reached the plan", not (planned & secrets))
check("filter: the ignored-but-tracked file did not either",
"build/forced.bin" not in planned)
check("filter: the lookalikes are still in the plan",
{".env.example", "certs/server.key.pub"} <= planned)
check("filter: a lockfile IS kept — a repo wants its lockfile",
"package-lock.json" in planned)
r = hg("export", *W, "--allow-untitled", "--commands")
# Exact tokens, not substrings: ".env" is inside ".env.example", so a
# substring test reports a leak every time the lookalike is kept —
# which is exactly the behaviour that is wanted.
added = {tok for line in r.stdout.split("\n") if line.startswith("git add -- ")
for tok in line[len("git add -- "):].split()}
check("filter: no secret reached the commands",
not (added & secrets), str(sorted(added & secrets)))
check("filter: the commands cover exactly the planned files",
added == planned, str(sorted(added ^ planned)))
check("filter: no secret reached the copy",
not any((out / "demo" / x).exists() for x in secrets))
out2 = Path(tmp) / "out2"
hg("run", "--source", str(source), "--out", str(out2))
r = hg("export", "--source", str(source), "--out", str(out2), "--allow-untitled")
check("filter: the guards still pass on the filtered set",
"tree matches source" in r.stdout, r.stdout + r.stderr)
tracked = subprocess.run(["git", "-C", str(out2 / "demo"), "ls-files"],
capture_output=True, text=True).stdout.split()
check("filter: the committed history holds no secret",
not (set(tracked) & secrets), str(tracked))
out3 = Path(tmp) / "out3"
hg("run", "--source", str(source), "--out", str(out3), "--keep-secrets")
index3 = json.loads((out3 / "index.json").read_text())
check("filter: --keep-secrets brings them back",
secrets <= set(index3["files"]))
# --- copy: the plain utility, no history involved ---
with tempfile.TemporaryDirectory(prefix="histgen-selftest-copy-") as tmp:
source, out = Path(tmp) / "demo", Path(tmp) / "out"
build(source)
for rel, body in SIFTABLE.items():
(source / rel).parent.mkdir(parents=True, exist_ok=True)
(source / rel).write_text(body)
env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
for cmd in (["init", "-q"], ["add", "-A", "-f", "."],
["commit", "-qm", "init"]):
subprocess.run(["git", "-C", str(source), *cmd], capture_output=True, env=env)
C = ["--source", str(source), "--out", str(out)]
r = hg("copy", *C)
check("copy: succeeds", r.returncode == 0, r.stderr.strip())
made = out / "demo"
check("copy: no .git came across", not (made / ".git").exists())
check("copy: nothing gitignored came across",
not any((made / i).exists() for i in IGNORED))
for rel in sorted(DROPPED):
check(f"copy: left behind {rel}", not (made / rel).exists())
for rel in sorted(KEPT_LOOKALIKES):
check(f"copy: kept {rel}", (made / rel).is_file())
check("copy: every drop is named in the output",
all(rel in r.stdout for rel in DROPPED), r.stdout)
check("copy: a manifest records what was left behind",
(out / "copied.md").is_file()
and all(rel in (out / "copied.md").read_text() for rel in DROPPED))
check("copy: refuses a destination that is not empty",
hg("copy", *C).returncode != 0)
preview = Path(tmp) / "preview"
r = hg("copy", "--source", str(source), "--out", str(preview), "--dry-run")
check("copy: --dry-run writes nothing", not preview.exists() and r.returncode == 0)
full = Path(tmp) / "full"
r = hg("copy", "--source", str(source), "--out", str(full),
"--all", "--keep-secrets")
check("copy: --all --keep-secrets keeps what it says",
(full / "demo" / "package-lock.json").is_file()
and (full / "demo" / ".env").is_file())
picky = Path(tmp) / "picky"
r = hg("copy", "--source", str(source), "--out", str(picky),
"--exclude", "*.png", "--include", "package-lock.json")
check("copy: --exclude drops by glob at any depth",
not (picky / "demo" / "assets" / "logo.png").exists())
check("copy: --include overrides the filters",
(picky / "demo" / "package-lock.json").is_file())
# --- config ---
with tempfile.TemporaryDirectory(prefix="histgen-selftest-cfg-") as tmp:
source, out = Path(tmp) / "demo", Path(tmp) / "out"
build(source)
cfg = Path(tmp) / "settings.json"
cfg.write_text(json.dumps({"source": str(source), "out": str(out)}))
r = hg("config", "--config", str(cfg))
check("config: a file supplies source and out",
r.returncode == 0 and str(out) in r.stdout, r.stderr.strip())
check("config: the command line wins",
"/tmp/override" in hg("config", "--config", str(cfg),
"--out", "/tmp/override").stdout)
cfg.write_text(json.dumps({"source": str(source), "outp": "typo"}))
check("config: a mistyped key is refused",
"unknown key" in hg("config", "--config", str(cfg)).stderr)
cfg.write_text(json.dumps({"repo": str(source), "out": str(out)}))
check("config: the old 'repo' key still works",
hg("config", "--config", str(cfg)).returncode == 0)
check("out inside source is refused",
"inside source" in hg("status", "--source", str(source),
"--out", str(source / "sub")).stderr)
print()
if failures:
print(f"{len(failures)} check(s) failed:")
for f in failures:
print(f" - {f}")
return 1
print("all checks passed")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,142 @@
"""
What to leave behind when copying a repo out of itself.
Three kinds of file get dropped, and they are dropped for three different
reasons. Keeping them apart matters, because only one of the three is safe to
decide silently.
derived something a build regenerates. Lockfiles, source maps, minified
output, compiled objects, cache directories. The list is ported
from `ppl/ctrl/distill.sh`, including the lesson written into its
comments: the line is **derived-vs-content, not text-vs-binary**.
That distinction was wrong there once and cost real files — a
logo, a font the site loads, a downloadable PDF — because none of
those can be regenerated from what is left, which is the only
thing that makes a file safe to drop. So images, fonts and
spreadsheets are content and are kept.
secret a private key, a credential store, an .env holding real values.
distill does not do this; .gitignore usually has, and where it has
not, the file is tracked and travels. That is not hypothetical —
soleprint's own notes record an API key that was tracked in a
tool's .env, and untracking it did not unpublish it.
ignored tracked, and yet matched by the repo's own ignore rules. Someone
ran `git add -f` once. Sometimes deliberate — a built artifact
committed on purpose — and sometimes a dump or a credentials file
that went in and was never noticed again. Reported by name either
way, because the repo is already contradicting itself about them.
oversize whatever --max-bytes says. Size is its own worry and gets its own
knob rather than being smuggled in as a guess about kind.
Every drop is reported. A file quietly missing from a copy is the same class of
failure as a file quietly missing from a history, and this tool exists because
that class of failure is expensive.
"""
import fnmatch
import re
from pathlib import Path
# ── derived ────────────────────────────────────────────────────────────────
# One list, one place to edit. `--all` turns it off wholesale.
NOISE = re.compile(
r'(^|/)(package-lock\.json|pnpm-lock\.yaml|npm-shrinkwrap\.json|yarn\.lock'
r'|bun\.lock|bun\.lockb|uv\.lock|poetry\.lock|Pipfile\.lock|Cargo\.lock'
r'|composer\.lock|Gemfile\.lock|go\.sum|\.DS_Store|Thumbs\.db)$'
r'|\.(map|min\.js|min\.css)$'
r'|\.(pyc|pyo|pyd|class|o|obj|a|lib|so|dylib|dll|wasm|pack|idx)$'
r'|(^|/)(__pycache__|\.pytest_cache|\.mypy_cache|\.ruff_cache|node_modules)/'
)
# ── secret ─────────────────────────────────────────────────────────────────
# Deliberately narrow. A pattern that catches a real key once a year and a
# needed file once a week gets turned off, and then it catches nothing.
SECRET = re.compile(
r'(^|/)\.env(\.[A-Za-z0-9_-]+)?$'
r'|(^|/)(id_rsa|id_dsa|id_ecdsa|id_ed25519)$'
r'|\.(pem|key|p12|pfx|jks|keystore|ppk|asc|gpg)$'
r'|(^|/)(\.netrc|\.npmrc|\.pypirc|\.htpasswd|\.dockercfg)$'
r'|(^|/)\.ssh/'
r'|(^|/)credentials(\.json|\.yaml|\.yml)?$'
r'|(^|/)service-account[-_A-Za-z0-9]*\.json$'
)
# The exceptions matter more than the rule. A committed .env.example is the
# documented way to say what the real one needs, and dropping it takes the
# documentation with the secret.
SECRET_OK = re.compile(
r'\.(example|sample|template|dist|tmpl)$'
r'|(^|/)\.env\.(example|sample|template)$'
r'|\.pub$'
)
def is_derived(rel: str) -> bool:
return bool(NOISE.search(rel))
def is_secret(rel: str) -> bool:
return bool(SECRET.search(rel)) and not SECRET_OK.search(rel)
def matches(rel: str, patterns) -> bool:
"""
Glob match, with distill's rule: a pattern holding no `/` also matches
basenames at any depth, so `--exclude '*.csv'` means what it looks like.
"""
name = Path(rel).name
for pattern in patterns or ():
if fnmatch.fnmatch(rel, pattern):
return True
if "/" not in pattern and fnmatch.fnmatch(name, pattern):
return True
if pattern.endswith("/") and (rel + "/").startswith(pattern.lstrip("/")):
return True
return False
def sift(source: Path, paths, keep_noise=False, keep_secrets=False,
max_bytes=None, exclude=(), include=(), ignored=()):
"""
Split the file list into what to copy and what to leave, with reasons.
`include` is checked first and wins outright: it is the way to say "yes, I
do want that lockfile" without turning the whole filter off.
"""
kept, dropped = [], []
for rel in paths:
if include and matches(rel, include):
kept.append(rel)
continue
if exclude and matches(rel, exclude):
dropped.append((rel, "excluded"))
continue
if not keep_secrets and is_secret(rel):
dropped.append((rel, "secret"))
continue
if rel in ignored:
dropped.append((rel, "ignored"))
continue
if not keep_noise and is_derived(rel):
dropped.append((rel, "derived"))
continue
if max_bytes:
try:
if (source / rel).stat().st_size > max_bytes:
dropped.append((rel, "oversize"))
continue
except OSError:
pass
kept.append(rel)
return kept, dropped
REASONS = {
"ignored": "tracked, but the ignore rules say they should not be",
"derived": "a build regenerates these",
"secret": "looks like a key or a credential",
"oversize": "larger than --max-bytes",
"excluded": "matched --exclude",
}

View File

@@ -0,0 +1,121 @@
"""
Copy a repo's files out of it, without the repo.
Named snapshot.py and not copy.py, which is what it was for about ten minutes.
A module called `copy` beside the code shadows the standard library's, and the
directory lands on sys.path whenever anything is run from inside it — so
`dataclasses` imported this file instead, and every command died on an import
error before parsing a single argument. The verb is still `copy`; the file
cannot be.
The plain utility underneath everything else: point it at a tree, get a folder
holding what the project actually is — no `.git`, nothing gitignored, nothing a
build regenerates, and nothing that looks like a key.
It is the thing to reach for when the history is not the point. Handing a
snapshot to someone, feeding a tree to something that should not see the
history, or getting a clean starting tree before planning one.
What is dropped is reported and written to a manifest, never assumed. A file
missing from a copy without a line saying so is the same failure this whole
tool exists to prevent, one directory earlier.
"""
import shutil
from pathlib import Path
from .census import file_set, ignored_but_tracked, is_git, state_dir
from .cli import fail
from .sift import REASONS, sift
MANIFEST = "copied.md"
def destination(source: Path, out: Path) -> Path:
"""`out/<name>`, so the folder keeps the name the thing already had."""
return Path(out) / source.name
def take(source: Path, out, keep_noise=False, keep_secrets=False, max_bytes=None,
exclude=(), include=(), force=False, dry_run=False, quiet=False):
dest = destination(source, out)
if dest.exists() and any(dest.iterdir()) and not force and not dry_run:
fail(f"{dest} already exists and is not empty.",
"Pass --force to write into it anyway, or point --out elsewhere.")
paths = file_set(source)
kept, dropped = sift(source, paths, keep_noise=keep_noise,
keep_secrets=keep_secrets, max_bytes=max_bytes,
exclude=exclude, include=include,
ignored=ignored_but_tracked(source, paths))
if not quiet:
print(f"{len(paths)} files tracked, {len(kept)} to copy, {len(dropped)} left behind.")
by_reason = {}
for rel, why in dropped:
by_reason.setdefault(why, []).append(rel)
for why in ("secret", "ignored", "derived", "oversize", "excluded"):
hits = by_reason.get(why)
if not hits:
continue
# Secrets are listed in full however many there are. The others are
# bulk and a count is enough; a key that got dropped is a thing you
# want to see the name of, because it means it was tracked.
shown = hits if why in ("secret", "ignored") else hits[:5]
print(f"\n {why}{REASONS[why]} ({len(hits)}):")
for rel in shown:
print(f" {rel}")
if len(hits) > len(shown):
print(f" ... and {len(hits) - len(shown)} more")
if dry_run:
if not quiet:
print(f"\nNothing written. Would copy to {dest}.")
return {"kept": kept, "dropped": dropped, "dest": dest}
dest.mkdir(parents=True, exist_ok=True)
for rel in kept:
target = dest / rel
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source / rel, target)
_manifest(source, out, dest, paths, kept, dropped)
if not quiet:
print(f"\nCopied to {dest}")
print(f" no .git — {'the source has one and it was not copied'
if is_git(source) else 'the source has none either'}")
print(f" what was left behind: {state_dir(out) / MANIFEST}")
return {"kept": kept, "dropped": dropped, "dest": dest}
def _manifest(source, out, dest, paths, kept, dropped):
"""A record of the decision, beside the copy rather than inside it."""
lines = [
f"# Copied from `{source}`", "",
f"- source: `{source}`",
f"- copy: `{dest}`",
f"- {len(paths)} files tracked, {len(kept)} copied, {len(dropped)} left behind",
"",
"No `.git` was copied. The file list is what git tracks, so nothing "
"untracked or ignored came across — except where a file was tracked "
"*despite* the ignore rules, which is listed below rather than assumed.",
"",
]
by_reason = {}
for rel, why in dropped:
by_reason.setdefault(why, []).append(rel)
for why in ("secret", "ignored", "derived", "oversize", "excluded"):
hits = by_reason.get(why)
if not hits:
continue
lines += [f"## {why}{REASONS[why]}", ""]
lines += [f"- `{rel}`" for rel in hits]
lines.append("")
if not dropped:
lines += ["Nothing was left behind.", ""]
path = state_dir(out) / MANIFEST
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("\n".join(lines))

View File

@@ -0,0 +1,77 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>histgen — station</title>
<link rel="stylesheet" href="/theme.css">
<style>
body { font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
background: var(--surface-0, #101418); color: var(--text-0, #d8dee4);
margin: 0; padding: 2rem; line-height: 1.55; }
h1 { margin: 0 0 .25rem; font-size: 1.4rem; }
p.lede { margin: 0 0 1.5rem; color: var(--text-1, #8b98a5); }
form { display: flex; gap: .5rem; margin-bottom: 1.5rem; }
input, button { font: inherit; padding: .45rem .7rem;
background: var(--surface-1, #161c22); color: inherit;
border: 1px solid var(--border, #2a333d); border-radius: 4px; }
button { cursor: pointer; }
.group { border: 1px solid var(--border, #2a333d); border-radius: 4px;
padding: .6rem .9rem; margin-bottom: .5rem;
background: var(--surface-1, #161c22); }
.n { color: var(--accent, #4fb3a6); }
.title { font-weight: 600; }
.untitled { color: var(--text-1, #8b98a5); font-style: italic; }
.paths { margin: .4rem 0 0; padding-left: 1.1rem; color: var(--text-1, #8b98a5);
font-size: .87rem; }
#summary { color: var(--text-1, #8b98a5); margin-bottom: 1rem; }
.err { color: #e08a5c; }
</style>
</head>
<body>
<h1>histgen</h1>
<p class="lede">A proposed history, before it is a history. Read-only —
<code>python -m station.tools.histgen apply &lt;repo&gt;</code> is what commits.</p>
<form onsubmit="load(event)">
<input id="repo" placeholder="path to a repo, e.g. rig" size="40" autofocus>
<button type="submit">Show plan</button>
</form>
<div id="summary"></div>
<div id="groups"></div>
<script>
async function load(event) {
event.preventDefault();
const repo = document.getElementById('repo').value.trim();
const summary = document.getElementById('summary');
const groups = document.getElementById('groups');
groups.innerHTML = ''; summary.textContent = 'Loading…';
try {
const res = await fetch(`/station/tools/histgen/api/plan?repo=${encodeURIComponent(repo)}`);
const data = await res.json();
if (!res.ok) throw new Error(data.detail || res.statusText);
summary.textContent =
`${data.commits} commits over ${data.files} files` +
(data.untitled.length ? `${data.untitled.length} still without a message` : '');
for (const g of data.groups) {
const el = document.createElement('div');
el.className = 'group';
const title = g.title
? `<span class="title">${escapeHtml(g.title)}</span>`
: `<span class="untitled">${escapeHtml(g.slug)} — no message yet</span>`;
el.innerHTML = `<span class="n">${String(g.n).padStart(2,'0')}</span> ${title}
<ul class="paths">${g.paths.map(p => `<li>${escapeHtml(p)}</li>`).join('')}</ul>`;
groups.appendChild(el);
}
} catch (e) {
summary.innerHTML = `<span class="err">${escapeHtml(e.message)}</span>`;
}
}
function escapeHtml(s) {
return String(s).replace(/[&<>"']/g, c =>
({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
}
</script>
</body>
</html>