Compare commits

..

8 Commits

Author SHA1 Message Date
a29e0708e8 new conventions 2026-09-14 06:13:22 -03:00
37c4d588ea Merge branch 'main' into docgen-graphgen 2026-09-13 21:41:29 -03:00
160ee31b8c docgen: drop the superseded first pass, port the style harvester
The IR supersedes both intermediate designs (requirements D6, D7):
station/tools/docgen and the graph model that briefly lived in graphgen.
Shipping them beside atlas2/docgen would mean two graph models, which is the
thing the architecture argues against.

Carried across rather than lost:
  - style/extract.py and tokens.py, the offline theme harvester (R31). Rewritten
    to emit a *theme* — a slot-to-hex binding — rather than a whole style file,
    since what harvesting recovers is which colour a slot should be, not what a
    kind should look like.
  - graphgen/README.md, rewritten for what graphgen actually is now: the
    schema explorer. It fixes the blank station-index entry at run.py:304.

Two bugs found while doing it:
  - Canvas and ink are the two lightness extremes, not the two most common
    values. In a Graphviz SVG every label carries a fill, so the ink outnumbers
    the canvas 87 to 43 and the old rule produced a theme whose text was
    invisible against its own background.
  - A name defined in both branches of an if/else produced a duplicate id, which
    failed validation on docgen's own source. Disambiguated by line.
2026-09-13 21:41:29 -03:00
358b98f826 site emitter 2026-09-12 07:08:00 -03:00
7cb892ccfe add cases for code, dbs, and outline notebook generation 2026-09-12 06:50:05 -03:00
542d704da4 docgen iter 2 2026-09-12 06:42:49 -03:00
49a9f8ee57 sanitized rig 2026-09-12 02:44:40 -03:00
966f8fc821 attemp to develop rig in spr without an actual use case 2026-08-26 07:27:12 -03:00
155 changed files with 22953 additions and 3055 deletions

4
.gitignore vendored
View File

@@ -40,7 +40,5 @@ cfg/dlt/
# not land here. They are versioned in their own repo.
#
# Anchored at the ROOT on purpose: a copy is a SIBLING of rig/, so a rule inside
# rig/.gitignore cannot see it. The negation must name the full path for the same
# reason — `*-rig/` is unanchored and matches at any depth, including rig/sample-rig.
# rig/.gitignore cannot see it.
*-rig/
!rig/sample-rig/

View File

@@ -129,7 +129,7 @@ Every script stays runnable on its own — the standalone rule holds:
```bash
python build.py --cfg amar # -> gen/amar/
cd gen/standalone && python run.py # bare-metal
./ctrl/kind-up.sh # still works directly
./ctrl/cluster.sh up # still runs directly; rig builds the cluster
cd gen/<room> && ./ctrl/start.sh # each room owns its lifecycle scripts
```

View File

@@ -13,7 +13,7 @@
# make component ARGS="publish soleprint-ui /tmp/out --dist"
# make deploy ARGS="--build"
#
# Every script stays runnable on its own (./ctrl/kind-up.sh still works, and each
# Every script stays runnable on its own (./ctrl/cluster.sh up still works, and each
# built room keeps its own gen/<room>/ctrl/*.sh) — the standalone rule holds, and
# this only saves typing.
#

View File

@@ -6,16 +6,49 @@
# ./ctrl/cluster.sh down # delete it (drops every room's namespace)
# ./ctrl/cluster.sh status # what's running on it
#
# One target, one script — the variants live here. The kind-*.sh files stay
# exactly as they are and remain runnable on their own; this only dispatches.
# spr depends on rig, never the other way round. Building and deleting a cluster
# is rig's job, so up and down hand straight to rig/ctrl/cluster.sh, carrying the
# four things that make this cluster spr's rather than rig's defaults:
#
# CLUSTER=spr rooms deploy into the kind-spr context
# KIND_CONFIG spr's own shape, which maps the rooms' gateway NodePorts
# REGISTRY_MODE=none rooms load images straight into the node
# PROFILE=minimal pinned here, so a change to rig's own ctrl/.env can never
# quietly add addons to spr's cluster
#
# status stays here: it answers a question about rooms, not about the cluster.
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RIG_CTRL="$SCRIPT_DIR/../rig/ctrl"
rig() {
CLUSTER=spr \
KIND_CONFIG="$SCRIPT_DIR/k8s/kind-config.yaml" \
REGISTRY_MODE=none \
PROFILE=minimal \
bash "$RIG_CTRL/cluster.sh" "$@"
}
case "${1:-status}" in
up) exec "$SCRIPT_DIR/kind-up.sh" ;;
down) exec "$SCRIPT_DIR/kind-down.sh" ;;
status) exec "$SCRIPT_DIR/kind-status.sh" ;;
up)
rig up
echo
echo "Per-room deploy:"
echo " cd gen/<room> && ./ctrl/k8s-up.sh"
;;
down)
rig down
;;
status)
if ! kind get clusters 2>/dev/null | grep -qx spr; then
echo "No 'spr' kind cluster — run: make cluster up"
exit 0
fi
kubectl --context kind-spr get namespaces -l soleprint-room
echo
kubectl --context kind-spr get pods -A -l soleprint-room
;;
*)
echo "Unknown subcommand: $1" >&2
echo "Usage: cluster.sh [up|down|status]" >&2

View File

@@ -3,9 +3,15 @@ apiVersion: kind.x-k8s.io/v1alpha4
# Single shared cluster for all soleprint rooms.
# Each room deploys into its own namespace; gateway Services pick a
# NodePort from the 30080-30099 range mapped here.
name: spr
#
# Built by rig, not by spr: ctrl/cluster.sh hands this file to
# rig/ctrl/cluster.sh, which substitutes CLUSTER and NODE_IMAGE (named without
# braces here so this comment survives the substitution). The shape is spr's —
# what its cluster needs is spr's business. Building it is rig's.
name: ${CLUSTER}
nodes:
- role: control-plane
image: ${NODE_IMAGE}
extraPortMappings:
# Room gateway NodePorts (one per active room).
- {containerPort: 30080, hostPort: 30080, protocol: TCP}

View File

@@ -1,12 +0,0 @@
#!/bin/bash
# Delete the shared `spr` kind cluster (drops every room's namespace too).
# Use `gen/<room>/ctrl/k8s-down.sh` instead if you only want to remove
# a single room's namespace.
set -e
if kind get clusters 2>/dev/null | grep -q '^spr$'; then
echo "Deleting kind cluster 'spr'..."
kind delete cluster --name spr
else
echo "No kind cluster 'spr' to delete."
fi

View File

@@ -1,12 +0,0 @@
#!/bin/bash
# Show what's running on the shared `spr` cluster.
set -e
if ! kind get clusters 2>/dev/null | grep -q '^spr$'; then
echo "No 'spr' kind cluster — run ctrl/kind-up.sh"
exit 0
fi
kubectl --context kind-spr get namespaces -l soleprint-room
echo
kubectl --context kind-spr get pods -A -l soleprint-room

View File

@@ -1,21 +0,0 @@
#!/bin/bash
# Create (or no-op) the single shared `spr` kind cluster used by every
# soleprint room. Per-room work happens inside namespaces — see
# `gen/<room>/ctrl/k8s-up.sh`.
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
KIND_CONFIG="$SCRIPT_DIR/k8s/kind-config.yaml"
if kind get clusters 2>/dev/null | grep -q '^spr$'; then
echo "Kind cluster 'spr' already exists."
else
echo "Creating kind cluster 'spr'..."
kind create cluster --config "$KIND_CONFIG"
fi
kubectl config use-context kind-spr >/dev/null
echo
echo "Cluster ready. Per-room deploy:"
echo " cd gen/<room> && ./ctrl/k8s-up.sh"

7
rig/.gitignore vendored
View File

@@ -11,10 +11,9 @@ ctrl/.env
arch/*.dot
ctrl/Tiltfile.gen
# binaries pulled by `make deps-bundle` for the air-gapped wizard image
# binaries pulled by `make deps-bundle` for the air-gapped installer image
vendor
# Client rigs are NOT ignored here. A copy is a SIBLING of this directory
# (spr/acme-rig), so a rule in this file cannot see it — the rules live in
# spr/.gitignore, anchored at spr's root, where `*-rig/` matches the siblings and
# `!rig/sample-rig/` keeps the committed stand-in.
# (../acme-rig), so a rule in this file cannot see it — the rules live in the
# parent repo's .gitignore, anchored at its root, where `*-rig/` matches them.

View File

@@ -4,9 +4,8 @@ The README says the prerequisite is Docker and nothing else. This is what that
actually looks like end to end: a bare Linux box, and a new project running under
Tilt at the end of it.
rig lives inside soleprint, at `spr/rig` — it is soleprint's cluster half, and
a client copy is a sibling (`spr/acme-rig`). Paths below are relative to
soleprint's checkout.
A copy of this directory is a sibling of it, named after the environment it
models (`acme-rig`). Paths below are relative to the parent checkout.
It spans three repos because the work does. **rig** prepares the machine — the
pinned toolchain, the cluster, the port arithmetic. **all** owns the shape a
@@ -50,7 +49,7 @@ isn't.
## Read the docs before installing anything
```bash
cd spr/rig
cd rig
make docs
```
@@ -67,11 +66,11 @@ persists; ctrl-c ends it.
## Ask what is wrong with this machine
```bash
make station
make check
cp ctrl/.env.example ctrl/.env
```
`station.sh` reports and instructs, and fixes nothing. It runs bare rather than
`check.sh` reports and instructs, and fixes nothing. It runs bare rather than
in a container because host detection only ever reads `/proc` and `/etc` — no
dependency beyond coreutils.
@@ -80,7 +79,7 @@ port rig binds derives from this directory's name, so the answer is specific to
this copy, and a clash here surfaces as an opaque `failed to bind host port` in
the middle of cluster creation if you skip it.
Copy the `.env` even though station only warns about it. It is gitignored, it is
Copy the `.env` even though the check only warns about it. It is gitignored, it is
where a machine-local override goes, and `ports.sh persist` expects it to exist.
@@ -88,33 +87,33 @@ where a machine-local override goes, and `ports.sh persist` expects it to exist.
This is the step where "nothing installed" stops being rhetorical.
`make deps` runs `ctrl/wizard.sh install` directly on the host, and the wizard
`make deps` runs `ctrl/deps.sh install` directly on the host, and the installer
fetches with `curl`. A stock `debian:trixie-slim` has no curl — detection runs
fine, then the first download dies with `curl: command not found` and an exit
code of 127. That is the bootstrap paradox `ctrl/Dockerfile.wizard` exists
to kill — the wizard carries its own toolchain so the host needs only Docker —
code of 127. That is the bootstrap paradox `ctrl/Dockerfile.deps` exists
to kill — the installer carries its own toolchain so the host needs only Docker —
but building the image and running it are two different things, and only the
build has a Makefile target today. **On a genuinely bare machine, run it by
hand:**
```bash
make wizard # builds rig-wizard:wizard
make deps-image # builds rig-deps:deps
mkdir -p ~/.local/bin
docker run --rm \
-v /:/host:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v "$HOME/.local/bin:/out/bin" \
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
rig-wizard:wizard install dev
rig-deps:deps install dev
```
The image name follows the directory, like everything else here: in `spr/rig`
it is `rig-wizard`, in a copy called `spr/acme-rig` it is `acme-rig-wizard`. The
tag is `wizard` (or `full`, below), not `latest`.
The image name follows the directory, like everything else here: in `rig`
it is `rig-deps`, in a copy called `acme-rig` it is `acme-rig-deps`. The
tag is `deps` (or `full`, below), not `latest`.
None of the four arguments are guessable, so:
- **`/:/host:ro`** — the wizard reads the *host's* `/etc/os-release` and
- **`/:/host:ro`** — the installer reads the *host's* `/etc/os-release` and
`/etc/wsl.conf`, not the container's. `HOST_ROOT=/host` is already baked into
the image; this is what it points at. Read-only, and it is the only reason
detection inside a container tells you anything about the machine.
@@ -122,7 +121,7 @@ None of the four arguments are guessable, so:
and how it counts kind clusters already running.
- **`/out/bin`** — the image's `OUT_BIN`. Whatever you mount here is where the
four binaries land.
- **`HOST_UID` / `HOST_GID`** — the wizard runs as root so it can reach that
- **`HOST_UID` / `HOST_GID`** — the installer runs as root so it can reach that
socket, which means everything it writes into a mounted volume is root-owned
and useless to you. These drive the `chown` back. Omit them and the install
looks like it worked.
@@ -131,18 +130,18 @@ None of the four arguments are guessable, so:
tooling — which is the right answer on a managed or corporate-issued machine and
is why the split exists.
Then put them on PATH, which the wizard will remind you about because it cannot
Then put them on PATH, which the installer will remind you about because it cannot
edit your shell for you:
```bash
export PATH="$HOME/.local/bin:$PATH" # and add the same line to ~/.bashrc
```
If something else on this machine already provides `kubectl`, the wizard says so
If something else on this machine already provides `kubectl`, the installer says so
by name rather than shadowing it quietly. `OUT_BIN=$PWD/def/bin` installs
somewhere private instead.
**Two variants worth knowing before you need them.** `make wizard full` bakes
**Two variants worth knowing before you need them.** `make deps-image full` bakes
every pinned binary into the image at build time (`DEPS_SOURCE=baked`), so
`docker save` gives you the entire installer as one file to carry into an
air-gapped network. And `DEPS_SOURCE=artifactory` with `DEPS_ARTIFACTORY_URL`
@@ -195,8 +194,8 @@ follows is only the mechanical part.
```bash
SLUG=<slug> # short, lowercase, no separators
cp -r ~/wdir/all/projects/templates/broad ~/wdir/"$SLUG"
cd ~/wdir/"$SLUG"
cp -r ~/wdir/semester/all/projects/templates/broad ~/wdir/semester/"$SLUG"
cd ~/wdir/semester/"$SLUG"
grep -rl '<slug>' ctrl | xargs sed -i "s/<slug>/$SLUG/g"
cp ctrl/k8s/.env.example ctrl/k8s/.env
git init && git add -A && git commit -m "scaffold $SLUG from broad"
@@ -214,7 +213,7 @@ scaffold's Makefile from the directory, so there is nothing to edit for either.
is already in use, so copying it unchanged puts two projects on one port:
```bash
grep -h '^TILT_PORT=' ~/wdir/*/ctrl/k8s/.env 2>/dev/null | sort
grep -h '^TILT_PORT=' ~/wdir/semester/*/ctrl/k8s/.env 2>/dev/null | sort
```
Choose a free one in `1030010399` — the range ALL reserves in
@@ -244,7 +243,7 @@ The workload is an nginx placeholder so a fresh copy reaches something that
answers; replace it. Keep `30080` in step between the overlay patch and
`kind-config.yaml`'s `containerPort` — the hostPort is this project's to pick.
Reachability is a plain kind port mapping: no ingress controller and no MetalLB.
Caddy maps `<slug>.local.ar` onto the host port (`~/wdir/ppl/local/Caddyfile`),
Caddy maps `<slug>.local.ar` onto the host port (`~/wdir/semester/ppl/local/Caddyfile`),
with `*.local.ar` resolving to 127.0.0.1 through dnsmasq. That is the whole chain.
**The one file the scaffold still does not ship is `ctrl/Tiltfile`**`make
@@ -270,9 +269,9 @@ delete-and-recreate for when a cluster wedges.
## Register it
The project exists; now it is findable. Add an entry to
`~/wdir/all/projects/index.json` and write its `projects/<slug>.md` beside the
`~/wdir/semester/all/projects/index.json` and write its `projects/<slug>.md` beside the
others. Structured fields in the index, prose in the markdown.
Putting it on the CI server and deploying it is `ppl`'s half, and it starts at
`~/wdir/ppl/ctrl/init-repo.sh` — gitea remote, then Woodpecker. That is a
`~/wdir/semester/ppl/ctrl/init-repo.sh` — gitea remote, then Woodpecker. That is a
different document.

View File

@@ -20,7 +20,7 @@ SLUG := $(shell echo '$(notdir $(CURDIR))' | tr '[:upper:]' '[:lower:]' | t
CLUSTER := $(or $(shell sed -n 's/^CLUSTER=//p' ctrl/.env 2>/dev/null),$(SLUG))
KCTX := --context kind-$(CLUSTER)
TILT_PORT := $(shell sed -n 's/^TILT_PORT=//p' ctrl/.env 2>/dev/null)
WIZARD := $(SLUG)-wizard
DEPSIMG := $(SLUG)-deps
# Words after the target become the script's subcommand. Make would otherwise
# treat them as goals of their own, so each gets a no-op rule.
@@ -35,7 +35,7 @@ $(eval $(ARGS):;@:)
.PHONY: $(ARGS)
endif
.PHONY: help setup station deps wizard cluster registry addons ports \
.PHONY: help setup check mem deps deps-image pins cluster registry addons ports \
newbox dockerhost docs tilt \
kind-up kind-down kind-reset tilt-up tilt-down
@@ -47,16 +47,22 @@ help: ## list targets
setup: ## prepare this machine [core] [--share-docker] [--cluster]
bash ctrl/setup.sh $(ARGS)
station: ## is this workstation ready? reports, never fixes
bash ctrl/station.sh
check: ## is this machine ready? reports, never fixes
bash ctrl/check.sh
mem: ## memory, and any cap holding it [status|backup|restore]
bash ctrl/mem.sh $(or $(ARGS),status)
deps: ## install the toolchain [core|dev] (default dev)
bash ctrl/wizard.sh install $(or $(ARGS),dev)
bash ctrl/deps.sh install $(or $(ARGS),dev)
pins: ## standalone/rigdeps.sh still installs what rig pins?
bash ctrl/pins.sh
wizard: ## build the installer image [full]
docker build -f ctrl/Dockerfile.wizard \
--target $(if $(filter full,$(ARGS)),wizard-full,wizard) \
-t $(WIZARD):$(if $(filter full,$(ARGS)),full,wizard) .
deps-image: ## build the installer image [full]
docker build -f ctrl/Dockerfile.deps \
--target $(if $(filter full,$(ARGS)),deps-full,deps) \
-t $(DEPSIMG):$(if $(filter full,$(ARGS)),full,deps) .
# ── cluster ────────────────────────────────────────────────────────────────

View File

@@ -9,6 +9,46 @@ topology, not the workloads.
**Docker.** Nothing else — no curl, no jq, no python, no apt repositories.
### Starting from plain Windows
Everything here is bash and runs *inside* a Linux shell, so on a Windows machine
that means WSL. Nothing in rig installs WSL, and nothing will: `wsl --install`
enables Windows features and requires a reboot, which is not something a script
should do to a machine on your behalf — and there is no tested undo for it.
From an elevated PowerShell or Command Prompt, once:
```powershell
wsl --install
```
Then reboot and open the Linux shell it installed.
**If you cloned this on the Windows side, copy it into WSL before carrying on.**
WSL can reach the Windows drives at `/mnt/c`, and working from there mostly
functions — slowly — but file watching does not: that filesystem raises no
inotify events, so anything watching for edits silently stops seeing them.
```bash
cp -r /mnt/c/Users/<you>/rig ~/rig
cd ~/rig
```
`make deps` reports it if you are running from `/mnt/...`. Then carry on below.
If it fails, the usual causes give unhelpful messages:
| symptom | cause |
| --- | --- |
| "the virtual machine could not be started" | virtualization disabled in BIOS/UEFI |
| the command is not recognised | Windows build too old — needs 2004 or later |
| the install starts, then nothing works | a reboot is still pending |
Running the scripts from **Git Bash, MSYS or Cygwin does not work** — those look
close enough to a Linux shell to get started and then fail without `/proc` or a
docker socket. `ctrl/deps.sh` detects that and says so rather than letting you
find out the slow way.
## Read the docs first
```bash
@@ -21,16 +61,43 @@ instructions for everything else. No cluster and no toolchain required.
## Then
```bash
make station # report host and config problems; changes nothing
make check # report host and config problems; changes nothing
make deps # install the toolchain (add `core` on a managed machine)
make cluster up # build the cluster for the active profile
```
`make cluster up` also starts this environment's local registry and wires it
into the node, so an image built locally is pullable by the cluster without
going near docker.io:
```bash
make registry status # prints: endpoint localhost:<port>
docker build -t localhost:<port>/app:1 .
docker push localhost:<port>/app:1
kubectl --context kind-$(basename $PWD) run app --image=localhost:<port>/app:1
```
The port block is derived from the directory name, so two copies of rig never
collide:
```bash
make ports show # HTTP / HTTPS / TILT / REGISTRY
make cluster list # every cluster on this machine, with memory
make cluster free # stop the others if memory is tight
make cluster down # remove this cluster and its registry
```
**`make tilt` has nothing to run yet.** The target and its `tilt-up` / `tilt-down`
aliases exist so rig answers to the same spelling as every other project here,
but rig ships no `Tiltfile` — it builds the estate, it is not itself a service
with a dev loop. Add a `ctrl/Tiltfile` and the target works; until then it fails
on the missing file, not on anything rig did.
`make help` lists every target.
On a machine where Docker really is the only thing installed, `make deps` has
nothing to download with — see [BOOTSTRAP.md](BOOTSTRAP.md), which runs the
toolchain through the wizard container and carries on to scaffolding and running
toolchain through the installer container and carries on to scaffolding and running
a new project.
## One directory is one environment
@@ -39,10 +106,10 @@ Copy this directory, rename it, run it. Cluster name, kubectl context, image
tags and the host port block all derive from the directory name, so copies never
collide and neither one's teardown can touch the other.
rig lives inside soleprint, at `spr/rig` — it is soleprint's cluster half, and a
copy is a **sibling**: `spr/acme-rig`. That is why the ignore rules for client
rigs sit in `spr/.gitignore` rather than here; a rule in this directory cannot
see a directory beside it.
A copy of this directory is a **sibling** of it, named after the environment it
models (`acme-rig`). That is why the ignore rules for copies sit in the *parent*
repo's `.gitignore` rather than here: a rule in this directory cannot see a
directory beside it.
## Profiles
@@ -54,7 +121,7 @@ apiserver audits. They live in `ctrl/env.d/`, and the active one is `PROFILE`.
| `minimal` | the default. One node, no addons, boots fast. |
| `client` | the regulated-estate shape — multi-node, audit on, registry mirror. |
| `offline` | air-gapped: everything from a preloaded local registry. |
| `data` | the dependency containers a soleprint room asks for. |
| `data` | the cabinets an environment asks for. |
```bash
PROFILE=data make cluster up
@@ -98,10 +165,10 @@ cluster does.
| `redis` | cache and broker |
| `airflow` | scheduled pipelines; needs postgres and redis |
The last three are the cluster half of **soleprint's cabinets**. A room declares
what it needs once, in `cfg/<room>/data/cabinets.json`; soleprint's `build.py`
composes those services into `docker-compose.yml` for a laptop, and these
install the same ones here. The names match on purpose each cabinet carries a
The last three are **cabinets**: a public service dropped in as-is, the upstream
image unmodified, reachable at a known address. A cabinet is declared once and
installs on either target — a `service.yml` composes it for a laptop, and these
install the same one here. The names match on purpose: each cabinet carries a
`rig_addon` field pointing at `ctrl/addons/<name>.sh`.
Plain manifests rather than helm charts, like every other addon: a chart repo is

View File

@@ -26,10 +26,10 @@ PROFILE=minimal
# MANIFESTS_DIR=../platform-manifests/overlays/dev
MANIFESTS_DIR=ctrl/k8s/overlays/dev
# Where the wizard fetches the pinned binaries from.
# Where the installer fetches the pinned binaries from.
# upstream GitHub releases / dl.k8s.io (needs internet)
# artifactory a generic repo — what a locked-down client usually allows
# baked already inside the wizard image; no network at all
# baked already inside the installer image; no network at all
DEPS_SOURCE=upstream
DEPS_ARTIFACTORY_URL=
@@ -43,7 +43,7 @@ REGISTRY_PASSWORD=
# Corporate root CA, if Artifactory is fronted by an internal CA (it usually is).
# Trust has to reach THREE places and nothing does it for you: the host docker
# daemon, every kind node's containerd, and any in-cluster client. registry.sh
# handles the first two; station.sh reports when it's configured but not trusted.
# handles the first two; check.sh reports when it's configured but not trusted.
# Symptom when missing: x509: certificate signed by unknown authority
REGISTRY_CA_FILE=

View File

@@ -1,18 +1,18 @@
# The installation wizard. It does NOT run the cluster — it installs a toolchain
# The toolchain installer image. It does NOT run the cluster — it installs a toolchain
# onto the host and gets out of the way.
#
# This exists to kill a bootstrap paradox: a plain bash installer needs curl, jq
# and sha256sum to already be present, and a minimal Debian has none of them.
# The wizard carries its own toolchain, so the only host prerequisite is Docker.
# It carries its own toolchain, so the only host prerequisite is Docker.
#
# Two variants from one file:
# docker build -f ctrl/Dockerfile.wizard --target wizard -t <slug>-wizard .
# docker build -f ctrl/Dockerfile.wizard --target wizard-full -t <slug>-wizard:full .
# docker build -f ctrl/Dockerfile.deps --target deps -t <slug>-deps .
# docker build -f ctrl/Dockerfile.deps --target deps-full -t <slug>-deps:full .
#
# wizard-full bakes every pinned binary in at build time. `docker save` it and
# deps-full bakes every pinned binary in at build time. `docker save` it and
# you have the whole installer as one file to carry into an air-gapped network.
FROM debian:trixie-slim AS wizard
FROM debian:trixie-slim AS deps
# ca-certificates + curl: fetch and verify. graphviz + python3: render diagrams
# and validate the arch model, so the host never needs an apt package.
@@ -26,21 +26,21 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
WORKDIR /work
COPY ctrl/versions.env /work/ctrl/versions.env
COPY ctrl/wizard.sh /work/ctrl/wizard.sh
RUN chmod +x /work/ctrl/wizard.sh
COPY ctrl/deps.sh /work/ctrl/deps.sh
RUN chmod +x /work/ctrl/deps.sh
# Defaults; every one is overridable with -e at run time.
ENV DEPS_SOURCE=upstream \
OUT_BIN=/out/bin \
HOST_ROOT=/host
ENTRYPOINT ["/work/ctrl/wizard.sh"]
ENTRYPOINT ["/work/ctrl/deps.sh"]
CMD ["install"]
# ---------------------------------------------------------------------------
# wizard-full — same wizard, binaries baked in, works with no network at all.
FROM wizard AS wizard-full
RUN /work/ctrl/wizard.sh fetch --to /opt/rig/bin
# deps-full — same image, binaries baked in, works with no network at all.
FROM deps AS deps-full
RUN /work/ctrl/deps.sh fetch --to /opt/rig/bin
ENV DEPS_SOURCE=baked \
BAKED_BIN=/opt/rig/bin

View File

@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Apache Airflow — the cluster half of soleprint's airflow cabinet.
# Apache Airflow — the cluster half of the airflow cabinet.
#
# Airflow needs a metadata database before it will start at all, so this refuses
# rather than rolls a pod that will CrashLoopBackOff while the real problem
@@ -7,7 +7,7 @@
#
# One pod on `standalone`, matching the compose cabinet: migration, admin user,
# scheduler and webserver in a single container. The official chart's five
# deployments model an installation; a room switching this on wants pipelines.
# deployments model an installation; switching this on means wanting pipelines.
set -euo pipefail
cd "$(dirname "$0")/.."

View File

@@ -1,9 +1,10 @@
#!/usr/bin/env bash
# PostgreSQL — the cluster half of soleprint's postgres cabinet.
# PostgreSQL — the cluster half of the postgres cabinet.
#
# A room declares the dependency once, in cfg/<room>/data/cabinets.json. On a
# laptop `build.py` composes it into docker-compose.yml; here it becomes a pod,
# so the same declaration works either way and nothing has to be remembered
# A cabinet is a public service dropped into the environment as-is — the
# upstream image, unmodified, reachable at a known address. This is the cluster
# half of it; the compose half is a `service.yml` beside a `cabinet.json`. The
# declaration is made once and both paths read it, so nothing is remembered
# twice.
#
# Plain manifests rather than a helm chart, matching the other addons: a chart
@@ -32,8 +33,8 @@ if $K get secret -n "$NS" postgres >/dev/null 2>&1; then
else
password=$(head -c 18 /dev/urandom | base64 | tr -d '/+=' | head -c 24)
$K create secret generic postgres -n "$NS" \
--from-literal=POSTGRES_DB="${POSTGRES_DB:-soleprint}" \
--from-literal=POSTGRES_USER="${POSTGRES_USER:-soleprint}" \
--from-literal=POSTGRES_DB="${POSTGRES_DB:-postgres}" \
--from-literal=POSTGRES_USER="${POSTGRES_USER:-postgres}" \
--from-literal=POSTGRES_PASSWORD="$password" >/dev/null
echo " generated a password (read it back with the command printed below)"
fi

View File

@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Redis — the cluster half of soleprint's redis cabinet.
# Redis — the cluster half of the redis cabinet.
#
# Cache, and the broker anything queue-shaped runs on. No persistence: a broker
# that loses its queue on restart is the honest local model, and a PVC here buys

221
rig/ctrl/check.sh Executable file
View File

@@ -0,0 +1,221 @@
#!/usr/bin/env bash
# Readiness check: is this machine ready to run rig?
#
# Reports and instructs; never silently fixes anything. Everything it finds is
# either already fine, or something a human has to decide on.
#
# Runs ctrl/deps.sh host detection in a container when Docker is the only thing
# installed, or directly when the toolchain is already present. Then adds the
# checks that need this repo's config: profile sanity, CA trust, port clashes.
set -euo pipefail
cd "$(dirname "$0")"
DEPS_IMAGE="${DEPS_IMAGE:-$(basename "$(cd .. && pwd)")-deps}"
# Host detection. Prefer running it bare — it needs no dependencies beyond
# coreutils — and fall back to the container only if this shell can't.
bash ./deps.sh detect
# ── repo-level checks ──────────────────────────────────────────────────────
source ./lib/config.sh
load_config
echo
echo "config"
echo " profile ${PROFILE_NAME} (nodes=${NODES} audit=${AUDIT})"
echo " cluster ${CLUSTER} (context ${KUBECONTEXT})"
echo " registry ${REGISTRY_MODE}"
echo " ingress ${INGRESS_MODE}"
if [ ! -f ./.env ]; then
echo " ! ctrl/.env missing — copy it: cp ctrl/.env.example ctrl/.env"
fi
# ── memory ─────────────────────────────────────────────────────────────────
#
# A profile on a box that is already full is the most common first failure, and
# it presents as pods stuck Pending rather than anything that says "memory".
# Warns; never blocks. Whether to try anyway is the user's call.
# A /proc/meminfo field in MB, 0 if absent. MEMINFO and OVERCOMMIT_FILE exist
# only so the tight and does-not-fit branches can be exercised against another
# machine's real numbers; in normal use they are the kernel's own files.
mb_of() {
awk -v k="$1:" '$1 == k { printf "%d", $2 / 1024; found = 1 }
END { if (!found) printf "0" }' "${MEMINFO:-/proc/meminfo}"
}
# What one node costs, measured rather than guessed. On 2026-09-11 a minimal
# control-plane node ran at 620 MiB idle and ~728 MiB with a small mock, plus
# 16 MiB for the local registry — ~745 MiB of working set. 800 rounds that up,
# and agrees with the 800 MB observed independently on a larger rig. Worker
# nodes carry no etcd or apiserver and are lighter, so for a multi-node shape
# this errs high. It is the cluster alone: whatever you deploy comes on top.
NODE_MB=800
# Every running container's working set in MB, tagged with the kind cluster it
# belongs to ('-' when it is not kind). docker stats reports usage minus page
# cache, which is what actually competes — cache is handed back under pressure.
# Counting only kind would hide the usual culprit on a managed workspace, where
# the memory is held by other containers entirely.
container_mb() {
docker info >/dev/null 2>&1 || return 0
awk -F'\t' '
FILENAME == ARGV[1] { cl[$1] = ($2 == "" ? "-" : $2); if ($2 != "") isc[$2] = 1; next }
{
grp = ($1 in cl ? cl[$1] : "-")
# A kind cluster'"'"'s local registry is a plain container with no kind
# label, named <cluster>-registry, so on its own it would read as a
# stranger. It belongs to its cluster — but only if that cluster exists:
# a registry whose cluster is gone is a genuine stray, and says so.
if (grp == "-" && $1 ~ /-registry$/) {
base = $1; sub(/-registry$/, "", base)
if (base in isc) grp = base
}
split($2, u, " "); v = u[1]; mb = 0
if (v ~ /GiB$/) { sub(/GiB$/, "", v); mb = v * 1024 }
else if (v ~ /MiB$/) { sub(/MiB$/, "", v); mb = v }
else if (v ~ /KiB$/) { sub(/KiB$/, "", v); mb = v / 1024 }
else if (v ~ /B$/) { sub(/B$/, "", v); mb = v / 1048576 }
printf "%d\t%s\t%s\n", mb, grp, $1
}
' <(docker ps --format '{{.Names}}\t{{.Label "io.x-k8s.kind.cluster"}}' 2>/dev/null) \
<(docker stats --no-stream --format '{{.Name}}\t{{.MemUsage}}' 2>/dev/null)
}
total_mb=$(mb_of MemTotal)
avail_mb=$(mb_of MemAvailable)
swap_used_mb=$(( $(mb_of SwapTotal) - $(mb_of SwapFree) ))
overcommit=$(cat "${OVERCOMMIT_FILE:-/proc/sys/vm/overcommit_memory}" 2>/dev/null || echo '?')
need_mb=$(( NODES * NODE_MB ))
rows=$(container_mb)
# Once this environment's own cluster is running, its real footprint is already
# out of MemAvailable and the per-node estimate stops being relevant. Subtracting
# the measurement from the estimate would count the same memory twice, and a
# running cluster that happens to sit under 800 MB would still "need" the gap.
ours_mb=$(awk -F'\t' -v c="$CLUSTER" '$2 == c { s += $1 } END { print s + 0 }' <<< "$rows")
still_mb=$(( ours_mb > 0 ? 0 : need_mb ))
echo
echo "memory"
printf " this profile ~%d MB %s node(s) x %d MB — the cluster alone, your workload on top\n" \
"$need_mb" "$NODES" "$NODE_MB"
if [ "$ours_mb" -gt 0 ]; then
printf " already held %d MB by '%s', which is up\n" "$ours_mb" "$CLUSTER"
fi
printf " available %d MB of %d MB\n" "$avail_mb" "$total_mb"
# The biggest things holding memory right now, other than this cluster: kind
# clusters summed per cluster, everything else by container name.
others=$(awk -F'\t' -v c="$CLUSTER" '
$2 != c && $2 != "-" && $2 != "" { k["kind cluster \x27" $2 "\x27"] += $1 }
$2 == "-" { k["container \x27" $3 "\x27"] += $1 }
END { for (n in k) printf "%d\t%s\n", k[n], n }' <<< "$rows" | sort -rn)
if [ -n "$others" ]; then
echo " held elsewhere:"
head -6 <<< "$others" | awk -F'\t' '{ printf " %6d MB %s\n", $1, $2 }'
n_others=$(wc -l <<< "$others")
if [ "$n_others" -gt 6 ]; then
echo " ... and $((n_others - 6)) more"
fi
fi
headroom=$(( avail_mb - still_mb ))
if [ "$still_mb" -eq 0 ]; then
if [ "$headroom" -ge 512 ]; then
printf " fits — already up; %d MB headroom for what you deploy\n" "$headroom"
else
printf " ! already up, but only %d MB headroom for anything you deploy\n" "$headroom"
fi
elif [ "$headroom" -ge 512 ]; then
printf " fits — %d MB headroom for what you deploy\n" "$headroom"
elif [ "$headroom" -ge 0 ]; then
printf " ! fits, but only %d MB headroom for anything you deploy\n" "$headroom"
else
printf " ! does not fit right now: ~%d MB needed, %d MB available\n" "$still_mb" "$avail_mb"
# Two failures with opposite fixes, and telling them apart is the point.
if [ "$still_mb" -le "$total_mb" ]; then
echo " The machine is big enough; something else is holding memory (above)."
echo " Stopping that is what helps — a bigger VM would not."
if grep -q 'kind cluster' <<< "$others"; then
echo " 'make cluster free' stops the other kind clusters. It stops, never deletes."
fi
else
echo " The machine itself is too small: ~${still_mb} MB needed, ${total_mb} MB total."
fi
fi
if [ "$swap_used_mb" -gt 0 ]; then
printf " ! %d MB already in swap — available memory does not count it, so expect a\n" "$swap_used_mb"
echo " cluster here to be slow well before it fails"
fi
if [ "$overcommit" = "1" ]; then
echo " ! overcommit=1: allocations never fail here, so read 'fits' as a ceiling."
echo " A cluster that starts cleanly can still lose processes to the OOM killer."
fi
# The CA reaches three places and only one of them is ours. Report the other two.
if [ -n "${REGISTRY_CA_FILE:-}" ]; then
echo
echo "registry CA"
if [ ! -r "$REGISTRY_CA_FILE" ]; then
echo " ! REGISTRY_CA_FILE not readable: $REGISTRY_CA_FILE"
else
echo " file $REGISTRY_CA_FILE"
host="${REGISTRY_REMOTE_URL#*://}"; host="${host%%/*}"
if [ -n "$host" ] && [ ! -f "/etc/docker/certs.d/${host}/ca.crt" ]; then
echo " ! the HOST docker daemon does not trust it yet:"
echo " sudo mkdir -p /etc/docker/certs.d/${host}"
echo " sudo cp ${REGISTRY_CA_FILE} /etc/docker/certs.d/${host}/ca.crt"
echo " (kind nodes are handled by registry.sh; in-cluster clients are the workload's job)"
fi
fi
fi
# Host ports this environment will try to bind. Checked before cluster creation
# because docker reports a clash halfway through, as an opaque
# "failed to bind host port ...: address already in use".
echo
echo "ports (block derived from the directory name — see 'make ports')"
port_busy() {
if command -v ss >/dev/null 2>&1; then
ss -ltn "sport = :$1" 2>/dev/null | grep -q LISTEN && return 0 || return 1
fi
# iproute2 is absent from a minimal Debian, so fall back to procfs rather
# than silently reporting everything as free.
local hex; hex=$(printf ':%04X' "$1")
grep -qi "^ *[0-9]*: [0-9A-F]*$hex " /proc/net/tcp /proc/net/tcp6 2>/dev/null
}
# A port held by THIS environment's own cluster is not a clash — it is the thing
# working. Reporting it as a problem every time the cluster is up would train
# people to ignore this section, which is the opposite of the point.
# Extract with a second grep rather than `tr -d ':->'`: in tr, ':->' is the
# character RANGE ':' to '>', which does not contain '-', so the trailing dash
# survives and nothing ever matches.
ours=$(docker ps --filter "label=io.x-k8s.kind.cluster=${CLUSTER}" \
--format '{{.Ports}}' 2>/dev/null | tr ',' '\n' \
| grep -oE ':[0-9]+->' | grep -oE '[0-9]+' || true)
clash=0
for entry in "HTTP:${HTTP_PORT}" "HTTPS:${HTTPS_PORT}" \
"TILT:${TILT_PORT}" "REGISTRY:${REGISTRY_PORT}"; do
name="${entry%%:*}"; p="${entry#*:}"
[ -n "$p" ] || continue
if ! port_busy "$p"; then
printf " %-9s %-6s free\n" "$name" "$p"
elif echo "$ours" | grep -qx "$p"; then
printf " %-9s %-6s in use by this environment's cluster\n" "$name" "$p"
else
printf " ! %-9s %-6s IN USE by something else\n" "$name" "$p"
clash=1
fi
done
if [ "$clash" -eq 1 ]; then
echo " override the clashing one in ctrl/.env, e.g. HTTP_PORT=21080"
echo " (or rename this directory — the whole block follows the name)"
fi

View File

@@ -25,7 +25,7 @@ up() {
# Say what this profile locks in BEFORE spending minutes building it:
# the audit policy is an apiserver flag and cannot be changed later.
echo "creating cluster '$CLUSTER' from profile '$PROFILE_NAME'"
echo " shape ctrl/k8s/$KIND_CONFIG"
echo " shape ${KIND_CONFIG_SHOWN}"
echo " nodes $NODES"
echo " image $NODE_IMAGE"
echo " audit $AUDIT"

View File

@@ -1,17 +1,22 @@
#!/usr/bin/env bash
# The installation wizard: detect the host, install a pinned toolchain onto it,
# then report what it could not do. It never runs the cluster and never mutates
# the host outside the directories mounted into it.
# Toolchain installer: detect the host, install a pinned toolchain onto it, then
# report what it could not do.
#
# Usage (normally via `make station` / `make deps`, or directly):
# wizard.sh detect # report host facts only, change nothing
# wizard.sh fetch [core|dev] [--to DIR] # download + verify into DIR
# wizard.sh install [core|dev] # detect, fetch, install, report
# It never runs the cluster, never uses sudo or apt, and writes only into
# $OUT_BIN (default ~/.local/bin). Everything that would touch the host proper —
# systemd, inotify limits, .wslconfig, docker group — is REPORTED for a human to
# decide on, never performed. That is what makes it safe to run on a machine that
# already has a working setup.
#
# Usage (normally via `make deps`, or directly):
# deps.sh detect # report host facts only, change nothing
# deps.sh fetch [core|dev] [--to DIR] # download + verify into DIR
# deps.sh install [core|dev] # detect, fetch, install, report
#
# Tiers: 'core' is kubectl + jq (talk to a cluster); 'dev' adds kind and tilt
# Default is dev.
#
# Runs both inside the wizard container and bare on a host. Inside the
# Runs both inside the installer container and bare on a host. Inside the
# container, host files are read through $HOST_ROOT (mount / as :ro); bare, it
# falls back to /.
@@ -44,6 +49,14 @@ MANUAL=()
# Host FILES (/etc/..., /mnt/c/...) must be read through the mount. Kernel-level
# facts (kernel version, meminfo, inotify) are shared with the container, so the
# container's own view is already the host's.
# A /proc/meminfo field in MB, 0 if the field is absent. MEMINFO exists so the
# tight and does-not-fit branches can be exercised against a real machine's
# numbers from somewhere else; in normal use it is always /proc/meminfo.
mb_of() {
awk -v k="$1:" '$1 == k { printf "%d", $2 / 1024; found = 1 }
END { if (!found) printf "0" }' "${MEMINFO:-/proc/meminfo}"
}
host_file() {
local p="${1#/}"
if [ "$HOST_ROOT" != "/" ] && [ -e "$HOST_ROOT/$p" ]; then
@@ -55,6 +68,29 @@ host_file() {
# ── detect ─────────────────────────────────────────────────────────────────
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
# the tooling. Detectable, so name it instead.
require_linux() {
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
cat >&2 <<'EOF'
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
wsl --install
That enables Windows features and needs a reboot, so it is not something this
script will do for you. Afterwards, open the Linux shell it installs and run
this from there.
See "Starting from plain Windows" in README.md.
EOF
exit 1 ;;
esac
}
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
detect() {
@@ -64,20 +100,34 @@ detect() {
local osr; osr=$(host_file /etc/os-release)
[ -r "$osr" ] && echo " distro $(sed -n 's/^PRETTY_NAME="\(.*\)"/\1/p' "$osr")"
local total_kb avail_kb
total_kb=$(awk '/^MemTotal:/{print $2}' /proc/meminfo)
avail_kb=$(awk '/^MemAvailable:/{print $2}' /proc/meminfo)
printf " memory %d GB total, %d GB available\n" \
$((total_kb / 1024 / 1024)) $((avail_kb / 1024 / 1024))
if [ $((avail_kb / 1024 / 1024)) -lt 4 ]; then
echo " ! under 4 GB available — a multi-node profile will struggle."
echo " 'make cluster list' shows the others; 'make cluster free' stops them."
# In MB. Whole gigabytes lose nearly half a GB on exactly the machines where
# it matters: 1874 MB available used to print as "1 GB". Facts only — whether
# that is enough depends on the profile, which check.sh knows and this does not.
local total_mb avail_mb swap_total_mb swap_used_mb om
total_mb=$(mb_of MemTotal)
avail_mb=$(mb_of MemAvailable)
swap_total_mb=$(mb_of SwapTotal)
swap_used_mb=$(( swap_total_mb - $(mb_of SwapFree) ))
printf " memory %d MB total, %d MB available\n" "$total_mb" "$avail_mb"
if [ "$swap_total_mb" -gt 0 ]; then
printf " swap %d MB used of %d MB\n" "$swap_used_mb" "$swap_total_mb"
fi
# How the kernel answers an allocation it cannot really satisfy. With 1 it
# always says yes and settles up later with the OOM killer, so a cluster that
# starts cleanly can still lose processes afterwards.
om=$(cat "${OVERCOMMIT_FILE:-/proc/sys/vm/overcommit_memory}" 2>/dev/null || echo '?')
case "$om" in
0) echo " overcommit 0 heuristic — allocations are granted on a guess" ;;
1) echo " overcommit 1 always — every allocation succeeds; the OOM killer is the only limit" ;;
2) echo " overcommit 2 strict — an allocation fails honestly instead of killing later" ;;
esac
detect_wsl
detect_filesystem
detect_docker
detect_inotify
detect_toolchain
}
detect_wsl() {
@@ -115,18 +165,46 @@ detect_wsl() {
if [ -n "$wcfg" ] && grep -qE '^\s*memory\s*=' "$wcfg"; then
echo " wslconfig memory set: $(grep -E '^\s*memory\s*=' "$wcfg" | tr -d ' ')"
else
MANUAL+=("Cap/raise the WSL VM memory — in %USERPROFILE%\\.wslconfig on Windows:
[wsl2]
memory=8GB
then from a WINDOWS terminal: wsl --shutdown")
MANUAL+=("Cap/raise the WSL VM memory — see what is set versus what booted:
make mem status
It prints the edit to make and the command to apply it.")
fi
}
# Not a path check: /mnt is an ordinary mount point and an ext4 disk mounted
# there is perfectly fine. What matters is the filesystem. The Windows drives
# arrive as 9p (WSL2) or drvfs (WSL1); network and fuse mounts behave the same
# way. None of them deliver inotify events, so anything watching files goes
# quiet without saying why.
watch_hostile_fs() {
local dir="$1" fstype
fstype=$(findmnt -no FSTYPE --target "$dir" 2>/dev/null || true)
[ -n "$fstype" ] || fstype=$(stat -f -c %T "$dir" 2>/dev/null || true)
case "$fstype" in
9p|v9fs|drvfs|cifs|smb3|nfs|nfs4|fuse.sshfs|fuseblk) echo "$fstype" ;;
*) echo "" ;;
esac
}
detect_filesystem() {
local root fstype
root=$(cd .. && pwd -P)
fstype=$(watch_hostile_fs "$root")
if [ -n "$fstype" ]; then
echo " ! this directory is on $fstype — file watching will not work"
MANUAL+=("Move this onto the local disk. Nothing watching files sees changes
on a $fstype mount, and everything else is slower:
cp -r \"$root\" ~/ && cd ~/$(basename "$root")")
else
echo " filesystem $root ($(findmnt -no FSTYPE --target "$root" 2>/dev/null || echo local))"
fi
}
detect_docker() {
# Reachability of the daemon is the real question, and the CLI is only how
# we ask it. Note that when this runs inside the wizard container, Docker
# we ask it. Note that when this runs inside the installer container, Docker
# necessarily exists on the host — otherwise nothing would be executing —
# so a missing CLI in here is a wizard packaging bug, not a host problem.
# so a missing CLI in here is an installer packaging bug, not a host problem.
if ! command -v docker >/dev/null 2>&1; then
if [ -S /var/run/docker.sock ]; then
echo " docker socket present (no cli in this context)"
@@ -230,7 +308,7 @@ fetch_tgz() {
chmod +x "$dest/$name"
}
# The wizard runs as root so it can reach the docker socket, which means
# The installer runs as root so it can reach the docker socket, which means
# everything it writes into a mounted volume lands root-owned and unusable from
# the host. Hand it back to whoever owns the mount point (the host user created
# that directory before mounting it).
@@ -257,7 +335,94 @@ fix_ownership() {
CORE_TOOLS="kubectl jq"
# No helm: every addon installs with `kubectl apply -f <url>`, so nothing here
# has ever invoked it. Add it back the day something actually needs a chart.
DEV_TOOLS="kind tilt"
#
# ctlptl is 'dev' rather than 'core' for the same reason kind is: core is "talk
# to a cluster someone else runs", and ctlptl builds them. It earns its place
# because it is what wires a cluster to a local registry — without one, an
# unqualified image name resolves to docker.io/library/<name> and there is
# nothing structural stopping a push there.
DEV_TOOLS="kind tilt ctlptl"
# ── what is already on this machine ───────────────────────────────────────
#
# A tool already on PATH at its pinned version is left where it is. Without
# this, install downloads a second copy into OUT_BIN and then reports the first
# one as shadowed — noise, and wrong, when both are the same version. That is
# the normal state of any machine someone set up by hand: the AWS Workspace
# keeps its toolchain in ~/wdir/bin, all five at exactly these pins.
pin_of() {
case "$1" in
kubectl) echo "$KUBECTL_VERSION" ;;
jq) echo "$JQ_VERSION" ;;
kind) echo "$KIND_VERSION" ;;
tilt) echo "$TILT_VERSION" ;;
ctlptl) echo "$CTLPTL_VERSION" ;;
esac
}
# The version string a binary reports. Each tool spells the question
# differently, and kubectl has to be told --client or it goes looking for a
# server to ask.
reported_version() {
local tool="$1" path="$2"
case "$tool" in
kubectl) "$path" version --client 2>/dev/null ;;
jq) "$path" --version 2>/dev/null ;;
*) "$path" version 2>/dev/null ;;
esac
}
# Does the binary at PATH report PIN? Matched as a whole version token, so
# 0.37.6 never matches 10.37.60, with the leading v optional either side: kind
# says v0.32.0, jq says jq-1.8.2, and tilt says v0.37.6 against a pin of 0.37.6.
#
# Bash's own regex rather than grep, deliberately. grep is not the same program
# on every machine — some builds reject patterns that others accept — and a
# failed grep inside a count reads exactly like a zero.
version_matches() {
local tool="$1" path="$2" pin="$3" out v re
out=$(reported_version "$tool" "$path") || return 1
v="${pin#v}"
v="${v//./\\.}"
re="(^|[^0-9.])v?${v}([^0-9.]|\$)"
[[ $out =~ $re ]]
}
# DEPS_ONLY narrows a fetch to the tools it names. Unset means the whole tier,
# which is what an explicit `deps.sh fetch` always gets: "download these into
# DIR" must not quietly skip something because this machine happens to have it.
# Only install() sets it, to what detect_toolchain found missing or mismatched.
want() { [ -z "${DEPS_ONLY:-}" ] || [[ " $DEPS_ONLY " == *" $1 "* ]]; }
# Every tool in the tier with its state, probed once and reported once. What
# still needs fetching is left in TOOLCHAIN_NEED for install() to act on.
TOOLCHAIN_NEED=""
detect_toolchain() {
local tier="${TIER:-dev}" b pin path found
TOOLCHAIN_NEED=""
echo
echo "toolchain (pinned, tier '$tier')"
for b in $(tier_tools "$tier"); do
pin=$(pin_of "$b")
path=$(command -v "$b" 2>/dev/null || true)
if [ -z "$path" ]; then
printf " - %-8s %-9s not found\n" "$b" "$pin"
TOOLCHAIN_NEED+="$b "
elif version_matches "$b" "$path" "$pin"; then
printf " %-8s %-9s %s\n" "$b" "$pin" "$path"
else
found=$(reported_version "$b" "$path" 2>/dev/null | head -1 || true)
printf " ! %-8s wants %s, %s reports '%s'\n" "$b" "$pin" "$path" "$found"
TOOLCHAIN_NEED+="$b "
fi
done
if [ -z "$TOOLCHAIN_NEED" ]; then
echo " every pinned tool is already on PATH — nothing to fetch"
else
echo " 'make deps' fetches only: ${TOOLCHAIN_NEED% }"
fi
}
fetch() {
local dest="$OUT_BIN" tier="${TIER:-dev}"
@@ -279,12 +444,17 @@ fetch() {
return
fi
if [ -n "${DEPS_ONLY:-}" ]; then
echo "fetching ${DEPS_ONLY% } (source: $DEPS_SOURCE)"
else
echo "fetching '$tier' toolchain (source: $DEPS_SOURCE)"
fetch_bin kubectl "$KUBECTL_URL" "$KUBECTL_SHA256" "$dest"
fetch_bin jq "$JQ_URL" "$JQ_SHA256" "$dest"
fi
if want kubectl; then fetch_bin kubectl "$KUBECTL_URL" "$KUBECTL_SHA256" "$dest"; fi
if want jq; then fetch_bin jq "$JQ_URL" "$JQ_SHA256" "$dest"; fi
if [ "$tier" = "dev" ]; then
fetch_bin kind "$KIND_URL" "$KIND_SHA256" "$dest"
fetch_tgz tilt "$TILT_URL" "$TILT_SHA256" "$dest" tilt 0
if want kind; then fetch_bin kind "$KIND_URL" "$KIND_SHA256" "$dest"; fi
if want tilt; then fetch_tgz tilt "$TILT_URL" "$TILT_SHA256" "$dest" tilt 0; fi
if want ctlptl; then fetch_tgz ctlptl "$CTLPTL_URL" "$CTLPTL_SHA256" "$dest" ctlptl 0; fi
fi
fix_ownership "$dest"
@@ -301,7 +471,7 @@ report_manual() {
echo "nothing left to do by hand."
return
fi
echo "host actions the wizard cannot perform (${#MANUAL[@]}):"
echo "host actions this cannot perform (${#MANUAL[@]}):"
echo
local n=1
for m in "${MANUAL[@]}"; do
@@ -326,6 +496,9 @@ warn_shadowing() {
command -v "$b" 2>/dev/null || true)
[ -n "$existing" ] || continue
[ "$existing" = "$OUT_BIN/$b" ] && continue
# The same version in both places is not a conflict: nothing changes for
# any other project whichever copy PATH happens to find first.
if version_matches "$b" "$existing" "$(pin_of "$b")"; then continue; fi
shadowed+=" $b $existing"$'\n'
done
@@ -348,31 +521,42 @@ warn_shadowing() {
}
install() {
local tier="${1:-dev}"
local tier="${1:-dev}" b
TIER="$tier"
detect
# detect_toolchain has already probed PATH. Fetch only what it found missing
# or at the wrong version; a tool already present at its pin stays where it is.
if [ -n "$TOOLCHAIN_NEED" ]; then
echo
fetch "$tier"
DEPS_ONLY="$TOOLCHAIN_NEED" fetch "$tier"
echo
echo "installed to $OUT_BIN ($tier):"
for b in $(tier_tools "$tier"); do
[ -x "$OUT_BIN/$b" ] && echo " $b"
for b in $TOOLCHAIN_NEED; do
if [ -x "$OUT_BIN/$b" ]; then echo " $b"; fi
done
if [ "$tier" = "core" ]; then
echo " (no kind/tilt — 'make deps dev' adds them)"
fi
warn_shadowing "$tier"
# Only worth saying when something actually landed in OUT_BIN. When every
# tool was satisfied elsewhere, OUT_BIN may reasonably be off PATH, and
# telling the user to add it would be advice to fix nothing.
case ":${PATH}:" in
*":$OUT_BIN:"*) ;;
*) MANUAL+=("Put the toolchain on your PATH — add to ~/.bashrc:
export PATH=\"${OUT_BIN}:\$PATH\"") ;;
esac
fi
warn_shadowing "$tier"
report_manual
}
# ── main ───────────────────────────────────────────────────────────────────
require_linux
case "${1:-install}" in
detect) detect; report_manual ;;
fetch) shift; fetch "$@" ;;

View File

@@ -19,7 +19,7 @@ DNS_MODE=hosts
# Opt in to the real ports below only when this is the ONLY environment and
# nothing else owns :80. They fail to bind otherwise, and docker reports it as an
# opaque "failed to bind host port 0.0.0.0:80/tcp: address already in use"
# halfway through cluster creation. `make station` checks before you spend the
# halfway through cluster creation. `make check` checks before you spend the
# time. Uncommenting also means only one environment can exist at a time.
# HTTP_PORT=80
# HTTPS_PORT=443

View File

@@ -1,11 +1,10 @@
# data — a cluster with the dependency containers a soleprint room asks for.
# data — a cluster with the cabinets an environment asks for.
#
# The point of this profile is that a room declares what it needs once, in
# cfg/<room>/data/cabinets.json, and gets it on either target: `build.py`
# composes those services into docker-compose.yml for a laptop, and the addons
# below install the same ones here. The names match deliberately —
# soleprint/station/cabinets/<name>/cabinet.json carries a `rig_addon` field
# pointing at ctrl/addons/<name>.sh.
# A cabinet is a public service dropped in as-is — the upstream image,
# unmodified, reachable at a known address. It is declared once and installs on
# either target: a `service.yml` composes it for a laptop, and the addons below
# install the same one here. The names match deliberately — each cabinet.json
# carries a `rig_addon` field pointing at ctrl/addons/<name>.sh.
#
# Everything lands in the `data` namespace (DATA_NAMESPACE to move it), so
# `make cluster reset` on the app namespace leaves the databases alone.
@@ -30,8 +29,8 @@ DATA_NAMESPACE=data
# Postgres identity. The password is not here: postgres.sh generates one on
# first install and keeps it across re-runs, so re-running the addon never
# rotates the credential out from under whatever is already connected.
POSTGRES_DB=soleprint
POSTGRES_USER=soleprint
POSTGRES_DB=app
POSTGRES_USER=app
POSTGRES_STORAGE=2Gi
AIRFLOW_ADMIN_USER=admin

View File

@@ -1,5 +1,5 @@
# offline — air-gapped. Everything comes from a local registry that was loaded
# ahead of time; nothing reaches the internet. Pair with the wizard-full image
# ahead of time; nothing reaches the internet. Pair with the deps-full image
# (DEPS_SOURCE=baked) so the toolchain install is offline too.
#
# The heavier addons are left out to keep first boot viable.

View File

@@ -1,8 +1,7 @@
# `ctrl/k8s` — cluster shape, and what runs on it
Same layout as every other project here (`unt`, `nvi`, `eth`, `mpr`, and
soleprint's generated rooms): a kind config, a kustomize `base/`, and an
`overlays/dev/` that patches it. See ALL `projects/templates/conventions.md`.
Same layout as every other project here: a kind config, a kustomize `base/`,
and an `overlays/dev/` that patches it.
```
kind-config*.yaml.tpl the cluster itself — nodes, ports, audit

View File

@@ -44,7 +44,7 @@ nodes:
- role: control-plane
image: ${NODE_IMAGE}
# hostPath is resolved by the HOST dockerd, so this must be a host path even
# when cluster.sh runs inside the wizard container. HOST_WORKDIR says where
# when cluster.sh runs inside the installer container. HOST_WORKDIR says where
# this rig lives on the host; bare on a host it is just the repo root.
extraMounts:
- hostPath: ${HOST_WORKDIR}/ctrl/k8s/audit-policy.yaml

View File

@@ -103,16 +103,26 @@ load_config() {
# The cluster's shape is a file in ctrl/k8s/, named by the profile. Adding a
# shape is adding a file; there is no dispatcher to edit.
#
# A host that needs its own shape — extra port mappings, more nodes — passes
# an absolute path instead, and rig renders it exactly like one of its own:
# ${CLUSTER} and ${NODE_IMAGE} are substituted either way. The shape stays in
# the host's tree, because what a host's cluster needs is the host's business;
# rig only knows how to build whatever it is handed.
KIND_CONFIG="${KIND_CONFIG:-kind-config.yaml.tpl}"
KIND_CONFIG_PATH="./k8s/${KIND_CONFIG}"
case "$KIND_CONFIG" in
/*) KIND_CONFIG_PATH="$KIND_CONFIG"; KIND_CONFIG_SHOWN="$KIND_CONFIG" ;;
*) KIND_CONFIG_PATH="./k8s/${KIND_CONFIG}"; KIND_CONFIG_SHOWN="ctrl/k8s/${KIND_CONFIG}" ;;
esac
if [ ! -f "$KIND_CONFIG_PATH" ]; then
echo "no such cluster shape: ctrl/k8s/${KIND_CONFIG}" >&2
echo "available: $(ls k8s/kind-config*.yaml.tpl 2>/dev/null | xargs -n1 basename | tr '\n' ' ')" >&2
echo "no such cluster shape: ${KIND_CONFIG_SHOWN}" >&2
echo "rig's own: $(ls k8s/kind-config*.yaml.tpl 2>/dev/null | xargs -n1 basename | tr '\n' ' ')" >&2
echo "or pass an absolute path to a shape of your own" >&2
exit 1
fi
# Read the shape back out of the YAML rather than trusting a profile to
# restate it. station.sh sizes the memory warning on NODES, and cluster.sh
# restate it. check.sh sizes the memory warning on NODES, and cluster.sh
# prints AUDIT before spending minutes building something that cannot be
# changed afterwards — both would mislead if the numbers drifted.
NODES=$(grep -c '^ - role:' "$KIND_CONFIG_PATH")
@@ -125,7 +135,7 @@ load_config() {
# depending on something the caller does not set.
#
# hostPath entries are resolved by the HOST dockerd, so HOST_WORKDIR must stay a
# host path even when this runs inside the wizard container.
# host path even when this runs inside the installer container.
render_kind_config() {
local host_workdir="${HOST_WORKDIR:-$(cd .. && pwd)}"
sed -e "s|\${CLUSTER}|${CLUSTER}|g" \

233
rig/ctrl/mem.sh Executable file
View File

@@ -0,0 +1,233 @@
#!/usr/bin/env bash
# What memory this machine has, what is left, and — where there is one — what
# cap is holding it there.
#
# Runs on native Linux and under WSL, because rig is developed on one and used
# on the other. The difference is not cosmetic: on WSL the memory you see is a
# VM allocation that can be raised, and the commonest failure is raising it
# without restarting, so the number on disk and the number in /proc disagree.
# On native Linux there is no such cap and pretending otherwise sends you to a
# file that does not exist.
#
# This reports and instructs. It never writes a .wslconfig — applying one costs
# a full VM restart that takes every shell, mount and container with it, and
# choosing that moment is yours.
#
# `backup` exists so `restore` has something to read: back up, hand-edit
# following the printed instruction, restore if it goes wrong. Both are
# WSL-only, because .wslconfig is the only thing here worth backing up.
#
# Usage: mem.sh status | backup | restore
set -euo pipefail
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
# the tooling. Detectable, so name it instead.
require_linux() {
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
cat >&2 <<'EOF'
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
wsl --install
That enables Windows features and needs a reboot, so it is not something this
script will do for you. Afterwards, open the Linux shell it installs and run
this from there.
See "Starting from plain Windows" in README.md.
EOF
exit 1 ;;
esac
}
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
require_wsl() {
if ! is_wsl; then
echo "$1 acts on .wslconfig, which only exists under WSL." >&2
echo "This is native Linux — there is no VM allocation to save or roll back." >&2
echo "Use 'mem.sh status' to see what the machine actually has." >&2
exit 1
fi
}
mb() { echo $(( $(awk "/^$1:/{print \$2}" /proc/meminfo) / 1024 )); }
# /mnt/c/Users can hold several real accounts — a renamed login leaves the old
# directory behind — so picking the first alphabetically is a coin toss. Ask
# Windows, then fall back to whichever profile actually owns a config.
wslconfig_path() {
local profile winpath found
profile=$(cmd.exe /c "echo %USERPROFILE%" 2>/dev/null | tr -d "\r\n" || true)
case "$profile" in
""|*%*) ;;
*) winpath=$(wslpath -u "$profile" 2>/dev/null || true)
if [ -n "$winpath" ] && [ -d "$winpath" ]; then
echo "$winpath/.wslconfig"; return
fi ;;
esac
found=$(ls -d /mnt/c/Users/*/.wslconfig 2>/dev/null | head -1 || true)
if [ -n "$found" ]; then echo "$found"; return; fi
echo "cannot tell which Windows profile owns .wslconfig. Candidates:" >&2
ls -d /mnt/c/Users/*/ 2>/dev/null \
| grep -viE "/(All Users|Default|Default User|Public)/$" | sed "s/^/ /" >&2
exit 1
}
configured_memory() {
[ -r "$1" ] || { echo ""; return; }
sed -n 's/^[[:space:]]*memory[[:space:]]*=[[:space:]]*//p' "$1" | tail -1 | tr -d '[:space:]'
}
# "9GB" / "8192MB" / "9G" -> MB, so it can be compared with /proc/meminfo.
to_mb() {
local v="${1^^}" n
n=$(echo "$v" | tr -dc '0-9')
[ -n "$n" ] || { echo ""; return; }
case "$v" in
*GB|*G) echo $(( n * 1024 )) ;;
*MB|*M) echo "$n" ;;
*) echo $(( n / 1024 / 1024 )) ;;
esac
}
hogs() {
echo "holding the most:"
ps -eo rss,comm --sort=-rss 2>/dev/null | awk 'NR>1 && NR<=6 {printf " %6.0f MB %s\n", $1/1024, $2}'
}
status() {
local total avail swap_total swap_free
total=$(mb MemTotal); avail=$(mb MemAvailable)
swap_total=$(mb SwapTotal); swap_free=$(mb SwapFree)
if is_wsl; then
local cfg conf conf_mb
cfg=$(wslconfig_path)
conf=$(configured_memory "$cfg")
echo "platform WSL"
echo "config $cfg"
if [ -n "$conf" ]; then
conf_mb=$(to_mb "$conf")
echo "configured $conf (${conf_mb} MB)"
else
conf_mb=""
echo "configured (no memory= set — WSL defaults to 50% of host RAM, or 8GB, whichever is less)"
fi
echo "booted ${total} MB"
echo "available ${avail} MB"
echo "swap ${swap_total} MB ($(( swap_total - swap_free )) MB used)"
if [ -n "$conf_mb" ]; then
# The VM reports a little less than allocated; 15% covers the kernel
# without calling every healthy machine a mismatch.
if [ "$total" -lt $(( conf_mb * 85 / 100 )) ]; then
echo
echo "! configured ${conf_mb} MB but booted ${total} MB."
echo " The change has not been applied. From a WINDOWS terminal:"
echo
echo " wsl --shutdown"
echo
echo " then start the distro again."
fi
else
echo
echo "To raise it, add to $cfg on the Windows side:"
echo
echo " [wsl2]"
echo " memory=8GB"
echo
echo "then, from a WINDOWS terminal: wsl --shutdown"
fi
local n
n=$(ls "$cfg".*.bak 2>/dev/null | wc -l)
[ "$n" -gt 0 ] && echo "backups $n (newest: $(ls -t "$cfg".*.bak 2>/dev/null | head -1))"
else
echo "platform native linux"
echo "total ${total} MB"
echo "available ${avail} MB"
echo "swap ${swap_total} MB ($(( swap_total - swap_free )) MB used)"
echo
echo "No VM allocation to raise here — this is the machine's own memory."
echo "If it is tight the levers are freeing something or adding swap."
fi
# Under a fifth left is worth naming wherever you are running.
if [ "$avail" -lt $(( total / 5 )) ]; then
echo
hogs
fi
return 0
}
backup() {
require_wsl backup
local cfg dest
cfg=$(wslconfig_path)
[ -r "$cfg" ] || { echo "nothing to back up: $cfg does not exist" >&2; exit 1; }
# Timestamped and never overwritten: a backup that can destroy itself on a
# second run is not a backup.
dest="${cfg}.$(date +%Y%m%d-%H%M%S).bak"
cp "$cfg" "$dest"
echo "backed up $dest"
echo
echo "Edit $cfg by hand, then from a WINDOWS terminal: wsl --shutdown"
}
restore() {
require_wsl restore
local cfg newest count
cfg=$(wslconfig_path)
newest=$(ls -t "$cfg".*.bak 2>/dev/null | head -1 || true)
[ -n "$newest" ] || { echo "no backups found beside $cfg" >&2; exit 1; }
echo "restoring $newest"
echo " -> $cfg"
echo
# Newest is the right default — undo the last edit — but if you backed up
# *after* editing, the state you want is older. Show the rest so a no-op
# restore is obviously a no-op rather than a mystery.
count=$(ls "$cfg".*.bak 2>/dev/null | wc -l)
if [ "$count" -gt 1 ]; then
echo "$count backups exist, newest first:"
ls -t "$cfg".*.bak | sed 's/^/ /'
echo " (restoring the newest; copy another by hand to pick an older one)"
echo
fi
if [ -r "$cfg" ]; then
echo "what changes:"
if diff "$cfg" "$newest" > /tmp/mem.diff 2>&1 && [ ! -s /tmp/mem.diff ]; then
echo " nothing — that backup is identical to the current config"
else
sed 's/^/ /' /tmp/mem.diff
fi
rm -f /tmp/mem.diff
echo
fi
printf "proceed? [y/N] "
read -r reply
case "$reply" in
y|Y|yes|Yes) ;;
*) echo "left alone"; return 0 ;;
esac
cp "$newest" "$cfg"
echo "restored. From a WINDOWS terminal: wsl --shutdown"
}
require_linux
case "${1:-status}" in
status) status ;;
backup) backup ;;
restore) restore ;;
*) echo "usage: $0 [status|backup|restore]" >&2; exit 1 ;;
esac

View File

@@ -58,10 +58,13 @@ require_wsl() {
cat >&2 <<'EOF'
newbox is WSL-only for now.
If WSL is not installed, run `wsl --install` from an elevated Windows prompt
first — see "Starting from plain Windows" in README.md.
On native Linux you do not need it: rig already isolates environments by
directory (own cluster, context, images and port block), so a second copy in a
second directory is the clean slate. To validate the installer itself against a
bare system, run the wizard against a stock Debian container instead.
bare system, run ctrl/deps.sh against a stock Debian container instead.
EOF
exit 1
fi
@@ -250,7 +253,7 @@ create() {
echo
echo "next:"
echo " make newbox shell # a shell inside it"
echo " then: cd ~/rig && make station && make deps && make cluster up"
echo " then: cd ~/rig && make check && make deps && make cluster up"
echo
echo "For a browser on Windows to resolve the hostnames, paste this into"
echo "C:\\Windows\\System32\\drivers\\etc\\hosts (it has no wildcard support):"

57
rig/ctrl/pins.sh Executable file
View File

@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Do the standalone scripts still install what rig pins?
#
# standalone/rigdeps.sh carries its toolchain pins inline, because it exists for
# a machine that will never have ctrl/versions.env. That makes two copies of the
# same versions and checksums, and two copies drift the day one is edited and
# the other forgotten. This is the check that notices.
#
# ctrl/versions.env is the source of truth. Only the keys rigdeps.sh itself
# defines are compared: versions.env also pins addon images (cert-manager,
# metallb, metrics-server) that rigdeps.sh never installs, and demanding those
# would make this fail forever for no reason.
#
# Exits non-zero on any mismatch — unlike the host checks, this one is a test.
#
# Usage: pins.sh
set -euo pipefail
cd "$(dirname "$0")"
SOURCE=./versions.env
COPY=../standalone/rigdeps.sh
[ -r "$COPY" ] || { echo "no $COPY to compare" >&2; exit 1; }
# KEY=value for the pin keys a file defines, quotes stripped. awk rather than a
# grep regex, which is not the same program everywhere.
pins() {
awk -F= '/^[A-Z_]+_(VERSION|SHA256)=/ {
v = substr($0, index($0, "=") + 1); gsub(/^["\x27]|["\x27]$/, "", v)
print $1 "=" v }' "$1"
}
echo "pins: standalone/rigdeps.sh against ctrl/versions.env"
bad=0
while IFS='=' read -r key copy_val; do
[ -n "$key" ] || continue
src_val=$(pins "$SOURCE" | sed -n "s/^${key}=//p" | head -1)
if [ -z "$src_val" ]; then
printf " ! %-16s in rigdeps.sh but not in versions.env\n" "$key"
bad=1
elif [ "$src_val" = "$copy_val" ]; then
printf " %-16s %s\n" "$key" "$( [ ${#src_val} -gt 20 ] && echo "${src_val:0:12}" || echo "$src_val" )"
else
printf " ! %-16s versions.env %s\n" "$key" "$src_val"
printf " %-16s rigdeps.sh %s\n" "" "$copy_val"
bad=1
fi
done < <(pins "$COPY")
echo
if [ "$bad" -eq 0 ]; then
echo "in step — rigdeps.sh installs exactly what rig pins."
else
echo "DRIFT. versions.env is the source of truth: copy the differing lines from it"
echo "into standalone/rigdeps.sh, taking checksums from the publisher's release list."
exit 1
fi

View File

@@ -43,7 +43,7 @@ K="kubectl --context ${KUBECONTEXT}"
# 2. every kind node's containerd — nodes do NOT inherit host trust
# 3. anything doing HTTPS from inside the cluster, in its own trust store
#
# We handle (2) here because it's ours to handle. (1) is reported by station.sh
# We handle (2) here because it's ours to handle. (1) is reported by check.sh
# since it needs root. (3) belongs to the workload.
install_ca_into_nodes() {
[ -n "${REGISTRY_CA_FILE:-}" ] || return 0

View File

@@ -73,11 +73,11 @@ record() {
step_host() {
local out
if ! out=$(bash ./wizard.sh detect 2>&1); then
if ! out=$(bash ./deps.sh detect 2>&1); then
record host fail "detection failed"
return
fi
# Anything the wizard flagged with '!' needs a human; surface the count here
# Anything flagged with '!' needs a human; surface the count here
# and the detail below rather than burying it.
local warns; warns=$(echo "$out" | grep -c '^\s*!' || true)
HOST_DETAIL="$out"
@@ -102,7 +102,7 @@ step_toolchain() {
return
fi
if bash ./wizard.sh install "$TIER" >/tmp/rig-deps.$$ 2>&1; then
if bash ./deps.sh install "$TIER" >/tmp/rig-deps.$$ 2>&1; then
local still=""
for b in $want; do
[ -x "${OUT_BIN:-$HOME/.local/bin}/$b" ] || still="$still $b"

View File

@@ -1,106 +0,0 @@
#!/usr/bin/env bash
# Station check: is this workstation ready to run rig?
#
# Reports and instructs; never silently fixes anything. Everything it finds is
# either already fine, or something a human has to decide on.
#
# Runs the wizard's host detection in a container when Docker is the only thing
# installed, or directly when the toolchain is already present. Then adds the
# checks that need this repo's config: profile sanity, CA trust, port clashes.
set -euo pipefail
cd "$(dirname "$0")"
WIZARD_IMAGE="${WIZARD_IMAGE:-$(basename "$(cd .. && pwd)")-wizard}"
# Host detection. Prefer running it bare — it needs no dependencies beyond
# coreutils — and fall back to the container only if this shell can't.
bash ./wizard.sh detect
# ── repo-level checks ──────────────────────────────────────────────────────
source ./lib/config.sh
load_config
echo
echo "config"
echo " profile ${PROFILE_NAME} (nodes=${NODES} audit=${AUDIT})"
echo " cluster ${CLUSTER} (context ${KUBECONTEXT})"
echo " registry ${REGISTRY_MODE}"
echo " ingress ${INGRESS_MODE}"
if [ ! -f ./.env ]; then
echo " ! ctrl/.env missing — copy it: cp ctrl/.env.example ctrl/.env"
fi
# A 3-node profile on a box that's already full is the most common first
# failure, and it presents as pods stuck Pending rather than anything obvious.
avail=$(awk '/^MemAvailable:/{printf "%d", $2/1024/1024}' /proc/meminfo)
need=$((NODES * 2))
if [ "$avail" -lt "$need" ]; then
echo " ! profile '${PROFILE_NAME}' wants ~${need} GB, ${avail} GB available"
echo " 'make cluster list' shows what else is running; 'make cluster free' stops it"
fi
# The CA reaches three places and only one of them is ours. Report the other two.
if [ -n "${REGISTRY_CA_FILE:-}" ]; then
echo
echo "registry CA"
if [ ! -r "$REGISTRY_CA_FILE" ]; then
echo " ! REGISTRY_CA_FILE not readable: $REGISTRY_CA_FILE"
else
echo " file $REGISTRY_CA_FILE"
host="${REGISTRY_REMOTE_URL#*://}"; host="${host%%/*}"
if [ -n "$host" ] && [ ! -f "/etc/docker/certs.d/${host}/ca.crt" ]; then
echo " ! the HOST docker daemon does not trust it yet:"
echo " sudo mkdir -p /etc/docker/certs.d/${host}"
echo " sudo cp ${REGISTRY_CA_FILE} /etc/docker/certs.d/${host}/ca.crt"
echo " (kind nodes are handled by registry.sh; in-cluster clients are the workload's job)"
fi
fi
fi
# Host ports this environment will try to bind. Checked before cluster creation
# because docker reports a clash halfway through, as an opaque
# "failed to bind host port ...: address already in use".
echo
echo "ports (block derived from the directory name — see 'make ports')"
port_busy() {
if command -v ss >/dev/null 2>&1; then
ss -ltn "sport = :$1" 2>/dev/null | grep -q LISTEN && return 0 || return 1
fi
# iproute2 is absent from a minimal Debian, so fall back to procfs rather
# than silently reporting everything as free.
local hex; hex=$(printf ':%04X' "$1")
grep -qi "^ *[0-9]*: [0-9A-F]*$hex " /proc/net/tcp /proc/net/tcp6 2>/dev/null
}
# A port held by THIS environment's own cluster is not a clash — it is the thing
# working. Reporting it as a problem every time the cluster is up would train
# people to ignore this section, which is the opposite of the point.
# Extract with a second grep rather than `tr -d ':->'`: in tr, ':->' is the
# character RANGE ':' to '>', which does not contain '-', so the trailing dash
# survives and nothing ever matches.
ours=$(docker ps --filter "label=io.x-k8s.kind.cluster=${CLUSTER}" \
--format '{{.Ports}}' 2>/dev/null | tr ',' '\n' \
| grep -oE ':[0-9]+->' | grep -oE '[0-9]+' || true)
clash=0
for entry in "HTTP:${HTTP_PORT}" "HTTPS:${HTTPS_PORT}" \
"TILT:${TILT_PORT}" "REGISTRY:${REGISTRY_PORT}"; do
name="${entry%%:*}"; p="${entry#*:}"
[ -n "$p" ] || continue
if ! port_busy "$p"; then
printf " %-9s %-6s free\n" "$name" "$p"
elif echo "$ours" | grep -qx "$p"; then
printf " %-9s %-6s in use by this environment's cluster\n" "$name" "$p"
else
printf " ! %-9s %-6s IN USE by something else\n" "$name" "$p"
clash=1
fi
done
if [ "$clash" -eq 1 ]; then
echo " override the clashing one in ctrl/.env, e.g. HTTP_PORT=21080"
echo " (or rename this directory — the whole block follows the name)"
fi

View File

@@ -1,4 +1,4 @@
# Pinned toolchain — the single manifest the wizard installs from.
# Pinned toolchain — the single manifest ctrl/deps.sh installs from.
# Every entry is a single binary; none of them needs an apt repo.
# kubectl fully static
# kind libc only
@@ -6,8 +6,19 @@
# jq upstream static build (Debian's is linked against libjq/libonig)
#
# Checksums are the upstream-published SHA256 of the linux/amd64 artifact.
# To bump: change the version, then re-run `bash ctrl/versions-refresh.sh` and
# commit the result — never hand-edit a checksum.
#
# To bump: change the version, then take the checksum from the release's own
# published list — never hand-edit or hand-copy one from a download you did.
# For anything hosted on GitHub releases that is:
#
# curl -sSL https://github.com/<org>/<repo>/releases/download/<tag>/checksums.txt \
# | grep linux.x86_64
#
# (kubectl publishes its own instead: <KUBECTL_URL>.sha256.)
#
# There was a `ctrl/versions-refresh.sh` named here that has never existed. If
# bumping stops being rare enough to do by hand, write it — but a comment
# pointing at a missing script is worse than no comment.
KIND_VERSION=v0.32.0
KIND_SHA256=50030de23cf40a18505f20426f6a8506bedf13c6e509244bd1fa9463721b0f54
@@ -21,6 +32,14 @@ TILT_VERSION=0.37.6
TILT_SHA256=e9672b8a18d43501f35dcfe98465969a7db0e436b36cf0c50c7e6f8d40de5fe6
TILT_URL=https://github.com/tilt-dev/tilt/releases/download/v${TILT_VERSION}/tilt.${TILT_VERSION}.linux.x86_64.tar.gz
# ctlptl — creates a kind cluster WITH a local registry wired in, which is what
# keeps images off docker.io (an unqualified name means docker.io/library/<name>).
# Same publisher and same archive shape as tilt: binary at the archive root, so
# fetch_tgz handles it with strip=0 and no special case.
CTLPTL_VERSION=0.9.4
CTLPTL_SHA256=c63a1ec28e60bc3faf6becb76f53355c5cf5e0143dafdd27ad85db5584fa6b1e
CTLPTL_URL=https://github.com/tilt-dev/ctlptl/releases/download/v${CTLPTL_VERSION}/ctlptl.${CTLPTL_VERSION}.linux.x86_64.tar.gz
JQ_VERSION=1.8.2
JQ_SHA256=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f
JQ_URL=https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-amd64
@@ -44,9 +63,9 @@ CERT_MANAGER_VERSION=v1.21.1
METRICS_SERVER_VERSION=v0.9.0
METALLB_VERSION=v0.16.0
# Dependency containers. These mirror soleprint's cabinets
# (soleprint/station/cabinets/), so a room that declares postgres gets the same
# thing whether it runs on compose or in the cluster. Pinned by tag rather than
# Cabinets — public services dropped in as-is, the upstream image unmodified.
# The same declaration installs on compose or in the cluster, so a dependency is
# named once and works either way. Pinned by tag rather than
# digest because they are ordinary upstream images with no supply chain claim
# attached — bump freely, and preload them for the offline profile.
POSTGRES_IMAGE=postgres:16-alpine

View File

@@ -20,13 +20,13 @@ digraph rig_install {
bin [label="~/.local/bin\nkind · kubectl · tilt\njq" fillcolor="#121829" shape=cylinder]
}
subgraph cluster_wizard {
subgraph cluster_installer {
label="Installer container (transient)"
style=dashed
color="#1e2a4a"
fontcolor="#8892a8"
wizard [label="wizard\ncurl · jq · python · graphviz" fillcolor="#121829"]
installer [label="deps installer\ncurl · jq · python · graphviz" fillcolor="#121829"]
detect [label="detect host\nWSL · memory · inotify · docker" fillcolor="#121829"]
fetch [label="fetch + verify\nSHA256, pinned versions" fillcolor="#121829"]
}
@@ -34,14 +34,14 @@ digraph rig_install {
upstream [label="upstream\nreleases / corporate mirror" fillcolor="#1a3a1a" fontcolor="#00c853" shape=octagon]
report [label="report what it\nCANNOT do" fillcolor="#3a1a1a" fontcolor="#ffc107"]
docker -> wizard [label="docker run"]
wizard -> detect
docker -> installer [label="docker run"]
installer -> detect
detect -> fetch
fetch -> upstream [label="pinned + checksummed" color="#00c853"]
fetch -> bin [label="install"]
detect -> report [style=dashed label="sudo / Windows-side steps" color="#ffc107"]
// The container is gone after this; nothing depends on it at run time.
wizard -> gone [style=dotted label="exits"]
installer -> gone [style=dotted label="exits"]
gone [label="(container discarded)" fillcolor="#0a0e17" fontcolor="#4a5568" color="#1e2a4a" style="filled,dashed"]
}

View File

@@ -16,7 +16,7 @@
<text xml:space="preserve" text-anchor="middle" x="1011.15" y="-170.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Your machine</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_wizard</title>
<title>cluster_installer</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="8,-95 8,-175 745.5,-175 745.5,-95 8,-95"/>
<text xml:space="preserve" text-anchor="middle" x="376.75" y="-155.8" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Installer container (transient)</text>
</g>
@@ -27,16 +27,16 @@
<text xml:space="preserve" text-anchor="middle" x="1011.15" y="-46.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">Docker</text>
<text xml:space="preserve" text-anchor="middle" x="1011.15" y="-32.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">(the one prerequisite)</text>
</g>
<!-- wizard -->
<!-- installer -->
<g id="node3" class="node">
<title>wizard</title>
<title>installer</title>
<polygon fill="#121829" stroke="#1e2a4a" points="181.25,-139 16,-139 16,-103 181.25,-103 181.25,-139"/>
<text xml:space="preserve" text-anchor="middle" x="98.62" y="-124.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">wizard</text>
<text xml:space="preserve" text-anchor="middle" x="98.62" y="-124.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">deps installer</text>
<text xml:space="preserve" text-anchor="middle" x="98.62" y="-110.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">curl · jq · python · graphviz</text>
</g>
<!-- docker&#45;&gt;wizard -->
<!-- docker&#45;&gt;installer -->
<g id="edge1" class="edge">
<title>docker&#45;&gt;wizard</title>
<title>docker&#45;&gt;installer</title>
<path fill="none" stroke="#4a5568" d="M906.12,-45.7C757.47,-50.51 475.98,-63.12 238.25,-94 223.38,-95.93 207.7,-98.48 192.45,-101.24"/>
<polygon fill="#4a5568" stroke="#4a5568" points="192.13,-97.74 182.93,-103.01 193.4,-104.63 192.13,-97.74"/>
<text xml:space="preserve" text-anchor="middle" x="506.62" y="-74.39" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">docker run</text>
@@ -57,9 +57,9 @@
<text xml:space="preserve" text-anchor="middle" x="333.62" y="-124.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">detect host</text>
<text xml:space="preserve" text-anchor="middle" x="333.62" y="-110.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">WSL · memory · inotify · docker</text>
</g>
<!-- wizard&#45;&gt;detect -->
<!-- installer&#45;&gt;detect -->
<g id="edge2" class="edge">
<title>wizard&#45;&gt;detect</title>
<title>installer&#45;&gt;detect</title>
<path fill="none" stroke="#4a5568" d="M181.44,-121C195.97,-121 211.28,-121 226.37,-121"/>
<polygon fill="#4a5568" stroke="#4a5568" points="226.33,-124.5 236.33,-121 226.33,-117.5 226.33,-124.5"/>
</g>
@@ -69,9 +69,9 @@
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="400.5,-47 266.75,-47 266.75,-11 400.5,-11 400.5,-47"/>
<text xml:space="preserve" text-anchor="middle" x="333.62" y="-25.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#4a5568">(container discarded)</text>
</g>
<!-- wizard&#45;&gt;gone -->
<!-- installer&#45;&gt;gone -->
<g id="edge7" class="edge">
<title>wizard&#45;&gt;gone</title>
<title>installer&#45;&gt;gone</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="1,5" d="M119.29,-102.62C138.26,-86 168.54,-62.29 199.25,-49.75 216.76,-42.6 236.49,-37.9 255.29,-34.81"/>
<polygon fill="#4a5568" stroke="#4a5568" points="255.67,-38.29 265.04,-33.35 254.64,-31.37 255.67,-38.29"/>
<text xml:space="preserve" text-anchor="middle" x="209.75" y="-52.45" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">exits</text>

Before

Width:  |  Height:  |  Size: 9.0 KiB

After

Width:  |  Height:  |  Size: 9.0 KiB

View File

@@ -20,7 +20,7 @@ digraph rig_environment {
cname [label="cluster name\nacmebank" fillcolor="#121829"]
ctx [label="kubectl context\nkind-acmebank" fillcolor="#121829"]
img [label="image tag\nacmebank-wizard" fillcolor="#121829"]
img [label="image tag\nacmebank-deps" fillcolor="#121829"]
ports [label="port block\n2130021309" fillcolor="#121829"]
reg [label="registry container\nacmebank-registry" fillcolor="#121829"]
}

View File

@@ -4,28 +4,28 @@
<!-- Generated by graphviz version 14.1.2 (0)
-->
<!-- Title: rig_environment Pages: 1 -->
<svg width="971pt" height="481pt"
viewBox="0.00 0.00 971.00 481.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<svg width="962pt" height="481pt"
viewBox="0.00 0.00 962.00 481.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 476.83)">
<title>rig_environment</title>
<polygon fill="#0a0e17" stroke="none" points="-4,4 -4,-476.83 967,-476.83 967,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="481.5" y="-453.63" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">One environment per directory — copies never collide</text>
<polygon fill="#0a0e17" stroke="none" points="-4,4 -4,-476.83 958,-476.83 958,4 -4,4"/>
<text xml:space="preserve" text-anchor="middle" x="477" y="-453.63" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#0066ff">One environment per directory — copies never collide</text>
<g id="clust1" class="cluster">
<title>cluster_derived</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="8,-65 8,-144.5 605,-144.5 605,-65 8,-65"/>
<text xml:space="preserve" text-anchor="middle" x="306.5" y="-125.3" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Everything below is derived from it</text>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="8,-65 8,-144.5 596,-144.5 596,-65 8,-65"/>
<text xml:space="preserve" text-anchor="middle" x="302" y="-125.3" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Everything below is derived from it</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_config</title>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="613,-65 613,-437.33 955,-437.33 955,-65 613,-65"/>
<text xml:space="preserve" text-anchor="middle" x="784" y="-418.13" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Configuration — weakest first, later wins</text>
<polygon fill="#0a0e17" stroke="#1e2a4a" stroke-dasharray="5,2" points="604,-65 604,-437.33 946,-437.33 946,-65 604,-65"/>
<text xml:space="preserve" text-anchor="middle" x="775" y="-418.13" font-family="Helvetica,sans-Serif" font-size="16.00" fill="#8892a8">Configuration — weakest first, later wins</text>
</g>
<!-- dirname -->
<g id="node1" class="node">
<title>dirname</title>
<polygon fill="#1f6feb" stroke="#1e2a4a" points="375.11,-197.44 375.11,-219.63 329.94,-235.33 266.06,-235.33 220.89,-219.63 220.89,-197.44 266.06,-181.75 329.94,-181.75 375.11,-197.44"/>
<text xml:space="preserve" text-anchor="middle" x="298" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">directory name</text>
<text xml:space="preserve" text-anchor="middle" x="298" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">e.g. acmebank/</text>
<polygon fill="#1f6feb" stroke="#1e2a4a" points="370.11,-197.44 370.11,-219.63 324.94,-235.33 261.06,-235.33 215.89,-219.63 215.89,-197.44 261.06,-181.75 324.94,-181.75 370.11,-197.44"/>
<text xml:space="preserve" text-anchor="middle" x="293" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">directory name</text>
<text xml:space="preserve" text-anchor="middle" x="293" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ffffff">e.g. acmebank/</text>
</g>
<!-- cname -->
<g id="node2" class="node">
@@ -37,8 +37,8 @@
<!-- dirname&#45;&gt;cname -->
<g id="edge1" class="edge">
<title>dirname&#45;&gt;cname</title>
<path fill="none" stroke="#4a5568" d="M232.76,-193.05C195.81,-183.01 149.78,-167.28 113,-144.5 101.37,-137.29 90.31,-127.09 81.33,-117.6"/>
<polygon fill="#4a5568" stroke="#4a5568" points="84.04,-115.37 74.73,-110.3 78.85,-120.07 84.04,-115.37"/>
<path fill="none" stroke="#4a5568" d="M228.68,-192.51C192.88,-182.36 148.52,-166.7 113,-144.5 101.4,-137.25 90.34,-127.04 81.36,-117.55"/>
<polygon fill="#4a5568" stroke="#4a5568" points="84.07,-115.32 74.76,-110.26 78.88,-120.02 84.07,-115.32"/>
</g>
<!-- ctx -->
<g id="node3" class="node">
@@ -50,120 +50,120 @@
<!-- dirname&#45;&gt;ctx -->
<g id="edge2" class="edge">
<title>dirname&#45;&gt;ctx</title>
<path fill="none" stroke="#4a5568" d="M269.64,-181.32C248.71,-161.98 220.42,-135.83 199.86,-116.83"/>
<polygon fill="#4a5568" stroke="#4a5568" points="202.4,-114.41 192.68,-110.19 197.65,-119.55 202.4,-114.41"/>
<path fill="none" stroke="#4a5568" d="M265.77,-181.32C245.77,-162.07 218.77,-136.06 199.05,-117.08"/>
<polygon fill="#4a5568" stroke="#4a5568" points="201.55,-114.63 191.91,-110.21 196.69,-119.67 201.55,-114.63"/>
</g>
<!-- img -->
<g id="node4" class="node">
<title>img</title>
<polygon fill="#121829" stroke="#1e2a4a" points="354,-109 242,-109 242,-73 354,-73 354,-109"/>
<text xml:space="preserve" text-anchor="middle" x="298" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">image tag</text>
<text xml:space="preserve" text-anchor="middle" x="298" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;wizard</text>
<polygon fill="#121829" stroke="#1e2a4a" points="344.12,-109 241.88,-109 241.88,-73 344.12,-73 344.12,-109"/>
<text xml:space="preserve" text-anchor="middle" x="293" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">image tag</text>
<text xml:space="preserve" text-anchor="middle" x="293" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;deps</text>
</g>
<!-- dirname&#45;&gt;img -->
<g id="edge3" class="edge">
<title>dirname&#45;&gt;img</title>
<path fill="none" stroke="#4a5568" d="M298,-181.32C298,-163.19 298,-139.07 298,-120.47"/>
<polygon fill="#4a5568" stroke="#4a5568" points="301.5,-120.67 298,-110.67 294.5,-120.67 301.5,-120.67"/>
<path fill="none" stroke="#4a5568" d="M293,-181.32C293,-163.19 293,-139.07 293,-120.47"/>
<polygon fill="#4a5568" stroke="#4a5568" points="296.5,-120.67 293,-110.67 289.5,-120.67 296.5,-120.67"/>
</g>
<!-- ports -->
<g id="node5" class="node">
<title>ports</title>
<polygon fill="#121829" stroke="#1e2a4a" points="460.38,-109 371.62,-109 371.62,-73 460.38,-73 460.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="416" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">port block</text>
<text xml:space="preserve" text-anchor="middle" x="416" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">2130021309</text>
<polygon fill="#121829" stroke="#1e2a4a" points="451.38,-109 362.62,-109 362.62,-73 451.38,-73 451.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="407" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">port block</text>
<text xml:space="preserve" text-anchor="middle" x="407" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">2130021309</text>
</g>
<!-- dirname&#45;&gt;ports -->
<g id="edge4" class="edge">
<title>dirname&#45;&gt;ports</title>
<path fill="none" stroke="#4a5568" d="M324.94,-181.53C336.66,-170.19 350.54,-156.71 363,-144.5 372.11,-135.57 382.06,-125.74 390.85,-117.02"/>
<polygon fill="#4a5568" stroke="#4a5568" points="393.15,-119.67 397.79,-110.14 388.22,-114.7 393.15,-119.67"/>
<path fill="none" stroke="#4a5568" d="M318.87,-181.32C337.78,-162.15 363.29,-136.3 382,-117.34"/>
<polygon fill="#4a5568" stroke="#4a5568" points="384.47,-119.81 389,-110.24 379.49,-114.9 384.47,-119.81"/>
</g>
<!-- reg -->
<g id="node6" class="node">
<title>reg</title>
<polygon fill="#121829" stroke="#1e2a4a" points="597.38,-109 478.62,-109 478.62,-73 597.38,-73 597.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="538" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">registry container</text>
<text xml:space="preserve" text-anchor="middle" x="538" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;registry</text>
<polygon fill="#121829" stroke="#1e2a4a" points="588.38,-109 469.62,-109 469.62,-73 588.38,-73 588.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="529" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">registry container</text>
<text xml:space="preserve" text-anchor="middle" x="529" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">acmebank&#45;registry</text>
</g>
<!-- dirname&#45;&gt;reg -->
<g id="edge5" class="edge">
<title>dirname&#45;&gt;reg</title>
<path fill="none" stroke="#4a5568" d="M356.87,-190.63C390.74,-179.74 433.49,-163.98 469,-144.5 483.3,-136.65 497.86,-126.04 509.9,-116.41"/>
<polygon fill="#4a5568" stroke="#4a5568" points="511.88,-119.31 517.39,-110.26 507.44,-113.9 511.88,-119.31"/>
<path fill="none" stroke="#4a5568" d="M350.86,-190.3C383.87,-179.35 425.43,-163.64 460,-144.5 474.27,-136.6 488.83,-125.98 500.87,-116.36"/>
<polygon fill="#4a5568" stroke="#4a5568" points="502.85,-119.26 508.36,-110.21 498.41,-113.84 502.85,-119.26"/>
</g>
<!-- cluster -->
<g id="node11" class="node">
<title>cluster</title>
<polygon fill="#1a1a3a" stroke="#1e2a4a" points="469.81,-10.54 469.81,-25.46 438.29,-36 393.71,-36 362.19,-25.46 362.19,-10.54 393.71,0 438.29,0 469.81,-10.54"/>
<text xml:space="preserve" text-anchor="middle" x="416" y="-14.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">kind cluster</text>
<polygon fill="#1a1a3a" stroke="#1e2a4a" points="460.81,-10.54 460.81,-25.46 429.29,-36 384.71,-36 353.19,-25.46 353.19,-10.54 384.71,0 429.29,0 460.81,-10.54"/>
<text xml:space="preserve" text-anchor="middle" x="407" y="-14.3" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#0066ff">kind cluster</text>
</g>
<!-- cname&#45;&gt;cluster -->
<g id="edge9" class="edge">
<title>cname&#45;&gt;cluster</title>
<path fill="none" stroke="#4a5568" d="M93.06,-72.62C99.54,-69.72 106.38,-67.02 113,-65 192.78,-40.7 288.45,-28.91 350.65,-23.42"/>
<polygon fill="#4a5568" stroke="#4a5568" points="350.69,-26.93 360.36,-22.6 350.1,-19.96 350.69,-26.93"/>
<path fill="none" stroke="#4a5568" d="M93.35,-72.51C99.75,-69.67 106.48,-67 113,-65 189.55,-41.49 281.19,-29.58 341.58,-23.85"/>
<polygon fill="#4a5568" stroke="#4a5568" points="341.71,-27.35 351.35,-22.96 341.07,-20.38 341.71,-27.35"/>
</g>
<!-- ports&#45;&gt;cluster -->
<g id="edge10" class="edge">
<title>ports&#45;&gt;cluster</title>
<path fill="none" stroke="#4a5568" d="M416,-72.81C416,-65.23 416,-56.1 416,-47.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="419.5,-47.54 416,-37.54 412.5,-47.54 419.5,-47.54"/>
<path fill="none" stroke="#4a5568" d="M407,-72.81C407,-65.23 407,-56.1 407,-47.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="410.5,-47.54 407,-37.54 403.5,-47.54 410.5,-47.54"/>
</g>
<!-- versions -->
<g id="node7" class="node">
<title>versions</title>
<polygon fill="#121829" stroke="#1e2a4a" points="759.38,-401.83 652.62,-401.83 652.62,-365.83 759.38,-365.83 759.38,-401.83"/>
<text xml:space="preserve" text-anchor="middle" x="706" y="-386.88" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">versions.env</text>
<text xml:space="preserve" text-anchor="middle" x="706" y="-373.38" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">pinned toolchain</text>
<polygon fill="#121829" stroke="#1e2a4a" points="750.38,-401.83 643.62,-401.83 643.62,-365.83 750.38,-365.83 750.38,-401.83"/>
<text xml:space="preserve" text-anchor="middle" x="697" y="-386.88" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">versions.env</text>
<text xml:space="preserve" text-anchor="middle" x="697" y="-373.38" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">pinned toolchain</text>
</g>
<!-- profile -->
<g id="node8" class="node">
<title>profile</title>
<polygon fill="#121829" stroke="#1e2a4a" points="790.5,-318.58 621.5,-318.58 621.5,-282.58 790.5,-282.58 790.5,-318.58"/>
<text xml:space="preserve" text-anchor="middle" x="706" y="-303.63" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">env.d/&lt;profile&gt;.env</text>
<text xml:space="preserve" text-anchor="middle" x="706" y="-290.13" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">nodes · CNI · audit · addons</text>
<polygon fill="#121829" stroke="#1e2a4a" points="781.5,-318.58 612.5,-318.58 612.5,-282.58 781.5,-282.58 781.5,-318.58"/>
<text xml:space="preserve" text-anchor="middle" x="697" y="-303.63" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">env.d/&lt;profile&gt;.env</text>
<text xml:space="preserve" text-anchor="middle" x="697" y="-290.13" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">nodes · CNI · audit · addons</text>
</g>
<!-- versions&#45;&gt;profile -->
<g id="edge6" class="edge">
<title>versions&#45;&gt;profile</title>
<path fill="none" stroke="#4a5568" d="M706,-365.59C706,-355.32 706,-342.03 706,-330.21"/>
<polygon fill="#4a5568" stroke="#4a5568" points="709.5,-330.58 706,-320.58 702.5,-330.58 709.5,-330.58"/>
<text xml:space="preserve" text-anchor="middle" x="737.5" y="-339.28" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
<path fill="none" stroke="#4a5568" d="M697,-365.59C697,-355.32 697,-342.03 697,-330.21"/>
<polygon fill="#4a5568" stroke="#4a5568" points="700.5,-330.58 697,-320.58 693.5,-330.58 700.5,-330.58"/>
<text xml:space="preserve" text-anchor="middle" x="728.5" y="-339.28" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
</g>
<!-- localenv -->
<g id="node9" class="node">
<title>localenv</title>
<polygon fill="#121829" stroke="#1e2a4a" points="761.88,-226.54 646.12,-226.54 646.12,-190.54 761.88,-190.54 761.88,-226.54"/>
<text xml:space="preserve" text-anchor="middle" x="704" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">ctrl/.env</text>
<text xml:space="preserve" text-anchor="middle" x="704" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">secrets, overrides</text>
<polygon fill="#121829" stroke="#1e2a4a" points="752.88,-226.54 637.12,-226.54 637.12,-190.54 752.88,-190.54 752.88,-226.54"/>
<text xml:space="preserve" text-anchor="middle" x="695" y="-211.59" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">ctrl/.env</text>
<text xml:space="preserve" text-anchor="middle" x="695" y="-198.09" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#e8eaf0">secrets, overrides</text>
</g>
<!-- profile&#45;&gt;localenv -->
<g id="edge7" class="edge">
<title>profile&#45;&gt;localenv</title>
<path fill="none" stroke="#4a5568" d="M705.61,-282.22C705.34,-269.76 704.96,-252.69 704.64,-238.23"/>
<polygon fill="#4a5568" stroke="#4a5568" points="708.14,-238.28 704.42,-228.36 701.14,-238.43 708.14,-238.28"/>
<text xml:space="preserve" text-anchor="middle" x="736.68" y="-256.03" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
<path fill="none" stroke="#4a5568" d="M696.61,-282.22C696.34,-269.76 695.96,-252.69 695.64,-238.23"/>
<polygon fill="#4a5568" stroke="#4a5568" points="699.14,-238.28 695.42,-228.36 692.14,-238.43 699.14,-238.28"/>
<text xml:space="preserve" text-anchor="middle" x="727.68" y="-256.03" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
</g>
<!-- shell -->
<g id="node10" class="node">
<title>shell</title>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="772.38,-109 623.62,-109 623.62,-73 772.38,-73 772.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="698" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">the environment</text>
<text xml:space="preserve" text-anchor="middle" x="698" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">PROFILE=client make …</text>
<polygon fill="#1a3a1a" stroke="#1e2a4a" points="763.38,-109 614.62,-109 614.62,-73 763.38,-73 763.38,-109"/>
<text xml:space="preserve" text-anchor="middle" x="689" y="-94.05" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">the environment</text>
<text xml:space="preserve" text-anchor="middle" x="689" y="-80.55" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#00c853">PROFILE=client make …</text>
</g>
<!-- localenv&#45;&gt;shell -->
<g id="edge8" class="edge">
<title>localenv&#45;&gt;shell</title>
<path fill="none" stroke="#00c853" d="M703.11,-190.49C702.16,-172.16 700.63,-142.72 699.49,-120.79"/>
<polygon fill="#00c853" stroke="#00c853" points="703,-120.81 698.99,-111.01 696.01,-121.18 703,-120.81"/>
<text xml:space="preserve" text-anchor="middle" x="733.21" y="-155.2" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
<path fill="none" stroke="#00c853" d="M694.11,-190.49C693.16,-172.16 691.63,-142.72 690.49,-120.79"/>
<polygon fill="#00c853" stroke="#00c853" points="694,-120.81 689.99,-111.01 687.01,-121.18 694,-120.81"/>
<text xml:space="preserve" text-anchor="middle" x="724.21" y="-155.2" font-family="Helvetica,sans-Serif" font-size="9.00" fill="#8892a8">overridden by</text>
</g>
<!-- shell&#45;&gt;cluster -->
<g id="edge11" class="edge">
<title>shell&#45;&gt;cluster</title>
<path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M637.29,-72.6C627.83,-69.99 618.17,-67.38 609,-65 562.72,-52.99 509.89,-40.48 471.22,-31.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="472.2,-28.18 461.67,-29.35 470.63,-35 472.2,-28.18"/>
<path fill="none" stroke="#4a5568" stroke-dasharray="5,2" d="M628.29,-72.6C618.83,-69.99 609.17,-67.38 600,-65 553.72,-52.99 500.89,-40.48 462.22,-31.54"/>
<polygon fill="#4a5568" stroke="#4a5568" points="463.2,-28.18 452.67,-29.35 461.63,-35 463.2,-28.18"/>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 11 KiB

After

Width:  |  Height:  |  Size: 11 KiB

View File

@@ -265,11 +265,11 @@
<h3>The only prerequisite</h3>
<p><b>Docker.</b> No curl, no jq, no python, no apt repositories to configure.</p>
<pre><code><span class="c"># then, in the environment directory:</span>
make station <span class="c"># is this workstation ready? reports, never fixes</span>
make check <span class="c"># is this machine ready? reports, never fixes</span>
make deps <span class="c"># install the pinned toolchain</span>
make cluster up <span class="c"># build the cluster for the active profile</span>
</code></pre>
<p>Read <code>make station</code> before <code>make deps</code>. It never changes
<p>Read <code>make check</code> before <code>make deps</code>. It never changes
anything — it prints what it found and, at the end, the steps it cannot perform
for you.</p>
</div>
@@ -280,13 +280,13 @@ make cluster up <span class="c"># build the cluster for the active profile</spa
<p class="lede">Start to finish, in order, with what each one actually does.</p>
<div class="prose">
<h3>1 &middot; make station</h3>
<p>Asks whether this workstation is ready. It <b>changes nothing</b> — it
<h3>1 &middot; make check</h3>
<p>Asks whether this machine is ready. It <b>changes nothing</b> — it
reports what it found and, at the end, the things only a human can do
(anything needing <code>sudo</code>, or a Windows-side restart). Read it
before installing anything; it is faster than discovering the same problems
one failure at a time.</p>
<pre><code>make station</code></pre>
<pre><code>make check</code></pre>
<h3>2 &middot; make setup</h3>
<p>Does the preparation that can be automated: installs the pinned
@@ -381,8 +381,8 @@ make setup core <span class="c"># same distinction, via setup</span>
wants only Docker.</p>
<h3>Air-gapped</h3>
<pre><code>make wizard full <span class="c"># bakes every binary into the image</span>
docker save …-wizard:full | gzip &gt; rig.tgz
<pre><code>make deps-image full <span class="c"># bakes every binary into the image</span>
docker save …-deps:full | gzip &gt; rig.tgz
<span class="c"># carry that one file in, then:</span>
docker load &lt; rig.tgz &amp;&amp; make cluster up PROFILE=offline
</code></pre>
@@ -485,7 +485,7 @@ docker load &lt; rig.tgz &amp;&amp; make cluster up PROFILE=offline
registry is usually behind an internal CA, and trust has to reach
<b>three</b> places: the host Docker daemon, every cluster node's containerd
(nodes do <i>not</i> inherit host trust), and any in-cluster client. Set
<code>REGISTRY_CA_FILE</code> and <code>make station</code> reports which is
<code>REGISTRY_CA_FILE</code> and <code>make check</code> reports which is
still missing. The symptom otherwise is an opaque
<code>x509: certificate signed by unknown authority</code>.</p></div>
@@ -528,11 +528,11 @@ docker load &lt; rig.tgz &amp;&amp; make cluster up PROFILE=offline
<h3>Tilt stops noticing file changes</h3>
<p>Almost always <code>inotify</code> limits, and it fails <i>silently</i>
nothing errors, changes just stop being picked up. Defaults on WSL are far too
low. <code>make station</code> reports it and prints the fix.</p>
low. <code>make check</code> reports it and prints the fix.</p>
<h3>Cluster creation dies halfway with a port error</h3>
<p>Docker reports <code>failed to bind host port … address already in use</code>
partway through creating the cluster. Run <code>make station</code> first — it
partway through creating the cluster. Run <code>make check</code> first — it
checks every port in this environment's block before anything is built.</p>
<h3>Every node stays NotReady</h3>

View File

@@ -1,9 +0,0 @@
# NOTE: generated/ is deliberately NOT ignored. The artifact IS the deliverable —
# the whole point is a folder you copy, apply and boot without generating
# anything first. Regenerate it with `make manifest` after editing bundle.json
# or app/serve.py, and commit the result.
#
# (This differs from rig's ctrl/k8s/generated, which is a local build artifact.)
__pycache__/
*.pyc

View File

@@ -1,50 +0,0 @@
# Thin control Makefile — one target per ctrl/ script, subcommand as an
# argument. Same shape as rig's, for the same reason: the logic lives in the
# script, never here.
#
# make up -> ctrl/bundle.sh up
# make bundle down
#
# This directory is a BUNDLE, not an installer. It needs a cluster, which rig
# owns:
#
# cd .. && make cluster up # kind cluster for this environment
# make up # then deploy this bundle into it
#
# Start with: make up
ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
ifneq ($(ARGS),)
$(eval $(ARGS):;@:)
endif
.DEFAULT_GOAL := help
.PHONY: help bundle manifest up down status url list dev
help: ## list targets
@grep -hE '^[a-z]+:.*?##' $(MAKEFILE_LIST) | sed 's/:.*##/\t/' | expand -t16
bundle: ## the bundle [manifest|up|down|status|url|list|dev]
bash ctrl/bundle.sh $(or $(ARGS),status)
# Shorthands for the ones used constantly.
manifest: ## regenerate generated/<slug>.yaml — no cluster needed
bash ctrl/bundle.sh manifest
up: ## deploy this rig (installs MetalLB if absent)
bash ctrl/bundle.sh up
down: ## remove this rig (leaves cluster, MetalLB, siblings)
bash ctrl/bundle.sh down
status: ## what is deployed for this rig
bash ctrl/bundle.sh status
url: ## the address MetalLB assigned
bash ctrl/bundle.sh url
list: ## every rig in this cluster, with addresses
bash ctrl/bundle.sh list
dev: ## run the UI locally with vite — no cluster needed
bash ctrl/bundle.sh dev

View File

@@ -1,140 +0,0 @@
# sample-rig
A minimal, non-sensitive bundle that proves an installation works and shows what
shipped. Copy it, rename it, and you have another rig.
```bash
make manifest # generate the artifact — no cluster, no kubectl needed
make up # deploy it into the local cluster
make list # every rig in this cluster, with addresses
make dev # run the UI locally with vite, no cluster at all
```
`make up` prints an address. Open it and the page says **IT WORKS**, then lists
the tools and rigs in the bundle.
## What it is for
Three jobs, in the order you hit them:
1. **Prove the install.** kind is there, a cluster exists, MetalLB hands out an
address, a `type: LoadBalancer` Service actually resolves, and a pod serves.
If all of that works, the environment is sound.
2. **Say what shipped.** The page renders [`bundle.json`](bundle.json) —
standalone tools and rigs, flat, with none of soleprint's internal hierarchy.
Editing that file is the only step needed to change the listing.
3. **Stand in for the real thing.** Nothing here is sensitive. The real
architecture connects separately, against a setup already known to work.
## The UI is a complement, not the product
`rig-ui/` is just a vite app. It complements a rig; a rig is complete and useful
without it, and nothing depends on it being there. It is deliberately **not**
generated by kind or tilt — you copy the folder into a rig after that rig is
pulled, and apply one manifest:
```bash
kubectl apply -n <namespace> -f rig-ui/k8s.yaml
```
That file is the whole integration: one Pod running `npm run dev` on
`node:22-alpine`, one Service. A bare Pod rather than a Deployment because this
is a dev-loop convenience, not a workload to keep alive.
The app and `bundle.json` arrive as a ConfigMap, so nothing is baked into an
image and editing the bundle is the entire update cycle. The container runs
`npm install` at start, which needs egress to a registry — on a locked-down
cluster point npm at the internal one, or bake an image instead. Nothing else
changes if you do.
## One artifact, two destinations
`ctrl/manifest.py` emits `generated/<slug>.yaml` — namespace, the app and
bundle embedded in a ConfigMap, Pod, Service. It is self-contained and applies
unmodified anywhere:
```bash
kubectl apply -f generated/sample-rig.yaml # local kind, or an external cluster
```
`make up` applies **that same file**. There is no separate local path, so what
works here cannot quietly differ from what is applied elsewhere.
This is what `type: LoadBalancer` buys. MetalLB answers it on kind; the AWS load
balancer controller answers it on EKS. NodePort would not survive the trip — it
is a single cluster-wide port range, so two rigs would have to negotiate numbers.
**VPC-agnostic on purpose.** The target is EKS, but the Service carries no
annotations — no `aws-load-balancer-subnets`, no security groups, no `-scheme`,
no `-type: nlb`. Each of those encodes a specific network layout, and one of them
appearing here would pin the artifact to the account and VPC it was written
against, which is precisely what stops it also working on kind. Subnet discovery
is the cluster's business: EKS resolves it from the tags its own subnets carry.
That leaves one thing genuinely environment-specific — internal versus
internet-facing. A bare `LoadBalancer` provisions internet-facing, which a
regulated account will usually refuse, and should. That belongs in a
per-environment overlay applied on top, never inlined into this artifact.
**MetalLB only — no ingress-nginx.** Its controller supports a narrow window of
Kubernetes versions, so depending on it constrains which k8s a rig can be built
with. That undercuts running trailing-edge control planes to model a legacy
estate, which is the reason `versions.env` pins v1_33..v1_36. MetalLB carries no
such constraint, so reachability costs nothing in version coverage.
## Several rigs, one cluster
Identity follows the **folder name**, the same rule rig uses for cluster
identity. The namespace is the folder; resource names are generic, and names only
have to be unique within a namespace.
```bash
cp -r sample-rig corporate-rig
cd corporate-rig && make up # its own namespace, its own address
```
No edits, no collisions, both in the same local cluster. `make list` shows them
together. `make down` removes only this one — siblings, MetalLB and the cluster
are left alone.
Client rigs are gitignored (`*-rig/`, with `sample-rig/` the deliberate
exception): a rig's k8s files spell out a real architecture, and that is exactly
what must not land in this repo.
## Staging workstations
`ctrl/manifest.py` is stdlib-only on purpose: it runs on a bare machine before
anything is installed. The toolchain itself is rig's job — `make deps` installs
the pinned kind and tilt binaries, which is what makes a staging AWS workspace
reachable from the same commands as a laptop.
## Layout
```
sample-rig/
├── Makefile # thin — one target per ctrl/ script
├── bundle.json # what shipped; the UI renders THIS
├── rig-ui/ # the vite app — optional, copied into a rig to enable it
│ ├── k8s.yaml # how to plug it in: one Pod, one Service
│ ├── index.html
│ ├── package.json
│ ├── vite.config.js
│ └── src/{main.js,style.css}
├── ctrl/
│ ├── manifest.py # emits the artifact
│ └── bundle.sh # generate / deploy / inspect
└── generated/ # the artifact — committed, this is the deliverable
```
Editing `bundle.json` or anything in `rig-ui/` means re-running `make manifest`.
The ConfigMap carries a checksum of everything embedded, so a stale deployment is
visible rather than silent.
## Not built, but not foreclosed
Everything derives from `bundle.json` plus a target namespace. A Pulumi or
Terraform emitter would sit beside `ctrl/manifest.py` consuming the same inputs;
nothing above it assumes the artifact is YAML.
Licence terms for the compiled UI component belong in the soleprint-generated
bundle, not here — this sample carries no proprietary component.

View File

@@ -1,51 +0,0 @@
{
"_comment": "What this bundle contains. Single source of truth — the landing page renders THIS file, so adding an entry here is the only edit needed. Deliberately FLAT: standalone tools and rigs, with none of soleprint's internal hierarchy (no artery/atlas/station layering). Nothing here is sensitive; the real architecture connects separately.",
"bundle": {
"name": "sample-rig",
"description": "Non-sensitive sample bundle. Proves the kind install works and shows what ships.",
"sensitive": false
},
"tools": [
{
"name": "modelgen",
"summary": "Generate models from config",
"standalone": true
},
{
"name": "datagen",
"summary": "Generate test data from rig-owned generators",
"standalone": true
},
{
"name": "graphgen",
"summary": "Generate navigable model graphs",
"standalone": true
},
{
"name": "tester",
"summary": "HTTP contract test runner — one suite, any environment",
"standalone": true
},
{
"name": "databrowse",
"summary": "SQL data browser",
"standalone": true
},
{
"name": "sbwrapper",
"summary": "Sandbox wrapper",
"standalone": true
}
],
"rigs": [
{
"name": "sample-rig",
"summary": "This bundle — a minimal, copyable environment",
"active": true
}
],
"next": [
"Point MANIFESTS_DIR at the real manifests to connect the actual architecture.",
"Real k8s files are versioned separately and are not part of this bundle."
]
}

View File

@@ -1,40 +0,0 @@
{
"_comment": "MOCKED cluster state. Nothing here is read from a live cluster — it exists so the UI can be shown when there is no cluster at all (a locked-down machine, a laptop with no memory to spare, a demo where kind will not start). The page labels it as mocked; a demo that looks live but is not is worse than one that says so. When a real cluster is present the same shapes come from kubectl.",
"mocked": true,
"cluster": {
"name": "sample-rig",
"context": "kind-sample-rig",
"provider": "kind",
"profile": "minimal",
"k8s": "v1.36.1",
"nodes": 1
},
"workloads": [
{
"name": "rig-ui",
"summary": "Pod · node:22-alpine · vite on :5173",
"state": "Running"
},
{
"name": "metallb-system/controller",
"summary": "Deployment · assigns LoadBalancer addresses",
"state": "Running"
},
{
"name": "metallb-system/speaker",
"summary": "DaemonSet · answers ARP in layer 2 mode",
"state": "Running"
}
],
"services": [
{
"name": "rig-ui",
"summary": "LoadBalancer · 80 -> 5173 · no annotations, so it resolves on kind and on EKS alike",
"state": "172.18.255.200"
}
]
}

View File

@@ -1,223 +0,0 @@
#!/usr/bin/env bash
# The rig bundle: generate it, deploy it, tear it down, find it.
#
# Usage: bundle.sh manifest | up | down | status | url | list | dev
#
# What `up` proves, in order: kind installed and a cluster exists, MetalLB can
# hand out an address, a Service of type LoadBalancer actually resolves, and a
# pod serves the bundle listing. If all of that works the installation is sound,
# and the only thing missing is the real architecture.
#
# ONE ARTIFACT
# `up` applies generated/<slug>.yaml — the same self-contained file you would
# hand to an external cluster. There is no separate local path, so what works
# here cannot quietly differ from the master deployment applied elsewhere.
#
# ONE CLUSTER, SEVERAL RIGS
# Identity follows the FOLDER NAME, exactly as rig's cluster identity does. This
# directory deploys into a namespace named after itself, so copying it to
# corporate-rig/ yields a second rig in the SAME local cluster with no edits and
# no collisions — different namespace, its own MetalLB address. `list` shows all
# of them. The cluster itself is rig's business; this only ever owns a namespace.
#
# MetalLB is installed by calling rig's own addon script rather than
# reimplementing it — deriving the pool from the kind Docker network is the
# fiddly part and there should be exactly one copy of it.
set -euo pipefail
cd "$(dirname "$0")/.."
BUNDLE_ROOT="$(pwd)"
RIG_CTRL="$(cd .. && pwd)/ctrl"
# The containing folder's name, reduced to a DNS label (same rule as rig's
# default_cluster_name and ctrl/manifest.py, so all three agree on the slug).
slug() {
local n
n=$(basename "$BUNDLE_ROOT")
n=$(echo "$n" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9-' '-')
n=$(echo "$n" | sed 's/^-*//; s/-*$//')
echo "${n:-rig-bundle}"
}
NS="$(slug)"
ARTIFACT="generated/${NS}.yaml"
# Resolved lazily, not at load time: `manifest` and `dev` deliberately work
# with no cluster and no kubectl at all, and a top-level check would break that.
#
# Follows whatever context rig's cluster.sh selected, so this bundle works in a
# copied-and-renamed environment without being told which cluster it is in.
init_kube() {
KUBECONTEXT="${KUBECONTEXT:-$(kubectl config current-context 2>/dev/null || true)}"
if [ -z "$KUBECONTEXT" ]; then
echo "no kubectl context — bring a cluster up first: (cd .. && make cluster up)" >&2
exit 1
fi
KCTX="kubectl --context ${KUBECONTEXT}"
K="kubectl --context ${KUBECONTEXT} --namespace ${NS}"
}
require_cluster() {
if ! $KCTX cluster-info >/dev/null 2>&1; then
echo "context '$KUBECONTEXT' does not reach a cluster" >&2
echo "bring one up: (cd .. && make cluster up)" >&2
exit 1
fi
}
ensure_metallb() {
if $KCTX get deployment -n metallb-system controller >/dev/null 2>&1; then
echo "metallb: present"
return 0
fi
# Only kind needs it. On a real cluster the cloud load balancer answers a
# `type: LoadBalancer` Service, and installing MetalLB there would be wrong.
case "$KUBECONTEXT" in
kind-*) ;;
*)
echo "metallb: skipped — '$KUBECONTEXT' is not a kind context"
echo " (a cloud load balancer answers LoadBalancer services there)"
return 0
;;
esac
if [ ! -f "$RIG_CTRL/addons/metallb.sh" ]; then
echo "metallb is not installed and rig's addon script was not found at" >&2
echo " $RIG_CTRL/addons/metallb.sh" >&2
echo "a Service of type LoadBalancer will sit at <pending> without it." >&2
exit 1
fi
# rig's addons derive their target cluster from RIG'S OWN folder name via
# load_config, so left alone this bundle would install into `kind-rig` —
# a cluster that need not exist — while deploying everything else into the
# context actually selected. CLUSTER is in load_config's overridable set,
# so passing it here points the addon at the same cluster we are using.
local target="${KUBECONTEXT#kind-}"
echo "metallb: installing via rig's addon into '$target'"
CLUSTER="$target" bash "$RIG_CTRL/addons/metallb.sh"
}
# Regenerate the artifact. No cluster and no kubectl required — this is the step
# a staging workstation runs before anything is installed.
manifest() {
mkdir -p generated
python3 ctrl/manifest.py "$NS" > "$ARTIFACT"
echo "wrote $ARTIFACT ($(wc -l < "$ARTIFACT") lines)"
echo " applies as-is anywhere: kubectl apply -f ${BUNDLE_ROOT}/${ARTIFACT}"
}
up() {
manifest
init_kube
require_cluster
ensure_metallb
echo
echo "applying '${NS}' to context '${KUBECONTEXT}'"
$KCTX apply -f "$ARTIFACT"
# `rollout status` does not work on a bare Pod — it only understands
# Deployments, StatefulSets and DaemonSets. Wait on the condition instead.
# This is the slow step: the container npm-installs before vite serves.
echo "waiting for the pod to be ready (npm install runs first)..."
$K wait --for=condition=Ready pod/rig-ui --timeout=300s
echo
url
}
down() {
init_kube
# Delete the namespace and everything in it goes with it. Scoped to THIS
# rig — a sibling rig in the same cluster is untouched.
$KCTX delete namespace "$NS" --ignore-not-found
echo "'${NS}' removed (cluster, metallb and any sibling rig are left alone)"
}
status() {
init_kube
require_cluster
if ! $KCTX get namespace "$NS" >/dev/null 2>&1; then
echo "'${NS}' is not deployed — run: make up"
return 0
fi
$K get pod,svc,configmap -o wide
}
# Every rig in this cluster, not just this one — the point of the namespace
# split is that several coexist, so there has to be a way to see them together.
list() {
init_kube
require_cluster
local names
names=$($KCTX get namespace -l rig.bundle/name \
-o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true)
if [ -z "$names" ]; then
echo "no rigs deployed in context '${KUBECONTEXT}'"
return 0
fi
printf "%-20s %-16s %s\n" RIG ADDRESS ""
local n ip
for n in $names; do
ip=$($KCTX -n "$n" get svc rig-ui \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true)
printf "%-20s %-16s %s\n" "$n" "${ip:-<pending>}" \
"$([ "$n" = "$NS" ] && echo '<- this one')"
done
}
# The address MetalLB (or a cloud load balancer) assigned. <pending> here is the
# classic silent failure: everything reports healthy and nothing is reachable.
url() {
init_kube
local ip
ip=$($K get svc rig-ui \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true)
if [ -z "$ip" ]; then
ip=$($K get svc rig-ui \
-o jsonpath='{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null || true)
fi
if [ -z "$ip" ]; then
echo "no external address yet — nothing has assigned one."
echo "on kind: kubectl --context $KUBECONTEXT -n metallb-system get pods"
return 1
fi
echo "IT WORKS -> http://${ip}/"
echo " bundle http://${ip}/bundle.json"
}
# Run the UI locally with no cluster at all — the fast way to iterate on
# bundle.json. Same vite command the pod runs, so what you see here is what
# gets served there.
dev() {
if ! command -v npm >/dev/null 2>&1; then
echo "npm not found — the UI needs node locally for this." >&2
echo "(in-cluster it runs on the node:22-alpine image instead)" >&2
exit 1
fi
# bundle.json lives one level up so it stays the rig's data rather than the
# app's; vite serves public/ at the root, which is where the app fetches it.
mkdir -p rig-ui/public
cp bundle.json rig-ui/public/bundle.json
# The mocked cluster is a DEMO asset and is deliberately not embedded in the
# deployed artifact — on a real rig the UI would then show canned values
# beside a live cluster, which is precisely the lie its banner warns about.
# It is served here, and in the static build for the public UI-only page.
cp cluster.mock.json rig-ui/public/cluster.mock.json
cd rig-ui
[ -d node_modules ] || npm install --no-audit --no-fund
VITE_RIG_NAME="$NS" npm run dev
}
case "${1:-status}" in
manifest) manifest ;;
up) up ;;
down) down ;;
status) status ;;
url) url ;;
list) list ;;
dev) dev ;;
*) echo "usage: $0 [manifest|up|down|status|url|list|dev]" >&2; exit 1 ;;
esac

View File

@@ -1,141 +0,0 @@
#!/usr/bin/env python3
"""Emit the complete, self-contained deployment for this rig.
python3 ctrl/manifest.py [namespace] > generated/<slug>.yaml
The output is the ARTIFACT. It carries everything — namespace, the vite app and
bundle.json embedded in a ConfigMap, the Pod and the Service — so it applies
unmodified to any cluster:
kubectl apply -f generated/sample-rig.yaml
On kind, MetalLB answers the `type: LoadBalancer` Service. On a real external
cluster the cloud load balancer does. Same file, no edits, no branch — which is
the point: what runs locally is byte-identical to the deployment applied
elsewhere, so local success actually means something.
`ctrl/bundle.sh up` applies this same generated output rather than a separate
code path, so the local convenience wrapper can never drift from the artifact.
Stdlib only, deliberately: this must run on a bare staging workstation before
anything is installed, so it cannot depend on PyYAML or a template engine.
Open seam — not built: everything here derives from bundle.json plus a target
namespace. A Pulumi or Terraform emitter would sit beside this file consuming the
same inputs; nothing above it assumes the artifact is YAML.
"""
import json
import re
import sys
import zlib
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
UI = ROOT / "rig-ui"
# Files embedded into the ConfigMap, mounted read-only at /src in the pod and
# copied into vite's layout at start (see rig-ui/k8s.yaml). Flat on purpose:
# ConfigMap keys cannot contain '/'.
EMBEDDED = {
"bundle.json": ROOT / "bundle.json",
"package.json": UI / "package.json",
"vite.config.js": UI / "vite.config.js",
"index.html": UI / "index.html",
"main.js": UI / "src" / "main.js",
"style.css": UI / "src" / "style.css",
}
def slug(name: str) -> str:
"""Reduce a folder name to a DNS label, matching ctrl/bundle.sh's rule."""
out = re.sub(r"[^a-z0-9-]", "-", name.lower()).strip("-")
return out or "rig-bundle"
def block(text: str, indent: int) -> str:
"""Indent a file's contents for a YAML literal block scalar.
Blank lines are emitted truly empty rather than as whitespace: trailing
spaces on an otherwise blank line are legal YAML but show up as diff noise
in a committed artifact.
"""
pad = " " * indent
return "\n".join(pad + line if line.strip() else "" for line in text.splitlines())
def checksum(parts: list[str]) -> str:
"""Stable content hash of everything embedded, stamped as a label.
A mounted ConfigMap updates in place without restarting anything, so without
a visible change nothing signals that the pod is serving stale content.
"""
return str(zlib.crc32("".join(parts).encode()) & 0xFFFFFFFF)
def build(namespace: str) -> str:
contents = {}
for key, path in EMBEDDED.items():
if not path.exists():
sys.exit(f"missing input: {path}")
contents[key] = path.read_text()
# Fail loudly here rather than shipping an artifact that renders an error.
try:
json.loads(contents["bundle.json"])
except json.JSONDecodeError as exc:
sys.exit(f"bundle.json is not valid JSON: {exc}")
app = (UI / "k8s.yaml").read_text()
app = app.replace("__RIG_NAME__", namespace)
data = "\n".join(
f" {key}: |\n{block(text, 4)}" for key, text in sorted(contents.items())
)
return f"""# GENERATED by ctrl/manifest.py — do not edit.
# Regenerate with: make manifest
#
# Self-contained: applies as-is to any cluster, local kind or external.
# kubectl apply -f this-file.yaml
#
# Namespace carries the identity, so several rigs coexist in one cluster.
apiVersion: v1
kind: Namespace
metadata:
name: {namespace}
labels:
rig.bundle/name: {namespace}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: rig-ui
namespace: {namespace}
labels:
rig.bundle/checksum: "{checksum(list(contents.values()))}"
data:
{data}
---
{_namespaced(app, namespace)}
"""
def _namespaced(doc: str, namespace: str) -> str:
"""Add `namespace:` to each resource so the artifact applies without -n.
rig-ui/k8s.yaml omits it on purpose — applied by hand it should land in
whatever namespace you choose. Pinning it belongs to the generated artifact,
which has to be self-contained.
"""
return re.sub(
r"^(metadata:\n(?:[ \t]+.*\n)*?)([ \t]+)(name: rig-ui)$",
lambda m: f"{m.group(1)}{m.group(2)}{m.group(3)}\n{m.group(2)}namespace: {namespace}",
doc.strip(),
flags=re.MULTILINE,
)
if __name__ == "__main__":
target = sys.argv[1] if len(sys.argv) > 1 else slug(ROOT.name)
sys.stdout.write(build(target))

View File

@@ -1,406 +0,0 @@
# GENERATED by ctrl/manifest.py — do not edit.
# Regenerate with: make manifest
#
# Self-contained: applies as-is to any cluster, local kind or external.
# kubectl apply -f this-file.yaml
#
# Namespace carries the identity, so several rigs coexist in one cluster.
apiVersion: v1
kind: Namespace
metadata:
name: sample-rig
labels:
rig.bundle/name: sample-rig
---
apiVersion: v1
kind: ConfigMap
metadata:
name: rig-ui
namespace: sample-rig
labels:
rig.bundle/checksum: "2074194964"
data:
bundle.json: |
{
"_comment": "What this bundle contains. Single source of truth — the landing page renders THIS file, so adding an entry here is the only edit needed. Deliberately FLAT: standalone tools and rigs, with none of soleprint's internal hierarchy (no artery/atlas/station layering). Nothing here is sensitive; the real architecture connects separately.",
"bundle": {
"name": "sample-rig",
"description": "Non-sensitive sample bundle. Proves the kind install works and shows what ships.",
"sensitive": false
},
"tools": [
{
"name": "modelgen",
"summary": "Generate models from config",
"standalone": true
},
{
"name": "datagen",
"summary": "Generate test data from rig-owned generators",
"standalone": true
},
{
"name": "graphgen",
"summary": "Generate navigable model graphs",
"standalone": true
},
{
"name": "tester",
"summary": "HTTP contract test runner — one suite, any environment",
"standalone": true
},
{
"name": "databrowse",
"summary": "SQL data browser",
"standalone": true
},
{
"name": "sbwrapper",
"summary": "Sandbox wrapper",
"standalone": true
}
],
"rigs": [
{
"name": "sample-rig",
"summary": "This bundle — a minimal, copyable environment",
"active": true
}
],
"next": [
"Point MANIFESTS_DIR at the real manifests to connect the actual architecture.",
"Real k8s files are versioned separately and are not part of this bundle."
]
}
index.html: |
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>IT WORKS</title>
</head>
<body>
<main id="app"></main>
<script type="module" src="/src/main.js"></script>
</body>
</html>
main.js: |
import "./style.css";
/* The IT WORKS page: renders bundle.json as the list of what shipped.
*
* Plain vite, no framework — this is a complement to the rig, not part of it,
* and it should stay small enough that nobody has to adopt a stack to read it.
*
* bundle.json is fetched at runtime rather than imported, so the same built app
* serves whatever rig it was copied into. Editing the ConfigMap changes the page
* without rebuilding.
*
* Styling is a handful of rules on purpose. The real visual identity lives in
* the soleprint UI package; nothing here should grow into a theme.
*/
const esc = (s) =>
String(s).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
const tag = (text, on = false) =>
`<span class="tag${on ? " on" : ""}">${esc(text)}</span>`;
function items(list, activeKey) {
if (!list?.length) return `<li><span class="summary">nothing listed</span></li>`;
return list
.map((it) => {
const tags = [
it.standalone ? tag("standalone") : "",
it.state ? tag(it.state) : "",
activeKey && it[activeKey] ? tag("active", true) : "",
].join("");
return `<li><span class="name">${esc(it.name ?? "?")}</span>
<span class="summary">${esc(it.summary ?? "")}</span>${tags}</li>`;
})
.join("");
}
/* Cluster state, when there is any to show.
*
* Fetched separately and allowed to fail: the bundle listing is the point, and a
* rig with no cluster reachable is a normal state, not an error. Renders nothing
* at all when absent.
*
* When the payload says `mocked`, say so loudly. This exists to demo the UI on a
* machine where kind will not run — and a demo that looks live but is not is
* worse than one that admits it. */
function clusterSection(c) {
if (!c) return "";
const m = c.cluster ?? {};
const banner = c.mocked
? `<p class="mock">mocked — no cluster was queried; these are canned values</p>`
: "";
const meta = [m.context, m.k8s, m.profile ? `profile ${m.profile}` : "",
m.nodes ? `${m.nodes} node${m.nodes > 1 ? "s" : ""}` : ""]
.filter(Boolean).join(" · ");
return `
<h2>Cluster${c.mocked ? " (mocked)" : ""}</h2>
${banner}
${meta ? `<p class="sub">${esc(meta)}</p>` : ""}
<ul>${items(c.workloads)}</ul>
<h2>Services (${c.services?.length ?? 0})</h2>
<ul>${items(c.services)}</ul>`;
}
function render(b, name, cluster) {
const meta = b.bundle ?? {};
const next = (b.next ?? []).map((n) => `<li>${esc(n)}</li>`).join("");
return `
<h1><span class="ok">IT WORKS</span> — ${esc(name || meta.name || "rig")}</h1>
<p class="sub">${esc(meta.description ?? "")}</p>
<h2>Tools (${b.tools?.length ?? 0})</h2>
<ul>${items(b.tools)}</ul>
<h2>Rigs (${b.rigs?.length ?? 0})</h2>
<ul>${items(b.rigs, "active")}</ul>
${clusterSection(cluster)}
${next ? `<div class="next"><ul>${next}</ul></div>` : ""}`;
}
const app = document.getElementById("app");
const json = (path, required) =>
fetch(path).then((r) => {
if (r.ok) return r.json();
if (required) throw new Error(`${path} -> HTTP ${r.status}`);
return null; // optional: absent is a normal state, not an error
}).catch((err) => {
if (required) throw err;
return null;
});
Promise.all([json("/bundle.json", true), json("/cluster.mock.json", false)])
// RIG_NAME is injected by vite from the pod env, so two rigs sharing a
// cluster are distinguishable even if a copied bundle.json kept its old name.
.then(([b, cluster]) => {
app.innerHTML = render(b, import.meta.env.VITE_RIG_NAME, cluster);
})
.catch((err) => {
app.innerHTML = `<h1 class="err">bundle unavailable</h1>
<p class="sub">${esc(err.message)}</p>`;
});
package.json: |
{
"name": "rig-ui",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite --host 0.0.0.0 --port 5173",
"build": "vite build",
"preview": "vite preview --host 0.0.0.0 --port 5173"
},
"devDependencies": {
"vite": "^6"
}
}
style.css: |
/* Minimal, self-contained. The real visual identity ships with the soleprint UI
package, which is a separate artifact — nothing here should grow into a theme. */
body {
margin: 0;
padding: 2.5rem 1.5rem;
background: #0d0d0f;
color: #e8e8f0;
font: 14px/1.6 ui-monospace, "JetBrains Mono", Menlo, monospace;
}
main { max-width: 52rem; margin: 0 auto; }
h1 { margin: 0; font-size: 1.6rem; letter-spacing: 0.02em; }
h1 .ok { color: #3ecf8e; }
h1.err { color: #f06565; }
.sub { color: #8888a0; margin: 0.35rem 0 2.25rem; }
h2 {
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.12em;
color: #8888a0;
margin: 2rem 0 0.75rem;
font-weight: 600;
}
ul { list-style: none; margin: 0; padding: 0; }
li {
display: flex;
gap: 0.75rem;
align-items: baseline;
padding: 0.5rem 0.75rem;
border: 1px solid #2e2e38;
border-radius: 6px;
margin-bottom: 0.4rem;
background: #16161a;
}
.name { font-weight: 600; min-width: 9rem; }
.summary { color: #8888a0; flex: 1; }
.tag {
font-size: 0.7rem;
padding: 0.1rem 0.45rem;
border-radius: 3px;
background: #26262f;
color: #8888a0;
white-space: nowrap;
}
.tag.on { background: #3ecf8e; color: #0d0d0f; }
.next {
color: #555568;
font-size: 0.8rem;
margin-top: 2.5rem;
border-top: 1px solid #2e2e38;
padding-top: 1rem;
}
.next li {
display: list-item;
border: 0;
background: none;
padding: 0.15rem 0;
margin: 0 0 0 1.1rem;
list-style: disc;
}
/* Mocked-data banner. Deliberately loud: this only appears when the cluster
payload is canned, and a demo that looks live but is not is worse than one
that says so. */
.mock {
margin: 0 0 0.75rem;
padding: 0.4rem 0.75rem;
border: 1px dashed #f5a623;
border-radius: 6px;
color: #f5a623;
font-size: 0.8rem;
}
vite.config.js: |
import { defineConfig } from "vite";
/* Serves on 0.0.0.0 so the pod is reachable through the Service, and allows any
* Host header because the address is assigned at runtime (MetalLB locally, a
* cloud load balancer on EKS) and is never known at build time. */
export default defineConfig({
server: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
preview: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
});
---
# How to plug the UI into whatever k8s you generated. THIS IS THE WHOLE THING:
# one Pod running the vite app, one Service to reach it.
#
# Optional by design. The UI complements a rig; it is not part of the end
# product, and a rig is complete and useful without it. Apply this only when you
# want the listing:
#
# kubectl apply -n <your-namespace> -f rig-ui/k8s.yaml
#
# A bare Pod, not a Deployment — this is a dev-loop convenience, not a workload
# to keep alive. If it dies you re-apply it; nothing depends on it staying up.
#
# The app and bundle.json arrive as a ConfigMap named `rig-ui`, which
# ctrl/manifest.py generates from the folder. Nothing is baked into an image, so
# editing bundle.json and re-applying is the whole update cycle.
apiVersion: v1
kind: Pod
metadata:
name: rig-ui
namespace: sample-rig
labels:
app: rig-ui
spec:
containers:
- name: vite
image: node:22-alpine
workingDir: /app
# npm install at start: no image to build and no registry to publish to,
# which is the point of a minimal plug-in. It needs egress to a registry —
# on a locked-down cluster point npm at the internal one, or bake an image
# instead. Nothing else here changes if you do.
command: ["sh", "-c"]
# A ConfigMap mounts flat (keys cannot contain '/'), so the files are
# placed into vite's expected layout here. bundle.json goes to public/
# because that is what vite serves at /bundle.json, which is where the
# app fetches it.
args:
- |
mkdir -p /app/src /app/public &&
cp /src/package.json /src/vite.config.js /src/index.html /app/ &&
cp /src/main.js /src/style.css /app/src/ &&
cp /src/bundle.json /app/public/ &&
npm install --no-audit --no-fund &&
npm run dev
env:
# Rendered in the heading so two rigs sharing a cluster stay
# distinguishable. Set from the namespace by ctrl/manifest.py.
- name: VITE_RIG_NAME
value: sample-rig
ports:
- name: http
containerPort: 5173
volumeMounts:
# /src is read-only from the ConfigMap; the app is copied to a writable
# /app because npm install has to create node_modules.
- name: rig-ui
mountPath: /src
- name: app
mountPath: /app
readinessProbe:
httpGet: { path: /, port: 5173 }
# npm install decides how long this takes, and it is the slow part.
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 30
resources:
requests: { memory: 128Mi, cpu: 50m }
limits: { memory: 512Mi }
volumes:
- name: rig-ui
configMap:
name: rig-ui
- name: app
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: rig-ui
namespace: sample-rig
labels:
app: rig-ui
# No annotations, deliberately — see k8s/app.yaml. The target is EKS but this
# stays VPC-agnostic: no subnets, no security groups, no -scheme, no -type.
# A bare LoadBalancer is what lets one manifest work on kind and on EKS.
spec:
type: LoadBalancer
selector:
app: rig-ui
ports:
- name: http
port: 80
targetPort: 5173
protocol: TCP
# Pinned, because a LoadBalancer Service also allocates a NodePort and
# this is the only address that works everywhere.
#
# On WSL the MetalLB address is on a docker bridge INSIDE the Linux VM,
# and Windows has no route to it — the page looks broken while the
# cluster is perfectly healthy. 30080 is what rig's `hostport` ingress
# mode publishes to the host, so this is reachable at
# localhost:$HTTP_PORT from a Windows browser with nothing configured.
#
# Costs nothing elsewhere: MetalLB still assigns an external IP on Linux,
# and on EKS the load balancer targets this NodePort anyway. One Service,
# no per-environment branch.
#
# A pinned NodePort is cluster-unique, so two rigs must live in separate
# clusters — which is how they are run anyway.
nodePort: 30080

View File

@@ -1,3 +0,0 @@
node_modules/
public/
dist/

View File

@@ -1,12 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>IT WORKS</title>
</head>
<body>
<main id="app"></main>
<script type="module" src="/src/main.js"></script>
</body>
</html>

View File

@@ -1,108 +0,0 @@
# How to plug the UI into whatever k8s you generated. THIS IS THE WHOLE THING:
# one Pod running the vite app, one Service to reach it.
#
# Optional by design. The UI complements a rig; it is not part of the end
# product, and a rig is complete and useful without it. Apply this only when you
# want the listing:
#
# kubectl apply -n <your-namespace> -f rig-ui/k8s.yaml
#
# A bare Pod, not a Deployment — this is a dev-loop convenience, not a workload
# to keep alive. If it dies you re-apply it; nothing depends on it staying up.
#
# The app and bundle.json arrive as a ConfigMap named `rig-ui`, which
# ctrl/manifest.py generates from the folder. Nothing is baked into an image, so
# editing bundle.json and re-applying is the whole update cycle.
apiVersion: v1
kind: Pod
metadata:
name: rig-ui
labels:
app: rig-ui
spec:
containers:
- name: vite
image: node:22-alpine
workingDir: /app
# npm install at start: no image to build and no registry to publish to,
# which is the point of a minimal plug-in. It needs egress to a registry —
# on a locked-down cluster point npm at the internal one, or bake an image
# instead. Nothing else here changes if you do.
command: ["sh", "-c"]
# A ConfigMap mounts flat (keys cannot contain '/'), so the files are
# placed into vite's expected layout here. bundle.json goes to public/
# because that is what vite serves at /bundle.json, which is where the
# app fetches it.
args:
- |
mkdir -p /app/src /app/public &&
cp /src/package.json /src/vite.config.js /src/index.html /app/ &&
cp /src/main.js /src/style.css /app/src/ &&
cp /src/bundle.json /app/public/ &&
npm install --no-audit --no-fund &&
npm run dev
env:
# Rendered in the heading so two rigs sharing a cluster stay
# distinguishable. Set from the namespace by ctrl/manifest.py.
- name: VITE_RIG_NAME
value: __RIG_NAME__
ports:
- name: http
containerPort: 5173
volumeMounts:
# /src is read-only from the ConfigMap; the app is copied to a writable
# /app because npm install has to create node_modules.
- name: rig-ui
mountPath: /src
- name: app
mountPath: /app
readinessProbe:
httpGet: { path: /, port: 5173 }
# npm install decides how long this takes, and it is the slow part.
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 30
resources:
requests: { memory: 128Mi, cpu: 50m }
limits: { memory: 512Mi }
volumes:
- name: rig-ui
configMap:
name: rig-ui
- name: app
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: rig-ui
labels:
app: rig-ui
# No annotations, deliberately — see k8s/app.yaml. The target is EKS but this
# stays VPC-agnostic: no subnets, no security groups, no -scheme, no -type.
# A bare LoadBalancer is what lets one manifest work on kind and on EKS.
spec:
type: LoadBalancer
selector:
app: rig-ui
ports:
- name: http
port: 80
targetPort: 5173
protocol: TCP
# Pinned, because a LoadBalancer Service also allocates a NodePort and
# this is the only address that works everywhere.
#
# On WSL the MetalLB address is on a docker bridge INSIDE the Linux VM,
# and Windows has no route to it — the page looks broken while the
# cluster is perfectly healthy. 30080 is what rig's `hostport` ingress
# mode publishes to the host, so this is reachable at
# localhost:$HTTP_PORT from a Windows browser with nothing configured.
#
# Costs nothing elsewhere: MetalLB still assigns an external IP on Linux,
# and on EKS the load balancer targets this NodePort anyway. One Service,
# no per-environment branch.
#
# A pinned NodePort is cluster-unique, so two rigs must live in separate
# clusters — which is how they are run anyway.
nodePort: 30080

File diff suppressed because it is too large Load Diff

View File

@@ -1,14 +0,0 @@
{
"name": "rig-ui",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite --host 0.0.0.0 --port 5173",
"build": "vite build",
"preview": "vite preview --host 0.0.0.0 --port 5173"
},
"devDependencies": {
"vite": "^6"
}
}

View File

@@ -1,136 +0,0 @@
import "./style.css";
/* The IT WORKS page: renders bundle.json as the list of what shipped.
*
* Plain vite, no framework — this is a complement to the rig, not part of it,
* and it should stay small enough that nobody has to adopt a stack to read it.
*
* Laid out like soleprint's templated vein pages, because it does the same job:
* name each component, list what it exposes, show what comes back. Tool chrome
* and output are styled apart on purpose (see style.css) — that separation is
* what tells you whether you are reading the tool or its result.
*
* bundle.json is fetched at runtime rather than imported, so the same built app
* serves whatever rig it was copied into. Editing the ConfigMap changes the page
* without rebuilding.
*/
const esc = (s) =>
String(s).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
const tag = (text, on = false) =>
`<span class="tag${on ? " on" : ""}">${esc(text)}</span>`;
/* Tool chrome: one bordered card per component. */
function components(list, activeKey) {
if (!list?.length)
return `<div class="component"><p>nothing listed</p></div>`;
return list
.map((it) => {
const tags = [
it.standalone ? tag("standalone") : "",
it.state ? tag(it.state) : "",
activeKey && it[activeKey] ? tag("active", true) : "",
].join("");
return `<div class="component">
<h4>${esc(it.name ?? "?")} ${tags}</h4>
<p>${esc(it.summary ?? "")}</p>
</div>`;
})
.join("");
}
/* Endpoint rows: path on the left, what it returns on the right. */
function endpoints(list) {
return list
.map(
(e) => `<li><code>${esc(e.path)}</code>
<span class="desc">${esc(e.desc)}</span></li>`
)
.join("");
}
/* Output: what the endpoint above actually returns, so the page demonstrates
itself rather than describing what a demonstration would look like. */
function example(bundle) {
const sample = {
bundle: bundle.bundle?.name,
tools: (bundle.tools ?? []).map((t) => t.name),
rigs: (bundle.rigs ?? []).map((r) => r.name),
};
return `<pre class="output">${esc(JSON.stringify(sample, null, 2))}</pre>`;
}
/* Cluster state, when there is any to show.
*
* Fetched separately and allowed to fail: the bundle listing is the point, and a
* rig with no cluster reachable is a normal state, not an error. Renders nothing
* at all when absent. When the payload says `mocked`, say so loudly. */
function clusterSection(c) {
if (!c) return "";
const m = c.cluster ?? {};
const meta = [m.context, m.k8s, m.profile && `profile ${m.profile}`,
m.nodes && `${m.nodes} node${m.nodes > 1 ? "s" : ""}`]
.filter(Boolean).join(" · ");
return `
<h2>Cluster${c.mocked ? " (mocked)" : ""}</h2>
${c.mocked ? `<p class="mock">mocked — no cluster was queried; these are canned values</p>` : ""}
${meta ? `<p class="tagline">${esc(meta)}</p>` : ""}
<div class="components">${components(c.workloads)}</div>
<h2>Services</h2>
<div class="components">${components(c.services)}</div>`;
}
function render(b, name, cluster) {
const meta = b.bundle ?? {};
const next = (b.next ?? []).map((n) => `<li>${esc(n)}</li>`).join("");
return `
<h1><span class="ok">IT WORKS</span> — ${esc(name || meta.name || "rig")}</h1>
<p class="tagline">${esc(meta.description ?? "")}</p>
<h2>Tools (${b.tools?.length ?? 0})</h2>
<div class="components">${components(b.tools)}</div>
<h2>Rigs (${b.rigs?.length ?? 0})</h2>
<div class="components">${components(b.rigs, "active")}</div>
<h2>Endpoints</h2>
<ul class="endpoints">${endpoints([
{ path: "/", desc: "this page" },
{ path: "/bundle.json", desc: "the manifest it renders" },
])}</ul>
<h2>Example — GET /bundle.json</h2>
${example(b)}
${clusterSection(cluster)}
${next ? `<div class="next"><ul>${next}</ul></div>` : ""}`;
}
const app = document.getElementById("app");
const json = (path, required) =>
fetch(path)
.then((r) => {
if (r.ok) return r.json();
if (required) throw new Error(`${path} -> HTTP ${r.status}`);
return null; // optional: absent is a normal state, not an error
})
.catch((err) => {
if (required) throw err;
return null;
});
Promise.all([json("/bundle.json", true), json("/cluster.mock.json", false)])
// RIG_NAME is injected by vite from the pod env, so two rigs sharing a
// cluster are distinguishable even if a copied bundle.json kept its old name.
.then(([b, cluster]) => {
app.innerHTML = render(b, import.meta.env.VITE_RIG_NAME, cluster);
})
.catch((err) => {
app.innerHTML = `<h1 class="err">bundle unavailable</h1>
<p class="tagline">${esc(err.message)}</p>`;
});

View File

@@ -1,139 +0,0 @@
/* Minimal and self-contained — no framework dependency.
*
* The visual language follows soleprint's templated vein pages, because this
* page does the same job: say what a component is, list what it exposes, and
* show what comes back. Two treatments, deliberately distinct:
*
* TOOL CHROME bordered cards on the darker background, accent-coloured
* titles, endpoint rows separated by rules.
* OUTPUT a lighter raised block, monospace, pre-wrap and selectable —
* it is data, not furniture, and should read as a payload.
*
* Keeping them apart matters more than either looks: it is what tells you at a
* glance whether you are reading the tool or the thing it produced. */
:root {
--bg: #0d0d0f;
--surface: #16161a;
--surface-raised: #1e1e24;
--border: #2e2e38;
--border-strong: #3d3d4a;
--text: #e8e8f0;
--muted: #8888a0;
--accent: #3ecf8e;
--accent-dim: #f5a623;
--mono: "JetBrains Mono", "Cascadia Mono", Consolas, ui-monospace, monospace;
}
body {
margin: 0;
padding: 2.5rem 1.5rem;
background: var(--bg);
color: var(--text);
font: 14px/1.6 var(--mono);
}
main { max-width: 56rem; margin: 0 auto; }
h1 { margin: 0; font-size: 1.6rem; letter-spacing: 0.02em; }
h1 .ok { color: var(--accent); }
h1.err { color: #f06565; }
.tagline { color: var(--muted); margin: 0.35rem 0 2.25rem; }
h2 {
font-size: 0.75rem;
text-transform: uppercase;
letter-spacing: 0.12em;
color: var(--muted);
margin: 2.25rem 0 0.75rem;
font-weight: 600;
}
/* ── tool chrome ────────────────────────────────────────────────────────── */
.components {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 0.75rem;
}
.component {
background: var(--bg);
border: 1px solid var(--border-strong);
border-radius: 8px;
padding: 0.75rem;
}
.component h4 {
margin: 0 0 0.25rem;
font-size: 0.95rem;
color: var(--accent);
}
.component p {
margin: 0;
font-size: 0.85rem;
color: var(--muted);
}
.endpoints { list-style: none; margin: 0; padding: 0; }
.endpoints li {
display: flex;
justify-content: space-between;
flex-wrap: wrap;
gap: 0.5rem;
padding: 0.6rem 0;
border-bottom: 1px solid var(--border-strong);
}
.endpoints li:last-child { border-bottom: none; }
.endpoints code {
background: var(--surface);
color: var(--accent);
padding: 0.25rem 0.5rem;
border-radius: 4px;
}
.endpoints .desc { color: var(--muted); font-size: 0.9rem; }
.tag {
font-size: 0.7rem;
padding: 0.1rem 0.45rem;
border-radius: 3px;
background: var(--border);
color: var(--muted);
white-space: nowrap;
}
.tag.on { background: var(--accent); color: var(--bg); }
/* ── output ─────────────────────────────────────────────────────────────── */
.output {
background: var(--surface);
border: 1px solid var(--border);
border-radius: 8px;
padding: 1rem;
font-size: 0.85rem;
white-space: pre-wrap;
word-break: break-word;
user-select: text;
color: var(--text);
margin: 0;
}
.output .k { color: var(--accent); }
/* Mocked-data banner. Loud on purpose: it only appears when the payload is
canned, and a demo that looks live but is not is worse than one that says so. */
.mock {
margin: 0 0 0.75rem;
padding: 0.4rem 0.75rem;
border: 1px dashed var(--accent-dim);
border-radius: 6px;
color: var(--accent-dim);
font-size: 0.8rem;
}
.next {
color: #555568;
font-size: 0.8rem;
margin-top: 2.5rem;
border-top: 1px solid var(--border);
padding-top: 1rem;
}
.next ul { margin: 0; padding: 0 0 0 1.1rem; }
.next li { list-style: disc; padding: 0.15rem 0; }

View File

@@ -1,9 +0,0 @@
import { defineConfig } from "vite";
/* Serves on 0.0.0.0 so the pod is reachable through the Service, and allows any
* Host header because the address is assigned at runtime (MetalLB locally, a
* cloud load balancer on EKS) and is never known at build time. */
export default defineConfig({
server: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
preview: { host: "0.0.0.0", port: 5173, strictPort: true, allowedHosts: true },
});

37
rig/standalone/README.md Normal file
View File

@@ -0,0 +1,37 @@
# standalone — single files for a machine the full rig is not going to
Each script here does one of rig's jobs without the rest of the tree. Copy one
file onto a machine, run it, read the output. Nothing to clone, nothing to
install first.
| file | does | full-rig equivalent |
| --- | --- | --- |
| `rigdeps.sh` | installs kind, kubectl, tilt, ctlptl and jq at rig's pins, checksum-verified, no sudo | `make deps` (`ctrl/deps.sh`) |
| `rigmini.sh` | reports how much memory the machine *advertises* and what caps it; `push` measures what it will actually *survive* | `make mem`, and the memory section of `make check` |
**These are transitional.** Where the full rig is installed, use its own
targets instead; they read `ctrl/versions.env` and the profile, which these
cannot.
## Why single files
`rigdeps.sh` carries its pins inline, because `ctrl/versions.env` is not on the
machine it is for. That makes two copies of the same versions and checksums.
`make pins` compares them and fails on any difference — `ctrl/versions.env` is
the source of truth.
`rigmini.sh` exists because on a container or managed workspace `/proc/meminfo`
reports the *host's* memory while a cgroup cap kills processes at a fraction of
it. `status` reads the caps; `push` allocates until something stops it.
## Use
```bash
bash rigdeps.sh detect # report, change nothing
bash rigdeps.sh install dev # install into ~/.local/bin
bash rigmini.sh status # advertised memory and caps; safe
bash rigmini.sh push # allocates until it stops — not on a machine you need
```
`rigmini.sh push` deliberately consumes memory. Run `status` first, and only run
`push` somewhere it is acceptable for other processes to be squeezed.

574
rig/standalone/rigdeps.sh Executable file
View File

@@ -0,0 +1,574 @@
#!/usr/bin/env bash
# Put kind, tilt and kubectl on a machine that has none of them.
#
# The single file companion to rigmini.sh, for the same reason: rig installs its
# toolchain from ctrl/deps.sh reading ctrl/versions.env, and neither of those is
# going to a fresh AWS WorkSpace. The pins live inline here instead.
#
# What it will not do, deliberately:
#
# * no sudo, no apt, no yum. It writes ONLY into $OUT_BIN (default
# ~/.local/bin). Everything needing root — installing Docker, joining the
# docker group, raising inotify limits — is REPORTED for you to decide on.
# That is what makes it safe to run on a machine that already works.
# * no unverified download. Every artifact is checked against a SHA256 taken
# from the publisher's own release list. A mismatch aborts.
# * no guessing at another architecture. See ARCHITECTURE below.
#
# Two tiers, because "install the toolchain" is not one decision:
#
# core kubectl, jq — talk to a cluster someone else runs. Nothing that
# creates one. The right answer on a managed or corporate machine.
# dev core plus kind, tilt and ctlptl — build clusters and hot-reload
# into them. The default, and what you want on a workspace of your own.
#
# Usage:
# rigdeps.sh detect report the host, change nothing
# rigdeps.sh list the pinned versions and where they come from
# rigdeps.sh install [core|dev] detect, download, verify, install, report
# rigdeps.sh fetch [core|dev] [--to DIR] download + verify only
# rigdeps.sh verify run what is installed and see if it works
set -euo pipefail
OUT_BIN="${OUT_BIN:-$HOME/.local/bin}"
# ── the pinned toolchain ───────────────────────────────────────────────────
#
# ARCHITECTURE. These checksums are the upstream-published SHA256 of the
# **linux/amd64** artifact and of nothing else. An arm64 WorkSpace bundle needs
# a different binary with a different checksum, and this script refuses rather
# than reusing these — a checksum that is merely plausible is worse than none,
# because it turns a verified download into a ceremony.
#
# To bump a version, or to add arm64: take the checksum from the release's own
# published list, never from a download you did.
#
# curl -sSL https://github.com/<org>/<repo>/releases/download/<tag>/checksums.txt
#
# kubectl publishes its own instead, at <KUBECTL_URL>.sha256.
KIND_VERSION=v0.32.0
KIND_SHA256=50030de23cf40a18505f20426f6a8506bedf13c6e509244bd1fa9463721b0f54
KIND_URL="https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64"
KUBECTL_VERSION=v1.36.3
KUBECTL_SHA256=ebbd080e7c2e275093b55915722043257eb24004363e20acb3c4d71919f88336
KUBECTL_URL="https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl"
TILT_VERSION=0.37.6
TILT_SHA256=e9672b8a18d43501f35dcfe98465969a7db0e436b36cf0c50c7e6f8d40de5fe6
TILT_URL="https://github.com/tilt-dev/tilt/releases/download/v${TILT_VERSION}/tilt.${TILT_VERSION}.linux.x86_64.tar.gz"
# ctlptl creates a kind cluster WITH a local registry wired in, which is what
# keeps images off docker.io — an unqualified image name resolves to
# docker.io/library/<name>, and there is nothing structural stopping a push there.
CTLPTL_VERSION=0.9.4
CTLPTL_SHA256=c63a1ec28e60bc3faf6becb76f53355c5cf5e0143dafdd27ad85db5584fa6b1e
CTLPTL_URL="https://github.com/tilt-dev/ctlptl/releases/download/v${CTLPTL_VERSION}/ctlptl.${CTLPTL_VERSION}.linux.x86_64.tar.gz"
# Upstream's static build. Debian's jq is linked against libjq/libonig, which is
# fine on Debian and not portable anywhere else.
JQ_VERSION=1.8.2
JQ_SHA256=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f
JQ_URL="https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-amd64"
CORE_TOOLS="kubectl jq"
DEV_TOOLS="kind tilt ctlptl"
# No helm: every rig addon installs with `kubectl apply -f`, so nothing has ever
# invoked it. Add it the day something actually needs a chart.
# Collected as we go, printed by report_manual() at the very end. Anything that
# needs root or a decision lands here instead of being done.
MANUAL=()
# ── platform ───────────────────────────────────────────────────────────────
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
# the tooling. Detectable, so name it instead.
require_linux() {
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
cat >&2 <<'EOF'
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
wsl --install
That enables Windows features and needs a reboot, so it is not something this
script will do for you. Afterwards, open the Linux shell it installs and run
this from there.
EOF
exit 1 ;;
Linux) ;;
*) echo "$(uname -s) is not Linux. These are linux binaries; nothing here" >&2
echo "would run even if it downloaded." >&2
exit 1 ;;
esac
}
arch() {
case "$(uname -m)" in
x86_64|amd64) echo amd64 ;;
aarch64|arm64) echo arm64 ;;
*) uname -m ;;
esac
}
# The pins above are amd64. Rather than download something that cannot execute
# and let it fail as "cannot execute binary file: Exec format error", say so
# here and hand over the commands that produce the right checksums.
require_amd64() {
local a; a=$(arch)
[ "$a" = "amd64" ] && return 0
cat >&2 <<EOF
This machine is ${a} ($(uname -m)); every pin in this script is linux/amd64.
Nothing here would run, so it does not download. To make an ${a} version, the
URLs need the ${a} artifact and the checksums need to come from each project's
own published list — not from these values, and not from a download you did:
curl -sSL https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/checksums.txt
curl -sSL https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${a}/kubectl.sha256
curl -sSL https://github.com/tilt-dev/tilt/releases/download/v${TILT_VERSION}/checksums.txt
curl -sSL https://github.com/tilt-dev/ctlptl/releases/download/v${CTLPTL_VERSION}/checksums.txt
curl -sSL https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/sha256sum.txt
Edit the pinned block at the top of this file with what those print.
EOF
exit 1
}
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
# ── the tools this script itself needs ─────────────────────────────────────
# A fresh minimal image may genuinely have neither curl nor wget. Find out once,
# up front, rather than half way through the first download.
DL=""
pick_downloader() {
if command -v curl >/dev/null 2>&1; then DL=curl
elif command -v wget >/dev/null 2>&1; then DL=wget
else
echo "neither curl nor wget is installed, so nothing can be downloaded." >&2
echo "Install one first: $(pkg_install_cmd curl)" >&2
exit 1
fi
}
download() {
local url="$1" out="$2"
case "$DL" in
curl) curl -fsSL --retry 3 -o "$out" "$url" ;;
wget) wget -q --tries=3 -O "$out" "$url" ;;
esac
}
# sha256sum is coreutils; shasum is the perl one that turns up on stripped
# images. Verification is not optional, so if neither exists that is fatal.
SHA=""
pick_sha() {
if command -v sha256sum >/dev/null 2>&1; then SHA=sha256sum
elif command -v shasum >/dev/null 2>&1; then SHA="shasum -a 256"
else
echo "no sha256sum and no shasum — downloads could not be verified." >&2
echo "Refusing to install unverified binaries." >&2
exit 1
fi
}
# ── package manager, for the instructions only ─────────────────────────────
# This never runs a package manager. It names one so the reported action is
# something you can paste, on the distro you are actually on — an apt line on
# Amazon Linux 2 is a wrong answer dressed up as help.
pkg_install_cmd() {
local pkg="$1"
if command -v apt-get >/dev/null 2>&1; then echo "sudo apt-get update && sudo apt-get install -y $pkg"
elif command -v dnf >/dev/null 2>&1; then echo "sudo dnf install -y $pkg"
elif command -v yum >/dev/null 2>&1; then echo "sudo yum install -y $pkg"
elif command -v zypper >/dev/null 2>&1; then echo "sudo zypper install -y $pkg"
elif command -v apk >/dev/null 2>&1; then echo "sudo apk add $pkg"
else echo "install '$pkg' with this system's package manager"
fi
}
docker_pkg() {
# Debian and Ubuntu call it docker.io; the RPM distros call it docker.
if command -v apt-get >/dev/null 2>&1; then echo docker.io; else echo docker; fi
}
# ── detect ─────────────────────────────────────────────────────────────────
detect() {
echo "host"
echo " kernel $(uname -r)"
echo " arch $(arch) ($(uname -m))"
[ -r /etc/os-release ] && \
echo " distro $(sed -n 's/^PRETTY_NAME="\(.*\)"/\1/p' /etc/os-release)"
if is_wsl; then echo " platform WSL"; else echo " platform native linux"; fi
local total_kb avail_kb
total_kb=$(awk '/^MemTotal:/{print $2}' /proc/meminfo)
avail_kb=$(awk '/^MemAvailable:/{print $2}' /proc/meminfo)
printf " memory %d GB total, %d GB available\n" \
$((total_kb / 1024 / 1024)) $((avail_kb / 1024 / 1024))
if [ $((avail_kb / 1024 / 1024)) -lt 4 ]; then
echo " ! under 4 GB available — a cluster will struggle here."
echo " rigmini.sh says how much this box will actually give you."
fi
echo " install to $OUT_BIN"
detect_libc
detect_prereqs
detect_docker
detect_inotify
return 0
}
# tilt is the one binary here that needs a recent glibc. MEASURED, not guessed:
# tilt 0.37.6 on Amazon Linux 2 (glibc 2.26) fails with
#
# /lib64/libc.so.6: version `GLIBC_2.34' not found (required by .../tilt)
#
# which names a symbol rather than the problem. Amazon Linux 2 is a stock
# WorkSpaces bundle, so this is the likely case, not an exotic one. Report the
# version now; `verify` catches the actual failure after installing.
detect_libc() {
local v=""
if command -v ldd >/dev/null 2>&1; then
v=$(ldd --version 2>/dev/null | head -1 | grep -oE '[0-9]+\.[0-9]+$' || true)
fi
if [ -z "$v" ]; then
echo " libc unknown (no ldd) — 'verify' is the real test"
return 0
fi
echo " libc glibc $v"
if [ "$(printf '%s\n2.34\n' "$v" | sort -V | head -1)" != "2.34" ]; then
echo " ! older than glibc 2.34, which tilt needs. kubectl, kind, jq and"
echo " ctlptl are static or libc-only and work here; tilt will not start."
echo " Install the core tier, or run tilt from a container."
fi
return 0
}
# What this script needs to do its own job. Reported here so `detect` answers
# "will install work?" instead of leaving you to find out one download in.
# Amazon Linux 2 ships without tar, which is exactly the surprise this catches.
detect_prereqs() {
local missing=""
if command -v curl >/dev/null 2>&1; then echo " download curl"
elif command -v wget >/dev/null 2>&1; then echo " download wget"
else echo " ! no curl and no wget — nothing can be downloaded"; missing+=" curl"
fi
if command -v sha256sum >/dev/null 2>&1 || command -v shasum >/dev/null 2>&1; then
echo " checksums ok"
else
echo " ! no sha256sum or shasum — downloads could not be verified"
missing+=" coreutils"
fi
if command -v tar >/dev/null 2>&1 && command -v gzip >/dev/null 2>&1; then
echo " archives tar + gzip"
else
echo " ! no tar/gzip — tilt and ctlptl ship as tarballs, so the dev tier"
echo " cannot be unpacked. The core tier is two bare binaries and is fine."
missing+=" tar gzip"
fi
if [ -n "$missing" ]; then
MANUAL+=("Install what this script needs to run at all:
$(pkg_install_cmd "${missing# }")")
fi
return 0
}
detect_docker() {
# kind builds a cluster out of containers. Without a reachable daemon,
# everything here installs perfectly and then does nothing.
if ! command -v docker >/dev/null 2>&1; then
if [ -S /var/run/docker.sock ]; then
echo " docker socket present, no cli"
return 0
fi
echo " ! docker not installed — kind has nothing to build a cluster in"
MANUAL+=("Install Docker. It is the one real prerequisite, and the only
thing here that needs root:
$(pkg_install_cmd "$(docker_pkg)")
sudo systemctl enable --now docker
sudo usermod -aG docker \"\$USER\"
then log out and back in, so the new group applies to your shell.")
return 0
fi
if docker info >/dev/null 2>&1; then
echo " docker $(docker version --format '{{.Server.Version}}' 2>/dev/null)"
local n
n=$(docker ps --filter "label=io.x-k8s.kind.cluster" --format '{{.Names}}' 2>/dev/null | wc -l)
# Must be an `if`, not `[ ] && echo`: as the last statement here the
# latter returns 1 when the count is zero, and `set -e` kills the
# caller. That is the fresh-machine case, where it does most harm.
if [ "$n" -gt 0 ]; then
echo " - $n kind node container(s) already running"
fi
else
echo " ! docker cli present but the daemon is unreachable"
MANUAL+=("Start Docker, or add yourself to the docker group:
sudo systemctl enable --now docker
sudo usermod -aG docker \"\$USER\" # then log out and back in")
fi
return 0
}
# kind and tilt both watch large trees. Distro defaults are far too low and the
# failure mode is silent: tilt simply stops noticing that files changed.
detect_inotify() {
local w i
w=$(cat /proc/sys/fs/inotify/max_user_watches 2>/dev/null || echo 0)
i=$(cat /proc/sys/fs/inotify/max_user_instances 2>/dev/null || echo 0)
echo " inotify watches=$w instances=$i"
if [ "$w" -lt 524288 ] || [ "$i" -lt 512 ]; then
echo " ! low — tilt will silently stop seeing file changes"
MANUAL+=("Raise the inotify limits (needs root):
echo -e 'fs.inotify.max_user_watches=524288\\nfs.inotify.max_user_instances=512' \\
| sudo tee /etc/sysctl.d/99-rig.conf
sudo sysctl --system")
fi
return 0
}
# ── fetch ──────────────────────────────────────────────────────────────────
verify_sha() {
local file="$1" want="$2" name="$3" got
got=$($SHA "$file" | awk '{print $1}')
if [ "$got" != "$want" ]; then
echo >&2
echo "CHECKSUM MISMATCH for $name — not installing it." >&2
echo " expected $want" >&2
echo " got $got" >&2
echo >&2
echo "Either the pin in this script is stale, or what arrived is not what" >&2
echo "the publisher released. Neither is worth guessing about." >&2
rm -f "$file"
exit 1
fi
}
# fetch_bin <name> <url> <sha256> <dest-dir> — a bare binary
fetch_bin() {
local name="$1" url="$2" sha="$3" dest="$4"
local tmp="$dest/.$name.tmp"
printf ' %-8s ' "$name"
download "$url" "$tmp"
verify_sha "$tmp" "$sha" "$name"
mv "$tmp" "$dest/$name"
chmod +x "$dest/$name"
echo "ok"
}
# fetch_tgz <name> <url> <sha256> <dest-dir> <path-inside> <strip>
# Archive layouts differ, so the caller says which. tilt and ctlptl both ship
# the binary at the archive root, hence strip=0.
fetch_tgz() {
local name="$1" url="$2" sha="$3" dest="$4" inner="$5" strip="$6"
local tmp="$dest/.$name.tgz"
printf ' %-8s ' "$name"
download "$url" "$tmp"
verify_sha "$tmp" "$sha" "$name"
# --no-same-owner: some archives ship as uid 1001, and extracting as root
# would otherwise restore an owner that is not you.
tar -xzf "$tmp" -C "$dest" --strip-components="$strip" --no-same-owner "$inner"
rm -f "$tmp"
chmod +x "$dest/$name"
echo "ok"
}
fetch() {
local dest="$OUT_BIN" tier="dev"
while [ $# -gt 0 ]; do
case "$1" in
--to) dest="${2:?--to needs a directory}"; shift 2 ;;
core|dev) tier="$1"; shift ;;
*) echo "unknown argument: $1" >&2; exit 1 ;;
esac
done
mkdir -p "$dest"
if ! command -v tar >/dev/null 2>&1 && [ "$tier" = "dev" ]; then
echo "tar is missing, and tilt and ctlptl ship as tarballs." >&2
echo " $(pkg_install_cmd tar)" >&2
echo "Or install the core tier, which is two bare binaries: $0 install core" >&2
exit 1
fi
echo "fetching '$tier' into $dest (verifying every checksum)"
fetch_bin kubectl "$KUBECTL_URL" "$KUBECTL_SHA256" "$dest"
fetch_bin jq "$JQ_URL" "$JQ_SHA256" "$dest"
if [ "$tier" = "dev" ]; then
fetch_bin kind "$KIND_URL" "$KIND_SHA256" "$dest"
fetch_tgz tilt "$TILT_URL" "$TILT_SHA256" "$dest" tilt 0
fetch_tgz ctlptl "$CTLPTL_URL" "$CTLPTL_SHA256" "$dest" ctlptl 0
fi
return 0
}
# ── verify ─────────────────────────────────────────────────────────────────
tier_tools() { [ "$1" = "core" ] && echo "$CORE_TOOLS" || echo "$CORE_TOOLS $DEV_TOOLS"; }
# Downloading a verified binary proves it is the right file, not that this
# machine can run it. On an old distro tilt fails here, with a linker error
# about a missing symbol, and finding that out now beats finding out during a
# first cluster build.
verify_tools() {
local tier="${1:-dev}" b bin out rc broke=0
echo "checking that each one actually runs"
for b in $(tier_tools "$tier"); do
bin="$OUT_BIN/$b"
if [ ! -x "$bin" ]; then
printf ' %-8s not installed\n' "$b"
continue
fi
rc=0
case "$b" in
kubectl) out=$("$bin" version --client 2>&1 | head -1) || rc=$? ;;
jq) out=$("$bin" --version 2>&1 | head -1) || rc=$? ;;
*) out=$("$bin" version 2>&1 | head -1) || rc=$? ;;
esac
if [ "$rc" -eq 0 ]; then
printf ' %-8s %s\n' "$b" "$out"
else
printf ' ! %-6s does not run here: %s\n' "$b" "$out"
broke=1
fi
done
if [ "$broke" -eq 1 ]; then
echo
echo " A binary that downloads and verifies but will not start is almost"
echo " always this distro's libc being older than the release needs."
echo " 'detect' prints the glibc version. The core tier (kubectl + jq)"
echo " has no such dependency and will work regardless."
fi
return 0
}
# ── install ────────────────────────────────────────────────────────────────
# Installing into a directory early in PATH silently replaces whatever the
# machine was already using, which on a shared or corporate machine can break
# unrelated work — kubectl more than one minor away from its cluster is the
# common one. Say so; never decide it.
warn_shadowing() {
local b existing shadowed="" tier="${1:-dev}"
case ":${PATH}:" in
*":$OUT_BIN:"*) ;;
*) return 0 ;; # not on PATH, so nothing is being shadowed yet
esac
for b in $(tier_tools "$tier"); do
[ -x "$OUT_BIN/$b" ] || continue
existing=$(PATH=$(echo "$PATH" | tr ':' '\n' | grep -vx "$OUT_BIN" | paste -sd:) \
command -v "$b" 2>/dev/null || true)
[ -n "$existing" ] || continue
[ "$existing" = "$OUT_BIN/$b" ] && continue
shadowed+=" $b $existing"$'\n'
done
[ -n "$shadowed" ] || return 0
echo
echo " ! these were already installed elsewhere and are now shadowed:"
printf '%s' "$shadowed"
MANUAL+=("Decide which toolchain wins. To keep the previous one:
rm -f $(for b in $(tier_tools "$tier"); do printf '%s ' "$OUT_BIN/$b"; done)
Or install somewhere private instead:
OUT_BIN=\$PWD/bin $0 install")
return 0
}
report_manual() {
echo
if [ ${#MANUAL[@]} -eq 0 ]; then
echo "nothing left to do by hand."
return 0
fi
echo "host actions this cannot perform (${#MANUAL[@]}):"
echo
local n=1 m
for m in "${MANUAL[@]}"; do
echo " $n. $m"
echo
n=$((n + 1))
done
return 0
}
install() {
local tier="${1:-dev}"
detect
echo
fetch "$tier"
echo
verify_tools "$tier"
warn_shadowing "$tier"
if [ "$tier" = "core" ]; then
echo
echo " core tier: no kind, tilt or ctlptl. '$0 install dev' adds them."
fi
case ":${PATH}:" in
*":$OUT_BIN:"*) ;;
*) MANUAL+=("Put the toolchain on your PATH — add to ~/.bashrc:
export PATH=\"${OUT_BIN}:\$PATH\"
then: source ~/.bashrc") ;;
esac
report_manual
if [ "$tier" = "dev" ]; then
echo "Once Docker is reachable and this is on PATH:"
echo
echo " kind create cluster --name scratch"
echo " kubectl cluster-info --context kind-scratch"
echo " kind delete cluster --name scratch"
echo
echo "That round trip is the real test that this machine can host a rig."
fi
return 0
}
list() {
echo "pinned, linux/amd64 only:"
printf ' %-8s %s\n' kubectl "$KUBECTL_VERSION"
printf ' %-8s %s\n' jq "$JQ_VERSION"
printf ' %-8s %s\n' kind "$KIND_VERSION"
printf ' %-8s %s\n' tilt "$TILT_VERSION"
printf ' %-8s %s\n' ctlptl "$CTLPTL_VERSION"
echo
echo " core = $CORE_TOOLS"
echo " dev = $CORE_TOOLS $DEV_TOOLS"
echo
echo "Checksums are pinned in the block at the top of this file. To bump one,"
echo "take the new checksum from the publisher's own release list — the header"
echo "comment has the exact commands."
return 0
}
# ── main ───────────────────────────────────────────────────────────────────
require_linux
case "${1:-install}" in
detect) detect; report_manual ;;
list) list ;;
verify) verify_tools "${2:-dev}" ;;
fetch) shift; require_amd64; pick_downloader; pick_sha; fetch "$@" ;;
install) shift; require_amd64; pick_downloader; pick_sha; install "${1:-dev}" ;;
*) echo "usage: $0 [detect|list|install|fetch|verify]" >&2
echo " install [core|dev] (default dev)" >&2
echo " fetch [core|dev] [--to DIR]" >&2
echo " OUT_BIN=<dir> overrides the install directory" >&2
exit 1 ;;
esac

635
rig/standalone/rigmini.sh Executable file
View File

@@ -0,0 +1,635 @@
#!/usr/bin/env bash
# How much memory this box will actually give you before something dies.
#
# rig answers this for a machine it is installed on. This is the single file
# version, for a machine rig is not going to: paste it onto a fresh AWS
# WorkSpace, an EC2 box or a container, run it, and get the same numbers in the
# same order so two machines can be read side by side.
#
# There are two numbers and they are rarely the same. `status` reports what the
# machine ADVERTISES and what is quietly capping it. `push` finds what it will
# SURVIVE, by allocating until it stops.
#
# The gap between them is the whole reason this exists. Under WSL the cap lives
# in .wslconfig; in a container or a managed workspace it is a cgroup limit, and
# there /proc/meminfo reports the HOST's memory while the kernel kills you at a
# fraction of it. A script that only read MemTotal would confidently report 32 GB
# on a box that OOMs at 2.
#
# Reports and instructs. It never raises a limit, frees anything, writes a
# config or installs a package — on a machine you are still evaluating, a probe
# that changes what it is measuring is worse than no probe.
#
# Usage:
# rigmini.sh status what it has, what caps it
# rigmini.sh push [--to GB] [--to-oom] climb until it stops
# rigmini.sh all [--budget GB] both, then the verdict
set -euo pipefail
# ── defaults ───────────────────────────────────────────────────────────────
STEP_MB=0 # per allocation; 0 means scale it to the ceiling. See push().
STEP_EXPLICIT=no # whether --step was given, which turns the scaling off.
TO_MB="" # --to: stop here regardless. Empty means no hard cap.
TO_OOM=no # --to-oom: opt in to running until the kernel intervenes.
BUDGET_GB=6 # what the rig data profile is assumed to want; see all().
BUDGET_EXPLICIT=no # whether --budget was given, which retires the guess below.
# ── platform ───────────────────────────────────────────────────────────────
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
# the tooling. Detectable, so name it instead.
require_linux() {
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
cat >&2 <<'EOF'
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
wsl --install
That enables Windows features and needs a reboot, so it is not something this
script will do for you. Afterwards, open the Linux shell it installs and run
this from there.
EOF
exit 1 ;;
esac
# Everything below reads /proc. Without it there is nothing to measure, and
# failing here beats printing a page of empty fields.
if [ ! -r /proc/meminfo ]; then
echo "no readable /proc/meminfo — this needs a Linux kernel." >&2
echo "On macOS or a BSD none of the numbers below exist." >&2
exit 1
fi
}
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
is_container() {
[ -f /.dockerenv ] && return 0
grep -qE '(docker|containerd|kubepods|lxc|podman)' /proc/1/cgroup 2>/dev/null
}
platform() {
if is_wsl; then echo WSL
elif is_container; then echo container
else echo "native linux"
fi
}
# ── reading memory ─────────────────────────────────────────────────────────
mb() { echo $(( $(awk "/^$1:/{print \$2}" /proc/meminfo) / 1024 )); }
# MemAvailable arrived in kernel 3.14. Older kernels — and they turn up on
# corporate images — need the estimate it replaced, which is worse but not wrong.
avail_meminfo_mb() {
if grep -q '^MemAvailable:' /proc/meminfo; then
mb MemAvailable
else
awk '/^(MemFree|Buffers|Cached):/{t+=$2} END{print int(t/1024)}' /proc/meminfo
fi
}
# Where a cgroup records this cgroup's own limit and usage. Set once by
# find_cgroup, because every later reading needs both and hunting for the files
# on each call would be the slow part of the poll loop.
CG_MAX_FILE=""
CG_CUR_FILE=""
CG_VERSION=""
find_cgroup() {
local rel
# Inside a container the cgroup namespace makes the top of the tree BE the
# container's own cgroup, so the unqualified path is already the right one.
# On a host it is the root cgroup, which is never limited — hence the second
# attempt via /proc/self/cgroup, which names the slice this shell is in.
if [ -r /sys/fs/cgroup/memory.max ]; then
CG_VERSION=v2
CG_MAX_FILE=/sys/fs/cgroup/memory.max
CG_CUR_FILE=/sys/fs/cgroup/memory.current
elif [ -r /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
CG_VERSION=v1
CG_MAX_FILE=/sys/fs/cgroup/memory/memory.limit_in_bytes
CG_CUR_FILE=/sys/fs/cgroup/memory/memory.usage_in_bytes
fi
rel=$(awk -F: '$1=="0"{print $3; exit}' /proc/self/cgroup 2>/dev/null || true)
if [ -n "$rel" ] && [ "$rel" != "/" ] && [ -r "/sys/fs/cgroup${rel}/memory.max" ]; then
CG_VERSION=v2
CG_MAX_FILE="/sys/fs/cgroup${rel}/memory.max"
CG_CUR_FILE="/sys/fs/cgroup${rel}/memory.current"
return 0
fi
rel=$(awk -F: '$2 ~ /(^|,)memory(,|$)/{print $3; exit}' /proc/self/cgroup 2>/dev/null || true)
if [ -n "$rel" ] && [ "$rel" != "/" ] \
&& [ -r "/sys/fs/cgroup/memory${rel}/memory.limit_in_bytes" ]; then
CG_VERSION=v1
CG_MAX_FILE="/sys/fs/cgroup/memory${rel}/memory.limit_in_bytes"
CG_CUR_FILE="/sys/fs/cgroup/memory${rel}/memory.usage_in_bytes"
fi
return 0
}
# The cap in MB, or "" when there is none worth reporting. v2 spells unlimited
# "max"; v1 spells it as a number near 2^63, which is why this compares against
# MemTotal rather than testing for a magic value — a "limit" above the machine's
# own memory is not a limit, however it is written.
cgroup_cap_mb() {
local raw cap
[ -n "$CG_MAX_FILE" ] && [ -r "$CG_MAX_FILE" ] || { echo ""; return 0; }
raw=$(cat "$CG_MAX_FILE" 2>/dev/null || echo max)
[ "$raw" = "max" ] && { echo ""; return 0; }
case "$raw" in ''|*[!0-9]*) echo ""; return 0 ;; esac
cap=$((raw / 1024 / 1024))
[ "$cap" -ge "$(mb MemTotal)" ] && { echo ""; return 0; }
echo "$cap"
}
cgroup_used_mb() {
local raw
[ -n "$CG_CUR_FILE" ] && [ -r "$CG_CUR_FILE" ] || { echo ""; return 0; }
raw=$(cat "$CG_CUR_FILE" 2>/dev/null || echo "")
case "$raw" in ''|*[!0-9]*) echo ""; return 0 ;; esac
echo $((raw / 1024 / 1024))
}
# ulimit -v is a per-process address-space cap. It stops YOU long before the box
# does, and because it is inherited from a login shell it is easy to hit without
# knowing it is set.
ulimit_v_mb() {
local v; v=$(ulimit -v 2>/dev/null || echo unlimited)
[ "$v" = "unlimited" ] && { echo ""; return 0; }
case "$v" in ''|*[!0-9]*) echo ""; return 0 ;; esac
echo $((v / 1024))
}
# The number everything else is about: the lowest of the things that can stop
# you. Printed at the end of `status` and used as the sanity bound in `push`.
effective_ceiling_mb() {
local c; c=$(mb MemTotal)
local cap; cap=$(cgroup_cap_mb)
local ul; ul=$(ulimit_v_mb)
[ -n "$cap" ] && [ "$cap" -lt "$c" ] && c="$cap"
[ -n "$ul" ] && [ "$ul" -lt "$c" ] && c="$ul"
echo "$c"
}
# How much room is left RIGHT NOW, from whichever accounting actually governs.
# In a capped container /proc/meminfo describes the host and is worse than
# useless for this — it would report tens of gigabytes free on a box that is one
# allocation from being killed.
headroom_mb() {
local cap used
cap=$(cgroup_cap_mb)
used=$(cgroup_used_mb)
if [ -n "$cap" ] && [ -n "$used" ]; then
echo $(( cap - used ))
else
avail_meminfo_mb
fi
}
# ── status ─────────────────────────────────────────────────────────────────
# /mnt/c/Users can hold several real accounts — a renamed login leaves the old
# directory behind — so picking the first alphabetically is a coin toss. Ask
# Windows, then fall back to whichever profile actually owns a config.
wslconfig_path() {
local profile winpath found
profile=$(cmd.exe /c "echo %USERPROFILE%" 2>/dev/null | tr -d "\r\n" || true)
case "$profile" in
""|*%*) ;;
*) winpath=$(wslpath -u "$profile" 2>/dev/null || true)
if [ -n "$winpath" ] && [ -d "$winpath" ]; then
echo "$winpath/.wslconfig"; return 0
fi ;;
esac
found=$(ls -d /mnt/c/Users/*/.wslconfig 2>/dev/null | head -1 || true)
[ -n "$found" ] && echo "$found"
return 0
}
hogs() {
echo " holding the most:"
ps -eo rss,comm --sort=-rss 2>/dev/null \
| awk 'NR>1 && NR<=6 {printf " %6.0f MB %s\n", $1/1024, $2}'
return 0
}
status() {
local total avail swap_total swap_free cap ul cur
echo "host"
echo " platform $(platform)"
echo " kernel $(uname -r)"
[ -r /etc/os-release ] && \
echo " distro $(sed -n 's/^PRETTY_NAME="\(.*\)"/\1/p' /etc/os-release)"
echo " cpu $(getconf _NPROCESSORS_ONLN 2>/dev/null || echo '?') online, load $(cut -d' ' -f1-3 /proc/loadavg)"
# ── the caps first, because they decide what the totals below are worth ──
echo
echo "caps"
cap=$(cgroup_cap_mb)
if [ -n "$cap" ]; then
cur=$(cgroup_used_mb)
echo " cgroup ${cap} MB (${CG_VERSION}, ${CG_CUR_FILE##*/} says ${cur:-?} MB used)"
echo " ! /proc/meminfo below describes the HOST, not this cgroup."
echo " $(mb MemTotal) MB total is not yours; ${cap} MB is."
elif [ -n "$CG_VERSION" ]; then
echo " cgroup none (${CG_VERSION} present, no memory limit set)"
else
echo " cgroup no memory controller found"
fi
ul=$(ulimit_v_mb)
if [ -n "$ul" ]; then
echo " ! ulimit -v ${ul} MB — a per-process cap, inherited from your shell"
echo " it stops this process long before the machine runs out"
else
echo " ulimit -v unlimited"
fi
# overcommit_memory=0 is the default heuristic: a large allocation is
# granted on a guess, and the reckoning arrives later as an OOM kill rather
# than as a failed malloc. It is why `push` touches every page it asks for.
local om or_
om=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null || echo '?')
or_=$(cat /proc/sys/vm/overcommit_ratio 2>/dev/null || echo '?')
case "$om" in
0) echo " overcommit 0 heuristic — allocations are granted on a guess," ;;
1) echo " overcommit 1 always — every allocation succeeds; the OOM killer is the only limit," ;;
2) echo " overcommit 2 strict (ratio ${or_}%) — allocation fails honestly instead of killing later," ;;
*) echo " overcommit ${om}" ;;
esac
[ "$om" != "?" ] && echo " so RSS is the number to trust, not what a process asked for"
# ── what it says it has ──
total=$(mb MemTotal); avail=$(avail_meminfo_mb)
swap_total=$(mb SwapTotal); swap_free=$(mb SwapFree)
echo
echo "memory"
echo " total ${total} MB"
echo " available ${avail} MB"
echo " swap ${swap_total} MB ($(( swap_total - swap_free )) MB used)"
if [ "$swap_total" -eq 0 ]; then
echo " - no swap: this box has no cushion. It goes from fine to OOM-killed"
echo " with nothing in between, which is the abrupt failure you get in a VM."
fi
# postgres puts its shared buffers in /dev/shm. Docker's default is 64 MB,
# and the resulting failure names neither shm nor the size.
if [ -d /dev/shm ]; then
local shm; shm=$(df -Pm /dev/shm 2>/dev/null | awk 'NR==2{print $2}')
if [ -n "$shm" ]; then
if [ "$shm" -le 64 ]; then
echo " ! /dev/shm ${shm} MB — postgres puts shared memory here and 64 MB"
echo " is docker's default. Raise it with --shm-size when the cabinet fails."
else
echo " /dev/shm ${shm} MB"
fi
fi
fi
echo
echo "disk"
local d
for d in / /tmp /var/lib/docker; do
[ -d "$d" ] || continue
df -Pm "$d" 2>/dev/null | awk -v p="$d" 'NR==2{printf " %-12s %s MB free of %s MB\n", p, $4, $2}'
done
# kind and Tilt both watch large trees, and the failure mode is silent:
# they simply stop noticing file changes. Cheap to report while we are here.
local w i
w=$(cat /proc/sys/fs/inotify/max_user_watches 2>/dev/null || echo 0)
i=$(cat /proc/sys/fs/inotify/max_user_instances 2>/dev/null || echo 0)
echo
echo "tooling"
echo " inotify watches=$w instances=$i"
if [ "$w" -lt 524288 ] || [ "$i" -lt 512 ]; then
echo " ! low — anything watching files will silently stop seeing changes"
fi
if ! command -v docker >/dev/null 2>&1; then
if [ -S /var/run/docker.sock ]; then
echo " docker socket present, no cli"
else
echo " docker not installed"
fi
elif docker info >/dev/null 2>&1; then
local n
n=$(docker ps -q 2>/dev/null | wc -l)
echo " docker $(docker version --format '{{.Server.Version}}' 2>/dev/null), ${n} container(s) running"
else
echo " ! docker cli present but the daemon is unreachable"
fi
# WSL keeps its cap on the Windows side, in a file this shell can read but
# not usefully apply — the change costs a full VM restart. Report it, and
# report the commonest mistake, which is editing it and not restarting.
if is_wsl; then
local cfg conf
cfg=$(wslconfig_path)
echo
echo "wsl"
if [ -z "$cfg" ]; then
echo " ! cannot tell which Windows profile owns .wslconfig"
else
echo " config $cfg"
conf=$(sed -n 's/^[[:space:]]*memory[[:space:]]*=[[:space:]]*//p' "$cfg" 2>/dev/null \
| tail -1 | tr -d '[:space:]')
if [ -n "$conf" ]; then
echo " configured $conf (booted ${total} MB)"
echo " - if those disagree the edit has not been applied."
echo " From a WINDOWS terminal: wsl --shutdown"
else
echo " configured no memory= set (WSL defaults to half the host RAM, or 8 GB,"
echo " whichever is less — which is where your Airflow ceiling comes from)"
fi
fi
fi
echo
echo "effective ceiling $(effective_ceiling_mb) MB"
echo " the lowest of MemTotal, the cgroup cap and ulimit -v. What the box"
echo " claims. 'push' measures what it will actually hand over."
[ "$avail" -lt $(( total / 5 )) ] && { echo; hogs; }
return 0
}
# ── push ───────────────────────────────────────────────────────────────────
STATE=""
CHILD=""
cleanup() {
if [ -n "$CHILD" ] && kill -0 "$CHILD" 2>/dev/null; then
kill -KILL "$CHILD" 2>/dev/null || true
wait "$CHILD" 2>/dev/null || true
fi
[ -n "$STATE" ] && rm -f "$STATE"
return 0
}
# The child allocates and stops itself; the parent only watches. That split is
# the point: under --to-oom the allocating process is expected to be killed, and
# something has to survive to say how far it got.
allocator() {
# Raise our own OOM score to the maximum so the kernel picks THIS process
# first. Raising needs no privilege (only lowering does). Without it, the
# kernel is free to choose your shell, your ssh session or dockerd — on a
# box you are still using, that is not an acceptable coin toss.
echo 1000 > "/proc/$BASHPID/oom_score_adj" 2>/dev/null || true
local arr=() held=0 i=0 rss swapped avail first_swap=0
local bytes=$((STEP_MB * 1024 * 1024))
local swap_used_start
swap_used_start=$(( $(mb SwapTotal) - $(mb SwapFree) ))
while :; do
# Written STRAIGHT INTO the array element. The obvious spelling —
# build one chunk and `arr+=("$chunk")` — costs three copies per step,
# not one: the template stays resident, expanding "$chunk" makes a
# temporary word, and the append makes the element. A 128 MB step then
# needs 384 MB transiently, and on a small box it is killed on the
# first append while reporting a third of the true ceiling.
#
# printf -v into a subscript also means every page is written, so it is
# resident rather than merely promised — the only kind of allocation
# that measures anything under heuristic overcommit.
printf -v "arr[$i]" '%*s' "$bytes" ''
i=$((i + 1)); held=$((held + STEP_MB))
rss=$(awk '/^VmRSS:/{print int($2/1024)}' "/proc/$BASHPID/status" 2>/dev/null || echo 0)
avail=$(headroom_mb)
swapped=$(( $(mb SwapTotal) - $(mb SwapFree) - swap_used_start ))
[ "$swapped" -lt 0 ] && swapped=0
printf '%8s MB held rss %7s MB headroom %7s MB swap +%s MB\n' \
"$held" "$rss" "$avail" "$swapped"
printf '%s %s %s %s\n' "$held" "$rss" "$avail" "$swapped" >> "$STATE"
# Worth calling out separately from the ceiling: this is where the box
# stops being fast and starts being unusable, which for a scheduler is
# a different and earlier problem than being killed.
if [ "$swapped" -gt 0 ] && [ "$first_swap" -eq 0 ]; then
first_swap=$held
echo " - first swap page at ${held} MB — past here it works but crawls"
echo "swapat $held" >> "$STATE"
fi
if [ -n "$TO_MB" ] && [ "$held" -ge "$TO_MB" ]; then
echo "stop reached-the-cap" >> "$STATE"; return 0
fi
if [ "$TO_OOM" = no ] && [ "$avail" -lt "$FLOOR_MB" ]; then
echo "stop floor" >> "$STATE"; return 0
fi
done
}
push() {
local total ceiling rc=0 last held rss swapat stop
total=$(mb MemTotal)
ceiling=$(effective_ceiling_mb)
# A step is worth about a sixty-fourth of the ceiling: enough resolution to
# find the edge, few enough lines to read, and small enough that the
# transient cost of one allocation never dominates a small box. A fixed
# size cannot do all three — 128 MB is fine on 16 GB and absurd on 512 MB.
if [ "$STEP_EXPLICIT" = no ]; then
STEP_MB=$(( ceiling / 64 ))
[ "$STEP_MB" -lt 4 ] && STEP_MB=4
[ "$STEP_MB" -gt 256 ] && STEP_MB=256
fi
# Stop with a cushion rather than riding it to the kill. How big a cushion
# depends on what it is protecting. Under a cgroup cap, running out kills
# only this container's own processes, so it need cover no more than the
# shell that prints the result — and a 512 MB cushion on a 1 GB box would
# halve the answer. On a host there is everything else to protect, and the
# OOM killer does not promise to pick the process that caused the problem.
if [ -n "$(cgroup_cap_mb)" ]; then FLOOR_MB=64; else FLOOR_MB=512; fi
[ $(( ceiling / 20 )) -gt "$FLOOR_MB" ] && FLOOR_MB=$(( ceiling / 20 ))
STATE=$(mktemp "${TMPDIR:-/tmp}/rigmini.XXXXXX")
trap cleanup EXIT
# INT kills the child and lets the summary below print anyway, so an
# impatient Ctrl-C still tells you how far it got — and, more importantly,
# still gives the memory back.
trap 'echo; echo " interrupted"; echo "stop interrupted" >> "$STATE"; [ -n "$CHILD" ] && kill -KILL "$CHILD" 2>/dev/null || true' INT
echo "push"
echo " step ${STEP_MB} MB per allocation, every page touched"
echo " ceiling ${ceiling} MB claimed"
if [ -n "$TO_MB" ]; then
echo " stopping at ${TO_MB} MB (--to)"
elif [ "$TO_OOM" = yes ]; then
echo " ! stopping only when the kernel stops it (--to-oom)"
echo " the allocating child is marked as the preferred OOM victim,"
echo " but nothing about an OOM kill is entirely polite. Not on a box"
echo " running anything you mind losing."
else
echo " stopping when headroom drops below ${FLOOR_MB} MB"
fi
echo
allocator &
CHILD=$!
wait "$CHILD" || rc=$?
CHILD=""
trap - INT
last=$(grep -E '^[0-9]' "$STATE" 2>/dev/null | tail -1 || true)
held=$(echo "$last" | awk '{print $1}')
rss=$(echo "$last" | awk '{print $2}')
swapat=$(awk '/^swapat/{print $2}' "$STATE" 2>/dev/null | head -1 || true)
stop=$(awk '/^stop/{print $2}' "$STATE" 2>/dev/null | head -1 || true)
echo
if [ -z "$held" ]; then
echo " ! nothing was allocated. Even one ${STEP_MB} MB chunk failed —"
echo " try a smaller --step, or check ulimit -v in 'status'."
return 1
fi
echo " reached ${rss:-$held} MB resident"
[ -n "$swapat" ] && echo " swapping from ${swapat} MB"
case "$stop" in
reached-the-cap)
echo " outcome stopped at the --to cap, not at a limit."
echo " The box held ${TO_MB} MB without complaint; there is more." ;;
floor)
echo " outcome stopped with a cushion intact, by choice."
echo " The real ceiling is higher — --to-oom finds it, at the"
echo " cost of an actual OOM kill." ;;
interrupted)
echo " outcome interrupted at ${rss:-$held} MB — where you stopped it,"
echo " not where the box did." ;;
*)
# No stop line means the child did not decide to stop: it was ended.
if [ "$rc" -ge 128 ]; then
echo " outcome the child was killed (signal $((rc - 128))) at ${rss:-$held} MB."
elif [ "$rc" -ne 0 ]; then
echo " outcome the allocation failed at ${rss:-$held} MB (exit ${rc})."
echo " bash could not get the next chunk — an honest malloc"
echo " failure rather than a kill. That is the strict-overcommit"
echo " or ulimit path."
else
echo " outcome ended at ${rss:-$held} MB."
fi
local ev
ev=$(dmesg 2>/dev/null | tail -80 | grep -iE 'oom-kill|killed process' | tail -1 || true)
if [ -n "$ev" ]; then
echo " kernel ${ev#*] }"
else
echo " - dmesg is unreadable here (dmesg_restrict, or no privilege),"
echo " so the kill cannot be confirmed from this side. The number stands."
fi ;;
esac
# The gap between the claim and the measurement is the finding — but only
# when the BOX chose where to stop. An empty $stop means the child was ended
# rather than deciding to end; anything else (--to, the floor) is a stop we
# asked for, and flagging those as short of the ceiling would put a warning
# on every deliberately small run.
local got="${rss:-$held}"
echo
if [ -z "$stop" ] && [ "$got" -lt $(( ceiling * 70 / 100 )) ]; then
echo " ! claimed ${ceiling} MB, gave up ${got} MB — under 70% of it."
echo " Something is taking the difference. 'status' names the candidates:"
echo " a cgroup cap, ulimit -v, or memory already resident."
fi
return 0
}
# ── all ────────────────────────────────────────────────────────────────────
all() {
status
echo
echo "────────────────────────────────────────────────────────────"
echo
push
local got budget_mb ceiling
budget_mb=$(( BUDGET_GB * 1024 ))
ceiling=$(effective_ceiling_mb)
got=$(grep -E '^[0-9]' "$STATE" 2>/dev/null | tail -1 | awk '{print $2}' || true)
[ -n "$got" ] || got=0
echo
echo "verdict"
echo " budget ${BUDGET_GB} GB for kind + postgres + redis + airflow"
# Only worth explaining while it is still a guess. Once --budget is given
# the number came from somewhere better than this reasoning, and repeating
# the derivation would describe a figure that is no longer in use.
if [ "$BUDGET_EXPLICIT" = no ]; then
echo " - that is 2 GB per kind node, which is rig's own figure, plus about"
echo " 4 GB for the three cabinets. THE 4 GB IS AN ESTIMATE, not something"
echo " measured. Re-run with --budget once you have watched the real thing."
fi
echo " measured ${got} MB handed over"
if [ "$got" -ge "$budget_mb" ]; then
echo " fits, with $(( got - budget_mb )) MB spare."
if [ "$got" -lt $(( budget_mb * 130 / 100 )) ]; then
echo " - under 30% spare is thin for a scheduler. Airflow's memory use"
echo " is spiky, and the spikes are what get killed."
fi
else
echo " ! short by $(( budget_mb - got )) MB."
if [ "$ceiling" -ge "$budget_mb" ]; then
echo " The box CLAIMS enough (${ceiling} MB) but did not deliver it."
echo " Free something, or read the caps section again."
else
echo " The box does not have it to give. A bigger bundle, or a smaller"
echo " profile: PROFILE=minimal drops the cabinets entirely."
fi
fi
return 0
}
# ── main ───────────────────────────────────────────────────────────────────
parse_flags() {
while [ $# -gt 0 ]; do
case "$1" in
--to) TO_MB=$(( ${2:?--to needs a value in GB} * 1024 )); shift 2 ;;
--to-mb) TO_MB="${2:?--to-mb needs a value in MB}"; shift 2 ;;
--step) STEP_MB="${2:?--step needs a value in MB}"; STEP_EXPLICIT=yes; shift 2 ;;
--to-oom) TO_OOM=yes; shift ;;
--budget) BUDGET_GB="${2:?--budget needs a value in GB}"; BUDGET_EXPLICIT=yes; shift 2 ;;
*) echo "unknown argument: $1" >&2; exit 1 ;;
esac
done
if [ "$TO_OOM" = yes ] && [ -n "$TO_MB" ]; then
echo "--to and --to-oom contradict each other: one stops early, the other" >&2
echo "refuses to stop at all. Pick one." >&2
exit 1
fi
return 0
}
require_linux
find_cgroup
cmd="${1:-status}"
[ $# -gt 0 ] && shift
case "$cmd" in
status) parse_flags "$@"; status ;;
push) parse_flags "$@"; push ;;
all) parse_flags "$@"; all ;;
*) echo "usage: $0 [status|push|all]" >&2
echo " push [--to GB] [--to-mb MB] [--step MB] [--to-oom]" >&2
echo " all [--budget GB]" >&2
exit 1 ;;
esac

4
soleprint/atlas2/docgen/.gitignore vendored Normal file
View File

@@ -0,0 +1,4 @@
# Everything this makes.
out/
__pycache__/
*.pyc

View File

@@ -0,0 +1,171 @@
# docgen — code to diagram, and to everything else the IR can feed.
#
# Derived from where this file sits, so the folder can be copied anywhere and
# renamed and still work. The logic lives in the Python, never here.
#
# make book SRC=../station the whole operation, measured at both ends
# make check prove docgen, on a tree it builds itself
# make check BOOK=out/book/x prove one book — its own level
# make ir SRC=../station extract -> out/ir.json (one step, on its own)
# make graph out/ir.json -> out/graph.svg
# make index out/ir.json -> out/index.md
# make self docgen's book of itself, then check it
# make doctor what this machine has
#
# Every target below is one step of a book and still works alone — that is the
# property the book spine exists to preserve, not to replace.
#
# The pipeline is three commands and they compose, which is the point:
#
# python3 -m docgen.extractors.python --root SRC -o ir.json
# python3 -m docgen.ops ir.json --overview -o view.json
# python3 -m docgen.emitters dot view.json -o graph.svg --theme dark
HERE := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
PKG := $(notdir $(HERE))
PARENT := $(patsubst %/,%,$(dir $(HERE)))
PY ?= python3
RUN := PYTHONPATH=$(PARENT) $(PY) -m
OUT ?= $(HERE)/out
SRC ?=
SCHEMA ?=
OPENAPI ?=
HAR ?=
STYLE ?= lucid
THEME ?=
DEPTH ?= 2
SCALE ?= 0.55
BOOK ?=
SLUG ?=
# NOT `LANG`: that is the shell's locale variable, so `?=` inherits
# en_US.UTF-8 from the environment and --lang rejects it.
READER ?= python
OVERLAY ?=
THEME_ARG := $(if $(THEME),--theme $(THEME))
SLUG_ARG := $(if $(SLUG),--slug $(SLUG))
OVER_ARG := $(if $(OVERLAY),--overlay $(OVERLAY))
.PHONY: help book check ir db code graph index site minimap explore docs view self doctor clean
help: ## List every target
@echo "docgen — static analysis of a tree, and the artifacts that fall out of it"
@echo
@grep -E '^[a-z-]+:.*?## .*$$' $(MAKEFILE_LIST) \
| awk 'BEGIN{FS=":.*?## "}{printf " \033[1m%-10s\033[0m %s\n", $$1, $$2}'
@echo
@echo " SRC=/path/to/tree what to read OUT=/path where output goes"
@echo " SCHEMA=schema.json a database instead STYLE=lucid THEME=dark|lucid"
@echo " OPENAPI=spec.yaml an API document HAR=session.har a recording"
@echo " BOOK=/path where a book goes, and which book to check"
@echo " READER=python|code ast, or tree-sitter SLUG=name what to call the book"
@echo " OVERLAY=overlay.json hand-written notebook additions, re-applied every build"
@echo " DEPTH=2 how deep to draw"
@echo
@echo " Three levels of test, by what they assert about:"
@echo " make doctor the machine. Never fails."
@echo " make check docgen. Exits 1."
@echo " make check BOOK=<dir> that book. Exits 1."
check: ## Prove docgen (or one book, with BOOK=<dir>)
@if [ -n "$(BOOK)" ]; then \
$(RUN) $(PKG).book.checks "$(BOOK)"; \
else \
$(PY) $(HERE)/selftest.py; \
fi
book: ## SRC (or SCHEMA/OPENAPI/HAR) -> one operation, measured at both ends
@test -n "$(SRC)$(SCHEMA)$(OPENAPI)$(HAR)" \
|| { echo "Error: set SRC=/path/to/tree (or SCHEMA=, OPENAPI=, HAR=)" >&2; exit 1; }
@$(RUN) $(PKG).book \
$(if $(SRC),--root "$(SRC)" --lang $(READER)) \
$(if $(SCHEMA),--schema "$(SCHEMA)") \
$(if $(OPENAPI),--openapi "$(OPENAPI)") \
$(if $(HAR),--har "$(HAR)") \
-o "$(if $(BOOK),$(BOOK),$(OUT)/book)" \
--style $(STYLE) $(THEME_ARG) $(SLUG_ARG) $(OVER_ARG)
ir: ## Extract SRC into OUT/ir.json
@test -n "$(SRC)" || { echo "Error: set SRC=/path/to/tree" >&2; exit 1; }
@mkdir -p $(OUT)
@$(RUN) $(PKG).extractors.python --root "$(SRC)" -o $(OUT)/ir.json
@$(RUN) $(PKG).ir $(OUT)/ir.json
db: ## Extract a graphgen-compatible SCHEMA into OUT/ir.json
@test -n "$(SCHEMA)" || { echo "Error: set SCHEMA=/path/to/schema.json" >&2; exit 1; }
@mkdir -p $(OUT)
@$(RUN) $(PKG).extractors --schema "$(SCHEMA)" -o $(OUT)/ir.json
@$(RUN) $(PKG).ir $(OUT)/ir.json
view: ## OUT/ir.json -> OUT/view.json, the default view for its source type
@$(RUN) $(PKG).ops $(OUT)/ir.json --overview -o $(OUT)/view.json
graph: view ## OUT/view.json -> whatever its structure asks for
@$(RUN) $(PKG).emitters auto $(OUT)/view.json -o $(OUT) \
--style $(STYLE) $(THEME_ARG)
site: view ## OUT/view.json -> a self-contained docs site in OUT/site
@$(RUN) $(PKG).emitters site $(OUT)/view.json -o $(OUT)/site \
--style $(STYLE) $(THEME_ARG)
@echo " open $(OUT)/site/index.html"
docs: ## Regenerate the figures in docs/ — docgen documented by docgen
@mkdir -p docs/img
@$(RUN) $(PKG).extractors.python --root $(HERE) -o /tmp/$(PKG)-docs.json >/dev/null
@$(RUN) $(PKG).ops /tmp/$(PKG)-docs.json --overview -o /tmp/$(PKG)-docs-view.json >/dev/null
@$(RUN) $(PKG).emitters dot /tmp/$(PKG)-docs-view.json -o docs/img/architecture.svg -q
@$(RUN) $(PKG).emitters minimap /tmp/$(PKG)-docs.json -o docs/img/minimap.svg --scale 0.5 --width 860
@$(RUN) $(PKG).emitters erd $(OUT)/ir.json -o docs/img/erd.svg 2>/dev/null \
|| echo " (erd figure kept — needs a schema IR at $(OUT)/ir.json to refresh)"
@PYTHONPATH=$(PARENT) $(PY) -c "from $(PKG).emitters.site import VIEWER, _slots, _fill; \
from $(PKG).style import Style; import pathlib; \
pathlib.Path('$(HERE)/docs/viewer.html').write_text( \
_fill(VIEWER.replace('__TITLE__', 'docgen docs'), _slots(Style.load('lucid'))))"
@echo " open $(HERE)/docs/index.html"
explore: ## OUT/ir.json -> OUT/explore/ — navigate on one side, explore on the other
@$(RUN) $(PKG).emitters explore $(OUT)/ir.json -o $(OUT)/explore \
--style $(STYLE) $(THEME_ARG) --scale $(SCALE)
@echo " open $(OUT)/explore/explore.html"
minimap: ## OUT/ir.json -> OUT/minimap.svg — what is where, read from the colours
@$(RUN) $(PKG).emitters minimap $(OUT)/ir.json -o $(OUT)/minimap.svg \
--style $(STYLE) $(THEME_ARG) --scale $(SCALE)
code: ## Extract C#/TypeScript from SRC (needs tree-sitter)
@test -n "$(SRC)" || { echo "Error: set SRC=/path/to/tree" >&2; exit 1; }
@mkdir -p $(OUT)
@$(RUN) $(PKG).extractors code --root "$(SRC)" -o $(OUT)/ir.json
@$(RUN) $(PKG).ir $(OUT)/ir.json
index: ## OUT/ir.json -> OUT/index.md and OUT/sidebar.json
@$(RUN) $(PKG).emitters index $(OUT)/ir.json -o $(OUT)/index.md
@$(RUN) $(PKG).emitters index $(OUT)/ir.json -o $(OUT)/sidebar.json
self: ## docgen's book of the widest tree it can see, then check it
@$(eval SELF_SRC := $(shell PYTHONPATH=$(PARENT) $(PY) -c "from $(PKG) import reference; \
r = reference.root(); print(r if r else '$(HERE)')"))
@echo " self-hosting on $(SELF_SRC)"
@$(MAKE) --no-print-directory book SRC=$(SELF_SRC) SLUG=self \
BOOK=$(OUT)/book/self OUT=$(OUT)
@echo
@$(MAKE) --no-print-directory check BOOK=$(OUT)/book/self
doctor: ## Report whether this machine can run it
@printf 'python : '; $(PY) --version 2>&1 || echo MISSING
@printf 'dot : '; (dot -V 2>&1) || echo 'MISSING — sudo apt install graphviz (only to render)'
@printf 'tree-sit : '; $(PY) -c 'import tree_sitter, tree_sitter_c_sharp, tree_sitter_typescript; print("ok — C# and TypeScript available")' 2>/dev/null || echo 'absent — Python only. pip install tree_sitter tree_sitter_c_sharp tree_sitter_typescript'
@printf 'networkx : '; $(PY) -c 'import networkx; print(networkx.__version__ + " — for lab/ experiments")' 2>/dev/null || echo 'absent — only used in lab/'
@printf 'package : %s (from %s)\n' '$(PKG)' '$(PARENT)'
@printf 'yaml : '; $(PY) -c 'import yaml; print("ok — needed only to read OpenAPI")' 2>/dev/null || echo 'absent — only used by the OpenAPI reader'
@printf 'reference: '; PYTHONPATH=$(PARENT) $(PY) -c "from $(PKG) import reference; print(reference.describe())"
@printf 'styles : '; $(RUN) $(PKG).style 2>/dev/null \
|| $(RUN) $(PKG) 2>/dev/null \
|| PYTHONPATH=$(PARENT) $(PY) -c "from $(PKG).style import Style; print(', '.join(Style.available()))"
@PYTHONPATH=$(PARENT) $(PY) -c "import $(PKG).ir, $(PKG).emitters.dot, $(PKG).ops" >/dev/null 2>&1 \
&& echo 'import : ok' || echo 'import : FAILED — is the folder intact?'
clean: ## Delete OUT. Nothing else is ever written to
@rm -rf "$(OUT)" && echo "Removed $(OUT)"

View File

@@ -0,0 +1,225 @@
# docgen
Static analysis of a tree, and the artifacts that fall out of it.
The point is not the diagram. The point is the format in the middle — diagrams
are one consumer of it, and not the one that reaches the most people.
```
extractors/ → graph IR (JSON) → emitters/
(per source type) (one schema) (per output target)
style/*.json
(consumed by emitters only)
```
```bash
make check # prove it, on a tree it builds itself
make self # run the whole thing over soleprint
make ir SRC=../../station/tools/histgen
make index && make graph
make help
```
Or as three composable commands, which is what the Makefile is wrapping:
```bash
python3 -m docgen.extractors.python --root SRC -o ir.json
python3 -m docgen.ops ir.json --drop-stdlib -o view.json
python3 -m docgen.emitters dot view.json -o graph.svg --theme dark
```
## DOT collapses three concerns; this separates them
| concern | question | owner |
|---|---|---|
| **structure** | what the graph *is* | `ir/schema.json` — versioned, golden-tested |
| **meaning** | what things *mean visually* | `style/*.json`, keyed on `kind` |
| **placement** | where things *go* | Graphviz defaults. Phase two |
An extractor has never heard of SVG, colours or layout. An emitter has never
heard of Python, `ast` or SQL. **The IR carries no visual information** — if a
field would change between light and dark theme, it does not belong in it.
`shape="cylinder"` is not a field; it is `kind="datastore"` plus a style rule,
which is what lets the same IR render in a theme that has no cylinders.
The selftest asserts all three of those, because they are the design rather than
a nicety and they are exactly what erodes first.
## The IR
```json
{
"meta": { "source": "python", "root": "app/", "schema_version": "1" },
"nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
"parent": "app.models",
"attrs": { "file": "app/models.py", "line": 12 } } ],
"edges": [ { "source": "app.models.User", "target": "app.db.Base",
"kind": "inherits", "attrs": {} } ]
}
```
- **`id`** is fully qualified and **stable across runs**. That is what makes two
graphs from two commits diffable.
- **`kind`** is the hinge, and the only field style and layout may key on.
- **`parent`** is containment. Relationships are edges.
- **`attrs`** is an open bag; `file`/`line` let a UI link a box to a line.
Stdlib dataclasses, not Pydantic. A format that needs a library installed to be
opened is not a format, it is an API. `ir/validate.py` is the check at the
boundary, and it reads the field lists out of `schema.json` so the two cannot
drift.
```bash
python3 -m docgen.ir ir.json
```
It catches what a schema cannot: an edge naming a node that does not exist, a
containment cycle, duplicate ids, and a visual field smuggled into `attrs`.
## Extraction is deterministic
**No LLM in the structural path.** A diagram from an AST cannot be out of date
with the code; one from a model's reading of the code is wrong the moment the
model has a bad day, which is the problem this exists to fix.
`ast` resolves nothing on its own — `class User(Base)` yields the literal string
`"Base"`. So there are two passes: one collects each module's definitions and
imports, the other resolves names against those tables.
```
from .db import Base ; class User(Base)
→ app.models.User --inherits--> app.db.Base not "Base"
```
**Unresolved names become `kind: "external"` nodes and keep their edges.**
Dropping them is the worse failure: the diagram looks complete and has quietly
lost a dependency. Gathered by the index emitter, they *are* the project's
dependency surface.
An unparseable file is recorded as a node with an `error` attr, not a crash —
one bad file must not cost you the other four hundred.
`calls` edges are deliberately **not** attempted. Resolving `self.foo()` needs
type inference, and a call graph that is quietly 60% right is worse than none
because it reads as authoritative.
### A second source
`extractors/db.py` reads the published `{models, relationships, source}`
contract that `modelgen` already emits and `graphgen` already consumes. Tables
become nodes, columns become contained nodes, foreign keys become edges — with
no new top-level field, which was the checkpoint on whether the schema was right.
Connecting to a live database is not here. `modelgen from-db --url ...` does
that and writes the schema this reads; the two-step also keeps credentials out
of this pipeline entirely.
## Views are not an emitter concern
The first real diagram out of this pipeline was a 3000px strip: four modules of
content and sixty `sys`/`json`/`typing` boxes, all peers. The emitter was
correct and the picture was useless. That is a **missing view**, and the fix
belongs to every consumer at once — the index, the diagram and the diff all want
"just this subsystem, two hops out, without the stdlib".
```bash
python3 -m docgen.ops ir.json --drop-stdlib --around docgen.ir --hops 2 -o view.json
```
`drop_stdlib`, `drop_external`, `only_kinds`, `drop_kinds`, `subtree`,
`neighbourhood`, `collapse_to_depth`. All IR→IR, all composable, each producing
a document that still validates.
Graph *algorithms* are not here. Transitive reduction, cycle detection and
dominators are `networkx`'s, and reimplementing them is the classic way to
acquire a quiet bug. `lab/` is where that dependency gets tried against real IRs
before anything depends on it — the aim being to learn which part of it is
actually attractive, rather than adopting all of it on faith.
## One colour language
A style rule names a **slot**, never a colour. `"border": "atlas"` is the rule;
the theme binds `atlas` to `#43A047` in print and `#15803d` on the docs site.
That indirection is the whole point. `common/theme/tokens.css`,
`docs/graphs/themes/*.gvpr` and `style/lucid.json` use the same slot names, so a
diagram and the page around it match by construction — which is the rule
`docs/graphs/README.md` already states. The `dark` theme's `artery`, `atlas` and
`station` slots are exactly the `--system-accent` values set in
`artery/index.html:30`, `atlas/index.html:25` and `station/index.html:29`, and
the selftest fails if they drift apart.
An unknown `kind` falls back to `default` rather than crashing, so a new
extractor renders plainly and legibly on day one instead of needing a style file
written first.
**How a container picks its colour without the IR naming one:** it does not. The
IR says which spr model a group belongs to (`attrs.domain` — semantic), and
`domain_slots` maps that to a slot. Same mechanism as `--system-accent`. With no
domain, the emitter assigns by sorted id, so two runs agree.
## Use DOT until it hits its limits
The emitter writes what DOT expresses natively and stops at the boundary rather
than growing machinery. The limits are recorded in `style/lucid.json` under
`limits` and reachable as `Style.limits()`:
| | |
|---|---|
| header bars | a cluster has a label and a fill, not a 100%-width header rectangle |
| `stroke-dasharray` | not parameterised — `4,4` and `5,5` collapse to one dash |
| corner radius | `rounded` is binary, so 4px and 6px are identical |
| icon above label | needs an HTML-like label table |
| sequence badges | `xlabel` carries the number; the circle does not exist |
Those mark where a richer emitter would begin. The style file carries the full
spec regardless, so that emitter needs no re-authoring.
One limit that *was* worth solving: DOT cannot use a cluster as an edge
endpoint, so every module-to-module import silently vanished. The native answer
is `compound=true` with `lhead`/`ltail` — draw between a representative leaf and
clip at the cluster border.
## The output is addressable
`id` and `kind` pass through to the SVG as the element's `id` and `class`, and
`attrs.file`/`attrs.line` become an `href`. A front end can bind behaviour to a
box and a box can link to the line it came from, without the emitter knowing
about either.
## Testing
```bash
make check # 59 checks, offline, nothing installed
```
**Golden tests go on the IR, never on the SVG.** Graphviz measures label text
with the host's fonts to size nodes, so identical input gives different geometry
on a machine with different fontconfig. The IR is deterministic; the SVG is not.
Self-hosting is the honest end-to-end check, and it is where the real bugs came
from — two name-resolution faults that no fixture had reached:
```bash
make self # extract soleprint, and read out/index.md
```
## Where this sits
`docgen` belongs to Atlas — documentation is whose concern it is. It is **not** a
station tool and is not under `station/tools/`; it *may depend on* station tools,
which is the permitted direction.
Atlas 2 is a successor, not a replacement. `soleprint/atlas/` is untouched: it
carries client information and an idea still worth extracting — deriving frontend
and backend tests from one source, which is the same shape as this pointed the
other way.
## Not here
No layout system, no positioning, no ELK. No HTML-like labels, no SVG post-pass.
No LLM in the structural path — annotation (summarising a module, naming a
cluster) is a later layer, cached to its own file keyed by node `id`, merged into
`attrs` at emit time, and extraction must work with it absent. No configuration
knobs until two real consumers disagree.

View File

@@ -0,0 +1 @@
"""Docgen — code to diagram. The IR is the product; diagrams are one consumer."""

View File

@@ -0,0 +1,315 @@
"""
A **book** — one docgen operation, measured at both ends.
larder ──► step ──► step ──► step ──► book
what each one usable what
came in by itself came out
The first step says what went into the operation. The last step says what came
out, and renders it for the web. Everything between is an ordinary artifact that
stands on its own — `ir.json` is still an `ir.json`, and `make ir` still works
without knowing this file exists.
## Why both ends, and why they are steps rather than a wrapper
Because the two numbers are only worth having **together**. "102 nodes" is not a
fact about anything; "49 files in, 102 nodes out, nothing lost" is. A book that
read 45 of 47 files and drew a clean diagram is lying by omission, and before
this there was no place for the other 2 to be mentioned.
They are steps, not a wrapper, because a wrapper is something you can forget to
apply. A step is in the sequence, and the sequence is the thing the notebook
emits — so the measure is in the document whether or not anyone remembered.
## The two things it is not
**Not a gate.** Running one step alone is still a book, just a short one. An
operation that cannot measure something says what it could not measure and
carries on. Gating would break the property that makes the intermediate
artifacts useful, and that property is the whole reason the sequence is worth
having.
**Not a new pipeline.** Everything here composes functions that already existed
— `ops.overview`, `emitters.dot`, `emitters.site`, `notebook.spec`. The spine
adds a ledger and two measurements. If it ever starts doing the work itself,
something has gone wrong.
## The web end is deliberately the loose one
It is last, so nothing depends on it, so it can be replaced wholesale without
touching a single thing upstream. That is what lets it rule what gets generated
without being a stable contract: the book measure is the promise, and the page
that displays it is free to change drastically and often.
"""
import json
from dataclasses import dataclass, field
from pathlib import Path
from .larder import Larder
# The two spine steps. Named here because the notebook, the site and the checks
# all have to agree on what they are called, and three string literals in three
# files is how they stop agreeing.
FIRST = "larder"
LAST = "book"
@dataclass
class Step:
"""One thing the operation did, and what it left behind."""
id: str
label: str
artifact: str | None = None # relative to the book directory
bytes: int = 0
note: str = ""
skipped: str = "" # why, when it did not run
def to_dict(self) -> dict:
out = {"id": self.id, "label": self.label}
if self.artifact:
out["artifact"] = self.artifact
out["bytes"] = self.bytes
if self.note:
out["note"] = self.note
if self.skipped:
out["skipped"] = self.skipped
return out
# How a larder's unit shows up on the output side. Per-kind because the relation
# genuinely differs, and because getting it wrong produces a check that passes
# for the wrong reason — which is what the first version of this did.
#
# relation "exact" one unit in, one node out. Fewer means input was dropped.
# "collapse" many units in, fewer nodes out, by design.
# noun what to call the output-side thing, in the claim
# represents whether a unit that FAILED still appears as a node. Where it does,
# that is checkable and is the whole-input form of the rule that an
# unresolved name becomes an `external` node rather than vanishing.
RELATION = {
"python": ("exact", "module", True),
"code": ("exact", "module", True),
"db": ("exact", "table", False),
"openapi": ("exact", "endpoint", False),
# A HAR entry with no URL has nothing to represent, so there is no node to
# look for. Its absence is the correct outcome and is recorded in `failed`.
"usage": ("collapse", "call", False),
}
def _unit_counts(ir: dict, larder: Larder) -> tuple[int, int]:
"""(nodes from units that were read, nodes from units that failed).
Split because a failed file still gets a module node — carrying
`attrs.error`, so the gap is visible in the graph rather than only in a log.
Counting them together made "2 files read produced 4 modules" pass a check
that was supposed to prove nothing had been dropped.
"""
nodes = ir.get("nodes") or []
if larder.kind in ("python", "code"):
mods = [n for n in nodes
if n["kind"] == "module" and (n.get("attrs") or {}).get("file")]
broken = {n["attrs"]["file"] for n in mods if (n.get("attrs") or {}).get("error")}
whole = {n["attrs"]["file"] for n in mods} - broken
return len(whole), len(broken)
if larder.kind == "db":
return sum(1 for n in nodes if n["kind"] == "table"), 0
if larder.kind == "openapi":
return len({
(n.get("attrs") or {}).get("path") for n in nodes
if n["kind"] == "endpoint" and (n.get("attrs") or {}).get("path")
}), 0
if larder.kind == "usage":
return sum(1 for n in nodes if n["kind"] in ("endpoint", "operation")), 0
return 0, 0
class Book:
"""A named operation, its ledger, and the two measures that bracket it."""
def __init__(self, slug: str, larder: Larder, out):
self.slug = slug
self.larder = larder
self.out = Path(out)
self.steps: list[Step] = []
self.ir: dict | None = None
self.notebook: str | None = None
# The first step, recorded before any work happens. Doing it here rather
# than at the end is the difference between a measure and a summary: it
# says what the operation *set out* to read, so a crash halfway leaves a
# book that still says what went in.
self.steps.append(Step(
id=FIRST,
label="what came in",
note=larder.line(),
))
# -- the ledger -------------------------------------------------------
def step(self, id: str, label: str, path=None, note: str = "",
skipped: str = "") -> Step:
"""Record a step. `path` is written already; this measures it."""
artifact, size = None, 0
if path is not None:
path = Path(path)
if path.exists():
artifact = str(path.relative_to(self.out)) if self.out in path.parents \
or path.parent == self.out else str(path)
size = path.stat().st_size if path.is_file() else _tree_bytes(path)
s = Step(id=id, label=label, artifact=artifact, bytes=size,
note=note, skipped=skipped)
self.steps.append(s)
return s
# -- the last step ----------------------------------------------------
def measure(self) -> dict:
"""What came out. Counted off the final IR and the ledger."""
ir = self.ir or {"nodes": [], "edges": []}
by_kind: dict[str, int] = {}
for n in ir.get("nodes") or []:
by_kind[n["kind"]] = by_kind.get(n["kind"], 0) + 1
edge_kinds: dict[str, int] = {}
for e in ir.get("edges") or []:
edge_kinds[e["kind"]] = edge_kinds.get(e["kind"], 0) + 1
artifacts = [s for s in self.steps if s.artifact]
return {
"nodes": sum(by_kind.values()),
"by_kind": {k: by_kind[k] for k in sorted(by_kind)},
"edges": sum(edge_kinds.values()),
"edges_by_kind": {k: edge_kinds[k] for k in sorted(edge_kinds)},
"external": by_kind.get("external", 0),
"steps": len(self.steps),
"artifacts": [{"path": s.artifact, "bytes": s.bytes} for s in artifacts],
"bytes": sum(s.bytes for s in artifacts),
}
def compare(self) -> list[dict]:
"""Reconcile the two ends. This is what having both is *for*.
Returns observations, each with a verdict, rather than raising: the book
is already built by the time anyone can compare, and refusing to write it
would destroy the evidence. `book/checks.py` turns these into pass/fail
at the book test level, and the CLI exits 1 when one of them is not ok.
"""
out = []
ir = self.ir or {}
produced, represented = _unit_counts(ir, self.larder)
relation, noun, represents = RELATION.get(
self.larder.kind, ("collapse", "node", False))
read, unit = self.larder.read, self.larder.unit
n_failed = len(self.larder.failed)
if relation == "exact":
ok = produced >= read
out.append({
"id": "units-accounted-for",
"ok": ok,
"claim": f"{read} {unit}(s) read produced {produced} {noun}(s)",
"why": (
"one unit in, one node out" if ok else
f"{read - produced} {unit}(s) were read and produced nothing — "
"the input was dropped between the extractor and the document, "
"which is the failure this measure exists to catch"
),
})
else:
out.append({
"id": "collapse-is-intended",
"ok": produced >= 1 or read == 0,
"claim": f"{read} {unit}(s) collapsed to {produced} {noun}(s)",
"why": "many recorded requests describe few endpoints — that is the point",
})
if n_failed:
out.append({
"id": "failures-surfaced",
"ok": True,
"claim": f"{n_failed} {unit}(s) could not be read",
"why": "named in the larder measure and on the landing page, not only in a log",
})
# The whole-input form of "an unresolved name becomes an `external` node".
# A file that failed to parse must still be in the graph, or the diagram
# shows a tree that is smaller than the tree on disk and says nothing.
if represents and n_failed:
out.append({
"id": "failures-still-in-the-graph",
"ok": represented >= n_failed,
"claim": f"{n_failed} unreadable {unit}(s) appear as {represented} "
f"marked {noun}(s)",
"why": (
"a gap that is drawn can be seen" if represented >= n_failed else
f"{n_failed - represented} unreadable {unit}(s) are missing from the "
"graph entirely — the picture is smaller than the source and does "
"not say so"
),
})
return out
# -- writing ----------------------------------------------------------
def close(self, *, site=None) -> Step:
"""Append the last step. Call once, after the web output is written.
Must be called BEFORE the notebook is built, because the notebook quotes
the book measure and the measure is not complete until this step exists.
The site is this step's *artifact*. The notebook is not: it is the whole
sequence's rendering rather than an item in it, and it is set on
`self.notebook` afterwards, by name only — a notebook cannot report its
own byte count without changing it.
"""
m = self.measure()
summary = " · ".join(f"{v} {k}" for k, v in
sorted(m["by_kind"].items(), key=lambda kv: -kv[1]))
artifact, size = None, 0
if site is not None:
site = Path(site)
if site.exists():
artifact, size = str(site.relative_to(self.out)), _tree_bytes(site)
self.steps.append(Step(
id=LAST,
label="what came out",
artifact=artifact,
bytes=size,
note=f"{summary} · {m['edges']} edges" if summary else "nothing was produced",
))
return self.steps[-1]
def to_dict(self) -> dict:
"""The book, as the ledger someone else can read.
`generated_at` is absent for the same reason the IR's is: an unchanged
larder must serialise to the same bytes, or nothing downstream can tell
a real change from a rebuild.
"""
out = {
"slug": self.slug,
"larder": self.larder.to_dict(),
"steps": [s.to_dict() for s in self.steps],
"book": self.measure(),
"reconciled": self.compare(),
}
if self.notebook:
out["notebook"] = self.notebook
return out
def write(self) -> Path:
self.out.mkdir(parents=True, exist_ok=True)
path = self.out / "book.json"
path.write_text(json.dumps(self.to_dict(), indent=2, sort_keys=False) + "\n")
return path
def _tree_bytes(root: Path) -> int:
return sum(p.stat().st_size for p in root.rglob("*") if p.is_file())

View File

@@ -0,0 +1,80 @@
""" python3 -m docgen.book --root ../station -o out/book/station
One book: larder measure, the steps, the web output, book measure. The step
artifacts land in `steps/` and are ordinary files — nothing here needs this
command to have been the thing that produced them.
"""
import argparse
import sys
from pathlib import Path
KINDS = ("python", "code", "db", "openapi", "usage")
def main(argv=None) -> int:
p = argparse.ArgumentParser(
prog="python3 -m docgen.book",
description="Run one docgen operation, measured at both ends.",
)
src = p.add_mutually_exclusive_group(required=True)
src.add_argument("--root", type=Path, help="A source tree (Python, or --lang code).")
src.add_argument("--schema", type=Path, help="A graphgen-compatible schema.json.")
src.add_argument("--openapi", type=Path, help="An OpenAPI document.")
src.add_argument("--har", type=Path, help="A HAR recording.")
p.add_argument("--lang", choices=("python", "code"), default="python",
help="With --root: the stdlib ast reader, or tree-sitter. Default python.")
p.add_argument("--output", "-o", type=Path, required=True,
help="The book directory. Created if absent.")
p.add_argument("--slug", help="What to call it. Defaults to the source's name.")
p.add_argument("--style", default="lucid")
p.add_argument("--theme", default=None, help="dark (default) or lucid.")
p.add_argument("--overlay", type=Path,
help="A hand-written overlay, re-applied on every build.")
p.add_argument("--exclude", action="append", default=[],
help="Directory name to skip. Repeatable.")
p.add_argument("--quiet", "-q", action="store_true")
args = p.parse_args(argv)
if args.root is not None:
kind, source = args.lang, args.root
elif args.schema is not None:
kind, source = "db", args.schema
elif args.openapi is not None:
kind, source = "openapi", args.openapi
else:
kind, source = "usage", args.har
if not Path(source).exists():
print(f"Error: {source} does not exist", file=sys.stderr)
return 1
overlay = None
if args.overlay:
if not args.overlay.exists():
# Absent is fine and is the documented default; named-but-missing is
# a typo, and quietly building without it would hide the typo.
print(f"Error: overlay {args.overlay} does not exist", file=sys.stderr)
return 1
from ..notebook import spec as spec_mod
overlay = spec_mod.load(args.overlay)
from .build import run
try:
book = run(kind, source, args.output, slug=args.slug, style=args.style,
theme=args.theme, exclude=tuple(args.exclude), overlay=overlay,
quiet=args.quiet)
except Exception as e: # noqa: BLE001 - the CLI reports, it does not traceback
print(f"Error: {type(e).__name__}: {e}", file=sys.stderr)
return 1
# Exit 1 when the two ends do not reconcile. The book is still written —
# the evidence is the point — but a build that lost input should fail a
# pipeline rather than pass quietly.
lost = [r for r in book.compare() if not r.get("ok")]
return 1 if lost else 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,350 @@
"""
Run a book: larder, the steps, the web output, the book measure.
python3 -m docgen.book --root ../station -o out/book/station
Composes functions that already exist. Nothing here parses, lays out or styles
anything — `ops.overview`, `emitters.dot`, `emitters.site` and `notebook.spec`
do all of it, and this decides the order and writes the ledger. If this file
ever starts doing the work, the seam has moved to the wrong place.
## The notebook is the sequence, not an item in it
So it is not a step. Its **first cell is the larder measure and its last cell is
the book measure**, which is what makes the two ends part of the document rather
than part of the tooling. Between them, one pair of cells per step: what the
step did, and a code cell that loads that step's artifact on its own.
That last part is the "usable by themselves" property made executable. Each cell
reads one artifact and prints one fact, so a reader can start anywhere in the
sequence, and `selftest.py` runs them.
"""
import json
from pathlib import Path
from . import Book
from .larder import Larder
# Loader for the notebook's step cells. Guarded on purpose: a notebook is
# opened from wherever somebody happens to open it, and a traceback on cell 2
# is a worse answer than a sentence saying which directory to run it from.
PRELUDE = '''from pathlib import Path
import json
BOOK = Path.cwd() # the book directory — change if you opened this elsewhere
def load(rel):
"""One step's artifact, on its own. Returns None when it is not here.
A step's artifact can be a directory — the explorer is one — so this returns
a file list for those rather than trying to read a directory as text.
"""
p = BOOK / rel
if not p.exists():
print(f"{rel} is not here — run: make book OUT={BOOK}")
return None
if p.is_dir():
return sorted(f.relative_to(p).as_posix() for f in p.rglob("*") if f.is_file())
return json.loads(p.read_text()) if p.suffix == ".json" else p.read_text()
'''
# Every extractor a book can run, as data rather than as an if/elif chain.
#
# This is the shape that gives the extension contract teeth. `selftest.py` loops
# over this dict and asserts each entry reports a larder measure — so adding an
# extractor makes the test start asking about it without anyone remembering to
# go and add a case. rig's `CONFIG_OVERRIDABLE` loop is the precedent, and it
# found two real bugs that way.
#
# kind -> (module, function, what the source is)
EXTRACTORS = {
"python": (".extractors.python", "extract", "tree"),
"code": (".extractors.code", "extract", "tree"),
"db": (".extractors.db", "extract", "file"),
"openapi": (".extractors.openapi", "extract", "file"),
"usage": (".extractors.usage", "extract", "file"),
}
# Extractors that walk a directory take an exclude list; the ones that read a
# single document have nothing to exclude.
TAKES_EXCLUDE = {"python", "code"}
def extract(kind: str, source, *, exclude=(), identity=None):
"""(IR dict, Larder) for one source type, dispatched through EXTRACTORS."""
from importlib import import_module
if kind not in EXTRACTORS:
raise ValueError(
f"no extractor named {kind!r} — have {', '.join(sorted(EXTRACTORS))}"
)
module_name, fn_name, _ = EXTRACTORS[kind]
# The package name is derived, not written: the Makefile takes PKG from the
# directory name so the folder can be copied anywhere and renamed, and a
# literal "docgen" here would quietly undo that.
root_pkg = __package__.rsplit(".", 1)[0]
fn = getattr(import_module(module_name, package=root_pkg), fn_name)
kwargs = {"identity": identity or str(source)}
if kind in TAKES_EXCLUDE:
kwargs["exclude"] = exclude
g = fn(source, **kwargs)
ir = g.to_dict()
measured = ir["meta"].get("larder")
if measured is None:
# Not fatal. An extractor that cannot count its input still produces a
# book; what it does not get to do is pretend it measured one.
return ir, None
return ir, Larder.from_dict(measured)
def spec_from(book: Book, ir: dict) -> dict:
"""The book's ledger -> a notebook spec, with the two measures at the ends.
Reuses `notebook.spec`'s step vocabulary rather than inventing one, so
`merge()` keeps working and a hand-written overlay can annotate a spine step
the same way it annotates any other.
"""
from ..notebook import spec as spec_mod
steps = [
spec_mod._step(
"larder", "md", title=f"{book.slug} — what came in",
text=(
f"`{book.larder.identity}`\n\n**{book.larder.line()}**\n\n"
"This is the first step of the book and the only measure of the "
"input. Everything below is derived from it, so a number here "
"that looks wrong makes everything below it suspect."
+ (
"\n\nCould not be read:\n\n"
+ "\n".join(f"- `{f['name']}` — {f['error']}"
for f in book.larder.failed)
if book.larder.failed else ""
)
),
),
spec_mod._step("prelude", "code", title="Reading a step on its own",
code=PRELUDE),
]
for s in book.steps:
if s.id in ("larder", "book") or not s.artifact:
continue
steps.append(spec_mod._step(
f"step-{s.id}", "md", title=s.label,
text=f"`{s.artifact}` — {s.bytes:,} bytes" + (f"\n\n{s.note}" if s.note else ""),
))
steps.append(spec_mod._step(
f"load-{s.id}", "code", code=_load_cell(s.id, s.artifact),
))
# Where the larder is API-shaped, the generated endpoint walkthrough slots in
# as further steps. For a tree of source there are no endpoints and this adds
# nothing, which is the correct amount for it to add.
if any(n["kind"] in ("endpoint", "operation") for n in ir.get("nodes") or []):
generated = spec_mod.from_ir(ir)
steps.extend(s for s in generated["steps"] if s["id"] not in ("intro",))
m = book.measure()
# `external` is reported on its own below, so it is dropped here rather than
# appearing twice in one line — which is how it read before.
summary = " · ".join(f"{v} {k}" for k, v in
sorted(m["by_kind"].items(), key=lambda kv: -kv[1])
if k != "external")
reconciled = "\n".join(
f"- {'' if r['ok'] else ''} {r['claim']}{r['why']}"
for r in book.compare()
)
steps.append(spec_mod._step(
"book", "md", title=f"{book.slug} — what came out",
text=(
f"**{summary} · {m['edges']} edges · {m['external']} external**\n\n"
f"{len(m['artifacts'])} artifact(s), {m['bytes']:,} bytes.\n\n"
f"Reconciled against what came in:\n\n{reconciled}\n\n"
"The web output is `site/index.html`."
),
))
return {"version": spec_mod.SPEC_VERSION, "steps": steps}
def _load_cell(step_id: str, artifact: str) -> str:
"""A cell that reads one artifact and prints one fact about it.
The fact has to suit the artifact. An earlier version printed "nodes, edges"
for every JSON file, so `sidebar.json` — which has neither — reported
"0 nodes, 0 edges", which is a true sentence about the wrong thing and worse
than saying nothing.
"""
if not artifact.endswith((".json", ".svg", ".md")):
# A directory, e.g. the explorer.
return (
f'files = load("{artifact}")\n'
'if files is not None:\n'
' print(f"{len(files)} file(s)")\n'
' print("\\n".join(files[:5]))'
)
if artifact.endswith(".json"):
return (
f'data = load("{artifact}")\n'
'if data:\n'
' if "nodes" in data:\n'
' print(f\'{len(data["nodes"])} nodes, {len(data.get("edges", []))} edges\')\n'
' else:\n'
' print(", ".join(f"{k}: {len(v) if isinstance(v, (list, dict)) else v}"\n'
' for k, v in data.items()))'
)
if artifact.endswith(".svg"):
# No IPython import, guarded or otherwise. A generated notebook is a
# build artifact and has to run wherever it is opened; requiring a
# kernel package in order to *load a file* would make the cell fail on
# the machine that produced it, which is where this was found.
return (
f'svg = load("{artifact}")\n'
'if svg:\n'
' print(f"{len(svg):,} bytes of SVG")\n'
' # In Jupyter: from IPython.display import SVG; SVG(svg)'
)
if artifact.endswith(".md"):
return (
f'text = load("{artifact}")\n'
'if text:\n'
' print(text[:400])'
)
return f'print(load("{artifact}") is not None)'
def run(kind: str, source, out, *, slug: str | None = None, style: str = "lucid",
theme: str | None = None, exclude=(), overlay=None, quiet: bool = False) -> Book:
"""The whole book, in one process. Returns it; `book.json` is written."""
from ..ir import check
from ..ops import classify, overview
from ..style import Style
out = Path(out)
slug = slug or Path(str(source)).name or "book"
steps_dir = out / "steps"
steps_dir.mkdir(parents=True, exist_ok=True)
def say(text):
if not quiet:
print(text)
ir, larder = extract(kind, source, exclude=exclude)
if larder is None:
larder = Larder(kind=kind, identity=str(source), unit="document", seen=0)
book = Book(slug=slug, larder=larder, out=out)
book.ir = ir
say(f" larder {larder.line()}")
problems = check(ir)
if problems:
# Recorded as a step rather than raised: a book that cannot be trusted
# should exist and say so, because the alternative is that nobody can
# see what went wrong.
book.step("validate", "the IR did not validate", note="; ".join(problems[:3]))
say(f" WARNING IR has {len(problems)} problem(s)")
p = steps_dir / "ir.json"
p.write_text(json.dumps(ir, indent=2) + "\n")
book.step("ir", "the graph, extracted", p, note=f"{len(ir['nodes'])} nodes")
view = overview(ir)
p = steps_dir / "view.json"
p.write_text(json.dumps(view, indent=2) + "\n")
verdict = classify(view)
book.step("view", "the default view for this source", p,
note=f"{verdict['kind']}{verdict['why']}")
style_obj = Style.load(style, theme=theme)
# The drawing, chosen by structure rather than by the caller. `classify`
# already decided; this runs what it named.
graph_rel = None
if verdict["emitter"] == "erd":
from ..emitters.erd import emit as erd_emit
p = steps_dir / "graph.svg"
p.write_text(erd_emit(view, style_obj))
graph_rel, label = "steps/graph.svg", "drawn as an ERD"
elif verdict["emitter"] == "index":
from ..emitters.index import to_markdown
p = steps_dir / "graph.md"
p.write_text(to_markdown(view))
graph_rel, label = None, "not a diagram — written as a list"
else:
from ..emitters.dot import emit as dot_emit, render
p = steps_dir / "graph.svg"
try:
p.write_bytes(render(dot_emit(view, style_obj,
rankdir=(verdict.get("options") or {}).get("rankdir"))))
graph_rel, label = "steps/graph.svg", f"drawn as a {verdict['kind']}"
except Exception as e: # noqa: BLE001 - graphviz may not be installed
p = None
label = f"not drawn — {type(e).__name__}"
book.step("graph", label, skipped=str(e)[:200])
if p is not None:
book.step("graph", label, p, note=verdict["why"])
from ..emitters.index import to_markdown, to_sidebar
p = steps_dir / "index.md"
p.write_text(to_markdown(ir))
book.step("index", "readable without a diagram", p)
p = steps_dir / "sidebar.json"
p.write_text(json.dumps(to_sidebar(ir), indent=2) + "\n")
book.step("sidebar", "navigation, for whatever renders it", p)
try:
from ..emitters.minimap import emit as mm_emit
p = steps_dir / "minimap.svg"
p.write_text(mm_emit(ir, style_obj))
book.step("minimap", "what is where, read from the colours", p)
except Exception as e: # noqa: BLE001
book.step("minimap", "minimap not drawn", skipped=f"{type(e).__name__}: {e}")
try:
from ..emitters.explore import write as exp_write
exp_write(ir, style_obj, out / "explore")
book.step("explore", "navigate on one side, explore on the other",
out / "explore")
except Exception as e: # noqa: BLE001
book.step("explore", "explorer not built", skipped=f"{type(e).__name__}: {e}")
# -- the last step ----------------------------------------------------
from ..emitters.site import write as site_write
site_dir = out / "site"
site_write(view, style_obj, site_dir,
graph=Path(graph_rel).name if graph_rel else None,
title=slug, book=book.to_dict())
if graph_rel and (out / graph_rel).exists():
(site_dir / Path(graph_rel).name).write_bytes((out / graph_rel).read_bytes())
# close() BEFORE the notebook spec is built, not after. The spec quotes the
# book measure, and the book measure only includes the site once the last
# step exists — build it the other way round and the notebook says 7
# artifacts while book.json says 8, which is exactly what it did.
book.close(site=site_dir)
from ..emitters.notebook import write as nb_write
from ..notebook import spec as spec_mod
spec, spec_problems = spec_mod.merge(spec_from(book, ir), overlay)
nb_write(spec, out / "notebook.ipynb")
# Recorded by name and not by size: a notebook cannot report its own byte
# count without changing it.
book.notebook = "notebook.ipynb"
for pr in spec_problems:
say(f" overlay {pr}")
path = book.write()
m = book.measure()
say(f" book {m['nodes']} nodes · {m['edges']} edges · "
f"{m['external']} external · {len(m['artifacts'])} artifacts")
for r in book.compare():
say(f" {'ok ' if r['ok'] else 'LOST'} {r['claim']}")
say(f" open {site_dir / 'index.html'}")
say(f" {path}")
return book

View File

@@ -0,0 +1,305 @@
"""
Does *this* book still hold — the third test level.
python3 -m docgen.book.checks out/book/station
docgen has three levels of test, and they differ by what they assert *about*.
The distinction is rig's, from `rig/ctrl/selftest.sh`, and it is worth keeping
because it decides what a failure means:
make doctor the MACHINE. Never fails; it reports.
make check DOCGEN. Exits 1. "docgen no longer does what it says."
make check BOOK=<dir>
THIS BOOK. Exits 1. "this book no longer holds."
The third is the one that reaches a project docgen has never seen. It is also
where **framework and custom checks live together**, at different levels:
generated the spine's own assertions, identical for every book. Both
measures present, the reconciliation holding, every artifact
where the ledger says it is, the notebook still executing.
custom hand-written, in the book's own `checks.py`, using the same
check/note/skip helpers — so a project's line and a framework
line read identically and fail identically.
Same split as the notebook's base and overlay, for the same reason: generation
alone cannot know what *this* project cares about, and hand-authoring alone rots.
## The discipline these are written in
Carried from rig verbatim, because it is the point of the whole level:
> Each check is ONE decision that has already been made, with the reason above
> it — not coverage, and deliberately not an exhaustive sweep of use cases. A
> rule without its reason gets overridden the first time it is inconvenient.
> Failing one should read as **"you are about to undo this"** rather than
> "something broke".
## Writing a book's own checks
Put a `checks.py` next to `book.json`:
def checks(book, check, note, skip):
note("what this project will not give up")
# Payments moved once already and the move broke three dashboards.
# If it is not here, something renamed it again.
check("payments is still a module", True,
any(n["id"] == "app.payments" for n in book.ir["nodes"]))
`book` carries `.data` (the parsed `book.json`), `.ir` (the extracted graph),
and `.dir`. `check` takes (name, expected, actual) — expected first, because a
failure report is only useful if it says what was wanted.
"""
import json
import sys
from pathlib import Path
from . import FIRST, LAST
class Loaded:
"""A book read back off disk, for checking rather than building."""
def __init__(self, directory):
self.dir = Path(directory)
path = self.dir / "book.json"
if not path.exists():
raise FileNotFoundError(
f"{path} does not exist — is {self.dir} a book directory?\n"
"Build one with: python3 -m docgen.book --root <src> -o <dir>"
)
self.data = json.loads(path.read_text())
ir_path = self.dir / "steps" / "ir.json"
self.ir = json.loads(ir_path.read_text()) if ir_path.exists() else None
nb_path = self.dir / (self.data.get("notebook") or "notebook.ipynb")
self.notebook = json.loads(nb_path.read_text()) if nb_path.exists() else None
@property
def larder(self) -> dict:
return self.data.get("larder") or {}
@property
def measure(self) -> dict:
return self.data.get("book") or {}
class Report:
"""rig's reporting shape: sections, one line per decision, nothing aborts."""
def __init__(self):
self.passed, self.failed, self.skipped = 0, [], 0
def note(self, text: str) -> None:
print(f"\n{text}")
def check(self, name: str, expected, actual) -> bool:
if expected == actual:
print(f" ok {name}")
self.passed += 1
return True
print(f" FAIL {name}\n expected: {expected!r}\n got: {actual!r}")
self.failed.append(name)
return False
def skip(self, name: str, why: str) -> None:
print(f" -- {name} ({why})")
self.skipped += 1
def total(self) -> int:
print()
# stdout is block-buffered when redirected and stderr is not, so the
# failure summary printed below would otherwise arrive BEFORE the checks
# it summarises — which is how this was found, piping to `tail`.
sys.stdout.flush()
if not self.failed:
print(f"{self.passed} checks passed — this book still holds"
+ (f", {self.skipped} skipped" if self.skipped else ""))
return 0
print(f"FAILED — {len(self.failed)} of {self.passed + len(self.failed)}: "
f"{', '.join(self.failed)}", file=sys.stderr)
print("Read the comment next to the check. A failure here means a decision "
"has drifted, not that a tool is broken.", file=sys.stderr)
return 1
def generated(book: Loaded, r: Report) -> None:
"""The spine's own assertions. Identical for every book, by design."""
r.note("the book is bracketed — both ends present")
# The one structural promise the whole design makes: an operation is
# measured at both ends. A book missing either end is not a short book, it
# is a book whose numbers cannot be reconciled against anything.
ids = [s["id"] for s in book.data.get("steps") or []]
r.check("the first step is the larder measure", FIRST, ids[0] if ids else None)
r.check("the last step is the book measure", LAST, ids[-1] if ids else None)
r.check("nothing is bracketed twice", [1, 1],
[ids.count(FIRST), ids.count(LAST)])
r.note("what came in was measured, not guessed")
larder = book.larder
for key in ("kind", "identity", "unit", "seen", "read", "failed"):
r.check(f"larder records {key}", True, key in larder)
# read is derived, so a stored value that disagrees means somebody wrote it
# by hand. This is the same arithmetic ir/validate.py enforces; asserted
# again here because a book can be assembled without going through the IR.
r.check("read == seen - failed", larder.get("read"),
max(0, larder.get("seen", 0) - len(larder.get("failed") or [])))
# The one field that could carry a secret out of a database URL.
r.check("the identity carries no password", True,
":***@" in larder.get("identity", "") or "@" not in larder.get("identity", ""))
r.note("the two ends reconcile")
reconciled = book.data.get("reconciled") or []
r.check("something was reconciled", True, len(reconciled) > 0)
for item in reconciled:
# Each of these is a claim the book makes about itself. A false one
# means the document below is smaller than the source and does not say
# so, which is the single failure this level exists to catch.
r.check(item["claim"], True, item["ok"])
r.note("the ledger describes files that exist")
for step in book.data.get("steps") or []:
artifact = step.get("artifact")
if not artifact:
continue
path = book.dir / artifact
# A ledger naming a file that is not there is worse than no ledger: it
# is a manifest somebody will build tooling against.
r.check(f"{artifact} is where the ledger says", True, path.exists())
if path.is_file():
r.check(f"{artifact} is {step['bytes']} bytes", step["bytes"],
path.stat().st_size)
r.note("the notebook carries the sequence, with the measures at its ends")
if book.notebook is None:
r.skip("notebook", "no notebook.ipynb in this book")
else:
cells = book.notebook.get("cells") or []
first = "".join(cells[0]["source"]) if cells else ""
last = "".join(cells[-1]["source"]) if cells else ""
r.check("its first cell is what came in", True, "what came in" in first)
r.check("its last cell is what came out", True, "what came out" in last)
# Compiling is not enough — see selftest.py. `json.dumps` writes
# `false`/`true`/`null`, which are valid Python *identifiers*, so a
# generated body full of them compiles and then raises NameError.
ran, failure = _run_cells(cells, book.dir)
r.check(f"its {ran} code cells run, not merely compile", None, failure)
r.note("the web output exists and shows both ends")
index = book.dir / "site" / "index.html"
if not index.exists():
r.skip("site", "no site/index.html in this book")
else:
html = index.read_text()
# The site is the last step precisely because it is the artifact somebody
# definitely opens. A page that shows the result without showing what
# went in is the thing this whole change corrects.
r.check("the page says what came in", True, "what came in" in html)
r.check("the page says what came out", True, "what came out" in html)
if larder.get("failed"):
r.check("the page admits the book is incomplete", True,
"This book is incomplete" in html)
r.note("the book is reproducible")
# A timestamp would make two builds of an unchanged larder differ, which
# destroys the only useful property a ledger has: that a diff means a real
# change. Same rule as the IR's `generated_at`.
r.check("no timestamp in the ledger", True, "generated_at" not in json.dumps(book.data))
def custom(book: Loaded, r: Report) -> None:
"""This book's own assertions, if it has any."""
path = book.dir / "checks.py"
if not path.exists():
r.note("this book's own checks")
r.skip("custom checks", f"no {path.name} — write one to assert what this project cares about")
return
namespace: dict = {"__file__": str(path), "__name__": "book_checks"}
try:
exec(compile(path.read_text(), str(path), "exec"), namespace)
except Exception as e: # noqa: BLE001 - report, do not traceback
r.note("this book's own checks")
r.check(f"{path.name} loads", None, f"{type(e).__name__}: {e}")
return
fn = namespace.get("checks")
if not callable(fn):
r.note("this book's own checks")
r.check(f"{path.name} defines checks(book, check, note, skip)", True, False)
return
try:
fn(book, r.check, r.note, r.skip)
except Exception as e: # noqa: BLE001
r.check(f"{path.name} ran to completion", None, f"{type(e).__name__}: {e}")
def _run_cells(cells: list, cwd: Path) -> tuple[int, str | None]:
"""Execute the notebook's code cells from the book directory."""
import contextlib
import io
import os
ns, ran, failure = {}, 0, None
previous = os.getcwd()
try:
os.chdir(cwd)
with contextlib.redirect_stdout(io.StringIO()):
for c in cells:
if c.get("cell_type") != "code":
continue
src = "".join(c["source"])
if "urlopen" in src or ("call(" in src and "def call" not in src):
# Anything that would reach the network is compiled, not run.
try:
compile(src, c["id"], "exec")
ran += 1
except SyntaxError as e:
failure = f"{c['id']}: {e}"
break
continue
try:
exec(compile(src, c["id"], "exec"), ns)
ran += 1
except Exception as e: # noqa: BLE001 - any failure is the finding
failure = f"{c['id']}: {type(e).__name__}: {e}"
break
finally:
os.chdir(previous)
return ran, failure
def main(argv=None) -> int:
import argparse
p = argparse.ArgumentParser(
prog="python3 -m docgen.book.checks",
description="Check one book: the spine's assertions, then its own.",
)
p.add_argument("book", type=Path, help="A book directory (holding book.json).")
p.add_argument("--only", choices=("generated", "custom"),
help="Run one level rather than both.")
args = p.parse_args(argv)
try:
book = Loaded(args.book)
except (FileNotFoundError, json.JSONDecodeError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
print(f"{book.dir}{book.data.get('slug', '?')}")
r = Report()
if args.only != "custom":
generated(book, r)
if args.only != "generated":
custom(book, r)
return r.total()
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,195 @@
"""
What went in — the first step of every book.
A **larder** is the stocked source together with what connects it: a repo, a
live database, an OpenAPI endpoint, a HAR capture. Deliberately not "bucket".
A bucket is somewhere bytes sit; a larder is stocked from outside, has an
inventory, and goes stale. All three of those are properties worth measuring,
and they are the reason this file exists.
## Why a measure at all
Every extractor knew what it read and threw the number away. `Meta` carried
`source` and `root` — which extractor ran, and what it was pointed at — and
nothing about what was actually consumed. So a run that read 45 of 47 files
produced exactly the same document as one that read all 47, and the diagram
looked complete either way.
That is the failure this prevents, and it is the same failure the IR already
guards against one level down: an unresolved name becomes an `external` node
rather than being dropped, because *silently losing a thing is worse than
recording an unresolved one*. A larder measure is that rule applied to the
input as a whole.
## seen, failed, read
seen how many units the larder offered
failed the ones that could not be consumed, by name, with the reason
read seen - len(failed), derived and never stored
Three fields where two numbers would do, on purpose: `read` as a stored value
invites the question "does it include the failures", and every reader answers it
differently. Derived, there is nothing to get wrong.
## Redaction
`identity` is the one field in this whole tool that can carry a credential —
a database DSN has the password in it. So it is scrubbed here at construction,
and `ir/validate.py` sweeps for the scrub having worked. That is the same
discipline as `VISUAL_KEYS`: the architectural rule is a check, not a convention.
Paths are left as the caller gave them rather than resolved. An absolute path is
not a secret but it is machine-specific, and `meta.root` is already a bare name
for exactly that reason.
"""
import re
from dataclasses import dataclass, field
# What `seen` counts, per source type. Closed, like the IR's `kind` vocabulary:
# a larder that counts something with no name here is a larder nobody can
# compare against another one.
UNITS = ("file", "table", "path", "entry", "document")
# Query-string and connection-string keys whose value is a secret. Matched
# case-insensitively, because ODBC writes `Password=` and URLs write `password=`.
SECRET_KEYS = (
"password", "passwd", "pwd", "secret", "token", "access_token", "refresh_token",
"api_key", "apikey", "key", "sig", "signature", "auth", "credentials",
)
MASK = "***"
def _count(label: str, n: int) -> str:
"""Agree a label with its count, both directions.
Two directions because both arise: the closed `UNITS` are singular and need
pluralising, while `extra` keys are written plural by the extractor that
knows them ("packages", "hosts") and need singularising at one. Naive `+ "s"`
gives "entrys"; naive nothing gives "1 hosts".
"""
if n == 1:
return label[:-3] + "y" if label.endswith("ies") else label.rstrip("s") or label
if label.endswith("y"):
return label[:-1] + "ies"
return label if label.endswith("s") else label + "s"
def redact(identity: str) -> str:
"""Strip credentials out of a path or connection string.
Three shapes, which is all of them in practice:
postgresql://user:hunter2@host:5432/db -> postgresql://user:***@host:5432/db
https://api/x?token=abc123 -> https://api/x?token=***
Driver=x;Server=y;Password=hunter2; -> Driver=x;Server=y;Password=***;
The user, host, port and database survive. Those are what someone reading
the measure needs in order to recognise which larder this was, and none of
them is a secret.
"""
if not identity:
return identity
# scheme://user:secret@host — the password is between the first colon after
# the scheme and the last @ of the authority.
identity = re.sub(
r"(?P<scheme>[a-zA-Z][a-zA-Z0-9+.\-]*://)(?P<user>[^:/@\s]+):(?P<secret>[^@/\s]*)@",
lambda m: f"{m.group('scheme')}{m.group('user')}:{MASK}@",
identity,
)
# key=value, in a query string or a semicolon-delimited connection string.
keys = "|".join(re.escape(k) for k in SECRET_KEYS)
identity = re.sub(
rf"(?i)\b(?P<key>{keys})(?P<sep>\s*=\s*)(?P<secret>[^&;\s]*)",
lambda m: f"{m.group('key')}{m.group('sep')}{MASK}",
identity,
)
return identity
@dataclass
class Larder:
"""What one source offered, and how much of it was consumed.
Lands in `meta.larder`. Never in `nodes` or `attrs` — the IR is structure,
and provenance is meta. That split is the same one that keeps colour out.
"""
kind: str # which extractor stocked it
identity: str # path or redacted DSN
unit: str # one of UNITS
seen: int = 0
failed: list[dict] = field(default_factory=list)
extra: dict = field(default_factory=dict) # per-source facts
def __post_init__(self):
self.identity = redact(self.identity)
if self.unit not in UNITS:
raise ValueError(
f"larder unit {self.unit!r} is not one of {UNITS}"
"a unit nobody else uses cannot be compared against another larder"
)
@property
def read(self) -> int:
"""`seen` minus what failed. Derived, so it cannot disagree with itself."""
return max(0, self.seen - len(self.failed))
def fail(self, name: str, error: str) -> None:
"""Record a unit that could not be consumed, by name and reason."""
self.failed.append({"name": name, "error": error})
def to_dict(self) -> dict:
"""Key order fixed so an unchanged larder serialises byte-identically."""
out = {
"kind": self.kind,
"identity": self.identity,
"unit": self.unit,
"seen": self.seen,
"read": self.read,
"failed": [{"name": f["name"], "error": f["error"]} for f in self.failed],
}
if self.extra:
out["extra"] = {k: self.extra[k] for k in sorted(self.extra)}
return out
@classmethod
def from_dict(cls, data: dict) -> "Larder":
"""Round-trip. `read` is dropped: it is derived, not carried."""
return cls(
kind=data["kind"],
identity=data["identity"],
unit=data["unit"],
seen=data.get("seen", 0),
failed=list(data.get("failed") or []),
extra=dict(data.get("extra") or {}),
)
def line(self) -> str:
"""The one-line human form — the notebook's first cell, and the CLI.
Reads as a sentence because it is the first thing anyone sees about a
book, and "47 files read, 2 failed" is the fact that decides whether the
rest of the document is worth trusting. Which is also why the grammar
gets attention it would not otherwise deserve: "2 entrys read, 1 hosts"
reads as a machine talking, and a measure nobody reads is not a measure.
"""
parts = [f"{self.read} {_count(self.unit, self.read)} read"]
if self.failed:
parts.append(f"{len(self.failed)} failed")
for key in sorted(self.extra):
value = self.extra[key]
label = key.replace("_", " ")
if isinstance(value, int):
parts.append(f"{value} {_count(label, value)}")
else:
parts.append(f"{value} {label}")
return f"{self.identity}" + ", ".join(parts)
def of(kind: str, identity: str, unit: str, **extra) -> Larder:
"""Shorthand for the common case: `of("db", dsn, "table", dialect=...)`."""
return Larder(kind=kind, identity=identity, unit=unit, extra=extra)

View File

@@ -0,0 +1,135 @@
/* docgen docs. The layout five demos under semester/ converged on: a sticky
sidebar beside a bounded content column. Colours are tokens.css values, the
same ones the diagrams are drawn in. */
:root {
--bg: #0a0a0a;
--surface: #141414;
--surface-2: #1a1a1a;
--border: #333333;
--border-strong: #4a4a4a;
--text: #e5e5e5;
--muted: #a3a3a3;
--dim: #666666;
--accent: #d4a574;
--station: #1d4ed8;
--atlas: #15803d;
--artery: #b91c1c;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--bg);
color: var(--text);
font-family: "Segoe UI", Inter, system-ui, -apple-system, Arial, sans-serif;
font-size: 14px;
line-height: 1.72;
}
.layout { display: flex; min-height: 100vh; }
/* ── sidebar ─────────────────────────────────────────────────────────── */
.sidebar {
width: 232px; flex-shrink: 0;
background: var(--surface);
border-right: 1px solid var(--border);
position: sticky; top: 0; height: 100vh; overflow-y: auto;
padding: 1.4rem 0 3rem;
scrollbar-width: none;
}
.sidebar::-webkit-scrollbar { display: none; }
.sidebar-header { padding: 0 1.15rem 1rem; border-bottom: 1px solid var(--border); }
.sidebar-header b { display: block; font-size: 14px; color: var(--text); }
.sidebar-header small { display: block; color: var(--dim); font-size: 11px; margin-top: 3px; }
.sidebar nav { padding-top: .75rem; }
.sidebar .group {
color: var(--dim); font-size: 9.5px; text-transform: uppercase;
letter-spacing: .07em; padding: 1rem 1.15rem .3rem;
}
.sidebar a {
display: block; padding: 3px 1.15rem;
color: var(--muted); text-decoration: none; font-size: 12.5px;
border-left: 2px solid transparent;
}
.sidebar a:hover { color: var(--text); background: var(--surface-2); }
.sidebar a.active { color: var(--accent); border-left-color: var(--accent); }
/* ── content ─────────────────────────────────────────────────────────── */
.content { flex: 1; min-width: 0; max-width: 820px; padding: 2.5rem 3.25rem 6rem; }
h1 { font-size: 26px; letter-spacing: -.01em; margin-bottom: .35rem; }
.lede { color: var(--muted); font-size: 15px; margin-bottom: 2.5rem; }
h2 {
font-size: 18px; margin: 3rem 0 .9rem; padding-top: 1.6rem;
border-top: 1px solid var(--border);
}
h2:first-of-type { border-top: none; padding-top: 0; }
h3 { font-size: 14px; margin: 1.9rem 0 .5rem; color: var(--text); }
h4 { font-size: 12.5px; margin: 1.3rem 0 .35rem; color: var(--muted); font-weight: 600; }
p { margin-bottom: .95rem; color: var(--muted); }
p strong, li strong { color: var(--text); font-weight: 600; }
em { color: var(--text); font-style: italic; }
ul, ol { margin: 0 0 1rem 1.15rem; color: var(--muted); }
li { margin-bottom: .3rem; }
a { color: var(--accent); text-decoration: none; }
a:hover { text-decoration: underline; }
code {
font-family: ui-monospace, "Cascadia Mono", Consolas, "SF Mono", monospace;
font-size: 12px; background: var(--surface); color: var(--text);
padding: 1px 5px; border-radius: 3px;
}
pre {
background: var(--surface); border: 1px solid var(--border);
border-radius: 7px; padding: 13px 16px; overflow-x: auto; margin: .9rem 0 1.2rem;
}
pre code { background: none; padding: 0; font-size: 12px; line-height: 1.62; color: var(--muted); }
pre .c { color: var(--dim); }
pre .k { color: var(--accent); }
table { border-collapse: collapse; margin: 1rem 0 1.4rem; width: 100%; font-size: 13px; }
th, td { text-align: left; padding: 7px 16px 7px 0; border-bottom: 1px solid var(--border);
color: var(--muted); vertical-align: top; }
th { color: var(--dim); font-size: 10px; text-transform: uppercase; letter-spacing: .06em;
font-weight: 600; }
td code { white-space: nowrap; }
blockquote {
border-left: 2px solid var(--border-strong); padding: .15rem 0 .15rem 1.1rem;
margin: 1.1rem 0; color: var(--dim); font-style: italic;
}
/* a claim worth not losing in the prose */
.note {
border: 1px solid var(--border); border-left: 3px solid var(--accent);
background: var(--surface); border-radius: 6px;
padding: .85rem 1.1rem; margin: 1.2rem 0; font-size: 13px;
}
.note b { color: var(--accent); }
.note.warn { border-left-color: var(--artery); }
.note.warn b { color: var(--artery); }
/* figures — inline and scaled, click for the viewer */
figure { margin: 1.3rem 0 1.7rem; }
figure a { display: block; border: 1px solid var(--border); border-radius: 8px;
overflow: hidden; background: var(--surface-2); }
figure a:hover { border-color: var(--accent); }
figure img { display: block; width: 100%; height: auto; }
figcaption { color: var(--dim); font-size: 11px; margin-top: .45rem; }
.pill {
display: inline-block; font-size: 10px; letter-spacing: .04em;
border: 1px solid var(--border-strong); border-radius: 20px;
padding: 1px 9px; color: var(--dim); margin-left: .5em; vertical-align: middle;
}
.pill.on { color: var(--atlas); border-color: var(--atlas); }
.pill.opt { color: var(--accent); border-color: var(--accent); }
.cols { display: flex; gap: 2rem; flex-wrap: wrap; }
.cols > div { flex: 1 1 250px; }

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 83 KiB

View File

@@ -0,0 +1,103 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="870pt" height="474pt" viewBox="0 0 870 474">
<rect width="870" height="474" fill="#0a0a0a"/>
<defs>
<marker id="fk" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#1d4ed8"/></marker>
</defs>
<g class="relationships">
<path d="M 330,103.0 C 280,103.0 300,155.0 250,155.0" fill="none" stroke="#1d4ed8" stroke-width="1.5" marker-end="url(#fk)" class="edge foreign_key"/>
<path d="M 620,155.0 C 570,155.0 590,155.0 540,155.0" fill="none" stroke="#1d4ed8" stroke-width="1.5" marker-end="url(#fk)" class="edge foreign_key"/>
<path d="M 250,369.0 C 300,369.0 280,155.0 330,155.0" fill="none" stroke="#1d4ed8" stroke-width="1.5" marker-end="url(#fk)" class="edge foreign_key"/>
</g>
<g class="table">
<rect x="40" y="40" width="210" height="154" rx="8" fill="#0a0a0a" stroke="#333333" stroke-width="1" data-id="Customer" data-kind="table" class="blk"/>
<path d="M 40,48 a 8,8 0 0 1 8,-8 h 194 a 8,8 0 0 1 8,8 v 42 h -210 z" fill="#1a1a1a"/>
<line x1="40" y1="90" x2="250" y2="90" stroke="#333333" stroke-width="1"/>
<text x="52" y="62" font-family="Helvetica,sans-Serif" font-size="12" font-weight="bold" fill="#e5e5e5">Customer</text>
<text x="52" y="78" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">A fixture customer (obviously…</text>
<text x="80" y="107" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">created_at</text>
<text x="238" y="107" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">datetime</text>
<line x1="41" y1="116" x2="249" y2="116" stroke="#1a1a1a" stroke-width="1"/>
<text x="80" y="133" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">email</text>
<text x="238" y="133" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">str</text>
<line x1="41" y1="142" x2="249" y2="142" stroke="#1a1a1a" stroke-width="1"/>
<text x="52" y="159" font-family="Helvetica,sans-Serif" font-size="8" font-weight="bold" fill="#d4a574">PK</text>
<text x="80" y="159" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">id</text>
<text x="238" y="159" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">int</text>
<line x1="41" y1="168" x2="249" y2="168" stroke="#1a1a1a" stroke-width="1"/>
<text x="80" y="185" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">name</text>
<text x="238" y="185" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">str</text>
</g>
<g class="table">
<rect x="330" y="40" width="210" height="206" rx="8" fill="#0a0a0a" stroke="#333333" stroke-width="1" data-id="Invoice" data-kind="table" class="blk"/>
<path d="M 330,48 a 8,8 0 0 1 8,-8 h 194 a 8,8 0 0 1 8,8 v 42 h -210 z" fill="#1a1a1a"/>
<line x1="330" y1="90" x2="540" y2="90" stroke="#333333" stroke-width="1"/>
<text x="342" y="62" font-family="Helvetica,sans-Serif" font-size="12" font-weight="bold" fill="#e5e5e5">Invoice</text>
<text x="342" y="78" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">An invoice issued to a custom…</text>
<text x="342" y="107" font-family="Helvetica,sans-Serif" font-size="8" font-weight="bold" fill="#1d4ed8">FK</text>
<text x="370" y="107" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">customer_id</text>
<text x="528" y="107" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">Customer</text>
<line x1="331" y1="116" x2="539" y2="116" stroke="#1a1a1a" stroke-width="1"/>
<text x="370" y="133" font-family="Helvetica,sans-Serif" font-size="10" fill="#a3a3a3">due_at</text>
<text x="528" y="133" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">datetime</text>
<line x1="331" y1="142" x2="539" y2="142" stroke="#1a1a1a" stroke-width="1"/>
<text x="342" y="159" font-family="Helvetica,sans-Serif" font-size="8" font-weight="bold" fill="#d4a574">PK</text>
<text x="370" y="159" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">id</text>
<text x="528" y="159" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">int</text>
<line x1="331" y1="168" x2="539" y2="168" stroke="#1a1a1a" stroke-width="1"/>
<text x="370" y="185" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">issued_at</text>
<text x="528" y="185" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">datetime</text>
<line x1="331" y1="194" x2="539" y2="194" stroke="#1a1a1a" stroke-width="1"/>
<text x="370" y="211" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">number</text>
<text x="528" y="211" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">str</text>
<line x1="331" y1="220" x2="539" y2="220" stroke="#1a1a1a" stroke-width="1"/>
<text x="370" y="237" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">status</text>
<text x="528" y="237" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">str</text>
</g>
<g class="table">
<rect x="620" y="40" width="210" height="180" rx="8" fill="#0a0a0a" stroke="#333333" stroke-width="1" data-id="LineItem" data-kind="table" class="blk"/>
<path d="M 620,48 a 8,8 0 0 1 8,-8 h 194 a 8,8 0 0 1 8,8 v 42 h -210 z" fill="#1a1a1a"/>
<line x1="620" y1="90" x2="830" y2="90" stroke="#333333" stroke-width="1"/>
<text x="632" y="62" font-family="Helvetica,sans-Serif" font-size="12" font-weight="bold" fill="#e5e5e5">LineItem</text>
<text x="632" y="78" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">A single billable line on an …</text>
<text x="660" y="107" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">description</text>
<text x="818" y="107" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">str</text>
<line x1="621" y1="116" x2="829" y2="116" stroke="#1a1a1a" stroke-width="1"/>
<text x="632" y="133" font-family="Helvetica,sans-Serif" font-size="8" font-weight="bold" fill="#d4a574">PK</text>
<text x="660" y="133" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">id</text>
<text x="818" y="133" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">int</text>
<line x1="621" y1="142" x2="829" y2="142" stroke="#1a1a1a" stroke-width="1"/>
<text x="632" y="159" font-family="Helvetica,sans-Serif" font-size="8" font-weight="bold" fill="#1d4ed8">FK</text>
<text x="660" y="159" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">invoice_id</text>
<text x="818" y="159" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">Invoice</text>
<line x1="621" y1="168" x2="829" y2="168" stroke="#1a1a1a" stroke-width="1"/>
<text x="660" y="185" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">quantity</text>
<text x="818" y="185" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">int</text>
<line x1="621" y1="194" x2="829" y2="194" stroke="#1a1a1a" stroke-width="1"/>
<text x="660" y="211" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">unit_price</text>
<text x="818" y="211" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">Decimal</text>
</g>
<g class="table">
<rect x="40" y="254" width="210" height="180" rx="8" fill="#0a0a0a" stroke="#333333" stroke-width="1" data-id="Payment" data-kind="table" class="blk"/>
<path d="M 40,262 a 8,8 0 0 1 8,-8 h 194 a 8,8 0 0 1 8,8 v 42 h -210 z" fill="#1a1a1a"/>
<line x1="40" y1="304" x2="250" y2="304" stroke="#333333" stroke-width="1"/>
<text x="52" y="276" font-family="Helvetica,sans-Serif" font-size="12" font-weight="bold" fill="#e5e5e5">Payment</text>
<text x="52" y="292" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">A payment recorded against an…</text>
<text x="80" y="321" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">amount</text>
<text x="238" y="321" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">Decimal</text>
<line x1="41" y1="330" x2="249" y2="330" stroke="#1a1a1a" stroke-width="1"/>
<text x="52" y="347" font-family="Helvetica,sans-Serif" font-size="8" font-weight="bold" fill="#d4a574">PK</text>
<text x="80" y="347" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">id</text>
<text x="238" y="347" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">int</text>
<line x1="41" y1="356" x2="249" y2="356" stroke="#1a1a1a" stroke-width="1"/>
<text x="52" y="373" font-family="Helvetica,sans-Serif" font-size="8" font-weight="bold" fill="#1d4ed8">FK</text>
<text x="80" y="373" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">invoice_id</text>
<text x="238" y="373" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">Invoice</text>
<line x1="41" y1="382" x2="249" y2="382" stroke="#1a1a1a" stroke-width="1"/>
<text x="80" y="399" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">method</text>
<text x="238" y="399" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">str</text>
<line x1="41" y1="408" x2="249" y2="408" stroke="#1a1a1a" stroke-width="1"/>
<text x="80" y="425" font-family="Helvetica,sans-Serif" font-size="10" fill="#e5e5e5">paid_at</text>
<text x="238" y="425" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">datetime</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 9.4 KiB

View File

@@ -0,0 +1,386 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg xmlns="http://www.w3.org/2000/svg" width="886pt" height="2251pt" viewBox="0 0 886 2251">
<rect width="886" height="2251" fill="#0a0a0a"/>
<rect x="28.0" y="46.0" width="74.0" height="247.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.site — 494 lines</title></rect>
<rect x="28.0" y="182.0" width="74.0" height="27.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.site._ledger" data-kind="function" class="blk"><title>_ledger — function, 54 lines</title></rect>
<rect x="28.0" y="210.0" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.site._slots" data-kind="function" class="blk"><title>_slots — function, 13 lines</title></rect>
<rect x="28.0" y="217.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.site._fill" data-kind="function" class="blk"><title>_fill — function, 4 lines</title></rect>
<rect x="28.0" y="220.5" width="74.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.site._sidebar" data-kind="function" class="blk"><title>_sidebar — function, 17 lines</title></rect>
<rect x="28.0" y="230.0" width="74.0" height="7.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.site._sections" data-kind="function" class="blk"><title>_sections — function, 15 lines</title></rect>
<rect x="28.0" y="238.5" width="74.0" height="48.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.site.emit" data-kind="function" class="blk"><title>emit — function, 96 lines</title></rect>
<rect x="28.0" y="287.5" width="74.0" height="5.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.site.write" data-kind="function" class="blk"><title>write — function, 10 lines</title></rect>
<rect x="110.0" y="46.0" width="74.0" height="158.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.explore — 316 lines</title></rect>
<rect x="110.0" y="65.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.explore._is_schema" data-kind="function" class="blk"><title>_is_schema — function, 4 lines</title></rect>
<rect x="110.0" y="68.5" width="74.0" height="24.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.explore._neighbourhood_svgs" data-kind="function" class="blk"><title>_neighbourhood_svgs — function, 48 lines</title></rect>
<rect x="117.0" y="75.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.explore._neighbourhood_svgs.render_one" data-kind="function" class="blk"><title>render_one — function, 2 lines</title></rect>
<rect x="117.0" y="78.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.explore._neighbourhood_svgs.render_one#L66" data-kind="function" class="blk"><title>render_one — function, 2 lines</title></rect>
<rect x="110.0" y="93.5" width="74.0" height="13.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.explore._facts" data-kind="function" class="blk"><title>_facts — function, 27 lines</title></rect>
<rect x="110.0" y="108.0" width="74.0" height="86.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.explore.emit" data-kind="function" class="blk"><title>emit — function, 172 lines</title></rect>
<rect x="110.0" y="195.0" width="74.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.explore.write" data-kind="function" class="blk"><title>write — function, 17 lines</title></rect>
<rect x="192.0" y="46.0" width="74.0" height="146.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.dot — 292 lines</title></rect>
<rect x="192.0" y="64.5" width="74.0" height="2.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.emitters.dot.RenderError" data-kind="class" class="blk"><title>RenderError — class, 2 lines</title></rect>
<rect x="192.0" y="66.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._esc" data-kind="function" class="blk"><title>_esc — function, 2 lines</title></rect>
<rect x="192.0" y="68.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._attrs" data-kind="function" class="blk"><title>_attrs — function, 3 lines</title></rect>
<rect x="192.0" y="71.0" width="74.0" height="4.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._style_words" data-kind="function" class="blk"><title>_style_words — function, 9 lines</title></rect>
<rect x="192.0" y="76.5" width="74.0" height="12.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._node_attrs" data-kind="function" class="blk"><title>_node_attrs — function, 24 lines</title></rect>
<rect x="192.0" y="89.5" width="74.0" height="52.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot.emit" data-kind="function" class="blk"><title>emit — function, 105 lines</title></rect>
<rect x="199.0" y="103.0" width="60.0" height="14.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.dot.emit.write" data-kind="function" class="blk"><title>write — function, 29 lines</title></rect>
<rect x="192.0" y="143.0" width="74.0" height="4.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._within" data-kind="function" class="blk"><title>_within — function, 8 lines</title></rect>
<rect x="192.0" y="148.0" width="74.0" height="20.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._endpoints" data-kind="function" class="blk"><title>_endpoints — function, 41 lines</title></rect>
<rect x="199.0" y="153.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.dot._endpoints.walk" data-kind="function" class="blk"><title>walk — function, 4 lines</title></rect>
<rect x="199.0" y="157.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.dot._endpoints.collapsed" data-kind="function" class="blk"><title>collapsed — function, 2 lines</title></rect>
<rect x="199.0" y="158.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.dot._endpoints.first_leaf" data-kind="function" class="blk"><title>first_leaf — function, 4 lines</title></rect>
<rect x="192.0" y="169.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._safe" data-kind="function" class="blk"><title>_safe — function, 2 lines</title></rect>
<rect x="192.0" y="171.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot._q" data-kind="function" class="blk"><title>_q — function, 2 lines</title></rect>
<rect x="192.0" y="175.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot.have_graphviz" data-kind="function" class="blk"><title>have_graphviz — function, 2 lines</title></rect>
<rect x="192.0" y="177.0" width="74.0" height="14.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.dot.render" data-kind="function" class="blk"><title>render — function, 29 lines</title></rect>
<rect x="274.0" y="46.0" width="74.0" height="139.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.notebook — 279 lines</title></rect>
<rect x="274.0" y="67.0" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook._cell" data-kind="function" class="blk"><title>_cell — function, 13 lines</title></rect>
<rect x="274.0" y="74.5" width="74.0" height="13.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook._example" data-kind="function" class="blk"><title>_example — function, 26 lines</title></rect>
<rect x="274.0" y="108.0" width="74.0" height="9.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook._params_cell" data-kind="function" class="blk"><title>_params_cell — function, 18 lines</title></rect>
<rect x="274.0" y="118.0" width="74.0" height="20.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook._call_cell" data-kind="function" class="blk"><title>_call_cell — function, 40 lines</title></rect>
<rect x="274.0" y="139.0" width="74.0" height="13.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook._call_md" data-kind="function" class="blk"><title>_call_md — function, 26 lines</title></rect>
<rect x="274.0" y="153.0" width="74.0" height="26.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook.build" data-kind="function" class="blk"><title>build — function, 52 lines</title></rect>
<rect x="274.0" y="180.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook.emit" data-kind="function" class="blk"><title>emit — function, 3 lines</title></rect>
<rect x="274.0" y="182.5" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.notebook.write" data-kind="function" class="blk"><title>write — function, 5 lines</title></rect>
<rect x="356.0" y="46.0" width="74.0" height="139.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.minimap — 278 lines</title></rect>
<rect x="356.0" y="77.5" width="74.0" height="28.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.minimap._files" data-kind="function" class="blk"><title>_files — function, 57 lines</title></rect>
<rect x="363.0" y="81.5" width="60.0" height="9.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.minimap._files.declared" data-kind="function" class="blk"><title>declared — function, 18 lines</title></rect>
<rect x="363.0" y="91.0" width="60.0" height="5.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.minimap._files.build" data-kind="function" class="blk"><title>build — function, 11 lines</title></rect>
<rect x="356.0" y="107.0" width="74.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.minimap._bands" data-kind="function" class="blk"><title>_bands — function, 7 lines</title></rect>
<rect x="356.0" y="111.5" width="74.0" height="9.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.minimap._blocks" data-kind="function" class="blk"><title>_blocks — function, 19 lines</title></rect>
<rect x="356.0" y="122.0" width="74.0" height="60.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.minimap.emit" data-kind="function" class="blk"><title>emit — function, 120 lines</title></rect>
<rect x="356.0" y="183.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.minimap.marks_to_labels" data-kind="function" class="blk"><title>marks_to_labels — function, 3 lines</title></rect>
<rect x="438.0" y="46.0" width="74.0" height="130.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.erd — 260 lines</title></rect>
<rect x="438.0" y="73.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.erd._truncate" data-kind="function" class="blk"><title>_truncate — function, 3 lines</title></rect>
<rect x="438.0" y="75.5" width="74.0" height="10.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.erd._tables" data-kind="function" class="blk"><title>_tables — function, 20 lines</title></rect>
<rect x="438.0" y="86.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.erd._card_height" data-kind="function" class="blk"><title>_card_height — function, 3 lines</title></rect>
<rect x="438.0" y="89.0" width="74.0" height="10.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.erd.layout" data-kind="function" class="blk"><title>layout — function, 20 lines</title></rect>
<rect x="438.0" y="100.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.erd._field_y" data-kind="function" class="blk"><title>_field_y — function, 3 lines</title></rect>
<rect x="438.0" y="102.5" width="74.0" height="73.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.erd.emit" data-kind="function" class="blk"><title>emit — function, 146 lines</title></rect>
<rect x="520.0" y="46.0" width="74.0" height="82.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.index — 164 lines</title></rect>
<rect x="520.0" y="63.0" width="74.0" height="4.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.index._tree" data-kind="function" class="blk"><title>_tree — function, 8 lines</title></rect>
<rect x="520.0" y="68.0" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.index._anchor" data-kind="function" class="blk"><title>_anchor — function, 5 lines</title></rect>
<rect x="520.0" y="71.5" width="74.0" height="42.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.index.to_markdown" data-kind="function" class="blk"><title>to_markdown — function, 84 lines</title></rect>
<rect x="527.0" y="83.0" width="60.0" height="13.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.index.to_markdown.walk" data-kind="function" class="blk"><title>walk — function, 27 lines</title></rect>
<rect x="520.0" y="114.5" width="74.0" height="13.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.index.to_sidebar" data-kind="function" class="blk"><title>to_sidebar — function, 26 lines</title></rect>
<rect x="527.0" y="118.0" width="60.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.emitters.index.to_sidebar.build" data-kind="function" class="blk"><title>build — function, 13 lines</title></rect>
<rect x="602.0" y="46.0" width="74.0" height="43.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.cli_dot — 87 lines</title></rect>
<rect x="602.0" y="52.5" width="74.0" height="36.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.cli_dot.main" data-kind="function" class="blk"><title>main — function, 73 lines</title></rect>
<rect x="684.0" y="46.0" width="74.0" height="40.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.auto — 80 lines</title></rect>
<rect x="684.0" y="57.5" width="74.0" height="28.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.auto.main" data-kind="function" class="blk"><title>main — function, 56 lines</title></rect>
<rect x="766.0" y="46.0" width="74.0" height="37.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.cli_notebook — 74 lines</title></rect>
<rect x="766.0" y="54.0" width="74.0" height="28.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.cli_notebook.main" data-kind="function" class="blk"><title>main — function, 57 lines</title></rect>
<rect x="28.0" y="331.0" width="74.0" height="37.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.cli_site — 74 lines</title></rect>
<rect x="28.0" y="339.0" width="74.0" height="28.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.cli_site.main" data-kind="function" class="blk"><title>main — function, 57 lines</title></rect>
<rect x="110.0" y="331.0" width="74.0" height="25.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.cli_erd — 51 lines</title></rect>
<rect x="110.0" y="337.0" width="74.0" height="19.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.cli_erd.main" data-kind="function" class="blk"><title>main — function, 38 lines</title></rect>
<rect x="192.0" y="331.0" width="74.0" height="25.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.cli_minimap — 51 lines</title></rect>
<rect x="192.0" y="337.5" width="74.0" height="18.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.cli_minimap.main" data-kind="function" class="blk"><title>main — function, 37 lines</title></rect>
<rect x="274.0" y="331.0" width="74.0" height="24.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.cli_explore — 48 lines</title></rect>
<rect x="274.0" y="337.0" width="74.0" height="17.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.cli_explore.main" data-kind="function" class="blk"><title>main — function, 35 lines</title></rect>
<rect x="356.0" y="331.0" width="74.0" height="22.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.cli_index — 44 lines</title></rect>
<rect x="356.0" y="336.5" width="74.0" height="16.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.cli_index.main" data-kind="function" class="blk"><title>main — function, 32 lines</title></rect>
<rect x="438.0" y="331.0" width="74.0" height="18.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters.__main__ — 37 lines</title></rect>
<rect x="438.0" y="333.5" width="74.0" height="13.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.emitters.__main__.main" data-kind="function" class="blk"><title>main — function, 27 lines</title></rect>
<rect x="538.0" y="331.0" width="74.0" height="1007.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.selftest — 2014 lines</title></rect>
<rect x="538.0" y="375.5" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest.check" data-kind="function" class="blk"><title>check — function, 5 lines</title></rect>
<rect x="538.0" y="379.0" width="74.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest._err" data-kind="function" class="blk"><title>_err — function, 7 lines</title></rect>
<rect x="538.0" y="383.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest.skip" data-kind="function" class="blk"><title>skip — function, 3 lines</title></rect>
<rect x="538.0" y="386.0" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest.build_tree" data-kind="function" class="blk"><title>build_tree — function, 5 lines</title></rect>
<rect x="538.0" y="515.5" width="74.0" height="97.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest._harvesting" data-kind="function" class="blk"><title>_harvesting — function, 194 lines</title></rect>
<rect x="538.0" y="858.0" width="74.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest._entry" data-kind="function" class="blk"><title>_entry — function, 7 lines</title></rect>
<rect x="538.0" y="1111.0" width="74.0" height="139.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest._books" data-kind="function" class="blk"><title>_books — function, 279 lines</title></rect>
<rect x="538.0" y="1255.5" width="74.0" height="76.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.selftest._standalone" data-kind="function" class="blk"><title>_standalone — function, 153 lines</title></rect>
<rect x="620.0" y="331.0" width="74.0" height="158.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.book — 316 lines</title></rect>
<rect x="620.0" y="360.0" width="74.0" height="10.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.book.Step" data-kind="class" class="blk"><title>Step — class, 20 lines</title></rect>
<rect x="627.0" y="365.0" width="60.0" height="5.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Step.to_dict" data-kind="function" class="blk"><title>to_dict — function, 10 lines</title></rect>
<rect x="620.0" y="381.5" width="74.0" height="15.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book._unit_counts" data-kind="function" class="blk"><title>_unit_counts — function, 30 lines</title></rect>
<rect x="620.0" y="397.5" width="74.0" height="89.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.book.Book" data-kind="class" class="blk"><title>Book — class, 178 lines</title></rect>
<rect x="627.0" y="399.0" width="60.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Book.__init__" data-kind="function" class="blk"><title>__init__ — function, 17 lines</title></rect>
<rect x="627.0" y="409.0" width="60.0" height="7.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Book.step" data-kind="function" class="blk"><title>step — function, 14 lines</title></rect>
<rect x="627.0" y="417.5" width="60.0" height="10.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Book.measure" data-kind="function" class="blk"><title>measure — function, 21 lines</title></rect>
<rect x="627.0" y="428.5" width="60.0" height="31.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Book.compare" data-kind="function" class="blk"><title>compare — function, 62 lines</title></rect>
<rect x="627.0" y="461.0" width="60.0" height="13.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Book.close" data-kind="function" class="blk"><title>close — function, 27 lines</title></rect>
<rect x="627.0" y="475.0" width="60.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Book.to_dict" data-kind="function" class="blk"><title>to_dict — function, 17 lines</title></rect>
<rect x="627.0" y="484.0" width="60.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.Book.write" data-kind="function" class="blk"><title>write — function, 5 lines</title></rect>
<rect x="620.0" y="487.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book._tree_bytes" data-kind="function" class="blk"><title>_tree_bytes — function, 2 lines</title></rect>
<rect x="702.0" y="331.0" width="74.0" height="97.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.style — 194 lines</title></rect>
<rect x="702.0" y="353.5" width="74.0" height="2.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.style.StyleError" data-kind="class" class="blk"><title>StyleError — class, 2 lines</title></rect>
<rect x="702.0" y="355.5" width="74.0" height="66.5" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.style.Style" data-kind="class" class="blk"><title>Style — class, 133 lines</title></rect>
<rect x="709.0" y="357.0" width="60.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.__init__" data-kind="function" class="blk"><title>__init__ — function, 17 lines</title></rect>
<rect x="709.0" y="367.5" width="60.0" height="6.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.load" data-kind="function" class="blk"><title>load — function, 12 lines</title></rect>
<rect x="709.0" y="374.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.available" data-kind="function" class="blk"><title>available — function, 2 lines</title></rect>
<rect x="709.0" y="376.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.themes" data-kind="function" class="blk"><title>themes — function, 2 lines</title></rect>
<rect x="709.0" y="378.5" width="60.0" height="16.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.validate" data-kind="function" class="blk"><title>validate — function, 32 lines</title></rect>
<rect x="709.0" y="396.0" width="60.0" height="6.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style._resolve" data-kind="function" class="blk"><title>_resolve — function, 12 lines</title></rect>
<rect x="709.0" y="402.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style._lookup" data-kind="function" class="blk"><title>_lookup — function, 3 lines</title></rect>
<rect x="709.0" y="404.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.node" data-kind="function" class="blk"><title>node — function, 2 lines</title></rect>
<rect x="709.0" y="406.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.group" data-kind="function" class="blk"><title>group — function, 2 lines</title></rect>
<rect x="709.0" y="407.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.edge" data-kind="function" class="blk"><title>edge — function, 2 lines</title></rect>
<rect x="709.0" y="409.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.graph" data-kind="function" class="blk"><title>graph — function, 2 lines</title></rect>
<rect x="709.0" y="410.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.geom" data-kind="function" class="blk"><title>geom — function, 2 lines</title></rect>
<rect x="709.0" y="412.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.slot" data-kind="function" class="blk"><title>slot — function, 2 lines</title></rect>
<rect x="709.0" y="413.5" width="60.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.domain_slot" data-kind="function" class="blk"><title>domain_slot — function, 13 lines</title></rect>
<rect x="709.0" y="420.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.Style.limits" data-kind="function" class="blk"><title>limits — function, 3 lines</title></rect>
<rect x="702.0" y="423.0" width="74.0" height="4.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.harvest" data-kind="function" class="blk"><title>harvest — function, 9 lines</title></rect>
<rect x="784.0" y="331.0" width="74.0" height="54.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.reference — 108 lines</title></rect>
<rect x="784.0" y="354.0" width="74.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.reference._candidates" data-kind="function" class="blk"><title>_candidates — function, 7 lines</title></rect>
<rect x="784.0" y="358.5" width="74.0" height="4.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.reference.root" data-kind="function" class="blk"><title>root — function, 9 lines</title></rect>
<rect x="784.0" y="364.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.reference.station_tools" data-kind="function" class="blk"><title>station_tools — function, 4 lines</title></rect>
<rect x="784.0" y="367.0" width="74.0" height="5.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.reference.describe" data-kind="function" class="blk"><title>describe — function, 10 lines</title></rect>
<rect x="784.0" y="373.0" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.reference.on_path" data-kind="function" class="blk"><title>on_path — function, 13 lines</title></rect>
<rect x="784.0" y="380.5" width="74.0" height="4.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.reference.missing" data-kind="function" class="blk"><title>missing — function, 8 lines</title></rect>
<rect x="28.0" y="1376.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.ops — 28 lines</title></rect>
<rect x="110.0" y="1376.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.notebook — 15 lines</title></rect>
<rect x="192.0" y="1376.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.emitters — 12 lines</title></rect>
<rect x="274.0" y="1376.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.lab — 11 lines</title></rect>
<rect x="356.0" y="1376.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.ir — 7 lines</title></rect>
<rect x="438.0" y="1376.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors — 2 lines</title></rect>
<rect x="538.0" y="1376.0" width="74.0" height="141.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.code — 282 lines</title></rect>
<rect x="538.0" y="1423.5" width="74.0" height="2.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.extractors.code.MissingParser" data-kind="class" class="blk"><title>MissingParser — class, 2 lines</title></rect>
<rect x="538.0" y="1425.5" width="74.0" height="10.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.code._parser" data-kind="function" class="blk"><title>_parser — function, 21 lines</title></rect>
<rect x="538.0" y="1437.0" width="74.0" height="5.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.code._name" data-kind="function" class="blk"><title>_name — function, 10 lines</title></rect>
<rect x="538.0" y="1443.0" width="74.0" height="17.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.code._walk" data-kind="function" class="blk"><title>_walk — function, 34 lines</title></rect>
<rect x="538.0" y="1461.0" width="74.0" height="13.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.code.extract_file" data-kind="function" class="blk"><title>extract_file — function, 27 lines</title></rect>
<rect x="538.0" y="1475.5" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.code._count_errors" data-kind="function" class="blk"><title>_count_errors — function, 5 lines</title></rect>
<rect x="538.0" y="1479.0" width="74.0" height="37.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.code.extract" data-kind="function" class="blk"><title>extract — function, 75 lines</title></rect>
<rect x="620.0" y="1376.0" width="74.0" height="134.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.usage — 269 lines</title></rect>
<rect x="620.0" y="1408.0" width="74.0" height="13.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.usage._template" data-kind="function" class="blk"><title>_template — function, 27 lines</title></rect>
<rect x="620.0" y="1422.5" width="74.0" height="5.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.usage._body" data-kind="function" class="blk"><title>_body — function, 10 lines</title></rect>
<rect x="620.0" y="1428.5" width="74.0" height="7.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.usage._shape" data-kind="function" class="blk"><title>_shape — function, 15 lines</title></rect>
<rect x="620.0" y="1437.0" width="74.0" height="5.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.usage._graphql" data-kind="function" class="blk"><title>_graphql — function, 11 lines</title></rect>
<rect x="620.0" y="1443.5" width="74.0" height="66.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.usage.extract" data-kind="function" class="blk"><title>extract — function, 133 lines</title></rect>
<rect x="702.0" y="1376.0" width="74.0" height="95.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.openapi — 191 lines</title></rect>
<rect x="702.0" y="1391.5" width="74.0" height="14.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.openapi._modelgen" data-kind="function" class="blk"><title>_modelgen — function, 28 lines</title></rect>
<rect x="702.0" y="1406.5" width="74.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.openapi._type_name" data-kind="function" class="blk"><title>_type_name — function, 6 lines</title></rect>
<rect x="702.0" y="1410.5" width="74.0" height="21.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.openapi._refs" data-kind="function" class="blk"><title>_refs — function, 43 lines</title></rect>
<rect x="702.0" y="1433.0" width="74.0" height="38.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.openapi.extract" data-kind="function" class="blk"><title>extract — function, 76 lines</title></rect>
<rect x="784.0" y="1376.0" width="74.0" height="80.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.db — 161 lines</title></rect>
<rect x="784.0" y="1395.5" width="74.0" height="41.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.db.from_schema_dict" data-kind="function" class="blk"><title>from_schema_dict — function, 82 lines</title></rect>
<rect x="784.0" y="1437.5" width="74.0" height="5.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.db._relation" data-kind="function" class="blk"><title>_relation — function, 10 lines</title></rect>
<rect x="784.0" y="1443.5" width="74.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.db._plain_type" data-kind="function" class="blk"><title>_plain_type — function, 6 lines</title></rect>
<rect x="784.0" y="1447.5" width="74.0" height="4.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.db._dedupe" data-kind="function" class="blk"><title>_dedupe — function, 9 lines</title></rect>
<rect x="784.0" y="1453.0" width="74.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.db.extract" data-kind="function" class="blk"><title>extract — function, 6 lines</title></rect>
<rect x="28.0" y="1555.0" width="74.0" height="20.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.code_main — 40 lines</title></rect>
<rect x="28.0" y="1559.0" width="74.0" height="15.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.code_main.main" data-kind="function" class="blk"><title>main — function, 31 lines</title></rect>
<rect x="110.0" y="1555.0" width="74.0" height="18.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.python — 37 lines</title></rect>
<rect x="110.0" y="1564.5" width="74.0" height="7.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.extract" data-kind="function" class="blk"><title>extract — function, 14 lines</title></rect>
<rect x="192.0" y="1555.0" width="74.0" height="16.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.usage_main — 33 lines</title></rect>
<rect x="192.0" y="1559.0" width="74.0" height="12.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.usage_main.main" data-kind="function" class="blk"><title>main — function, 24 lines</title></rect>
<rect x="274.0" y="1555.0" width="74.0" height="16.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.db_main — 32 lines</title></rect>
<rect x="274.0" y="1560.0" width="74.0" height="10.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.db_main.main" data-kind="function" class="blk"><title>main — function, 21 lines</title></rect>
<rect x="356.0" y="1555.0" width="74.0" height="15.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.openapi_main — 30 lines</title></rect>
<rect x="356.0" y="1559.0" width="74.0" height="10.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.openapi_main.main" data-kind="function" class="blk"><title>main — function, 21 lines</title></rect>
<rect x="438.0" y="1555.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.__main__ — 26 lines</title></rect>
<rect x="438.0" y="1557.7" width="74.0" height="8.6" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.__main__.main" data-kind="function" class="blk"><title>main — function, 16 lines</title></rect>
<rect x="538.0" y="1555.0" width="74.0" height="175.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.book.build — 351 lines</title></rect>
<rect x="538.0" y="1592.5" width="74.0" height="13.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.build.extract" data-kind="function" class="blk"><title>extract — function, 27 lines</title></rect>
<rect x="538.0" y="1607.0" width="74.0" height="34.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.build.spec_from" data-kind="function" class="blk"><title>spec_from — function, 68 lines</title></rect>
<rect x="538.0" y="1642.0" width="74.0" height="22.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.build._load_cell" data-kind="function" class="blk"><title>_load_cell — function, 44 lines</title></rect>
<rect x="538.0" y="1665.0" width="74.0" height="65.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.build.run" data-kind="function" class="blk"><title>run — function, 130 lines</title></rect>
<rect x="545.0" y="1671.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.build.run.say" data-kind="function" class="blk"><title>say — function, 3 lines</title></rect>
<rect x="620.0" y="1555.0" width="74.0" height="153.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.book.checks — 306 lines</title></rect>
<rect x="620.0" y="1585.5" width="74.0" height="13.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.book.checks.Loaded" data-kind="class" class="blk"><title>Loaded — class, 26 lines</title></rect>
<rect x="627.0" y="1587.0" width="60.0" height="7.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Loaded.__init__" data-kind="function" class="blk"><title>__init__ — function, 15 lines</title></rect>
<rect x="627.0" y="1595.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Loaded.larder" data-kind="function" class="blk"><title>larder — function, 2 lines</title></rect>
<rect x="627.0" y="1597.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Loaded.measure" data-kind="function" class="blk"><title>measure — function, 2 lines</title></rect>
<rect x="620.0" y="1599.5" width="74.0" height="18.5" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.book.checks.Report" data-kind="class" class="blk"><title>Report — class, 37 lines</title></rect>
<rect x="627.0" y="1601.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Report.__init__" data-kind="function" class="blk"><title>__init__ — function, 2 lines</title></rect>
<rect x="627.0" y="1602.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Report.note" data-kind="function" class="blk"><title>note — function, 2 lines</title></rect>
<rect x="627.0" y="1604.0" width="60.0" height="4.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Report.check" data-kind="function" class="blk"><title>check — function, 8 lines</title></rect>
<rect x="627.0" y="1608.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Report.skip" data-kind="function" class="blk"><title>skip — function, 3 lines</title></rect>
<rect x="627.0" y="1610.5" width="60.0" height="7.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.checks.Report.total" data-kind="function" class="blk"><title>total — function, 15 lines</title></rect>
<rect x="620.0" y="1619.0" width="74.0" height="41.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.checks.generated" data-kind="function" class="blk"><title>generated — function, 83 lines</title></rect>
<rect x="620.0" y="1661.5" width="74.0" height="13.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.checks.custom" data-kind="function" class="blk"><title>custom — function, 26 lines</title></rect>
<rect x="620.0" y="1675.5" width="74.0" height="16.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.checks._run_cells" data-kind="function" class="blk"><title>_run_cells — function, 33 lines</title></rect>
<rect x="620.0" y="1693.0" width="74.0" height="12.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.checks.main" data-kind="function" class="blk"><title>main — function, 25 lines</title></rect>
<rect x="702.0" y="1555.0" width="74.0" height="98.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.book.larder — 196 lines</title></rect>
<rect x="702.0" y="1586.5" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.larder._count" data-kind="function" class="blk"><title>_count — function, 13 lines</title></rect>
<rect x="702.0" y="1594.0" width="74.0" height="16.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.larder.redact" data-kind="function" class="blk"><title>redact — function, 32 lines</title></rect>
<rect x="702.0" y="1611.5" width="74.0" height="38.5" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.book.larder.Larder" data-kind="class" class="blk"><title>Larder — class, 77 lines</title></rect>
<rect x="709.0" y="1618.5" width="60.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.larder.Larder.__post_init__" data-kind="function" class="blk"><title>__post_init__ — function, 7 lines</title></rect>
<rect x="709.0" y="1623.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.larder.Larder.read" data-kind="function" class="blk"><title>read — function, 3 lines</title></rect>
<rect x="709.0" y="1625.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.larder.Larder.fail" data-kind="function" class="blk"><title>fail — function, 3 lines</title></rect>
<rect x="709.0" y="1627.0" width="60.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.larder.Larder.to_dict" data-kind="function" class="blk"><title>to_dict — function, 13 lines</title></rect>
<rect x="709.0" y="1634.5" width="60.0" height="5.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.larder.Larder.from_dict" data-kind="function" class="blk"><title>from_dict — function, 10 lines</title></rect>
<rect x="709.0" y="1640.0" width="60.0" height="10.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.book.larder.Larder.line" data-kind="function" class="blk"><title>line — function, 20 lines</title></rect>
<rect x="702.0" y="1651.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.larder.of" data-kind="function" class="blk"><title>of — function, 3 lines</title></rect>
<rect x="784.0" y="1555.0" width="74.0" height="40.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.book.__main__ — 81 lines</title></rect>
<rect x="784.0" y="1562.0" width="74.0" height="31.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.book.__main__.main" data-kind="function" class="blk"><title>main — function, 62 lines</title></rect>
<rect x="28.0" y="1768.5" width="74.0" height="118.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.python.collect — 236 lines</title></rect>
<rect x="28.0" y="1785.0" width="74.0" height="5.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect.Definition" data-kind="class" class="blk"><title>Definition — class, 10 lines</title></rect>
<rect x="28.0" y="1791.5" width="74.0" height="6.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect.Module" data-kind="class" class="blk"><title>Module — class, 12 lines</title></rect>
<rect x="28.0" y="1798.5" width="74.0" height="33.5" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect._Collector" data-kind="class" class="blk"><title>_Collector — class, 67 lines</title></rect>
<rect x="35.0" y="1800.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.extractors.python.collect._Collector.__init__" data-kind="function" class="blk"><title>__init__ — function, 3 lines</title></rect>
<rect x="35.0" y="1803.0" width="60.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.extractors.python.collect._Collector._define" data-kind="function" class="blk"><title>_define — function, 17 lines</title></rect>
<rect x="35.0" y="1812.0" width="60.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.extractors.python.collect._Collector.visit_ClassDef" data-kind="function" class="blk"><title>visit_ClassDef — function, 5 lines</title></rect>
<rect x="35.0" y="1815.0" width="60.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.extractors.python.collect._Collector.visit_FunctionDef" data-kind="function" class="blk"><title>visit_FunctionDef — function, 5 lines</title></rect>
<rect x="35.0" y="1820.5" width="60.0" height="4.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.extractors.python.collect._Collector.visit_Import" data-kind="function" class="blk"><title>visit_Import — function, 8 lines</title></rect>
<rect x="35.0" y="1825.0" width="60.0" height="7.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.extractors.python.collect._Collector.visit_ImportFrom" data-kind="function" class="blk"><title>visit_ImportFrom — function, 14 lines</title></rect>
<rect x="28.0" y="1833.0" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect._first_line" data-kind="function" class="blk"><title>_first_line — function, 5 lines</title></rect>
<rect x="28.0" y="1836.5" width="74.0" height="7.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect._name_of" data-kind="function" class="blk"><title>_name_of — function, 15 lines</title></rect>
<rect x="28.0" y="1845.0" width="74.0" height="8.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect._resolve_relative" data-kind="function" class="blk"><title>_resolve_relative — function, 16 lines</title></rect>
<rect x="28.0" y="1854.0" width="74.0" height="13.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect.module_name" data-kind="function" class="blk"><title>module_name — function, 26 lines</title></rect>
<rect x="28.0" y="1868.0" width="74.0" height="10.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect.collect_file" data-kind="function" class="blk"><title>collect_file — function, 21 lines</title></rect>
<rect x="28.0" y="1879.5" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.collect.collect" data-kind="function" class="blk"><title>collect — function, 13 lines</title></rect>
<rect x="110.0" y="1768.5" width="74.0" height="92.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.python.resolve — 184 lines</title></rect>
<rect x="110.0" y="1782.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.resolve._id_for" data-kind="function" class="blk"><title>_id_for — function, 2 lines</title></rect>
<rect x="110.0" y="1784.5" width="74.0" height="16.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.resolve._resolve" data-kind="function" class="blk"><title>_resolve — function, 32 lines</title></rect>
<rect x="110.0" y="1801.5" width="74.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.resolve.larder_of" data-kind="function" class="blk"><title>larder_of — function, 17 lines</title></rect>
<rect x="110.0" y="1811.0" width="74.0" height="49.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.resolve.to_ir" data-kind="function" class="blk"><title>to_ir — function, 98 lines</title></rect>
<rect x="117.0" y="1840.5" width="60.0" height="4.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.extractors.python.resolve.to_ir._point_at" data-kind="function" class="blk"><title>_point_at — function, 9 lines</title></rect>
<rect x="192.0" y="1768.5" width="74.0" height="19.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.extractors.python.__main__ — 38 lines</title></rect>
<rect x="192.0" y="1773.5" width="74.0" height="11.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.extractors.python.__main__.main" data-kind="function" class="blk"><title>main — function, 23 lines</title></rect>
<rect x="292.0" y="1768.5" width="74.0" height="156.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.ir.validate — 313 lines</title></rect>
<rect x="292.0" y="1790.0" width="74.0" height="2.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.ir.validate.IRError" data-kind="class" class="blk"><title>IRError — class, 2 lines</title></rect>
<rect x="292.0" y="1792.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate._schema" data-kind="function" class="blk"><title>_schema — function, 2 lines</title></rect>
<rect x="292.0" y="1794.0" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate._props" data-kind="function" class="blk"><title>_props — function, 5 lines</title></rect>
<rect x="292.0" y="1797.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate._fields" data-kind="function" class="blk"><title>_fields — function, 4 lines</title></rect>
<rect x="292.0" y="1800.5" width="74.0" height="54.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate.check" data-kind="function" class="blk"><title>check — function, 108 lines</title></rect>
<rect x="292.0" y="1862.0" width="74.0" height="29.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate._check_larder" data-kind="function" class="blk"><title>_check_larder — function, 58 lines</title></rect>
<rect x="292.0" y="1892.0" width="74.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate.validate" data-kind="function" class="blk"><title>validate — function, 6 lines</title></rect>
<rect x="292.0" y="1896.0" width="74.0" height="11.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate.check_model_matches_schema" data-kind="function" class="blk"><title>check_model_matches_schema — function, 23 lines</title></rect>
<rect x="292.0" y="1908.5" width="74.0" height="16.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ir.validate.main" data-kind="function" class="blk"><title>main — function, 32 lines</title></rect>
<rect x="374.0" y="1768.5" width="74.0" height="82.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.ir.model — 165 lines</title></rect>
<rect x="374.0" y="1785.0" width="74.0" height="19.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.ir.model.Meta" data-kind="class" class="blk"><title>Meta — class, 38 lines</title></rect>
<rect x="381.0" y="1795.0" width="60.0" height="9.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Meta.to_dict" data-kind="function" class="blk"><title>to_dict — function, 18 lines</title></rect>
<rect x="374.0" y="1805.5" width="74.0" height="10.5" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.ir.model.Node" data-kind="class" class="blk"><title>Node — class, 21 lines</title></rect>
<rect x="381.0" y="1810.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Node.__post_init__" data-kind="function" class="blk"><title>__post_init__ — function, 3 lines</title></rect>
<rect x="381.0" y="1812.0" width="60.0" height="4.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Node.to_dict" data-kind="function" class="blk"><title>to_dict — function, 8 lines</title></rect>
<rect x="374.0" y="1817.5" width="74.0" height="7.5" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.ir.model.Edge" data-kind="class" class="blk"><title>Edge — class, 15 lines</title></rect>
<rect x="381.0" y="1821.5" width="60.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Edge.to_dict" data-kind="function" class="blk"><title>to_dict — function, 7 lines</title></rect>
<rect x="374.0" y="1826.5" width="74.0" height="24.0" rx="2" fill="#1d4ed8" stroke="none" opacity="0.95" data-id="docgen.ir.model.Graph" data-kind="class" class="blk"><title>Graph — class, 48 lines</title></rect>
<rect x="381.0" y="1831.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Graph.node" data-kind="function" class="blk"><title>node — function, 4 lines</title></rect>
<rect x="381.0" y="1833.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Graph.edge" data-kind="function" class="blk"><title>edge — function, 4 lines</title></rect>
<rect x="381.0" y="1836.0" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Graph.has" data-kind="function" class="blk"><title>has — function, 2 lines</title></rect>
<rect x="381.0" y="1838.5" width="60.0" height="8.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Graph.to_dict" data-kind="function" class="blk"><title>to_dict — function, 16 lines</title></rect>
<rect x="381.0" y="1847.5" width="60.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ir.model.Graph.from_dict" data-kind="function" class="blk"><title>from_dict — function, 6 lines</title></rect>
<rect x="456.0" y="1768.5" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.ir.__main__ — 9 lines</title></rect>
<rect x="556.0" y="1768.5" width="74.0" height="242.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.ops.filter — 485 lines</title></rect>
<rect x="556.0" y="1786.5" width="74.0" height="28.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter._rebuild" data-kind="function" class="blk"><title>_rebuild — function, 57 lines</title></rect>
<rect x="563.0" y="1791.0" width="60.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ops.filter._rebuild.surviving_parent" data-kind="function" class="blk"><title>surviving_parent — function, 5 lines</title></rect>
<rect x="563.0" y="1798.0" width="60.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ops.filter._rebuild.lift" data-kind="function" class="blk"><title>lift — function, 6 lines</title></rect>
<rect x="556.0" y="1816.0" width="74.0" height="7.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.drop_kinds" data-kind="function" class="blk"><title>drop_kinds — function, 14 lines</title></rect>
<rect x="556.0" y="1824.0" width="74.0" height="7.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.only_kinds" data-kind="function" class="blk"><title>only_kinds — function, 14 lines</title></rect>
<rect x="556.0" y="1832.0" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.drop_stdlib" data-kind="function" class="blk"><title>drop_stdlib — function, 13 lines</title></rect>
<rect x="556.0" y="1839.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.drop_external" data-kind="function" class="blk"><title>drop_external — function, 3 lines</title></rect>
<rect x="556.0" y="1842.0" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.subtree" data-kind="function" class="blk"><title>subtree — function, 13 lines</title></rect>
<rect x="556.0" y="1849.5" width="74.0" height="22.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.neighbourhood" data-kind="function" class="blk"><title>neighbourhood — function, 45 lines</title></rect>
<rect x="556.0" y="1873.0" width="74.0" height="9.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.collapse_to_depth" data-kind="function" class="blk"><title>collapse_to_depth — function, 18 lines</title></rect>
<rect x="563.0" y="1878.0" width="60.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ops.filter.collapse_to_depth.level" data-kind="function" class="blk"><title>level — function, 6 lines</title></rect>
<rect x="556.0" y="1883.0" width="74.0" height="7.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.drop_builtins" data-kind="function" class="blk"><title>drop_builtins — function, 14 lines</title></rect>
<rect x="556.0" y="1891.0" width="74.0" height="17.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.overview" data-kind="function" class="blk"><title>overview — function, 35 lines</title></rect>
<rect x="556.0" y="1909.5" width="74.0" height="36.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.shape" data-kind="function" class="blk"><title>shape — function, 72 lines</title></rect>
<rect x="563.0" y="1933.5" width="60.0" height="4.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.ops.filter.shape.rank_of" data-kind="function" class="blk"><title>rank_of — function, 9 lines</title></rect>
<rect x="556.0" y="1946.5" width="74.0" height="12.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.split" data-kind="function" class="blk"><title>split — function, 24 lines</title></rect>
<rect x="556.0" y="1959.5" width="74.0" height="51.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.filter.classify" data-kind="function" class="blk"><title>classify — function, 102 lines</title></rect>
<rect x="638.0" y="1768.5" width="74.0" height="51.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.ops.__main__ — 102 lines</title></rect>
<rect x="638.0" y="1775.0" width="74.0" height="42.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.ops.__main__.main" data-kind="function" class="blk"><title>main — function, 84 lines</title></rect>
<rect x="738.0" y="1768.5" width="74.0" height="111.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.style.extract — 222 lines</title></rect>
<rect x="738.0" y="1799.0" width="74.0" height="8.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.extract._values_from" data-kind="function" class="blk"><title>_values_from — function, 17 lines</title></rect>
<rect x="738.0" y="1808.5" width="74.0" height="22.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.extract._normalise" data-kind="function" class="blk"><title>_normalise — function, 45 lines</title></rect>
<rect x="738.0" y="1832.0" width="74.0" height="12.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.extract._svg_files" data-kind="function" class="blk"><title>_svg_files — function, 25 lines</title></rect>
<rect x="738.0" y="1845.5" width="74.0" height="21.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.extract.harvest" data-kind="function" class="blk"><title>harvest — function, 43 lines</title></rect>
<rect x="738.0" y="1868.0" width="74.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.extract.write" data-kind="function" class="blk"><title>write — function, 7 lines</title></rect>
<rect x="738.0" y="1872.5" width="74.0" height="6.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.extract.summarise" data-kind="function" class="blk"><title>summarise — function, 13 lines</title></rect>
<rect x="28.0" y="2049.0" width="74.0" height="105.5" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.style.tokens — 211 lines</title></rect>
<rect x="28.0" y="2077.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.tokens._top" data-kind="function" class="blk"><title>_top — function, 2 lines</title></rect>
<rect x="28.0" y="2079.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.tokens._mode" data-kind="function" class="blk"><title>_mode — function, 3 lines</title></rect>
<rect x="28.0" y="2081.5" width="74.0" height="3.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.tokens._luminance" data-kind="function" class="blk"><title>_luminance — function, 6 lines</title></rect>
<rect x="28.0" y="2085.5" width="74.0" height="50.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.tokens.derive" data-kind="function" class="blk"><title>derive — function, 101 lines</title></rect>
<rect x="35.0" y="2107.5" width="60.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.style.tokens.derive.accent" data-kind="function" class="blk"><title>accent — function, 2 lines</title></rect>
<rect x="28.0" y="2137.0" width="74.0" height="7.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.tokens.geometry" data-kind="function" class="blk"><title>geometry — function, 14 lines</title></rect>
<rect x="28.0" y="2145.0" width="74.0" height="3.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.tokens.write" data-kind="function" class="blk"><title>write — function, 7 lines</title></rect>
<rect x="28.0" y="2149.5" width="74.0" height="4.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.style.tokens.from_folder" data-kind="function" class="blk"><title>from_folder — function, 9 lines</title></rect>
<rect x="128.0" y="2049.0" width="74.0" height="14.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen — 2 lines</title></rect>
<rect x="228.0" y="2049.0" width="74.0" height="75.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.lab.pg_probe — 150 lines</title></rect>
<rect x="228.0" y="2084.5" width="74.0" height="15.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.lab.pg_probe.probe" data-kind="function" class="blk"><title>probe — function, 30 lines</title></rect>
<rect x="228.0" y="2106.5" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.lab.pg_probe._simplify" data-kind="function" class="blk"><title>_simplify — function, 3 lines</title></rect>
<rect x="228.0" y="2109.0" width="74.0" height="12.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.lab.pg_probe.main" data-kind="function" class="blk"><title>main — function, 25 lines</title></rect>
<rect x="328.0" y="2049.0" width="74.0" height="132.0" rx="2" fill="#141414" stroke="#333333" stroke-width="1"><title>docgen.notebook.spec — 264 lines</title></rect>
<rect x="328.0" y="2077.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.notebook.spec._step" data-kind="function" class="blk"><title>_step — function, 4 lines</title></rect>
<rect x="328.0" y="2080.0" width="74.0" height="54.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.notebook.spec.from_ir" data-kind="function" class="blk"><title>from_ir — function, 108 lines</title></rect>
<rect x="335.0" y="2087.0" width="60.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.8" data-id="docgen.notebook.spec.from_ir._order" data-kind="function" class="blk"><title>_order — function, 5 lines</title></rect>
<rect x="328.0" y="2135.0" width="74.0" height="10.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.notebook.spec.scaffold" data-kind="function" class="blk"><title>scaffold — function, 20 lines</title></rect>
<rect x="328.0" y="2146.0" width="74.0" height="29.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.notebook.spec.merge" data-kind="function" class="blk"><title>merge — function, 58 lines</title></rect>
<rect x="328.0" y="2176.0" width="74.0" height="2.0" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.notebook.spec.load" data-kind="function" class="blk"><title>load — function, 2 lines</title></rect>
<rect x="328.0" y="2178.0" width="74.0" height="2.5" rx="2" fill="#15803d" stroke="none" opacity="0.95" data-id="docgen.notebook.spec.dump" data-kind="function" class="blk"><title>dump — function, 5 lines</title></rect>
<text x="28" y="40" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.emitters</text>
<text x="28" y="325" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.emitters</text>
<text x="538" y="325" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen</text>
<text x="28" y="1370" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen</text>
<text x="538" y="1370" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.extractors</text>
<text x="28" y="1549" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.extractors</text>
<text x="538" y="1549" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.book</text>
<text x="28" y="1762" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.extractors.python</text>
<text x="292" y="1762" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.ir</text>
<text x="556" y="1762" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.ops</text>
<text x="738" y="1762" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.style</text>
<text x="28" y="2043" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.style</text>
<text x="128" y="2043" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">(root)</text>
<text x="228" y="2043" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.lab</text>
<text x="328" y="2043" font-family="Helvetica,sans-Serif" font-size="10" font-weight="bold" fill="#a3a3a3">docgen.notebook</text>
<text x="28" y="302" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">site</text>
<text x="110" y="213" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">explore</text>
<text x="192" y="201" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">dot</text>
<text x="274" y="194" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">notebook</text>
<text x="356" y="194" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">minimap</text>
<text x="438" y="185" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">erd</text>
<text x="520" y="137" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">index</text>
<text x="602" y="98" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">cli_dot</text>
<text x="684" y="95" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">auto</text>
<text x="766" y="92" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">cli_noteboo</text>
<text x="28" y="377" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">cli_site</text>
<text x="110" y="366" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">cli_erd</text>
<text x="192" y="366" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">cli_minimap</text>
<text x="274" y="364" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">cli_explore</text>
<text x="356" y="362" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">cli_index</text>
<text x="438" y="358" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">__main__</text>
<text x="538" y="1347" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">selftest</text>
<text x="620" y="498" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">book</text>
<text x="702" y="437" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">style</text>
<text x="784" y="394" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">reference</text>
<text x="28" y="1399" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">ops</text>
<text x="110" y="1399" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">notebook</text>
<text x="192" y="1399" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">emitters</text>
<text x="274" y="1399" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">lab</text>
<text x="356" y="1399" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">ir</text>
<text x="438" y="1399" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">extractors</text>
<text x="538" y="1526" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">code</text>
<text x="620" y="1520" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">usage</text>
<text x="702" y="1480" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">openapi</text>
<text x="784" y="1466" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">db</text>
<text x="28" y="1584" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">code_main</text>
<text x="110" y="1582" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">python</text>
<text x="192" y="1580" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">usage_main</text>
<text x="274" y="1580" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">db_main</text>
<text x="356" y="1579" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">openapi_mai</text>
<text x="438" y="1578" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">__main__</text>
<text x="538" y="1740" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">build</text>
<text x="620" y="1717" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">checks</text>
<text x="702" y="1662" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">larder</text>
<text x="784" y="1604" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">__main__</text>
<text x="28" y="1896" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">collect</text>
<text x="110" y="1870" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">resolve</text>
<text x="192" y="1796" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">__main__</text>
<text x="292" y="1934" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">validate</text>
<text x="374" y="1860" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">model</text>
<text x="456" y="1792" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">__main__</text>
<text x="556" y="2020" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">filter</text>
<text x="638" y="1828" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">__main__</text>
<text x="738" y="1888" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">extract</text>
<text x="28" y="2164" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">tokens</text>
<text x="128" y="2072" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">docgen</text>
<text x="228" y="2133" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">pg_probe</text>
<text x="328" y="2190" font-family="Helvetica,sans-Serif" font-size="7" fill="#666666">spec</text>
<rect x="28" y="2230.0" width="9" height="9" rx="2" fill="#1a1a1a"/>
<text x="41" y="2238.0" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">module</text>
<rect x="86" y="2230.0" width="9" height="9" rx="2" fill="#1d4ed8"/>
<text x="99" y="2238.0" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">class</text>
<rect x="138" y="2230.0" width="9" height="9" rx="2" fill="#d4a574"/>
<text x="151" y="2238.0" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">interface</text>
<rect x="214" y="2230.0" width="9" height="9" rx="2" fill="#15803d"/>
<text x="227" y="2238.0" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">function</text>
<text x="858" y="2238.0" text-anchor="end" font-family="Helvetica,sans-Serif" font-size="9" fill="#666666">53 files · 9,752 lines · 1px ≈ 2.0 lines</text>
</svg>

After

Width:  |  Height:  |  Size: 73 KiB

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,119 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>docgen docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { background: #0a0a0a; overflow: hidden; width: 100vw; height: 100vh;
font-family: "Segoe UI", Inter, system-ui, -apple-system, Arial, sans-serif; }
#container { width: 100vw; height: 100vh; overflow: hidden; cursor: grab; }
#container.dragging { cursor: grabbing; }
img { transform-origin: 0 0; user-select: none; -webkit-user-drag: none; }
#hud {
position: fixed; bottom: 14px; left: 14px; display: flex; gap: 8px;
align-items: center; font-size: 11px; color: #a3a3a3;
background: #141414; border: 1px solid #333333;
border-radius: 6px; padding: 5px 9px; user-select: none;
}
#hud b { color: #e5e5e5; font-weight: 600; font-variant-numeric: tabular-nums; }
#hud span { opacity: .7; }
a.back { position: fixed; top: 14px; left: 14px; font-size: 11px;
color: #a3a3a3; text-decoration: none; background: #141414;
border: 1px solid #333333; border-radius: 6px; padding: 5px 9px; }
a.back:hover { color: #e5e5e5; }
</style>
</head>
<body>
<div id="container"><img id="img" alt=""></div>
<a class="back" href="index.html">&larr; docs</a>
<div id="hud"><b id="pct">100%</b><span id="mode">fit</span><span>&middot; click 1:1 &middot; drag &middot; wheel</span></div>
<script>
var src = new URLSearchParams(location.search).get('src');
var img = document.getElementById('img');
var container = document.getElementById('container');
var pct = document.getElementById('pct');
var modeEl = document.getElementById('mode');
if (src) { img.src = src; document.title = src + ' — docgen docs'; }
var scale = 1, x = 0, y = 0, fitScale = 1, mode = 'fit';
var dragging = false, moved = false, startX, startY, startPanX, startPanY;
function apply() {
img.style.transform = 'translate(' + x + 'px,' + y + 'px) scale(' + scale + ')';
pct.textContent = Math.round(scale * 100) + '%';
modeEl.textContent = mode;
}
function fit() {
var sw = window.innerWidth / img.naturalWidth;
var sh = window.innerHeight / img.naturalHeight;
fitScale = Math.min(sw, sh) * 0.95;
scale = fitScale;
x = (window.innerWidth - img.naturalWidth * scale) / 2;
y = (window.innerHeight - img.naturalHeight * scale) / 2;
mode = 'fit';
apply();
}
// Zoom about a point in the viewport, so what is under the cursor stays there.
function zoomAt(px, py, factor) {
x = px - (px - x) * factor;
y = py - (py - y) * factor;
scale *= factor;
mode = Math.abs(scale - fitScale) < 0.001 ? 'fit'
: (Math.abs(scale - 1) < 0.001 ? '1:1' : 'free');
apply();
}
img.onload = fit;
window.addEventListener('resize', function () { if (mode === 'fit') fit(); });
container.addEventListener('wheel', function (e) {
e.preventDefault();
var rect = container.getBoundingClientRect();
zoomAt(e.clientX - rect.left, e.clientY - rect.top, e.deltaY < 0 ? 1.12 : 0.89);
}, { passive: false });
container.addEventListener('mousedown', function (e) {
if (e.button !== 0) return;
dragging = true; moved = false;
startX = e.clientX; startY = e.clientY; startPanX = x; startPanY = y;
container.classList.add('dragging');
e.preventDefault();
});
window.addEventListener('mousemove', function (e) {
if (!dragging) return;
if (Math.abs(e.clientX - startX) > 3 || Math.abs(e.clientY - startY) > 3) moved = true;
x = startPanX + (e.clientX - startX);
y = startPanY + (e.clientY - startY);
apply();
});
window.addEventListener('mouseup', function (e) {
if (!dragging) return;
dragging = false;
container.classList.remove('dragging');
// A click that moved the mouse was a drag, and must not also toggle.
if (moved) return;
if (mode === '1:1') { fit(); return; }
// Toggle to actual size about the point clicked, so the thing you aimed at
// is the thing you end up looking at.
var rect = container.getBoundingClientRect();
zoomAt(e.clientX - rect.left, e.clientY - rect.top, 1 / scale);
mode = '1:1';
apply();
});
container.addEventListener('dblclick', fit);
window.addEventListener('keydown', function (e) {
if (e.key === '0' || e.key === 'f') fit();
if (e.key === '1') { var r = container.getBoundingClientRect();
zoomAt(r.width / 2, r.height / 2, 1 / scale); mode = '1:1'; apply(); }
if (e.key === 'Escape') location.href = 'index.html';
});
</script>
</body>
</html>

View File

@@ -0,0 +1,11 @@
"""
Emitters: IR -> an artifact. None of them has heard of Python, `ast` or SQL.
dot .dot -> Graphviz -> SVG static docs, embedding
index markdown / sidebar JSON no graph literacy required
diff two IRs -> what changed review
notebook .ipynb a runnable document
The non-visual ones matter most for reach. A sorted, described index of what
exists is readable by people who will never open a diagram.
"""

View File

@@ -0,0 +1,36 @@
""" python3 -m docgen.emitters <emitter> <ir.json> [options]"""
import sys
def main(argv=None):
argv = sys.argv[1:] if argv is None else argv
if not argv:
print("usage: python3 -m docgen.emitters <auto|dot|index|erd|notebook|site|minimap|explore> <ir.json> [-o OUT] [--style NAME] [--theme NAME]",
file=sys.stderr)
return 2
name, rest = argv[0], argv[1:]
if name == "dot":
from .cli_dot import main as run
elif name == "index":
from .cli_index import main as run
elif name == "erd":
from .cli_erd import main as run
elif name == "auto":
from .auto import main as run
elif name == "notebook":
from .cli_notebook import main as run
elif name == "site":
from .cli_site import main as run
elif name == "minimap":
from .cli_minimap import main as run
elif name == "explore":
from .cli_explore import main as run
else:
print(f"Error: no emitter {name!r} — have: auto, dot, index, erd, notebook, site, minimap, explore", file=sys.stderr)
return 1
return run(rest)
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,79 @@
"""
Draw it the way its structure asks to be drawn.
python3 -m docgen.emitters auto ir.json -o out/
`ops.classify` reads the structure and names an emitter; this runs it. The whole
point is that nobody should have to know that a schema wants cards and a module
graph wants ranks — or discover it from a 235:1 image.
When the answer is "this is not a diagram", it says so and writes the index,
because that *is* the right artifact for a flat list of peers.
"""
import argparse
import json
import sys
from pathlib import Path
from ..ir import check
from ..ops import classify
from ..style import Style, StyleError
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters auto")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path, help="Directory to write into.")
p.add_argument("--style", default="lucid")
p.add_argument("--theme", default=None)
p.add_argument("--force", help="Use this emitter regardless of what fits.")
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)}):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
verdict = classify(data)
chosen = args.force or verdict["emitter"]
print(f" {verdict['kind']:<8} -> {chosen}")
print(f" {verdict['why']}")
out_dir = args.output or Path(".")
out_dir.mkdir(parents=True, exist_ok=True)
stem = args.ir.stem
try:
style = Style.load(args.style, theme=args.theme)
except StyleError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
if chosen == "erd":
from .erd import emit as erd_emit
path = out_dir / f"{stem}.svg"
path.write_text(erd_emit(data, style))
elif chosen == "index":
from .index import to_markdown
path = out_dir / f"{stem}.md"
path.write_text(to_markdown(data))
else:
from .dot import emit as dot_emit, render
path = out_dir / f"{stem}.svg"
opts = verdict.get("options") or {}
path.write_bytes(render(dot_emit(data, style, rankdir=opts.get("rankdir"))))
print(f" {path}")
return 0

View File

@@ -0,0 +1,86 @@
""" python3 -m docgen.emitters dot <ir.json> [-o out.svg] [--style lucid] [--theme dark]"""
import argparse
import json
import sys
from pathlib import Path
from ..ir import check
from ..ops import shape
from ..style import Style, StyleError
from .dot import RenderError, emit, render
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters dot")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path, help="Write here. .dot or .svg by suffix.")
p.add_argument("--style", default="lucid")
p.add_argument("--theme", default=None)
p.add_argument("--max-depth", type=int, default=None)
p.add_argument("--quiet", "-q", action="store_true", help="Do not warn about shape.")
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)} problem(s)):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
try:
style = Style.load(args.style, theme=args.theme)
except StyleError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
# Aspect ratio is a property of the graph, not of the renderer: a layered
# engine puts one dependency level in one row, so the widest level is the
# width. Say so before writing the file, because the alternative is finding
# out from a 13671pt image — and the fix is never a layout flag, it is a
# smaller question.
if not args.quiet:
sh = shape(data)
if sh["widest_level"] > 20 or sh["nodes"] > 60:
est = sh["widest_level"] / max(sh["levels"], 1)
print(
f" note: {sh['nodes']} nodes, {sh['levels']} levels, widest level "
f"{sh['widest_level']} — this will render roughly {est:.0f}:1.",
file=sys.stderr,
)
print(
" Around 20 nodes is where it stops being a diagram. Try "
"`ops --split`,\n `--around <id> --hops 2`, or `--subtree <id>`. "
"Layout flags will not fix it.",
file=sys.stderr,
)
if sh["isolated"] > sh["nodes"] // 3:
print(
f" {sh['isolated']} of {sh['nodes']} nodes have no edges; they are "
"laid out side by side.",
file=sys.stderr,
)
dot_text = emit(data, style, max_depth=args.max_depth)
if not args.output:
sys.stdout.write(dot_text)
return 0
args.output.parent.mkdir(parents=True, exist_ok=True)
if args.output.suffix == ".dot":
args.output.write_text(dot_text)
else:
try:
args.output.write_bytes(render(dot_text, fmt=args.output.suffix.lstrip(".") or "svg"))
except RenderError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
print(f" {args.style}/{style.theme:6} {args.output}")
return 0

View File

@@ -0,0 +1,50 @@
""" python3 -m docgen.emitters erd <ir.json> [-o out.svg] [--theme dark]"""
import argparse
import json
import sys
from pathlib import Path
from ..ir import check
from ..style import Style, StyleError
from .erd import emit
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters erd")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path)
p.add_argument("--style", default="lucid")
p.add_argument("--theme", default=None)
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)} problem(s)):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
try:
style = Style.load(args.style, theme=args.theme)
svg = emit(data, style)
except (StyleError, ValueError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(svg)
import re
m = re.search(r'width="(\d+)pt" height="(\d+)pt"', svg)
size = f"{m.group(1)}x{m.group(2)} {int(m.group(1))/int(m.group(2)):.1f}:1" if m else ""
print(f" erd/{style.theme:6} {args.output} {size}")
else:
sys.stdout.write(svg)
return 0

View File

@@ -0,0 +1,47 @@
""" python3 -m docgen.emitters explore <ir.json> -o DIR [--scale 0.55] [--hops 1]"""
import argparse
import json
import sys
from pathlib import Path
from ..ir import check
from ..style import Style, StyleError
from .explore import write
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters explore")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path, required=True, help="Directory.")
p.add_argument("--style", default="lucid")
p.add_argument("--theme", default=None)
p.add_argument("--scale", type=float, default=0.55)
p.add_argument("--width", type=int, default=1100)
p.add_argument("--hops", type=int, default=1)
p.add_argument("--title", default="")
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)}):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
try:
style = Style.load(args.style, theme=args.theme)
path = write(data, style, args.output, scale=args.scale, width=args.width,
hops=args.hops, title=args.title)
except (StyleError, ValueError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
graphs = len(list((args.output / "graphs").glob("*.svg"))) if (args.output / "graphs").exists() else 0
print(f" explore {path} {graphs} neighbourhood diagram(s)")
return 0

View File

@@ -0,0 +1,43 @@
""" python3 -m docgen.emitters index <ir.json> [-o out.md|out.json]"""
import argparse
import json
import sys
from pathlib import Path
from ..ir import check
from .index import to_markdown, to_sidebar
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters index")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path, help=".md for the document, .json for a sidebar.")
p.add_argument("--title", default="")
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)} problem(s)):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
if args.output and args.output.suffix == ".json":
text = json.dumps(to_sidebar(data), indent=2) + "\n"
else:
text = to_markdown(data, title=args.title)
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(text)
print(f" index {args.output}")
else:
sys.stdout.write(text)
return 0

View File

@@ -0,0 +1,50 @@
""" python3 -m docgen.emitters minimap <ir.json> [-o out.svg] [--scale 0.55]"""
import argparse
import json
import re
import sys
from pathlib import Path
from ..ir import check
from ..style import Style, StyleError
from .minimap import emit
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters minimap")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path)
p.add_argument("--style", default="lucid")
p.add_argument("--theme", default=None)
p.add_argument("--scale", type=float, default=0.55, help="Pixels per source line.")
p.add_argument("--width", type=int, default=1180, help="Wrap a shelf past this.")
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)}):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
try:
style = Style.load(args.style, theme=args.theme)
svg = emit(data, style, scale=args.scale, target_width=args.width)
except (StyleError, ValueError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(svg)
m = re.search(r'width="(\d+)pt" height="(\d+)pt"', svg)
print(f" minimap {args.output} {m.group(1)}x{m.group(2)}" if m else "")
else:
sys.stdout.write(svg)
return 0

View File

@@ -0,0 +1,73 @@
""" python3 -m docgen.emitters notebook <ir.json> [-o out.ipynb] [--overlay f.json]
--spec-out FILE write the generated spec (the base), for reading/diffing
--scaffold FILE write a blank overlay listing every step id
"""
import argparse
import json
import sys
from pathlib import Path
from ..ir import check
from ..notebook import dump, from_ir, merge, scaffold
from .notebook import emit
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters notebook")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path)
p.add_argument("--overlay", type=Path, help="Hand-written additions, re-applied.")
p.add_argument("--spec-out", type=Path, help="Write the generated spec too.")
p.add_argument("--scaffold", type=Path, help="Write a blank overlay and stop.")
p.add_argument("--base-url", default="https://api.example.invalid")
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)}):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
base = from_ir(data, base_url=args.base_url)
if args.scaffold:
dump(scaffold(base), args.scaffold)
print(f" overlay {args.scaffold} {len(base['steps'])} step(s), none filled in")
return 0
overlay = None
if args.overlay:
if args.overlay.exists():
overlay = json.loads(args.overlay.read_text())
else:
print(f" note: no overlay at {args.overlay} — generating the base only",
file=sys.stderr)
spec, drift = merge(base, overlay)
for d in drift:
# The base moved under the overlay. Worth saying out loud; not a reason
# to refuse to build the document.
print(f" drift: {d}", file=sys.stderr)
if args.spec_out:
dump(spec, args.spec_out)
print(f" spec {args.spec_out}")
text = emit(spec)
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(text)
n = len(json.loads(text)["cells"])
extra = f", {len(drift)} drift" if drift else ""
print(f" notebook {args.output} {len(spec['steps'])} steps, {n} cells{extra}")
else:
sys.stdout.write(text)
return 0

View File

@@ -0,0 +1,73 @@
""" python3 -m docgen.emitters site <ir.json> -o DIR [--theme lucid]
Writes index.html, viewer.html, site.css and the graph — self-contained, offline.
"""
import argparse
import json
import sys
from pathlib import Path
from ..ir import check
from ..ops import classify
from ..style import Style, StyleError
from .site import write
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.emitters site")
p.add_argument("ir", type=Path)
p.add_argument("--output", "-o", type=Path, required=True, help="Directory.")
p.add_argument("--style", default="lucid")
p.add_argument("--theme", default=None)
p.add_argument("--title", default="")
p.add_argument("--no-graph", action="store_true")
args = p.parse_args(argv)
try:
data = json.loads(args.ir.read_text())
except (OSError, json.JSONDecodeError) as e:
print(f"Error: could not read {args.ir}: {e}", file=sys.stderr)
return 1
problems = check(data)
if problems:
print(f"Error: {args.ir} is not a valid IR ({len(problems)}):", file=sys.stderr)
for pr in problems[:5]:
print(f" {pr}", file=sys.stderr)
return 1
try:
style = Style.load(args.style, theme=args.theme)
except StyleError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
args.output.mkdir(parents=True, exist_ok=True)
graph_name = None
if not args.no_graph:
# Whatever the structure asks for, so the page carries the right picture.
verdict = classify(data)
if verdict["emitter"] == "erd":
from .erd import emit as draw
(args.output / "graph.svg").write_text(draw(data, style))
graph_name = "graph.svg"
elif verdict["emitter"] == "dot":
from .dot import emit as dot_emit, have_graphviz, render
if have_graphviz():
opts = verdict.get("options") or {}
(args.output / "graph.svg").write_bytes(
render(dot_emit(data, style, rankdir=opts.get("rankdir")))
)
graph_name = "graph.svg"
else:
print(" note: graphviz absent — the site is text only", file=sys.stderr)
else:
print(f" note: {verdict['kind']}{verdict['why']}", file=sys.stderr)
print(" no diagram on the page; the index is the artifact", file=sys.stderr)
files = write(data, style, args.output, graph=graph_name, title=args.title)
for f in files:
print(f" site {f}")
if graph_name:
print(f" site {args.output / graph_name}")
return 0

View File

@@ -0,0 +1,291 @@
"""
IR + style -> DOT -> SVG.
This module has never heard of Python, `ast` or SQL. It walks nodes, looks up a
rule by `kind`, and writes attributes. That is deliberately the whole algorithm:
if it starts making decisions about what something *is*, the decision belongs in
an extractor, and if it starts making decisions about what something *looks
like*, it belongs in a style file.
from docgen.emitters.dot import emit, render
svg = render(emit(ir, Style.load("lucid")))
## Containment becomes clusters
A node with children is a `subgraph cluster_*`; a leaf is a node. That is the
only structural interpretation made here, and it follows from `parent` meaning
containment and nothing else.
## The SVG is addressable
`id` and `kind` are written through to the SVG as the element's `id` and
`class`, and `attrs.file`/`attrs.line` become an `href`. So a front end can
attach behaviour to a box, and a box can link to the line it came from, without
this emitter knowing anything about either.
## Known limits
Recorded in `style/lucid.json` under `limits` and reachable as `Style.limits()`.
DOT is used until it genuinely cannot express a rule, and then it stops rather
than growing machinery — an HTML-like label table for header bars, a post-pass
for badge circles. Those mark where a richer emitter begins.
"""
import shutil
import subprocess
class RenderError(RuntimeError):
"""Graphviz is absent, or refused the graph."""
def _esc(text: str) -> str:
return str(text).replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
def _attrs(pairs: dict) -> str:
inner = " ".join(f'{k}="{_esc(v)}"' for k, v in pairs.items() if v not in (None, "", []))
return f" [{inner}]" if inner else ""
def _style_words(rule: dict, *, filled: bool = True) -> str:
words = ["filled"] if filled else []
# `record` and `plaintext` ignore rounding; asking for it warns and changes
# nothing, which is noise in the build output.
if rule.get("rounded") and rule.get("shape") not in ("record", "Mrecord", "plaintext"):
words.append("rounded")
if rule.get("dashed"):
words.append("dashed")
return ",".join(words)
def _node_attrs(node, rule: dict, style) -> dict:
a = {
"label": node.get("label") or node["id"],
"shape": rule.get("shape", "box"),
"style": _style_words(rule),
"fillcolor": rule.get("fill"),
"color": rule.get("border"),
"fontcolor": rule.get("text"),
"penwidth": style.geom("hairline"),
"fontname": style.geom("font-bold" if rule.get("bold") else "font"),
"fontsize": rule.get("font-size") or style.geom("font-size-base"),
"margin": style.geom("padding"),
# Addressability: through to the SVG, for whatever reads it later.
"id": node["id"],
"class": node["kind"],
}
attrs = node.get("attrs") or {}
if attrs.get("file"):
line = attrs.get("line")
a["href"] = f"{attrs['file']}#L{line}" if line else attrs["file"]
a["tooltip"] = attrs.get("doc") or node["id"]
elif attrs.get("doc"):
a["tooltip"] = attrs["doc"]
return a
def emit(ir: dict, style, *, max_depth: int | None = None,
rankdir: str | None = None) -> str:
"""IR document (a dict) + Style -> DOT text."""
nodes = {n["id"]: n for n in ir["nodes"]}
children: dict[str | None, list[str]] = {}
for n in ir["nodes"]:
children.setdefault(n.get("parent"), []).append(n["id"])
g = style.graph()
out = [
"digraph ir {",
f' bgcolor="{g.get("bgcolor", "transparent")}"',
f' rankdir={rankdir or g.get("rankdir", "TB")}',
f' nodesep="{g.get("nodesep", 0.5)}"',
f' ranksep="{g.get("ranksep", 0.6)}"',
f' pad="{g.get("pad", 0.3)}"',
f' fontname="{style.geom("font")}"',
" compound=true",
"",
]
# Deterministic: groups are numbered by sorted id, so the rotation of
# domain colours is the same on every run.
group_index = {
nid: i for i, nid in enumerate(sorted(k for k in children if k is not None))
}
def write(node_id: str, depth: int) -> None:
node = nodes[node_id]
kids = sorted(children.get(node_id, []))
too_deep = max_depth is not None and depth >= max_depth
pad = " " * (depth + 1)
if not kids or too_deep:
out.append(f"{pad}{_q(node_id)}{_attrs(_node_attrs(node, style.node(node['kind']), style))}")
return
rule = style.group(node["kind"])
domain = (node.get("attrs") or {}).get("domain")
border = rule.get("border") or style.slot(
style.domain_slot(domain, group_index.get(node_id, 0))
)
out.append(f"{pad}subgraph cluster_{_safe(node_id)} {{")
out.append(f'{pad} label="{_esc(node.get("label") or node_id)}"')
out.append(f'{pad} style="{_style_words(rule)}"')
out.append(f'{pad} color="{border}"')
out.append(f'{pad} fillcolor="{rule.get("fill", "transparent")}"')
out.append(f'{pad} fontcolor="{rule.get("text", "")}"')
out.append(f'{pad} fontname="{style.geom("font-bold" if rule.get("bold") else "font")}"')
out.append(f'{pad} fontsize="{style.geom("font-size-header")}"')
out.append(f'{pad} labeljust=l')
out.append(f'{pad} id="{_esc(node_id)}"')
out.append(f'{pad} class="{node["kind"]}"')
for kid in kids:
write(kid, depth + 1)
out.append(f"{pad}}}")
for root in sorted(children.get(None, [])):
write(root, 0)
out.append("")
# DOT cannot use a cluster as an edge endpoint. The native answer is
# `compound=true` plus lhead/ltail: draw between a representative leaf
# inside each cluster and clip the line at the cluster boundary. Without
# this, every module-to-module import silently disappears — which is most of
# the graph a Python extractor produces.
endpoint = _endpoints(nodes, children, max_depth)
for e in ir["edges"]:
src, dst = endpoint.get(e["source"]), endpoint.get(e["target"])
if not src or not dst or src[0] == dst[0]:
continue
if src[1] and src[1] == dst[1]:
continue # both collapsed into the same cluster
# A package importing its own submodule gives an edge whose head sits
# inside its tail's cluster. Graphviz warns and draws it oddly; clipping
# to the enclosing boundary is meaningless there, so drop that side's
# clip and let the line run to the box.
ltail, lhead = src[1], dst[1]
if ltail and _within(ltail, dst[0], nodes):
ltail = None
if lhead and _within(lhead, src[0], nodes):
lhead = None
rule = style.edge(e["kind"])
out.append(
f" {_q(src[0])} -> {_q(dst[0])}"
+ _attrs(
{
"color": rule.get("color"),
"fontcolor": rule.get("text"),
"penwidth": style.geom("hairline"),
"arrowhead": rule.get("arrowhead", "normal"),
"arrowsize": rule.get("arrowsize", 0.7),
"style": "dashed" if rule.get("dashed") else None,
"fontname": style.geom("font"),
"fontsize": style.geom("font-size-sm"),
"label": (e.get("attrs") or {}).get("label"),
"ltail": ltail,
"lhead": lhead,
"class": e["kind"],
}
)
)
out.append("}")
return "\n".join(out) + "\n"
def _within(cluster_name: str, node_id: str, nodes) -> bool:
"""Is `node_id` inside the cluster named `cluster_name`?"""
cur = node_id
while cur:
if f"cluster_{_safe(cur)}" == cluster_name:
return True
cur = nodes.get(cur, {}).get("parent")
return False
def _endpoints(nodes, children, max_depth):
"""id -> (leaf to draw from, cluster to clip to or None).
A leaf is its own endpoint. A node that became a cluster is represented by
its first leaf descendant in sorted order — deterministic, so the same graph
twice produces the same DOT — with `ltail`/`lhead` naming the cluster so the
line stops at its border instead of burrowing to the inner box.
"""
depth_of: dict[str, int] = {}
def walk(nid, depth):
depth_of[nid] = depth
for kid in children.get(nid, []):
walk(kid, depth + 1)
for root in children.get(None, []):
walk(root, 0)
def collapsed(nid: str) -> bool:
return max_depth is not None and depth_of.get(nid, 0) >= max_depth
def first_leaf(nid: str) -> str:
while children.get(nid) and not collapsed(nid):
nid = sorted(children[nid])[0]
return nid
out: dict[str, tuple[str, str | None]] = {}
for nid in nodes:
cur = nid
# Anything past the depth limit is represented by the ancestor that
# survived it.
while max_depth is not None and depth_of.get(cur, 0) > max_depth:
parent = nodes[cur].get("parent")
if not parent:
break
cur = parent
if children.get(cur) and not collapsed(cur):
out[nid] = (first_leaf(cur), f"cluster_{_safe(cur)}")
else:
out[nid] = (cur, None)
return out
def _safe(text: str) -> str:
return "".join(c if c.isalnum() else "_" for c in text)
def _q(text: str) -> str:
return f'"{_esc(text)}"'
# -- render ----------------------------------------------------------------
def have_graphviz(engine: str = "dot") -> bool:
return shutil.which(engine) is not None
def render(dot_text: str, fmt: str = "svg", engine: str = "dot") -> bytes:
"""DOT -> bytes, via the graphviz binary.
The binary, not a wrapper library: it is what the render hosts have and what
`docs/graphs/render.sh` already shells out to.
Note for anything reading geometry back out: Graphviz is y-up in points and
the SVG backend flips it with a wrapper `<g transform="...">`, and layout
measures label text with the host's fonts — so the same graph on a machine
with different fontconfig produces different coordinates. Pin golden tests
to the IR, never to the SVG.
"""
if not have_graphviz(engine):
raise RenderError(
f"{engine!r} not found — install with: sudo apt install graphviz\n"
"(already-rendered files keep working; this is only needed to re-render)"
)
proc = subprocess.run([engine, f"-T{fmt}"], input=dot_text.encode(), capture_output=True)
if proc.returncode != 0:
raise RenderError(
f"{engine} -T{fmt} failed ({proc.returncode}):\n"
+ proc.stderr.decode("utf-8", "replace").strip()
)
if proc.stderr.strip():
# Graphviz warns and still renders — a missing font, an ignored
# attribute. Worth seeing, not worth failing on.
for line in proc.stderr.decode("utf-8", "replace").strip().splitlines():
print(f" graphviz: {line}")
return proc.stdout

View File

@@ -0,0 +1,259 @@
"""
A schema, as an entity-relationship diagram. SVG written directly — no Graphviz.
**This is the answer to the aspect-ratio problem, and it is not a layout engine.**
A layered engine puts every node at one dependency level into one row, so the
widest level is the width; soleprint's 7-by-109 overview rendered 14:1 and no
Graphviz flag helped. A schema is not layered anyway — tables are peers that
reference each other — so laying it out in ranks was the wrong shape from the
start.
The design is lifted from `station/tools/graphgen/templates/index.html`, the
Supabase-style schema explorer already in this repo. It had solved this:
const cols = Math.max(2, Math.ceil(Math.sqrt(sorted.length * 1.2)));
**Columns from the square root of the table count.** The result is near-square
whatever the size — 4 tables or 400 — because the aspect ratio is chosen rather
than emergent. That is the one thing a rank-based engine cannot do.
Three more things it gets right that a generic node-edge drawing does not:
- **A table is a card**, not a box: a header and a list of its columns. That is
what a schema *is*, and it is the form every ER tool has converged on.
- **An edge starts at the column that holds the key** and ends on the target's
primary key, rather than joining two box centres. That is what makes a
foreign key readable rather than merely present.
- **The geometry is computed, never measured.** Card width and row height are
constants, so this produces identical bytes on any machine — unlike Graphviz,
which measures label text with the host's fonts and so renders differently
wherever fontconfig differs.
Colours come from the same style slots as every other emitter, so an ER diagram
and a code diagram are still one visual language.
"""
from xml.sax.saxutils import escape
# Geometry, from the explorer. Fixed on purpose: nothing here measures text, so
# the output is deterministic and can be golden-tested.
CARD_W = 210
HDR_H = 36
HDR_H_DOC = 50
FIELD_H = 26
COL_GAP = 80
ROW_GAP = 60
PAD = 40
BADGE_W = 28
RADIUS = 8
# ~6.2px per character at 11px in a humanist sans. An estimate, and it only
# decides where a long name is cut — never where anything is placed.
CHAR_W = 6.2
def _truncate(text: str, px: float) -> str:
limit = max(3, int(px / CHAR_W))
return text if len(text) <= limit else text[: limit - 1] + ""
def _tables(ir: dict) -> list[dict]:
"""Tables with their columns, in a stable order."""
columns: dict[str, list] = {}
for n in ir["nodes"]:
if n["kind"] == "column" and n.get("parent"):
columns.setdefault(n["parent"], []).append(n)
out = []
for n in ir["nodes"]:
if n["kind"] != "table":
continue
fields = columns.get(n["id"], [])
out.append(
{
"id": n["id"],
"name": n.get("label") or n["id"],
"doc": (n.get("attrs") or {}).get("doc"),
"fields": fields,
}
)
return sorted(out, key=lambda t: t["id"])
def _card_height(table: dict, columns: bool = True) -> int:
header = HDR_H_DOC if table["doc"] else HDR_H
return header + (len(table["fields"]) * FIELD_H if columns else 0)
def layout(tables: list[dict], edges: list[dict], columns: bool = True) -> dict[str, tuple[int, int]]:
"""Place cards in √n columns, referenced tables first.
Sorting by "is the target of a foreign key" puts the tables everything
points at into the left columns, so the majority of edges run left to right
and stop crossing each other. Same trick the explorer uses.
"""
referenced = {e["target"] for e in edges}
ordered = sorted(tables, key=lambda t: (t["id"] not in referenced, t["id"]))
cols = max(2, int((len(ordered) * 1.2) ** 0.5 + 0.999))
col_w = CARD_W + COL_GAP
cursor = [PAD] * cols
pos: dict[str, tuple[int, int]] = {}
for i, table in enumerate(ordered):
col = i % cols
pos[table["id"]] = (col * col_w + PAD, cursor[col])
cursor[col] += _card_height(table, columns) + ROW_GAP
return pos
def _field_y(table: dict, index: int, top: int) -> float:
header = HDR_H_DOC if table["doc"] else HDR_H
return top + header + (max(index, 0) + 0.5) * FIELD_H
def emit(ir: dict, style, *, columns: bool = True) -> str:
"""IR (a db document) + Style -> SVG text.
`columns=False` draws the header of every card and none of its contents —
the whole schema at a glance, which is what you want before you know which
table you care about. Two hundred tables with their columns is a reference;
two hundred names is a map.
"""
tables = _tables(ir)
if not tables:
raise ValueError(
"no tables in this IR — erd draws a schema, and this one has none. "
"Was it extracted with the python reader?"
)
edges = [e for e in ir["edges"] if e["kind"] in ("foreign_key", "references")]
by_id = {t["id"]: t for t in tables}
pos = layout(tables, edges, columns)
s = style.slot
width = max(x for x, _ in pos.values()) + CARD_W + PAD
height = max(y + _card_height(by_id[t], columns) for t, (_, y) in pos.items()) + PAD
out = [
'<?xml version="1.0" encoding="UTF-8" standalone="no"?>',
f'<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" '
f'width="{width}pt" height="{height}pt" viewBox="0 0 {width} {height}">',
f'<rect width="{width}" height="{height}" fill="{s("surface-0")}"/>',
"<defs>",
f'<marker id="fk" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" '
f'markerHeight="6" orient="auto-start-reverse">'
f'<path d="M 0 0 L 10 5 L 0 10 z" fill="{s("station")}"/></marker>',
"</defs>",
]
# Edges first, so cards sit on top of them where they meet.
out.append('<g class="relationships">')
for e in edges:
src, dst = by_id.get(e["source"]), by_id.get(e["target"])
if not src or not dst:
continue
sx, sy_top = pos[src["id"]]
dx_, dy_top = pos[dst["id"]]
label = (e.get("attrs") or {}).get("label")
from_idx = next(
(i for i, f in enumerate(src["fields"]) if f.get("label") == label), 0
)
to_idx = next(
(i for i, f in enumerate(dst["fields"]) if (f.get("attrs") or {}).get("pk")), 0
)
# Leave from whichever side faces the target, so a line never crosses
# its own card to get out.
leaving_right = dx_ >= sx
x1 = sx + CARD_W if leaving_right else sx
x2 = dx_ if leaving_right else dx_ + CARD_W
if columns:
y1 = _field_y(src, from_idx, sy_top)
y2 = _field_y(dst, to_idx, dy_top)
else:
y1 = sy_top + _card_height(src, False) / 2
y2 = dy_top + _card_height(dst, False) / 2
ctrl = min(max(abs(x2 - x1) * 0.5, 50), 180)
c1 = x1 + ctrl if leaving_right else x1 - ctrl
c2 = x2 - ctrl if leaving_right else x2 + ctrl
out.append(
f'<path d="M {x1},{y1:.1f} C {c1},{y1:.1f} {c2},{y2:.1f} {x2},{y2:.1f}" '
f'fill="none" stroke="{s("station")}" stroke-width="1.5" '
f'marker-end="url(#fk)" class="edge {e["kind"]}"/>'
)
out.append("</g>")
# Cards.
for table in tables:
x, y = pos[table["id"]]
header = HDR_H_DOC if table["doc"] else HDR_H
h = _card_height(table, columns)
out.append(f'<g class="table">')
out.append(
f'<rect x="{x}" y="{y}" width="{CARD_W}" height="{h}" rx="{RADIUS}" '
f'fill="{s("surface-0")}" stroke="{s("border")}" stroke-width="1" '
f'data-id="{escape(table["id"])}" data-kind="table" class="blk"/>'
)
# Header band, clipped to the card's rounded top by drawing a rounded
# rect and squaring its bottom with a second one.
out.append(
f'<path d="M {x},{y + RADIUS} a {RADIUS},{RADIUS} 0 0 1 {RADIUS},{-RADIUS} '
f'h {CARD_W - 2 * RADIUS} a {RADIUS},{RADIUS} 0 0 1 {RADIUS},{RADIUS} '
f'v {header - RADIUS} h {-CARD_W} z" fill="{s("surface-2")}"/>'
)
if columns:
out.append(
f'<line x1="{x}" y1="{y + header}" x2="{x + CARD_W}" y2="{y + header}" '
f'stroke="{s("border")}" stroke-width="1"/>'
)
out.append(
f'<text x="{x + 12}" y="{y + 22}" font-family="Helvetica,sans-Serif" '
f'font-size="12" font-weight="bold" fill="{s("text")}">'
f'{escape(_truncate(table["name"], CARD_W - 24))}</text>'
)
if table["doc"]:
out.append(
f'<text x="{x + 12}" y="{y + 38}" font-family="Helvetica,sans-Serif" '
f'font-size="9" fill="{s("text-dim")}">'
f'{escape(_truncate(table["doc"], CARD_W - 24))}</text>'
)
for i, field in enumerate(table["fields"] if columns else []):
fy = y + header + i * FIELD_H
attrs = field.get("attrs") or {}
name = field.get("label") or field["id"].rsplit(".", 1)[-1]
if attrs.get("pk"):
badge, badge_fill = "PK", s("accent")
elif attrs.get("references"):
badge, badge_fill = "FK", s("station")
else:
badge, badge_fill = "", s("text-dim")
if i:
out.append(
f'<line x1="{x + 1}" y1="{fy}" x2="{x + CARD_W - 1}" y2="{fy}" '
f'stroke="{s("surface-2")}" stroke-width="1"/>'
)
if badge:
out.append(
f'<text x="{x + 12}" y="{fy + 17}" font-family="Helvetica,sans-Serif" '
f'font-size="8" font-weight="bold" fill="{badge_fill}">{badge}</text>'
)
out.append(
f'<text x="{x + 12 + BADGE_W}" y="{fy + 17}" '
f'font-family="Helvetica,sans-Serif" font-size="10" '
f'fill="{s("text") if not attrs.get("nullable") else s("text-muted")}">'
f'{escape(_truncate(name, 96))}</text>'
)
type_text = attrs.get("references") or attrs.get("type", "")
if type_text:
out.append(
f'<text x="{x + CARD_W - 12}" y="{fy + 17}" text-anchor="end" '
f'font-family="Helvetica,sans-Serif" font-size="9" '
f'fill="{s("text-dim")}">{escape(_truncate(str(type_text), 60))}</text>'
)
out.append("</g>")
out.append("</svg>")
return "\n".join(out) + "\n"

View File

@@ -0,0 +1,315 @@
"""
Two panes: a minimap to navigate by, and a detail pane to explore with.
The minimap on its own showed shape and no meaning — a block said "a 30-line
class" and not *which* class, or what it touched. It is not the artifact. It is
the **selector**.
left the whole tree as coloured blocks. Scan, then click.
right what that block is, what it reaches, what reaches it — and the
neighbourhood drawn, small enough to read.
**This is also what solves the 14:1 problem.** The whole-graph diagram was
unusable because 109 nodes sat at one dependency level, and no engine draws that
well. Here the whole graph is never drawn: the minimap carries the overview, and
only the neighbourhood of a selection is rendered — a handful of nodes, which
lays out fine every time. Overview and detail stop competing for one picture.
## Selecting for an LLM
The other reason to navigate a tree quickly is to decide what to feed a model.
Blocks can be added to a basket, and the basket is a copyable list of file paths
plus a line count — enough to hand to `distill` or paste, and enough to see that
the selection got too big before spending the context on it.
## Offline and static
The neighbourhood diagrams are rendered at build time, one small SVG per module,
so the page needs no layout engine, no server and no network. Everything is
computed here and read there.
"""
import json
import re
from html import escape
from pathlib import Path
MAX_NEIGHBOURS = 24 # past this, a list reads better than a picture
def _is_schema(ir: dict) -> bool:
return (ir.get("meta") or {}).get("source") == "db" or any(
n["kind"] == "table" for n in ir["nodes"]
)
def _neighbourhood_svgs(ir: dict, style, out_dir: Path, hops: int) -> dict[str, str]:
"""One small diagram per navigable thing, pre-rendered. id -> filename.
A schema walks table by table: the selected table **with its columns**, plus
the tables its keys reach, each clickable to step further. A codebase walks
module by module through its imports. Same operation, different drawing —
which is the `classify` rule applied one level down.
"""
from ..ops import neighbourhood
schema = _is_schema(ir)
if schema:
from .erd import emit as draw
def render_one(view):
return draw(view, style).encode()
wanted = "table"
else:
from .dot import emit as dot_emit, have_graphviz, render
if not have_graphviz():
return {}
def render_one(view):
return render(dot_emit(view, style))
wanted = "module"
graphs_dir = out_dir / "graphs"
graphs_dir.mkdir(parents=True, exist_ok=True)
made: dict[str, str] = {}
for node in ir["nodes"]:
if node["kind"] != wanted:
continue
# A table without its columns is not a table, so a schema's
# neighbourhood carries contents; a module's does not, because its
# contents are the hundred functions that made the sheet unreadable.
view = neighbourhood(ir, node["id"], hops=hops, with_contents=schema)
if len(view["nodes"]) < 2:
continue
if not schema and len(view["nodes"]) > MAX_NEIGHBOURS:
continue
if schema and sum(1 for n in view["nodes"] if n["kind"] == "table") > 12:
continue
name = re.sub(r"[^A-Za-z0-9_.-]", "_", node["id"]) + ".svg"
try:
(graphs_dir / name).write_bytes(render_one(view))
except Exception: # noqa: BLE001 - one bad graph must not cost the page
continue
made[node["id"]] = f"graphs/{name}"
return made
def _facts(ir: dict) -> dict:
"""Everything the detail pane needs, keyed by id."""
out: dict[str, dict] = {}
for n in ir["nodes"]:
a = n.get("attrs") or {}
out[n["id"]] = {
"label": n.get("label") or n["id"],
"kind": n["kind"],
"parent": n.get("parent"),
"file": a.get("file"),
"line": a.get("line"),
"lines": a.get("lines"),
"doc": a.get("doc"),
"error": a.get("error"),
"out": [],
"in": [],
"members": [],
}
for n in ir["nodes"]:
if n.get("parent") in out:
out[n["parent"]]["members"].append(n["id"])
for e in ir["edges"]:
if e["source"] in out:
out[e["source"]]["out"].append([e["target"], e["kind"]])
if e["target"] in out:
out[e["target"]]["in"].append([e["source"], e["kind"]])
return out
def emit(ir: dict, style, minimap_svg: str, graphs: dict[str, str],
title: str = "") -> str:
s = style.slot
meta = ir.get("meta", {})
name = title or meta.get("root", "explore")
facts = _facts(ir)
# The minimap goes inline rather than in an <img>: a block has to be
# clickable, and an image is one opaque rectangle.
inner = minimap_svg[minimap_svg.index("<svg"):]
return f"""<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{escape(name)} — explore</title>
<style>
:root {{
--bg: {s("surface-0")}; --surface: {s("surface-1", s("surface-2"))};
--surface-2: {s("surface-2")}; --border: {s("border")};
--text: {s("text")}; --muted: {s("text-muted")}; --dim: {s("text-dim")};
--accent: {s("accent")};
}}
* {{ margin: 0; padding: 0; box-sizing: border-box; }}
body {{ background: var(--bg); color: var(--text); overflow: hidden;
font-family: "Segoe UI", Inter, system-ui, -apple-system, Arial, sans-serif;
font-size: 12px; line-height: 1.6; }}
.split {{ display: flex; height: 100vh; }}
/* navigate */
.nav {{ flex: 1 1 58%; overflow: auto; padding: 14px; position: relative; }}
.nav svg {{ display: block; }}
.blk {{ cursor: pointer; }}
.blk:hover {{ stroke: var(--accent); stroke-width: 1.5; }}
.blk.sel {{ stroke: var(--accent); stroke-width: 2; }}
.blk.basket {{ stroke: var(--accent); stroke-width: 1; stroke-dasharray: 2 2; }}
/* explore */
.side {{ flex: 0 0 42%; max-width: 560px; border-left: 1px solid var(--border);
background: var(--surface); overflow: auto; padding: 18px 20px; }}
.side h2 {{ font-size: 15px; margin-bottom: 2px; }}
.side .kind {{ color: var(--accent); font-size: 10px; text-transform: uppercase;
letter-spacing: .05em; }}
.side .path {{ color: var(--dim); font-size: 11px; margin: 6px 0 12px;
font-family: ui-monospace, Consolas, monospace; }}
.side .doc {{ color: var(--muted); margin-bottom: 14px; }}
.side h3 {{ font-size: 10px; text-transform: uppercase; letter-spacing: .05em;
color: var(--dim); margin: 16px 0 6px; }}
.side ul {{ list-style: none; }}
.side li {{ padding: 2px 0; color: var(--muted); }}
.side a {{ color: var(--muted); text-decoration: none; cursor: pointer; }}
.side a:hover {{ color: var(--accent); }}
.side .rel {{ color: var(--dim); font-size: 10px; margin-left: .4em; }}
.side img {{ width: 100%; border: 1px solid var(--border); border-radius: 6px;
background: var(--bg); margin-top: 6px; }}
.empty {{ color: var(--dim); }}
.bar {{ position: sticky; top: 0; background: var(--surface);
border-bottom: 1px solid var(--border); margin: -18px -20px 14px;
padding: 10px 20px; display: flex; gap: 10px; align-items: center; }}
button {{ background: var(--surface-2); color: var(--muted); cursor: pointer;
border: 1px solid var(--border); border-radius: 5px; padding: 4px 9px;
font-size: 11px; font-family: inherit; }}
button:hover {{ color: var(--text); border-color: var(--accent); }}
#basket {{ color: var(--accent); }}
textarea {{ width: 100%; height: 120px; background: var(--bg); color: var(--muted);
border: 1px solid var(--border); border-radius: 6px; padding: 8px;
font-family: ui-monospace, Consolas, monospace; font-size: 10px; }}
</style>
</head>
<body>
<div class="split">
<div class="nav" id="nav">{inner}</div>
<aside class="side">
<div class="bar">
<b>{escape(name)}</b>
<span class="rel" id="basket">0 selected</span>
<button onclick="showBasket()">selection</button>
<button onclick="clearBasket()">clear</button>
</div>
<div id="detail"><p class="empty">Click a block. Shift-click adds it to the
selection, for feeding somewhere else.</p></div>
</aside>
</div>
<script>
var FACTS = {json.dumps(facts)};
var GRAPHS = {json.dumps(graphs)};
var basket = [];
function esc(t) {{ return String(t).replace(/[&<>"]/g, function (c) {{
return {{'&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;'}}[c]; }}); }}
function link(id, rel) {{
var f = FACTS[id];
var label = f ? f.label : id;
return '<li><a onclick="select(\\'' + id.replace(/'/g, "\\\\'") + '\\')">' +
esc(label) + '</a>' + (rel ? '<span class="rel">' + esc(rel) + '</span>' : '') +
'<span class="rel">' + esc(id) + '</span></li>';
}}
function group(pairs) {{
if (!pairs.length) return '<p class="empty">none</p>';
return '<ul>' + pairs.slice(0, 60).map(function (p) {{ return link(p[0], p[1]); }}).join('') + '</ul>';
}}
function select(id) {{
var f = FACTS[id];
if (!f) return;
document.querySelectorAll('.blk.sel').forEach(function (b) {{ b.classList.remove('sel'); }});
document.querySelectorAll('[data-id="' + CSS.escape(id) + '"]').forEach(function (b) {{
b.classList.add('sel');
}});
var h = '<h2>' + esc(f.label) + '</h2><div class="kind">' + esc(f.kind) + '</div>';
var where = f.file ? f.file + (f.line ? ':' + f.line : '') : id;
h += '<div class="path">' + esc(where) + (f.lines ? ' · ' + f.lines + ' lines' : '') + '</div>';
if (f.doc) h += '<div class="doc">' + esc(f.doc) + '</div>';
if (f.error) h += '<div class="doc">⚠ ' + esc(f.error) + '</div>';
if (GRAPHS[id]) {{
h += '<h3>neighbourhood</h3><img src="' + GRAPHS[id] + '" alt="">';
}}
h += '<h3>reaches (' + f.out.length + ')</h3>' + group(f.out);
h += '<h3>reached by (' + f['in'].length + ')</h3>' + group(f['in']);
if (f.members.length) {{
h += '<h3>contains (' + f.members.length + ')</h3>' +
group(f.members.map(function (m) {{ return [m, FACTS[m] ? FACTS[m].kind : '']; }}));
}}
document.getElementById('detail').innerHTML = h;
}}
function toggleBasket(id) {{
var i = basket.indexOf(id);
if (i >= 0) basket.splice(i, 1); else basket.push(id);
document.querySelectorAll('[data-id="' + CSS.escape(id) + '"]').forEach(function (b) {{
b.classList.toggle('basket', basket.indexOf(id) >= 0);
}});
document.getElementById('basket').textContent = basket.length + ' selected';
}}
function showBasket() {{
// A copyable list of paths and a line count — enough to hand to distill, and
// enough to see the selection got too big before spending context on it.
var files = [], lines = 0, seen = {{}};
basket.forEach(function (id) {{
var f = FACTS[id];
if (!f) return;
var p = f.file || id;
if (!seen[p]) {{ seen[p] = 1; files.push(p); lines += (f.lines || 0); }}
}});
document.getElementById('detail').innerHTML =
'<h2>Selection</h2><div class="kind">' + files.length + ' files · ~' + lines +
' lines</div><div class="path">Paste, or feed to distill.</div>' +
'<textarea readonly>' + esc(files.join('\\n')) + '</textarea>';
}}
function clearBasket() {{
basket = [];
document.querySelectorAll('.blk.basket').forEach(function (b) {{ b.classList.remove('basket'); }});
document.getElementById('basket').textContent = '0 selected';
document.getElementById('detail').innerHTML = '<p class="empty">Cleared.</p>';
}}
document.getElementById('nav').addEventListener('click', function (e) {{
var b = e.target.closest('.blk');
if (!b) return;
if (e.shiftKey) toggleBasket(b.dataset.id); else select(b.dataset.id);
}});
</script>
</body>
</html>
"""
def write(ir: dict, style, out_dir, *, scale: float = 0.55, width: int = 1100,
hops: int = 1, title: str = "") -> Path:
from .minimap import emit as minimap_emit
out_dir = Path(out_dir)
out_dir.mkdir(parents=True, exist_ok=True)
if _is_schema(ir):
# The whole schema, no columns: a map rather than a reference. The
# minimap's line-span geometry means nothing for a table.
from .erd import emit as erd_emit
svg = erd_emit(ir, style, columns=False)
else:
svg = minimap_emit(ir, style, scale=scale, target_width=width)
graphs = _neighbourhood_svgs(ir, style, out_dir, hops)
path = out_dir / "explore.html"
path.write_text(emit(ir, style, svg, graphs, title=title))
return path

View File

@@ -0,0 +1,163 @@
"""
IR -> an index. Markdown for reading, JSON for a sidebar.
**This is the emitter that matters most for reach.** A sorted, described list of
what exists is readable by people who will never open a diagram — a PM checking
that a feature has a home, QA looking for the surface to test, someone new
trying to find where anything is. A diagram asks for graph literacy and a
screen; this asks for neither.
It is also the checkpoint on the whole design. If the IR were secretly
diagram-shaped, this emitter would be awkward to write — it would be reaching
for positions, or re-deriving containment from edges. It is not, because
`parent` is containment and `kind` is meaning, and that is all a table of
contents needs.
python3 -m docgen.emitters index ir.json # markdown to stdout
python3 -m docgen.emitters index ir.json -o x.json # sidebar JSON
## What it reports that a diagram cannot
- **what depends on what is outside**, gathered in one place. `external` nodes
are the project's real dependency surface, and in a diagram they are scattered
boxes.
- **what could not be parsed.** A file the extractor choked on is a hole in the
analysis; it is listed rather than quietly absent.
"""
import json
from collections import Counter, defaultdict
# Order matters for reading, not for correctness: containers before contents.
KIND_ORDER = ["module", "class", "function", "table", "column", "external"]
def _tree(ir: dict):
children = defaultdict(list)
for n in ir["nodes"]:
children[n.get("parent")].append(n)
for kids in children.values():
kids.sort(key=lambda n: (KIND_ORDER.index(n["kind"]) if n["kind"] in KIND_ORDER else 99,
n["id"]))
return children
def _anchor(node: dict) -> str:
attrs = node.get("attrs") or {}
if not attrs.get("file"):
return ""
return f"{attrs['file']}:{attrs['line']}" if attrs.get("line") else attrs["file"]
def to_markdown(ir: dict, title: str = "") -> str:
"""A document. Headings for containers, a list for their contents."""
children = _tree(ir)
nodes = {n["id"]: n for n in ir["nodes"]}
meta = ir.get("meta", {})
counts = Counter(n["kind"] for n in ir["nodes"])
edge_counts = Counter(e["kind"] for e in ir["edges"])
out = [f"# {title or meta.get('root', 'index')}", ""]
out.append(
f"Extracted from `{meta.get('root', '?')}` by the `{meta.get('source', '?')}` "
f"reader. {len(ir['nodes'])} nodes, {len(ir['edges'])} edges."
)
out.append("")
out.append("| | |")
out.append("|---|---|")
for kind, n in sorted(counts.items(), key=lambda kv: -kv[1]):
out.append(f"| {kind} | {n} |")
for kind, n in sorted(edge_counts.items(), key=lambda kv: -kv[1]):
out.append(f"| *{kind}* (edges) | {n} |")
out.append("")
# -- the contents -----------------------------------------------------
def walk(node: dict, depth: int):
kids = [k for k in children.get(node["id"], [])]
doc = (node.get("attrs") or {}).get("doc")
anchor = _anchor(node)
if depth == 0:
out.append(f"## {node['label']} <small>{node['kind']}</small>")
out.append("")
if doc:
out.append(doc)
out.append("")
if anchor:
out.append(f"`{anchor}`")
out.append("")
else:
bullet = " " * (depth - 1) + "-"
parts = [f"**{node['label']}**", f"*{node['kind']}*"]
if doc:
parts.append(f"{doc}")
if anchor:
parts.append(f"`{anchor}`")
out.append(f"{bullet} {' '.join(parts)}")
for kid in kids:
walk(kid, depth + 1)
if depth == 0 and kids:
out.append("")
roots = [n for n in children.get(None, []) if n["kind"] != "external"]
for root in roots:
walk(root, 0)
# -- what is outside --------------------------------------------------
externals = sorted(n["id"] for n in ir["nodes"] if n["kind"] == "external")
if externals:
out.append("## Depends on, outside this tree")
out.append("")
out.append(
"Names that could not be resolved to anything in the source. This is the "
"dependency surface — third-party imports, and anything reached dynamically."
)
out.append("")
incoming = Counter(e["target"] for e in ir["edges"] if e["target"] in set(externals))
for eid in sorted(externals, key=lambda e: (-incoming[e], e)):
n = incoming[eid]
out.append(f"- `{eid}`" + (f" — referenced {n}×" if n > 1 else ""))
out.append("")
# -- holes in the analysis --------------------------------------------
broken = [n for n in ir["nodes"] if (n.get("attrs") or {}).get("error")]
if broken:
out.append("## Not parsed")
out.append("")
out.append("These files were skipped, so anything they define is missing below.")
out.append("")
for n in sorted(broken, key=lambda n: n["id"]):
out.append(f"- `{(n.get('attrs') or {}).get('file', n['id'])}` — "
f"{(n.get('attrs') or {}).get('error')}")
out.append("")
return "\n".join(out).rstrip() + "\n"
def to_sidebar(ir: dict) -> dict:
"""Nested JSON for a navigation pane.
Shaped for a UI to render directly: `label`, `kind`, `href`, `children`.
"""
children = _tree(ir)
def build(node: dict) -> dict:
attrs = node.get("attrs") or {}
item = {"id": node["id"], "label": node["label"], "kind": node["kind"]}
if attrs.get("doc"):
item["doc"] = attrs["doc"]
if attrs.get("file"):
item["href"] = (
f"{attrs['file']}#L{attrs['line']}" if attrs.get("line") else attrs["file"]
)
kids = [build(k) for k in children.get(node["id"], [])]
if kids:
item["children"] = kids
return item
return {
"meta": ir.get("meta", {}),
"items": [build(n) for n in children.get(None, []) if n["kind"] != "external"],
"external": sorted(n["id"] for n in ir["nodes"] if n["kind"] == "external"),
}

View File

@@ -0,0 +1,277 @@
"""
IR -> a structural minimap. What is where, readable without reading.
Sublime's minimap shrinks the *characters*. This draws the *structure* at full
scale: one file is one column, one line is a fixed number of pixels, and every
construct is a block sized by the lines it actually occupies and coloured by
what it is. Nothing is summarised and no text is rendered — the point is to see
the shape of a codebase without reading a line of it.
a tall solid block one long class
a column of thin stripes many small functions
a wide pale gap module-level code, imports, comments
one block filling a file the 800-line thing everybody avoids
**The pattern has to come from the colours alone**, so `kind` is the only thing
that varies and nesting is drawn by inset rather than by hue: a method inside a
class is the class's colour, indented. Scanning a hundred files then shows which
are class-shaped, which are a pile of loose functions, and which are one block.
Not a node-edge graph, and deliberately not forced into one — it answers "what
is where", which a dependency diagram never does.
## Layout
Files are grouped by their package, packed left to right into shelves, and each
shelf is as tall as its tallest file. Grouping by package is what makes *where*
legible: the shape of a subsystem is the shape of its band.
## Geometry
Computed, never measured — `lines × SCALE`. Deterministic, portable, and
independent of any font, which is the same property `erd` has and `dot` does
not.
"""
from html import escape
COL_W = 74 # one file
COL_GAP = 8
SCALE = 0.55 # pixels per line of source
MIN_H = 14
PAD = 28
LABEL_H = 18
ROW_GAP = 26
PKG_GAP = 18
INSET = 7 # per level of nesting
TARGET_W = 1180 # wrap a shelf past this
# Which slot each kind is drawn in. Everything else lands on `default`, so an
# unfamiliar vocabulary still renders rather than vanishing.
KIND_SLOT = {
"class": "station",
"interface": "accent",
"function": "atlas",
"module": "surface-2",
"table": "station",
"endpoint": "accent",
"operation": "accent",
"task": "station",
"external": "muted",
}
def _files(ir: dict) -> list[dict]:
"""Modules with their constructs, nested, each carrying a line span."""
by_id = {n["id"]: n for n in ir["nodes"]}
kids: dict[str, list] = {}
for n in ir["nodes"]:
if n.get("parent"):
kids.setdefault(n["parent"], []).append(n)
def declared(node: dict) -> list:
"""A node's drawable children, with wrappers dissolved.
A C# `namespace` and a TypeScript `module` come through as `kind:
"module"` nested inside a file. Drawing them adds a level of inset to
everything without adding information — and worse, the file's own
contents then appear twice. They are descended through and not drawn.
"""
out = []
for c in sorted(kids.get(node["id"], []),
key=lambda c: ((c.get("attrs") or {}).get("line", 0))):
if not (c.get("attrs") or {}).get("line"):
continue
if c["kind"] == "module":
out.extend(declared(c)) # a wrapper: keep what is inside it
else:
out.append(c)
return out
def build(node: dict, depth: int) -> dict:
a = node.get("attrs") or {}
return {
"id": node["id"],
"label": node.get("label") or node["id"],
"kind": node["kind"],
"line": a.get("line", 1),
"lines": max(1, a.get("lines", 1)),
"depth": depth,
"children": [build(c, depth + 1) for c in declared(node)],
}
out = []
for n in ir["nodes"]:
if n["kind"] != "module":
continue
a = n.get("attrs") or {}
total = a.get("lines")
if not total:
continue
# A file, not a namespace. Both arrive as `module`; only a file has a
# length without a starting line, because a file starts at the start.
if a.get("line"):
continue
node = build(n, 0)
node["total"] = total
node["package"] = n["id"].rsplit(".", 1)[0] if "." in n["id"] else ""
node["error"] = a.get("error")
out.append(node)
return out
def _bands(files: list[dict]) -> list[tuple[str, list]]:
groups: dict[str, list] = {}
for f in files:
groups.setdefault(f["package"] or "(root)", []).append(f)
for members in groups.values():
members.sort(key=lambda f: -f["total"])
return sorted(groups.items(), key=lambda kv: (-len(kv[1]), kv[0]))
def _blocks(node: dict, top: float, height: float, x: float, out: list, style, file_lines: int):
"""Place a construct and everything inside it."""
slot = KIND_SLOT.get(node["kind"], "border")
inset = INSET * max(0, node["depth"] - 1)
out.append({
"x": x + inset,
"y": top,
"w": COL_W - 2 * inset,
"h": max(2.0, height),
"fill": style.slot(slot, style.slot("border")),
"depth": node["depth"],
"id": node["id"],
"kind": node["kind"],
"title": f'{node["label"]}{node["kind"]}, {node["lines"]} lines',
})
for child in node["children"]:
offset = (child["line"] - node["line"]) / max(node["lines"], 1)
child_h = child["lines"] / max(node["lines"], 1) * height
_blocks(child, top + offset * height, child_h, x, out, style, file_lines)
def emit(ir: dict, style, *, scale: float = SCALE, target_width: int = TARGET_W) -> str:
"""IR + Style -> SVG text. No Graphviz, no font measurement."""
files = _files(ir)
if not files:
raise ValueError(
"nothing to map — a minimap needs modules carrying `attrs.lines`. "
"Was this extracted with the python reader?"
)
s = style.slot
placed, labels, marks = [], [], []
x, y, shelf_h, max_x = PAD, PAD + LABEL_H, 0, 0
current_package = None
# One continuous flow, files ordered by package, wrapping at the target
# width — rather than a row per package. A real tree has many small
# packages (soleprint: 234 files in 70) and a row each gave a 1196x10165
# ribbon. The aspect ratio has to be chosen, not left to emerge; the
# grouping stays legible because a package's files are still adjacent.
ordered = [(pkg, f) for pkg, members in _bands(files) for f in members]
for package, f in ordered:
height = max(MIN_H, f["total"] * scale)
starts_package = package != current_package
gap = COL_GAP + (PKG_GAP if starts_package and x > PAD else 0)
if x + gap + COL_W > target_width and x > PAD:
y += shelf_h + ROW_GAP
x, shelf_h, gap = PAD, 0, 0
starts_package = True # a wrap re-labels, so a row is readable alone
elif x > PAD:
x += gap
if starts_package:
marks.append({"x": x, "y": y - 6, "text": package})
current_package = package
placed.append({
"x": x, "y": y, "w": COL_W, "h": height,
"fill": s("surface-1", s("surface-2")), "depth": -1,
"title": f'{f["id"]}{f["total"]} lines'
+ (f' — NOT PARSED: {f["error"]}' if f["error"] else ""),
"outline": s("border"),
})
for child in f["children"]:
offset = (child["line"] - 1) / max(f["total"], 1)
_blocks(child, y + offset * height,
child["lines"] / max(f["total"], 1) * height,
x, placed, style, f["total"])
labels.append({"x": x, "y": y + height + 9, "text": f["label"][:11], "band": False})
shelf_h = max(shelf_h, height + 12)
max_x = max(max_x, x + COL_W)
x += COL_W
y += shelf_h + PAD
labels = marks_to_labels(marks) + labels
width = max(max_x + PAD, 420)
height = y + 30
out = [
'<?xml version="1.0" encoding="UTF-8" standalone="no"?>',
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width:.0f}pt" '
f'height="{height:.0f}pt" viewBox="0 0 {width:.0f} {height:.0f}">',
f'<rect width="{width:.0f}" height="{height:.0f}" fill="{s("surface-0")}"/>',
]
for b in placed:
extra = (f' stroke="{b["outline"]}" stroke-width="1"' if b.get("outline")
else ' stroke="none"')
# Nested blocks sit on their parent, so a little transparency keeps the
# containment readable instead of hiding it.
opacity = "" if b["depth"] < 0 else f' opacity="{0.95 if b["depth"] <= 1 else 0.8}"'
ident = (f' data-id="{escape(b["id"])}" data-kind="{b.get("kind", "")}" '
f'class="blk"' if b.get("id") else "")
out.append(
f'<rect x="{b["x"]:.1f}" y="{b["y"]:.1f}" width="{b["w"]:.1f}" '
f'height="{b["h"]:.1f}" rx="2" fill="{b["fill"]}"{extra}{opacity}{ident}>'
f'<title>{escape(b["title"])}</title></rect>'
)
for lab in labels:
if lab["band"]:
out.append(
f'<text x="{lab["x"]:.0f}" y="{lab["y"]:.0f}" font-family="Helvetica,sans-Serif" '
f'font-size="10" font-weight="bold" fill="{s("text-muted")}">'
f'{escape(lab["text"])}</text>'
)
else:
out.append(
f'<text x="{lab["x"]:.0f}" y="{lab["y"]:.0f}" font-family="Helvetica,sans-Serif" '
f'font-size="7" fill="{s("text-dim")}">{escape(lab["text"])}</text>'
)
# A legend, because the whole claim is that the colours carry the meaning.
lx = PAD
ly = height - 14
for kind in ("module", "class", "interface", "function"):
slot = KIND_SLOT.get(kind, "border")
out.append(
f'<rect x="{lx}" y="{ly - 7}" width="9" height="9" rx="2" '
f'fill="{s(slot, s("border"))}"/>'
)
out.append(
f'<text x="{lx + 13}" y="{ly + 1}" font-family="Helvetica,sans-Serif" '
f'font-size="9" fill="{s("text-dim")}">{kind}</text>'
)
lx += 22 + len(kind) * 6
if lx > width - 220:
ly += 13 # the legend ran into the summary; put it on its own line
out.append(
f'<text x="{width - PAD:.0f}" y="{ly + 1}" text-anchor="end" '
f'font-family="Helvetica,sans-Serif" font-size="9" fill="{s("text-dim")}">'
f'{len(files)} files · {sum(f["total"] for f in files):,} lines · '
f'1px ≈ {1 / scale:.1f} lines</text>'
)
out.append("</svg>")
return "\n".join(out) + "\n"
def marks_to_labels(marks: list) -> list:
"""Package names, as band labels above the first file of each group."""
return [{"x": m["x"], "y": m["y"], "text": m["text"], "band": True} for m in marks]

View File

@@ -0,0 +1,278 @@
"""
IR -> a Jupyter notebook. Generated, never hand-authored.
## The frame, because it is the whole argument
A notebook is normally a **source file** that someone confects by hand — prose,
code and stored output braided together, diffing badly, drifting from whatever
it documents the moment either moves, with no way to tell by looking. jupytext
addresses the diffing and leaves the rest: it makes the notebook editable as
text, so you still hand-author it.
Here a notebook is a **build artifact**. The source is the OpenAPI document —
the same file the server is built from — and the notebook is regenerated from
it. Nobody edits the `.ipynb`, the same way nobody edits a `.o` file. "Is this
document current" stops being a question about somebody's diligence and becomes
a question about whether the build ran.
That is also why it suits a mixed audience rather than a data-science one. The
endpoints, their methods, their payload shapes and their status codes are facts
taken from the spec, so a PM reading it is reading the API, not somebody's
recollection of it.
## Reproducible in the strict sense
Cell ids come from position, `execution_count` is null, `outputs` is empty, and
the JSON is key-sorted. **The same IR produces byte-identical bytes.** A
notebook that changes on every build cannot be reviewed, and one that cannot be
reviewed will not be trusted.
Written against the nbformat 4 schema directly. `nbformat` is not installed on
the machines this runs on, and the schema has six required keys.
"""
import json
from pathlib import Path
METADATA = {
"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
"language_info": {"name": "python"},
}
def _cell(index: int, kind: str, text: str) -> dict:
lines = text.split("\n")
source = [ln + "\n" for ln in lines[:-1]] + ([lines[-1]] if lines[-1] else [])
cell = {
"cell_type": "markdown" if kind == "md" else "code",
"id": f"cell-{index:03d}",
"metadata": {},
"source": source,
}
if kind == "code":
cell["execution_count"] = None
cell["outputs"] = []
return cell
def _example(fields: list[dict]) -> str:
"""A request body shaped like the schema, with placeholder values."""
sample = {}
for f in fields:
if f.get("pk"):
continue # the server assigns it
t = str(f.get("type", "str")).lower()
name = f["name"]
if "int" in t:
sample[name] = 0
elif "float" in t or "decimal" in t:
sample[name] = 0.0
elif "bool" in t:
sample[name] = False
elif "date" in t or "time" in t:
sample[name] = "2026-01-01T00:00:00Z"
elif "list" in t:
sample[name] = []
else:
sample[name] = f"<{name}>"
# A Python literal, not JSON. `json.dumps` writes `false`/`true`/`null`,
# which are valid *identifiers* in Python — so the cell compiles and then
# raises NameError the moment anyone runs it. Compiling is not enough of a
# check; the notebook selftest executes these cells for exactly this reason.
body = ",\n".join(f" {k!r}: {v!r}" for k, v in sorted(sample.items()))
return "{\n" + body + ",\n}" if body else "{}"
CLIENT = '''def call(method, path, params=None, body=None):
"""One request. Returns (status, parsed_body).
A non-2xx is returned rather than raised: the error body usually names the
field that was wrong, and an exception throws that away.
"""
url = BASE_URL.rstrip("/") + "/" + path.lstrip("/")
if params:
url += "?" + urllib.parse.urlencode(params)
data = json.dumps(body).encode() if body is not None else None
headers = {"Accept": "application/json", **AUTH}
if data:
headers["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=data, headers=headers, method=method)
print(f"-> {method} {url}")
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
status, raw = r.status, r.read()
except urllib.error.HTTPError as e:
status, raw = e.code, e.read()
except urllib.error.URLError as e:
print(f"<- unreachable: {e.reason}")
return None, None
try:
parsed = json.loads(raw) if raw else None
except json.JSONDecodeError:
parsed = raw.decode("utf-8", "replace")
print(f"<- {status}")
return status, parsed
def show(result, limit=1500):
status, body = result
if status is None:
return
text = body if isinstance(body, str) else json.dumps(body, indent=2)
print(text[:limit] + (f"\\n… {len(text) - limit} more" if len(text) > limit else ""))'''
def _params_cell(step: dict) -> str:
env = step.get("env_var", "API_TOKEN")
return f'''import json
import os
import urllib.error
import urllib.parse
import urllib.request
BASE_URL = os.environ.get("API_BASE_URL", "{step.get("base_url", "")}")
TOKEN = os.environ.get("{env}", "")
TIMEOUT = 30
# Read from the environment, never written here: a token pasted into a cell
# travels with every copy of this notebook from then on.
AUTH = {{"Authorization": f"Bearer {{TOKEN}}"}} if TOKEN else {{}}
print("base ", BASE_URL)
print("token", f"set ({{len(TOKEN)}} chars)" if TOKEN else "NOT SET — export {env}=…")'''
def _call_cell(step: dict) -> str:
"""The generated call. An overlay's `code` replaces this wholesale."""
method, path = step.get("method", "GET"), step.get("path", "/")
params = step.get("path_params") or []
if step.get("graphql"):
# One endpoint carries every operation, so the operation name is the
# thing worth showing, not the path.
variables = {f["name"]: f"<{f['name']}>" for f in (step.get("body_fields") or [])}
return (
f'QUERY = """{step.get("title", "query")} {{ ... }}""" '
"# fill in the selection set\n"
+ (f"VARIABLES = {variables!r}\n" if variables else "")
+ f'show(call("POST", "{path}", body={{"query": QUERY'
+ (", \"variables\": VARIABLES" if variables else "")
+ '}))'
)
lines = [f'{p.upper()} = "<{p}>" # path parameter' for p in params]
call_path = path
for p in params:
call_path = call_path.replace("{" + p + "}", f'" + str({p.upper()}) + "')
# Trim the empty concatenations a placeholder at either end leaves behind.
expr = f'"{call_path}"' if params else f'"{path}"'
expr = expr.replace(' + ""', "").replace('"" + ', "")
# Parameters that were *always* sent are not optional in practice, whatever
# the spec calls them.
always = step.get("params_always") or []
if always:
lines.append("PARAMS = " + repr({p: f"<{p}>" for p in always}))
arg = ", params=PARAMS" if always else ""
if step.get("body_fields"):
lines.append("BODY = " + _example(step["body_fields"]))
lines.append("")
lines.append(f'show(call("{method}", {expr}{arg}, body=BODY))')
else:
if lines:
lines.append("")
lines.append(f'show(call("{method}", {expr}{arg}))')
return "\n".join(lines)
def _call_md(step: dict) -> str:
out = [f'## {step.get("method", "GET")} `{step.get("path", "/")}`']
if step.get("summary"):
out += ["", step["summary"]]
facts = []
if step.get("response_model"):
facts.append(f'returns **{step["response_model"]}**'
+ (" (a list)" if step.get("returns_list") else ""))
if step.get("request_model"):
facts.append(f'accepts **{step["request_model"]}**')
if step.get("status"):
facts.append(f'expects `{step["status"]}`')
if step.get("statuses"):
# What really came back, which is usually more than the spec promises.
facts.append("seen: " + ", ".join(f'`{c}`' for c in step["statuses"]))
if step.get("calls"):
facts.append(f'called {step["calls"]}×')
if step.get("params_sometimes"):
facts.append("sometimes sends " + ", ".join(f'`{p}`' for p in step["params_sometimes"]))
if step.get("id_formats"):
facts.append("id as " + "/".join(step["id_formats"]))
if facts:
out += ["", " · ".join(facts)]
if step.get("note"):
out += ["", step["note"]]
return "\n".join(out)
def build(spec: dict) -> dict:
"""A merged notebook spec -> the notebook, as a dict."""
blocks: list[tuple[str, str]] = []
for step in spec["steps"]:
kind = step.get("kind", "md")
before = step.get("before")
if before:
blocks.append(("md", before))
if kind == "md":
text = []
if step.get("title") and step["id"] != "intro":
text.append(f'## {step["title"]}')
elif step.get("title"):
text.append(f'# {step["title"]}')
if step.get("text"):
text += ["", step["text"]]
if step.get("table"):
text += ["", "| | fields |", "|---|---|"]
for row in step["table"]:
names = ", ".join(f'`{f}`' for f in row.get("fields", []))
text.append(f'| **{row["name"]}** | {names or ""} |')
blocks.append(("md", "\n".join(text)))
elif kind == "params":
if step.get("title"):
blocks.append(("md", f'## {step["title"]}'))
blocks.append(("code", step.get("code") or _params_cell(step)))
elif kind == "code":
if step.get("title"):
blocks.append(("md", f'## {step["title"]}'))
code = step.get("code")
if code is None and step.get("builtin") == "client":
code = CLIENT
blocks.append(("code", code or ""))
elif kind == "call":
blocks.append(("md", _call_md(step)))
blocks.append(("code", step.get("code") or _call_cell(step)))
after = step.get("after_text")
if after:
blocks.append(("md", after))
return {
"cells": [_cell(i, k, t) for i, (k, t) in enumerate(blocks)],
"metadata": METADATA,
"nbformat": 4,
"nbformat_minor": 5,
}
def emit(spec: dict) -> str:
"""The notebook as text. Key-sorted, so the same spec gives the same bytes."""
return json.dumps(build(spec), indent=1, sort_keys=True, ensure_ascii=False) + "\n"
def write(spec: dict, path) -> Path:
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(emit(spec))
return path

View File

@@ -0,0 +1,493 @@
"""
IR -> a self-contained documentation site: sidebar, content, graph viewer.
Not invented here. Five demos under `semester/` already converged on the same
two files, and this generates that arrangement rather than a sixth variant:
docs/index.html a 220px sticky sidebar beside a max-800px content column
docs/viewer.html `?src=` → fit to window, wheel-zoom at cursor, drag to pan
`sms`, `mpr`, `cht`, `unt` and `eth` each carry a copy of that viewer. They are
**the same 97 lines**, differing only in comments and one background colour —
which is the same eight-ways-to-do-one-thing this whole tool exists to end.
The handoff between them is already a convention, in both `spr/docs/docs.js:229`
and `sms/docs/index.html:311`, arrived at independently:
<a href="viewer.html?src=X.svg"><img src="X.svg" title="Click to expand"></a>
Inline and scaled to the column; click for the full thing.
## What is added
**A 1:1 toggle.** The copied viewer fits on load and resets to fit on
double-click, and has no way to say "actual size" — which is the one thing you
want the moment a diagram has small text in it. A click toggles fit ↔ 100%,
with the current scale shown in the corner so it is never ambiguous which you
are looking at. A click that moved the mouse is a drag and does not toggle.
## Colours
Baked from the same style slots as every diagram, so the page and the graph on
it match by construction. `--theme lucid` produces a light site and a light
diagram together; nothing has to be kept in sync by hand.
Self-contained and offline: no CDN, no build step, opens over `file://`.
"""
import json
from html import escape
from pathlib import Path
SIDEBAR_W = 220
CONTENT_W = 800
# The viewer, with the toggle the copied ones lack. Kept as one string because
# it is one file and its whole value is that there is exactly one of it.
VIEWER = """<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>__TITLE__</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { background: __BG__; overflow: hidden; width: 100vw; height: 100vh;
font-family: __FONT__; }
#container { width: 100vw; height: 100vh; overflow: hidden; cursor: grab; }
#container.dragging { cursor: grabbing; }
img { transform-origin: 0 0; user-select: none; -webkit-user-drag: none; }
#hud {
position: fixed; bottom: 14px; left: 14px; display: flex; gap: 8px;
align-items: center; font-size: 11px; color: __MUTED__;
background: __SURFACE__; border: 1px solid __BORDER__;
border-radius: 6px; padding: 5px 9px; user-select: none;
}
#hud b { color: __TEXT__; font-weight: 600; font-variant-numeric: tabular-nums; }
#hud span { opacity: .7; }
a.back { position: fixed; top: 14px; left: 14px; font-size: 11px;
color: __MUTED__; text-decoration: none; background: __SURFACE__;
border: 1px solid __BORDER__; border-radius: 6px; padding: 5px 9px; }
a.back:hover { color: __TEXT__; }
</style>
</head>
<body>
<div id="container"><img id="img" alt=""></div>
<a class="back" href="index.html">&larr; docs</a>
<div id="hud"><b id="pct">100%</b><span id="mode">fit</span><span>&middot; click 1:1 &middot; drag &middot; wheel</span></div>
<script>
var src = new URLSearchParams(location.search).get('src');
var img = document.getElementById('img');
var container = document.getElementById('container');
var pct = document.getElementById('pct');
var modeEl = document.getElementById('mode');
if (src) { img.src = src; document.title = src + ' — __TITLE__'; }
var scale = 1, x = 0, y = 0, fitScale = 1, mode = 'fit';
var dragging = false, moved = false, startX, startY, startPanX, startPanY;
function apply() {
img.style.transform = 'translate(' + x + 'px,' + y + 'px) scale(' + scale + ')';
pct.textContent = Math.round(scale * 100) + '%';
modeEl.textContent = mode;
}
function fit() {
var sw = window.innerWidth / img.naturalWidth;
var sh = window.innerHeight / img.naturalHeight;
fitScale = Math.min(sw, sh) * 0.95;
scale = fitScale;
x = (window.innerWidth - img.naturalWidth * scale) / 2;
y = (window.innerHeight - img.naturalHeight * scale) / 2;
mode = 'fit';
apply();
}
// Zoom about a point in the viewport, so what is under the cursor stays there.
function zoomAt(px, py, factor) {
x = px - (px - x) * factor;
y = py - (py - y) * factor;
scale *= factor;
mode = Math.abs(scale - fitScale) < 0.001 ? 'fit'
: (Math.abs(scale - 1) < 0.001 ? '1:1' : 'free');
apply();
}
img.onload = fit;
window.addEventListener('resize', function () { if (mode === 'fit') fit(); });
container.addEventListener('wheel', function (e) {
e.preventDefault();
var rect = container.getBoundingClientRect();
zoomAt(e.clientX - rect.left, e.clientY - rect.top, e.deltaY < 0 ? 1.12 : 0.89);
}, { passive: false });
container.addEventListener('mousedown', function (e) {
if (e.button !== 0) return;
dragging = true; moved = false;
startX = e.clientX; startY = e.clientY; startPanX = x; startPanY = y;
container.classList.add('dragging');
e.preventDefault();
});
window.addEventListener('mousemove', function (e) {
if (!dragging) return;
if (Math.abs(e.clientX - startX) > 3 || Math.abs(e.clientY - startY) > 3) moved = true;
x = startPanX + (e.clientX - startX);
y = startPanY + (e.clientY - startY);
apply();
});
window.addEventListener('mouseup', function (e) {
if (!dragging) return;
dragging = false;
container.classList.remove('dragging');
// A click that moved the mouse was a drag, and must not also toggle.
if (moved) return;
if (mode === '1:1') { fit(); return; }
// Toggle to actual size about the point clicked, so the thing you aimed at
// is the thing you end up looking at.
var rect = container.getBoundingClientRect();
zoomAt(e.clientX - rect.left, e.clientY - rect.top, 1 / scale);
mode = '1:1';
apply();
});
container.addEventListener('dblclick', fit);
window.addEventListener('keydown', function (e) {
if (e.key === '0' || e.key === 'f') fit();
if (e.key === '1') { var r = container.getBoundingClientRect();
zoomAt(r.width / 2, r.height / 2, 1 / scale); mode = '1:1'; apply(); }
if (e.key === 'Escape') location.href = 'index.html';
});
</script>
</body>
</html>
"""
CSS = """/* Generated by docgen. The layout five demos converged on: a sticky sidebar
beside a bounded content column. Colours are baked from the style's theme, so
the page and the diagrams on it are one visual language. */
:root {
--bg: __BG__;
--surface: __SURFACE__;
--surface-2: __SURFACE2__;
--border: __BORDER__;
--text: __TEXT__;
--muted: __MUTED__;
--dim: __DIM__;
--accent: __ACCENT__;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--bg); color: var(--text);
font-family: __FONT__; font-size: 13px; line-height: 1.65;
}
.layout { display: flex; min-height: 100vh; }
.sidebar {
width: __SIDEBAR__px; flex-shrink: 0; background: var(--surface);
border-right: 1px solid var(--border);
position: sticky; top: 0; height: 100vh; overflow-y: auto;
padding: 1.25rem 0; scrollbar-width: none;
}
.sidebar::-webkit-scrollbar { display: none; }
.sidebar-header { padding: 0 1rem 1rem; border-bottom: 1px solid var(--border); }
.sidebar-header b { color: var(--text); font-size: 13px; }
.sidebar-header small { display: block; color: var(--dim); font-size: 10px; margin-top: 2px; }
.sidebar ul { list-style: none; }
/* Every link, however deep — a link inside a <summary> is still a link, and
selecting `li > a` quietly missed all of them. */
.sidebar a {
display: block; padding: 3px 1rem; color: var(--muted);
text-decoration: none; font-size: 12px; border-left: 2px solid transparent;
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
}
.sidebar a:hover { color: var(--text); background: var(--surface-2); }
.sidebar a.active { color: var(--accent); border-left-color: var(--accent); }
.sidebar .k { color: var(--dim); font-size: 9px; text-transform: uppercase;
letter-spacing: .04em; margin-left: .4em; font-weight: 400; }
/* Indent by nesting depth rather than by element, so it keeps working however
deep the tree goes. */
.sidebar ul ul a { padding-left: 1.8rem; }
.sidebar ul ul ul a { padding-left: 2.6rem; }
.sidebar ul ul ul ul a { padding-left: 3.4rem; }
.sidebar details > summary {
cursor: pointer; list-style: none; display: flex; align-items: center;
}
.sidebar details > summary::-webkit-details-marker { display: none; }
.sidebar details > summary::before {
content: ""; color: var(--dim); flex: 0 0 auto;
margin-left: .55rem; font-size: 9px; transition: transform .12s;
}
.sidebar details[open] > summary::before { transform: rotate(90deg); }
.sidebar details > summary > a { flex: 1 1 auto; padding-left: .45rem; }
.sidebar details > summary:hover::before { color: var(--text); }
.content { flex: 1; min-width: 0; max-width: __CONTENT__px; padding: 2rem 3rem; }
.content h1 { font-size: 22px; margin-bottom: .25rem; }
.content h2 { font-size: 15px; margin: 2rem 0 .5rem; padding-top: 1rem;
border-top: 1px solid var(--border); }
.content h3 { font-size: 13px; margin: 1.25rem 0 .35rem; color: var(--muted); }
.content p { margin-bottom: .75rem; color: var(--muted); }
.content code { font-family: ui-monospace, "Cascadia Mono", Consolas, monospace;
font-size: 11px; background: var(--surface); padding: 1px 5px;
border-radius: 3px; color: var(--text); }
.content table { border-collapse: collapse; margin: .75rem 0; font-size: 12px; }
.content th, .content td { text-align: left; padding: 4px 14px 4px 0;
border-bottom: 1px solid var(--border); color: var(--muted); }
.content th { color: var(--dim); font-weight: 600; font-size: 10px;
text-transform: uppercase; letter-spacing: .04em; }
.lede { color: var(--dim); font-size: 12px; margin-bottom: 1.5rem; }
/* The convention both spr/docs and sms/docs arrived at independently:
inline and scaled to the column, click for the full thing. */
.figure { margin: 1rem 0 1.5rem; }
.figure a { display: block; border: 1px solid var(--border); border-radius: 8px;
overflow: hidden; background: var(--surface); }
.figure a:hover { border-color: var(--accent); }
.figure img { display: block; width: 100%; height: auto; }
.figure figcaption { color: var(--dim); font-size: 10px; margin-top: .4rem; }
/* Both ends of the book, side by side and above the diagram. Above, because a
reader who scrolls past the picture has already formed an impression, and
"2 files could not be read" has to arrive before that and not after. */
.ledger { display: flex; gap: 1px; background: var(--border); border: 1px solid var(--border);
border-radius: 8px; overflow: hidden; margin: 0 0 1.5rem; }
.ledger > div { flex: 1 1 0; background: var(--surface); padding: .7rem .9rem; min-width: 0; }
.ledger dt { color: var(--dim); font-size: 9.5px; text-transform: uppercase;
letter-spacing: .07em; margin-bottom: .3rem; }
.ledger dd { margin: 0; color: var(--text); font-size: 12.5px; }
.ledger dd .sub { display: block; color: var(--muted); font-size: 11px; margin-top: .2rem;
overflow-wrap: anywhere; }
.ledger .lost { color: var(--artery); }
.ledger .kept { color: var(--ok); }
.gap { border: 1px solid var(--artery); border-left-width: 3px; border-radius: 6px;
background: var(--surface); padding: .7rem .9rem; margin: 0 0 1.5rem; font-size: 12px; }
.gap b { color: var(--artery); }
.gap ul { margin: .4rem 0 0 1.1rem; color: var(--muted); }
.gap code { font-size: 11px; }
"""
def _ledger(book: dict) -> str:
"""The two measures, and the gap between them if there is one.
This is the whole reason the site is the book's last step rather than just
another emitter: it is the one artifact somebody definitely opens, so it is
where "45 of 47 files" has to appear. A diagram cannot say it, and a log
nobody reads does not count as having said it.
"""
larder = book.get("larder") or {}
measure = book.get("book") or {}
failed = larder.get("failed") or []
kinds = measure.get("by_kind") or {}
out_summary = " · ".join(f"{v} {k}" for k, v in
sorted(kinds.items(), key=lambda kv: -kv[1])[:4])
unit = larder.get("unit", "unit")
read, seen = larder.get("read", 0), larder.get("seen", 0)
plural = unit if read == 1 else (unit[:-1] + "ies" if unit.endswith("y") else unit + "s")
in_line = f"{read} {plural} read"
if failed:
in_line += f' <span class="lost"{len(failed)} of {seen} could not be</span>'
panel = (
'<div class="ledger">'
f'<div><dt>what came in</dt><dd>{in_line}'
f'<span class="sub">{escape(larder.get("identity", "?"))}</span></dd></div>'
f'<div><dt>what came out</dt><dd>{escape(out_summary) or "nothing"}'
f'<span class="sub">{measure.get("edges", 0)} edges · '
f'{measure.get("external", 0)} external · '
# "step artifacts", not "artifacts": this page is written before the
# book's last step closes, so it cannot count itself. Saying `step`
# makes the number true rather than one short of book.json's.
f'{len(measure.get("artifacts") or [])} step artifacts</span></dd></div>'
"</div>"
)
# A reconciliation that failed is not a footnote. It means the document
# below is incomplete in a way the document below cannot show.
lost = [r for r in (book.get("reconciled") or []) if not r.get("ok")]
if lost or failed:
items = "".join(f"<li>{escape(r['claim'])}{escape(r['why'])}</li>" for r in lost)
items += "".join(
f"<li><code>{escape(f['name'])}</code> — {escape(f['error'])}</li>"
for f in failed[:12]
)
if len(failed) > 12:
items += f"<li>and {len(failed) - 12} more</li>"
panel += (
'<div class="gap"><b>This book is incomplete.</b> '
"What is drawn below is everything that could be read, which is not "
f"everything there is.<ul>{items}</ul></div>"
)
return panel
def _slots(style) -> dict:
s = style.slot
return {
"__BG__": s("surface-0"),
"__SURFACE__": s("surface-1") or s("surface-2"),
"__SURFACE2__": s("surface-2"),
"__BORDER__": s("border"),
"__TEXT__": s("text"),
"__MUTED__": s("text-muted"),
"__DIM__": s("text-dim"),
"__ACCENT__": s("accent"),
"__FONT__": '"Segoe UI", Inter, system-ui, -apple-system, Arial, sans-serif',
}
def _fill(template: str, values: dict) -> str:
for key, value in values.items():
template = template.replace(key, str(value))
return template
def _sidebar(items: list, depth: int = 0) -> str:
out = ["<ul>"]
for item in items:
label = escape(item.get("label", item["id"]))
kind = escape(item.get("kind", ""))
anchor = escape(item["id"])
link = f'<a href="#{anchor}" data-id="{anchor}">{label}<span class="k">{kind}</span></a>'
kids = item.get("children") or []
if kids:
out.append(
f"<li><details{' open' if depth == 0 else ''}>"
f"<summary>{link}</summary>{_sidebar(kids, depth + 1)}</details></li>"
)
else:
out.append(f"<li>{link}</li>")
out.append("</ul>")
return "".join(out)
def _sections(items: list, depth: int = 0) -> str:
out = []
for item in items:
tag = "h2" if depth == 0 else "h3"
attrs = item.get("attrs") or {}
out.append(f'<{tag} id="{escape(item["id"])}">{escape(item.get("label", ""))}'
f'<span class="k"> {escape(item.get("kind", ""))}</span></{tag}>')
if item.get("doc"):
out.append(f"<p>{escape(item['doc'])}</p>")
if item.get("href"):
out.append(f'<p><code>{escape(item["href"])}</code></p>')
kids = item.get("children") or []
if kids:
out.append(_sections(kids, depth + 1))
return "".join(out)
def emit(ir: dict, style, *, graph: str | None = None, title: str = "",
book: dict | None = None) -> dict:
"""IR + Style -> {filename: text}. Write them next to each other.
`book` is a book ledger (`book/__init__.py`). When present the page opens
with both measures — what went in, what came out — because a page that
shows only the result is the thing the measure exists to correct.
Optional, so the site emitter still works on a bare IR.
"""
from .index import to_sidebar
meta = ir.get("meta", {})
name = title or meta.get("root", "docs")
side = to_sidebar(ir)
values = _slots(style)
counts: dict[str, int] = {}
for n in ir["nodes"]:
counts[n["kind"]] = counts.get(n["kind"], 0) + 1
summary = " · ".join(f"{v} {k}" for k, v in sorted(counts.items(), key=lambda kv: -kv[1]))
ledger = _ledger(book) if book else ""
figure = ""
if graph:
figure = (
'<figure class="figure">'
f'<a href="viewer.html?src={escape(graph)}" title="Click to open — then click again for 1:1">'
f'<img src="{escape(graph)}" alt="{escape(name)}"></a>'
"<figcaption>Click to open the viewer · click again for actual size</figcaption>"
"</figure>"
)
external = side.get("external") or []
ext_html = ""
if external:
rows = "".join(f"<tr><td><code>{escape(e)}</code></td></tr>" for e in external[:40])
ext_html = (
'<h2 id="__external">Depends on, outside this tree</h2>'
"<p>Names that could not be resolved here — the dependency surface.</p>"
f"<table>{rows}</table>"
)
index = f"""<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{escape(name)}</title>
<link rel="stylesheet" href="site.css">
</head>
<body>
<div class="layout">
<nav class="sidebar">
<div class="sidebar-header"><b>{escape(name)}</b><small>{escape(summary)}</small></div>
{_sidebar(side["items"])}
</nav>
<main class="content">
<h1>{escape(name)}</h1>
<p class="lede">Generated from <code>{escape(meta.get("source", "?"))}</code> ·
{escape(summary)}. Regenerated, not edited.</p>
{ledger}
{figure}
{_sections(side["items"])}
{ext_html}
</main>
</div>
<script>
// Highlight the section being read. No dependency, no build step.
var links = [].slice.call(document.querySelectorAll('.sidebar a[data-id]'));
var byId = {{}};
links.forEach(function (a) {{ byId[a.dataset.id] = a; }});
var obs = new IntersectionObserver(function (entries) {{
entries.forEach(function (en) {{
var a = byId[en.target.id];
if (!a) return;
if (en.isIntersecting) {{
links.forEach(function (l) {{ l.classList.remove('active'); }});
a.classList.add('active');
}}
}});
}}, {{ rootMargin: '-10% 0px -80% 0px' }});
document.querySelectorAll('h2[id], h3[id]').forEach(function (h) {{ obs.observe(h); }});
</script>
</body>
</html>
"""
return {
"index.html": index,
"viewer.html": _fill(VIEWER.replace("__TITLE__", escape(name)), values),
"site.css": _fill(
CSS.replace("__SIDEBAR__", str(SIDEBAR_W)).replace("__CONTENT__", str(CONTENT_W)),
values,
),
}
def write(ir: dict, style, out_dir, *, graph: str | None = None, title: str = "",
book: dict | None = None) -> list[Path]:
out_dir = Path(out_dir)
out_dir.mkdir(parents=True, exist_ok=True)
written = []
for name, text in emit(ir, style, graph=graph, title=title, book=book).items():
path = out_dir / name
path.write_text(text)
written.append(path)
return written

View File

@@ -0,0 +1,67 @@
// Drive the explorer's logic under a stub DOM: select, walk, basket.
const fs = require('fs');
const html = fs.readFileSync(process.argv[2], 'utf8');
const script = html.split('<script>').pop().split('</script>')[0];
const detail = { innerHTML: '' };
const basketEl = { textContent: '' };
const nav = { addEventListener() {} };
const ids = { detail, basket: basketEl, nav };
global.document = {
getElementById: (i) => ids[i],
querySelectorAll: () => [],
};
global.CSS = { escape: (s) => s };
const api = new Function(script + '; return {select, toggleBasket, showBasket, clearBasket, FACTS, GRAPHS};')();
let ok = true;
const check = (name, cond) => {
console.log(` ${cond ? 'ok ' : 'FAIL'} ${name}`);
if (!cond) ok = false;
};
const ids_ = Object.keys(api.FACTS);
const withEdges = ids_.filter((i) => api.FACTS[i].out.length || api.FACTS[i]['in'].length);
check('facts are embedded for every node', ids_.length > 0);
check('some node has relationships', withEdges.length > 0);
const target = withEdges[0];
api.select(target);
check('selecting renders a detail pane', detail.innerHTML.includes('<h2>'));
check('it names what the thing is', detail.innerHTML.includes(api.FACTS[target].kind));
check('it lists what it reaches', detail.innerHTML.includes('reaches ('));
check('it lists what reaches it', detail.innerHTML.includes('reached by ('));
// Walking forward: every link in the pane must be a selectable id.
const links = [...detail.innerHTML.matchAll(/select\('([^']+)'\)/g)].map((m) => m[1]);
check('neighbours are offered as links to walk to', links.length > 0);
const reachable = links.every((i) => api.FACTS[i] !== undefined);
check('every link resolves to a real node', reachable);
if (links.length) {
api.select(links[0]);
check('walking forward re-renders on the neighbour',
detail.innerHTML.includes(api.FACTS[links[0]].label));
}
const withGraph = ids_.find((i) => api.GRAPHS[i]);
if (withGraph) {
api.select(withGraph);
check('a neighbourhood diagram is shown when one exists',
detail.innerHTML.includes('<img src="graphs/'));
} else {
check('neighbourhood diagrams were generated', false);
}
api.toggleBasket(target);
check('the basket counts a selection', basketEl.textContent === '1 selected');
api.showBasket();
check('the basket is a copyable list of paths', detail.innerHTML.includes('<textarea'));
check('...with a line count, to see it got too big',
/~\d+\s*lines/.test(detail.innerHTML));
api.toggleBasket(target);
check('toggling again removes it', basketEl.textContent === '0 selected');
process.exit(ok ? 0 : 1);

View File

@@ -0,0 +1 @@
"""Extractors: source artifacts -> IR. None of them has heard of SVG."""

View File

@@ -0,0 +1,25 @@
""" python3 -m docgen.extractors <db|openapi|usage|code> [options]"""
import sys
def main(argv=None):
argv = sys.argv[1:] if argv is None else argv
if argv and argv[0] == "openapi":
from .openapi_main import main as run
return run(argv[1:])
if argv and argv[0] == "code":
from .code_main import main as run
return run(argv[1:])
if argv and argv[0] == "usage":
from .usage_main import main as run
return run(argv[1:])
if argv and argv[0] == "db":
from .db_main import main as run
return run(argv[1:])
from .db_main import main as run # bare form stays the db one, as before
return run(argv)
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,281 @@
"""
Source in several languages -> IR structure, via **tree-sitter**.
python3 -m docgen.extractors code --root src/ --lang auto -o ir.json
Adopts a parser rather than writing one, which is the rule the brief sets and
the reason this handles generics, strings containing braces, nested types and
`#region` without any of them being a special case. `pip install tree_sitter
tree_sitter_c_sharp tree_sitter_typescript tree_sitter_python`.
**Optional, never required.** Python still goes through the stdlib `ast`
extractor, which does two-pass name resolution tree-sitter would have to
reimplement. Without tree-sitter installed, docgen loses C# and TypeScript and
nothing else — the import is lazy and the error says what to install.
## Structure only, and that is the point
This produces what is *where*: declarations, their nesting, and the lines each
occupies. It produces **no edges**. Resolving a C# `using` or a TypeScript
`import` to the thing it names is a different and much larger job, and the
consumer that needs this — the minimap — needs none of it.
Saying so matters: an extractor that quietly produced half a dependency graph
would be worse than one that produces none, because the half would look whole.
## Vocabulary
module a file, or a namespace
class class, struct, record, enum — a type with members
interface kept separate from class on purpose: in C# and TypeScript the
distinction is most of what reading a file tells you, and the
minimap's whole claim is that the pattern comes from the colours
function method, constructor, property, function, arrow function
"""
import sys
from pathlib import Path
from ..ir import Graph, Meta
# Extension -> (grammar module, language function). `.tsx` needs its own parser:
# the TSX grammar is a different language, not an option on the TypeScript one.
LANGUAGES = {
".cs": ("tree_sitter_c_sharp", "language"),
".ts": ("tree_sitter_typescript", "language_typescript"),
".mts": ("tree_sitter_typescript", "language_typescript"),
".tsx": ("tree_sitter_typescript", "language_tsx"),
".py": ("tree_sitter_python", "language"),
}
# Declaration node types, per grammar, mapped onto the IR's small vocabulary.
DECLARATIONS = {
"c_sharp": {
"namespace_declaration": "module",
"file_scoped_namespace_declaration": "module",
"class_declaration": "class",
"struct_declaration": "class",
"record_declaration": "class",
"record_struct_declaration": "class",
"enum_declaration": "class",
"interface_declaration": "interface",
"method_declaration": "function",
"constructor_declaration": "function",
"destructor_declaration": "function",
"property_declaration": "function",
"operator_declaration": "function",
"local_function_statement": "function",
},
"typescript": {
"module": "module",
"internal_module": "module",
"class_declaration": "class",
"abstract_class_declaration": "class",
"enum_declaration": "class",
"interface_declaration": "interface",
"type_alias_declaration": "interface",
"function_declaration": "function",
"generator_function_declaration": "function",
"method_definition": "function",
"public_field_definition": "function",
},
"python": {
"class_definition": "class",
"function_definition": "function",
"decorated_definition": None, # descend; the real node is inside
},
}
GRAMMAR_FAMILY = {
"tree_sitter_c_sharp": "c_sharp",
"tree_sitter_typescript": "typescript",
"tree_sitter_python": "python",
}
class MissingParser(ImportError):
"""tree-sitter, or one of its grammars, is not installed."""
def _parser(suffix: str):
"""(Parser, family) for a file extension. Lazy, so the import is optional."""
if suffix not in LANGUAGES:
raise MissingParser(f"no grammar registered for {suffix!r}")
module_name, fn = LANGUAGES[suffix]
try:
from tree_sitter import Language, Parser
except ImportError:
raise MissingParser(
"tree-sitter is not installed — C# and TypeScript need it.\n"
" pip install tree_sitter tree_sitter_c_sharp tree_sitter_typescript\n"
"Python does not: it uses the stdlib `ast` extractor."
) from None
try:
grammar = __import__(module_name)
except ImportError:
raise MissingParser(
f"{module_name} is not installed — needed for {suffix} files.\n"
f" pip install {module_name.replace('_', '-')}"
) from None
return Parser(Language(getattr(grammar, fn)())), GRAMMAR_FAMILY[module_name]
def _name(node, source: bytes) -> str | None:
"""A declaration's name, or None when it has none worth recording."""
field = node.child_by_field_name("name")
if field is not None:
return source[field.start_byte:field.end_byte].decode("utf-8", "replace")
# C# properties and TS fields sometimes carry the name as an identifier child.
for child in node.children:
if child.type in ("identifier", "property_identifier", "type_identifier"):
return source[child.start_byte:child.end_byte].decode("utf-8", "replace")
return None
def _walk(node, source: bytes, family: str, out: list, parent: str | None, prefix: str,
seen_ids: set | None = None):
"""Collect declarations, depth-first, keeping nesting in the id."""
table = DECLARATIONS[family]
seen_ids = seen_ids if seen_ids is not None else set()
for child in node.children:
kind = table.get(child.type, ...)
if kind is None:
# A wrapper — a decorated definition, say. Descend without naming it.
_walk(child, source, family, out, parent, prefix, seen_ids)
continue
if kind is ...:
_walk(child, source, family, out, parent, prefix, seen_ids)
continue
name = _name(child, source)
if not name:
_walk(child, source, family, out, parent, prefix, seen_ids)
continue
nid = f"{prefix}.{name}" if prefix else name
if nid in seen_ids:
# Same reason as the Python extractor: an overload, a partial class,
# or a name declared twice in different branches. The line keeps the
# id unique without making it unstable.
nid = f"{nid}#L{child.start_point[0] + 1}"
seen_ids.add(nid)
out.append({
"id": nid,
"kind": kind,
"label": name,
"parent": parent,
"line": child.start_point[0] + 1,
"lines": max(1, child.end_point[0] - child.start_point[0] + 1),
})
_walk(child, source, family, out, nid, nid, seen_ids)
def extract_file(path: Path, root: Path) -> tuple[dict, list] | None:
"""One file -> (module node, declarations). None when unparseable."""
parser, family = _parser(path.suffix)
try:
source = path.read_bytes()
except OSError as e:
return {"error": str(e)}, []
rel = path.relative_to(root)
module_id = ".".join([*rel.parts[:-1], rel.stem])
tree = parser.parse(source)
decls: list = []
_walk(tree.root_node, source, family, decls, module_id, module_id)
module = {
"id": module_id,
"kind": "module",
"label": rel.stem,
"parent": ".".join(rel.parts[:-1]) or None,
"lines": source.count(b"\n") + 1,
"file": rel.as_posix(),
# tree-sitter never fails to parse; it produces ERROR nodes instead. That
# is more useful than an exception, and worth recording rather than
# silently accepting a partial tree.
"errors": _count_errors(tree.root_node),
}
return module, decls
def _count_errors(node) -> int:
n = 1 if node.type == "ERROR" or node.is_missing else 0
for child in node.children:
n += _count_errors(child)
return n
def extract(root, suffixes=None, exclude=(), source: str = "code",
identity: str | None = None) -> Graph:
"""Walk a tree and map its structure. No edges — see the module docstring."""
root = Path(root).resolve()
if not root.is_dir():
raise NotADirectoryError(f"not a directory: {root}")
wanted = tuple(suffixes) if suffixes else tuple(LANGUAGES)
skip = {"__pycache__", ".git", ".venv", "venv", "node_modules", "dist", "build",
"bin", "obj", *exclude}
from ..book.larder import Larder
g = Graph(Meta(source=source, root=root.name))
packages: set[str] = set()
files = [
p for p in sorted(root.rglob("*"))
if p.suffix in wanted and p.is_file()
and not any(part in skip for part in p.relative_to(root).parts)
]
# The measure is built alongside the walk rather than recomputed after it.
# tree-sitter never raises on bad syntax — it produces ERROR nodes — so
# "failed" here means a file that could not be handled at all, and the
# partially-parsed ones are counted separately under `unparsed_regions`.
larder = Larder(kind="code", identity=identity or root.name, unit="file",
seen=len(files))
unparsed = 0
for path in files:
try:
module, decls = extract_file(path, root)
except MissingParser:
raise
except Exception as e: # noqa: BLE001 - one bad file must not cost the run
rel = path.relative_to(root)
larder.fail(rel.as_posix(), f"{type(e).__name__}: {e}")
g.node(".".join([*rel.parts[:-1], rel.stem]), "module", rel.stem,
attrs={"file": rel.as_posix(), "error": f"{type(e).__name__}: {e}"})
continue
parent = module["parent"]
if parent:
packages.add(parent)
attrs = {"file": module["file"], "lines": module["lines"]}
if module["errors"]:
attrs["error"] = f"{module['errors']} unparsed region(s)"
unparsed += 1
g.node(module["id"], "module", module["label"], parent=parent, attrs=attrs)
for d in decls:
g.node(d["id"], d["kind"], d["label"], parent=d["parent"],
attrs={"file": module["file"], "line": d["line"], "lines": d["lines"]})
# Directories that hold files but are not themselves files still need to
# exist, or every module in them is an orphan.
known = {n.id for n in g.nodes}
for pkg in sorted(packages):
parts = pkg.split(".")
for i in range(1, len(parts) + 1):
pid = ".".join(parts[:i])
if pid not in known:
g.node(pid, "module", parts[i - 1],
parent=".".join(parts[: i - 1]) or None,
attrs={"lines": 0, "directory": True})
known.add(pid)
larder.extra["languages"] = len({p.suffix for p in files})
if unparsed:
# Worth its own number rather than folding into `failed`: a file with an
# unparsed region still contributed structure, so calling it a failure
# would understate what the book contains.
larder.extra["unparsed_regions"] = unparsed
g.meta.larder = larder.to_dict()
return g

View File

@@ -0,0 +1,39 @@
""" python3 -m docgen.extractors code --root src/ [--ext .cs] [-o ir.json]"""
import argparse
import json
import sys
from pathlib import Path
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.extractors code")
p.add_argument("--root", "-s", required=True, type=Path)
p.add_argument("--output", "-o", type=Path)
p.add_argument("--ext", action="append", default=[],
help="Limit to these extensions. Default: every registered one.")
p.add_argument("--exclude", action="append", default=[])
args = p.parse_args(argv)
from .code import LANGUAGES, MissingParser, extract
try:
ir = extract(args.root, suffixes=args.ext or None, exclude=tuple(args.exclude))
except MissingParser as e:
print(f"Error: {e}", file=sys.stderr)
return 1
except (NotADirectoryError, OSError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
text = json.dumps(ir.to_dict(), indent=2) + "\n"
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(text)
from collections import Counter
c = Counter(n.kind for n in ir.nodes)
print(f"{len(ir.nodes)} nodes -> {args.output} "
+ " · ".join(f"{v} {k}" for k, v in sorted(c.items(), key=lambda kv: -kv[1])))
else:
sys.stdout.write(text)
return 0

View File

@@ -0,0 +1,160 @@
"""
A database schema -> IR.
Reads the **published** `{models, relationships, source}` contract rather than
importing modelgen's Python. That contract is already emitted by
`modelgen/generator/jsonschema.py`, consumed by `graphgen/schema.py` and
`datagen`, and asserted by two modelgen tests — so it is the stable surface, and
reading it means this extractor works for every source modelgen supports
(Django, SQLAlchemy, OpenAPI, CSV/ODS, a live database) without knowing about
any of them.
python3 -m docgen.extractors.db --schema cfg/sample/.../graphgen/schema.json
Connecting to a live database is **not here**. `modelgen from-db --url ...`
reflects via SQLAlchemy's Inspector across dialects and writes the schema.json
this reads; that is its job and it is already done. The two-step is also the
safer one — the URL, and therefore the credentials, never enters this pipeline.
## The schema checkpoint
The brief requires the IR to survive a second domain without new top-level
fields. It does, and the mapping is not a squeeze:
table -> node, kind "table" column -> node, kind "column"
column -> parent is its table FK -> edge, kind "foreign_key"
M2M -> edge, kind "references"
`kind` carries the domain vocabulary, `parent` carries containment, `attrs`
carries what only this domain cares about — `pk`, `nullable`, the column type.
Nothing needed a field that `module`/`class`/`function` did not also use, which
is the result the checkpoint was there to confirm.
"""
import json
from pathlib import Path
from ..ir import Graph, Meta
def from_schema_dict(data: dict, root: str = "schema", source: str = "db",
identity: str | None = None) -> Graph:
"""A graphgen-compatible schema dict -> IR.
Accepts both spellings of the contract: the on-disk `schema.json` form,
where `models` is a mapping of name to definition, and the loaded form that
`graphgen.schema.load_graph_schema` returns, where it is a list. They are
the same data and callers have both.
"""
from ..book.larder import Larder
g = Graph(Meta(source=source, root=root))
models = data.get("models", {})
listed = models.values() if isinstance(models, dict) else models
names = (
set(models)
if isinstance(models, dict)
else {m.get("id") or m.get("name") for m in models}
)
for name, model in (
models.items() if isinstance(models, dict)
else ((m.get("id") or m.get("name"), m) for m in models)
):
attrs = {}
if model.get("doc"):
attrs["doc"] = model["doc"]
g.node(name, "table", name, attrs=attrs)
fields = model.get("fields", {})
pairs = fields.items() if isinstance(fields, dict) else (
(f.get("name"), f) for f in fields
)
for field_name, field in pairs:
type_str = field.get("type", "str")
target, kind = _relation(type_str, field)
a = {"type": _plain_type(type_str)}
if field.get("pk"):
a["pk"] = True
if field.get("nullable"):
a["nullable"] = True
if target:
# Kept on the column so the index can say what it points at
# without walking the edge list.
a["references"] = target
g.node(f"{name}.{field_name}", "column", field_name, parent=name, attrs=a)
if target and kind:
# The edge is table -> table: a diagram of forty columns joined
# column-to-column is unreadable, and the relationship is
# between the tables. Which column carries it is in `attrs`.
g.edge(name, target, kind, attrs={"label": field_name})
# Some producers give `relationships` alongside the fields; take them too,
# and let the dedupe below settle it.
for rel in data.get("relationships", []):
src, dst = rel.get("from_model"), rel.get("to_model")
if src in names and dst in names:
kind = "references" if rel.get("type") == "M2M" else "foreign_key"
g.edge(src, dst, kind, attrs={"label": rel.get("from_field", "")})
# A schema is a published contract, so there is nothing here that can fail
# to be read — every table it declares is a table. `seen == read` always,
# and saying so is more useful than omitting the measure: it distinguishes
# "no failures" from "not measured".
larder = Larder(kind="db", identity=identity or root, unit="table",
seen=len(names))
rels = data.get("relationships") or []
larder.extra["relationships"] = len(rels)
if data.get("source"):
larder.extra["dialect"] = str(data["source"])
g.meta.larder = larder.to_dict()
_dedupe(g)
for missing in sorted(
{e.target for e in g.edges} - {n.id for n in g.nodes}
):
# A foreign key naming a table the schema does not define. Recorded
# rather than dropped, for the same reason an unresolved import is.
g.node(missing, "external", missing, attrs={"unresolved": True})
return g
def _relation(type_str: str, field: dict) -> tuple[str | None, str | None]:
"""(target table, edge kind) for a field, or (None, None)."""
if isinstance(type_str, str):
if type_str.startswith("FK:"):
return type_str[3:], "foreign_key"
if type_str.startswith("M2M:"):
return type_str[4:], "references"
if field.get("fk"):
return field["fk"], "references" if field.get("m2m") else "foreign_key"
return None, None
def _plain_type(type_str) -> str:
if isinstance(type_str, str):
for prefix in ("FK:", "M2M:"):
if type_str.startswith(prefix):
return prefix.rstrip(":")
return type_str
def _dedupe(g: Graph) -> None:
seen, keep = set(), []
for e in g.edges:
key = (e.source, e.target, e.kind)
if key in seen:
continue
seen.add(key)
keep.append(e)
g.edges = keep
def extract(schema_path, source: str = "db", identity: str | None = None) -> Graph:
"""Read a schema.json from disk."""
path = Path(schema_path)
data = json.loads(path.read_text())
return from_schema_dict(data, root=path.parent.name or path.stem, source=source,
identity=identity or str(schema_path))

View File

@@ -0,0 +1,31 @@
""" python3 -m docgen.extractors.db --schema path/to/schema.json [-o ir.json]"""
import argparse
import json
import sys
from pathlib import Path
from .db import extract
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.extractors.db")
p.add_argument("--schema", "-s", required=True, type=Path,
help="A graphgen-compatible schema.json, as modelgen emits.")
p.add_argument("--output", "-o", type=Path)
args = p.parse_args(argv)
try:
ir = extract(args.schema)
except (OSError, json.JSONDecodeError, KeyError) as e:
print(f"Error: could not read {args.schema}: {e}", file=sys.stderr)
return 1
text = json.dumps(ir.to_dict(), indent=2) + "\n"
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(text)
print(f"{len(ir.nodes)} nodes, {len(ir.edges)} edges -> {args.output}")
else:
sys.stdout.write(text)
return 0

View File

@@ -0,0 +1,190 @@
"""
An OpenAPI document -> IR: the endpoints, and the shapes they carry.
Adapts `modelgen/loader/extract/openapi.py`, which already parses OpenAPI 3.x
and Swagger 2.0 and resolves `$ref`. This turns its output into IR nodes; it
does not re-parse anything.
python3 -m docgen.extractors.openapi --spec petstore.yaml -o ir.json
## Why this one matters more than it looks
A hand-written API notebook is the thing nobody can keep current: the spec moves
and the document does not, and there is no way to tell by looking. Extracting
the endpoints means the document is **generated from the same file the server is
built from**, so "is this current" becomes a question about a build rather than
about somebody's diligence.
It emits schemas as `table`/`column`, the same vocabulary the database extractor
uses. That is deliberate: an API's data model and a database's are the same kind
of thing, so the ER emitter draws either without knowing which it got. Endpoints
are a separate `kind`, so a view can ask for one or the other.
only_kinds(ir, {"table", "column"}) -> the data model, as an ER diagram
only_kinds(ir, {"endpoint"}) -> the surface, as a notebook
"""
from pathlib import Path
from ..ir import Graph, Meta
def _modelgen():
"""modelgen's OpenAPI reader, from wherever the reference repo is.
Imported lazily and by path rather than as a hard dependency. This is the
**only** seam between docgen and the wider repo — see `reference.py`, which
resolves it from `$DOCGEN_REFERENCE` or by walking up. Every other extractor
and every emitter works with nothing above `docgen/`.
Not reimplemented here on purpose: modelgen already parses the spec and
resolves `$ref`, and a second OpenAPI reader in the same repo is two things
to keep correct.
"""
from .. import reference
if reference.on_path() is None:
raise reference.missing(
"modelgen",
"OpenAPI is read through station/tools/modelgen/loader/extract/"
"openapi.py, which parses the spec and resolves $ref",
)
try:
from modelgen.loader.extract.openapi import OpenAPIExtractor
except ImportError as e:
raise reference.missing(
"modelgen.loader.extract.openapi",
f"the reference repo was found but the module did not import ({e})",
) from None
return OpenAPIExtractor
def _type_name(hint) -> str:
if hint is None:
return "Any"
if isinstance(hint, str):
return hint
return getattr(hint, "__name__", str(hint))
def _refs(path: Path) -> dict[str, dict[str, str]]:
"""{schema: {field: referenced schema}} — the relationships, recovered.
modelgen resolves an inter-schema `$ref` to the literal string `dict`, so by
the time its fields reach us the *target* is gone. Without this, an API's
data model draws as disconnected cards: three tables, no foreign keys, and
nothing saying the relationships were lost. Which is exactly the failure
this tool is otherwise built to prevent.
So one key is read directly, and one only: `$ref` under a schema's
`properties`. That is not parsing OpenAPI — no paths, no bodies, no
responses, no `$ref` resolution, no composition keywords. modelgen still
does all of the reading that matters, and this recovers the single fact its
type mapping cannot carry.
Returns `{}` on anything unexpected. A missing relationship is a worse
diagram; a raised exception here would be no diagram at all.
"""
try:
import yaml # available: modelgen just used it
except ImportError:
return {}
try:
doc = yaml.safe_load(path.read_text()) or {}
schemas = ((doc.get("components") or {}).get("schemas")) or {}
except Exception: # noqa: BLE001 - see the docstring
return {}
out: dict[str, dict[str, str]] = {}
for name, schema in schemas.items():
if not isinstance(schema, dict):
continue
for field, spec in (schema.get("properties") or {}).items():
if not isinstance(spec, dict):
continue
# A direct reference, or an array of them — `lines: [OrderLine]` is
# the same relationship as `order: Order`, pointing the other way.
ref = spec.get("$ref")
if not ref and isinstance(spec.get("items"), dict):
ref = spec["items"].get("$ref")
if isinstance(ref, str) and ref.startswith("#/components/schemas/"):
out.setdefault(name, {})[field] = ref.rsplit("/", 1)[-1]
return out
def extract(spec_path, source: str = "openapi", identity: str | None = None) -> Graph:
"""An OpenAPI file -> IR."""
from ..book.larder import Larder
OpenAPIExtractor = _modelgen()
path = Path(spec_path)
extractor = OpenAPIExtractor(path)
models, enums = extractor.extract()
endpoints = extractor.endpoints()
g = Graph(Meta(source=source, root=path.name))
known = {m.name for m in models}
refs = _refs(path)
for model in models:
attrs = {}
if getattr(model, "docstring", None):
attrs["doc"] = model.docstring.strip().split("\n")[0]
g.node(model.name, "table", model.name, attrs=attrs)
for field in model.fields:
a = {"type": _type_name(field.type_hint)}
if getattr(field, "optional", False):
a["nullable"] = True
if field.name in ("id", "uuid"):
a["pk"] = True
# The recovered $ref target wins over the mapped type name: modelgen
# says `dict` where the spec said which schema.
target = refs.get(model.name, {}).get(field.name) or _type_name(field.type_hint)
if target in known and target != model.name:
a["references"] = target
g.edge(model.name, target, "foreign_key", attrs={"label": field.name})
g.node(
f"{model.name}.{field.name}", "column", field.name,
parent=model.name, attrs=a,
)
for e in endpoints:
eid = f"{e.method} {e.path}"
attrs = {
"method": e.method,
"path": e.path,
"status": getattr(e, "status", None) or 200,
}
for key in ("summary", "operation_id", "envelope_key"):
value = getattr(e, key, None)
if value:
attrs[key] = value
if getattr(e, "response_is_list", False):
attrs["returns_list"] = True
if getattr(e, "path_params", None):
attrs["path_params"] = list(e.path_params)
if getattr(e, "request_model", None):
attrs["request_model"] = e.request_model
if getattr(e, "response_model", None):
attrs["response_model"] = e.response_model
g.node(eid, "endpoint", eid, attrs=attrs)
# `accepts` and `returns` rather than one `uses`: which direction a
# shape travels is the thing a reader wants to know.
if getattr(e, "request_model", None) in known:
g.edge(eid, e.request_model, "accepts")
if getattr(e, "response_model", None) in known:
g.edge(eid, e.response_model, "returns")
# A spec is the larder here, and `path` is the unit because that is what a
# spec is an inventory of. Schemas and enums are counted separately: a spec
# with 40 schemas and 3 endpoints is a data model, and the measure should
# make that visible before the diagram does.
larder = Larder(kind="openapi", identity=identity or path.name, unit="path",
seen=len({e.path for e in endpoints}))
larder.extra["operations"] = len(endpoints)
larder.extra["schemas"] = len(models)
if enums:
larder.extra["enums"] = len(enums)
g.meta.larder = larder.to_dict()
return g

View File

@@ -0,0 +1,29 @@
""" python3 -m docgen.extractors.openapi --spec petstore.yaml [-o ir.json]"""
import argparse
import json
import sys
from pathlib import Path
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.extractors.openapi")
p.add_argument("--spec", "-s", required=True, type=Path)
p.add_argument("--output", "-o", type=Path)
args = p.parse_args(argv)
from .openapi import extract
try:
ir = extract(args.spec)
except (OSError, ImportError, ValueError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
text = json.dumps(ir.to_dict(), indent=2) + "\n"
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(text)
eps = sum(1 for n in ir.nodes if n.kind == "endpoint")
print(f"{len(ir.nodes)} nodes ({eps} endpoints), {len(ir.edges)} edges -> {args.output}")
else:
sys.stdout.write(text)
return 0

View File

@@ -0,0 +1,36 @@
"""
Python source -> IR, by `ast`.
from docgen.extractors.python import extract
ir = extract(Path("app/"))
Deterministic. A diagram built from this cannot be out of date with the code,
which is the whole reason the structural path has no model in it.
Two passes, because `ast` resolves nothing on its own — see `collect.py` and
`resolve.py`.
"""
from pathlib import Path
from .collect import collect
from .resolve import to_ir
def extract(root, exclude=(), source="python", identity=None):
"""Walk `root`, return an IR Graph. Never raises on a bad file.
`identity` is what the larder measure calls this source — the path as the
caller wrote it, which is what they will recognise. It defaults to the
directory name rather than the resolved path, for the same reason
`meta.root` does: an absolute path is not a secret but it is machine
specific, and the measure should mean the same thing on two machines.
"""
root = Path(root).resolve()
if not root.is_dir():
raise NotADirectoryError(f"not a directory: {root}")
modules = collect(root, exclude=exclude)
return to_ir(modules, root=root.name, source=source, identity=identity)
__all__ = ["extract", "collect", "to_ir"]

View File

@@ -0,0 +1,37 @@
""" python3 -m docgen.extractors.python --root PATH [--exclude NAME ...]"""
import argparse
import json
import sys
from pathlib import Path
from . import extract
def main(argv=None):
p = argparse.ArgumentParser(prog="python3 -m docgen.extractors.python")
p.add_argument("--root", "-s", required=True, type=Path, help="Tree to read.")
p.add_argument("--output", "-o", type=Path, help="Where to write. Default stdout.")
p.add_argument("--exclude", action="append", default=[], help="Directory name to skip.")
args = p.parse_args(argv)
try:
ir = extract(args.root, exclude=tuple(args.exclude))
except (NotADirectoryError, OSError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
text = json.dumps(ir.to_dict(), indent=2) + "\n"
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(text)
skipped = sum(1 for n in ir.nodes if n.attrs.get("error"))
print(f"{len(ir.nodes)} nodes, {len(ir.edges)} edges -> {args.output}"
+ (f" ({skipped} file(s) unparsed)" if skipped else ""))
else:
sys.stdout.write(text)
return 0
if __name__ == "__main__":
sys.exit(main())

Some files were not shown because too many files have changed in this diff Show More