251 lines
13 KiB
Markdown
251 lines
13 KiB
Markdown
# berth
|
|
|
|
spr's deploy half. A rig is a mobile installation; a **berth** is the allocated, paid-for
|
|
place where it is moored and actually operates. Local rig → remote berth.
|
|
|
|
```bash
|
|
make check # is this estate coherent? reports, never fixes
|
|
make estate show # the description, resolved
|
|
make vpn check # the overlay: addresses, routing, bindings, key hygiene
|
|
make services render aws
|
|
make ports verify # does the local map still agree with rig?
|
|
```
|
|
|
|
`rm -rf berth/` is the uninstall.
|
|
|
|
---
|
|
|
|
## What berth is
|
|
|
|
**One description of an estate, with a swappable executor.** The description is the artifact;
|
|
the tool that runs it is a rendering.
|
|
|
|
| layer | what berth uses | note |
|
|
| --- | --- | --- |
|
|
| overlay | **WireGuard**, config generated per peer | GPL-2.0, in-kernel, and **no coordination server** |
|
|
| infra | **OpenTofu** — one executor, not a pair | plain `.tf`; `terraform` works identically |
|
|
| gateway | Caddy locally, nginx on the box | a projection with per-target rules, not a format conversion |
|
|
| pipeline | Woodpecker | Actions/GitLab reachable from the same description; not built |
|
|
|
|
The estate's facts — domain, hosts, ports, instance size, firewall rules, services — live in
|
|
**one file every rendering reads** (`estate/<name>.json`), rather than being restated in
|
|
one tool's language and again in another's. **Swappability is bought by the description, not
|
|
by maintaining two renderings** — a rendering you *can* produce, not one you *must* keep in
|
|
step. Two live renderings cost you every resource twice, forever, with nothing enforcing that
|
|
they agree.
|
|
|
|
**The seam belongs in a script, not in a tool.** A tool's schema is a ceiling you do not
|
|
control.
|
|
|
|
*(This once leaned on a specific precedent, since withdrawn — `✖ B2` in [STALE.md](STALE.md).)*
|
|
|
|
**OpenTofu, and only OpenTofu.** Terraform has been BUSL-licensed since 2023; OpenTofu is the
|
|
CLI-compatible MPL-2.0 fork (Linux Foundation). Write plain `.tf` that runs under both; the
|
|
scripts say `tofu`, and `terraform` works identically.
|
|
|
|
**Pipelines are the second executor axis, and are not built.** Woodpecker is the
|
|
self-hosted rendering; Actions and GitLab CI are the standards that must be reachable from
|
|
the same description. Named here so the infra seam is not designed in a way that forecloses
|
|
it.
|
|
|
|
---
|
|
|
|
## The overlay is berth's network layer
|
|
|
|
Two instances in different clouds cannot share a VPC. A WireGuard overlay gives them one flat
|
|
address space that berth owns and can reproduce on any provider — and, under a flaky
|
|
environment, a second layer beneath whatever the provider offers.
|
|
|
|
That inverts the usual cloud pattern. Instead of a VPC with security groups and private
|
|
subnets, each instance gets a public IP, opens **only** the WireGuard port, and carries
|
|
everything else inside the tunnel. **The security boundary moves out of the provider's VPC
|
|
and into a layer that is identical on AWS, on GCP, and on a laptop behind NAT.**
|
|
|
|
**Plain WireGuard, not Tailscale/Headscale/NetBird.** Tailscale's client is open but its
|
|
coordination plane is proprietary SaaS; Headscale and NetBird are open but add a control plane
|
|
to run. Plain WireGuard needs no server at all.
|
|
|
|
**The cost is real and accepted:** no NAT traversal, no relay, no peer discovery. A peer behind
|
|
NAT must dial one with a public endpoint. Fine here — the instances have public IPs and the dev
|
|
box roams — but two roaming peers cannot reach each other. That is why `PersistentKeepalive` is
|
|
a checked invariant rather than a detail.
|
|
|
|
**Keys never enter the description.** Private keys are generated on the peer that owns them
|
|
(`make vpn keygen`) into `ctrl/.secrets/` and injected only at render time; public keys live in
|
|
the estate, because a config cannot be built without them. `vpn.sh` **refuses to write** either
|
|
a key or a rendered config until `git check-ignore` confirms the path is ignored — this repo
|
|
has already been bitten once by a `.gitignore` pattern anchoring to the wrong directory.
|
|
|
|
This also decides something about the IaC layer: **OpenTofu must never generate a WireGuard
|
|
private key**, because Terraform-lineage state stores every resource attribute in plaintext.
|
|
|
|
---
|
|
|
|
## Two rules that are not style preferences
|
|
|
|
### 1. Every default is the read-only verb
|
|
|
|
```
|
|
make estate -> show make certs -> status
|
|
make dns -> list make host -> status
|
|
make estate apply / destroy -> print the plan, then refuse without --yes
|
|
```
|
|
|
|
rig's `make cluster` defaults to `up`, because every rig verb is safe — a kind cluster is
|
|
disposable. berth's are not: `tofu destroy` costs money and takes live DNS with it. **A
|
|
tool where every verb is safe must not grow verbs that are not.**
|
|
|
|
The failure this prevents is not hypothetical. `ppl/ctrl/certs.sh:42` is `CMD="${1:-all}"`,
|
|
so a bare `./ctrl/certs.sh` there issues a real Let's Encrypt cert, rsyncs it to the gateway,
|
|
and reloads nginx. berth's `certs` defaults to `status`.
|
|
|
|
### 2. berth and rig share a convention, not code
|
|
|
|
Neither imports the other. They match on **shape** — the `make <noun> <verb>` dispatch, the
|
|
four-source config layering, the key names — and consistency is verified by
|
|
**recomputation**: `make ports verify` recomputes rig's `20000 + (cksum(name) % 200) * 10`
|
|
to check the local map, rather than sourcing rig's `lib/config.sh`.
|
|
|
|
Copying three stable lines is the whole cost of not coupling them. A shared library would
|
|
put something outside `rig/` on rig's path, and rig's promise is that
|
|
`grep -rIn -iE 'soleprint|\bspr\b'` across it returns nothing.
|
|
|
|
**rig is also unaware that berth exists.** `ppl/local/Caddyfile` is berth's to generate; rig
|
|
must not reference `local.ar` — its handover scrub refuses the string.
|
|
|
|
---
|
|
|
|
## The gateway doctrine
|
|
|
|
Practised across this codebase for a long time and never written down, so: written down.
|
|
|
|
- **Caddy where routing is dynamic and config-driven** — the in-cluster gateway that
|
|
multiplexes by Host header, and the host-side `.local.ar` name→port map.
|
|
- **nginx where it is a static server or a plain long-running compose service on the box.**
|
|
- **Envoy in `mpr`** — a deliberate one-off, not a third pattern.
|
|
- **`ingress-nginx` only as a kind addon** — a different thing again from either gateway.
|
|
|
|
### Rendering is a projection, not a format conversion
|
|
|
|
The local Caddyfile and the box's nginx are not two spellings of the same content:
|
|
|
|
| | local (Caddy) | cloud (nginx) |
|
|
| --- | --- | --- |
|
|
| granularity | one file | one file per vhost |
|
|
| blocks per service | one | two (`:80` redirect + `:443` server) |
|
|
| TLS | none; every address needs an explicit `:80` | one shared wildcard cert |
|
|
| upstream | `localhost:<port>` | container name + `resolver 127.0.0.11` |
|
|
| name depth | free | constrained by the cert |
|
|
| ambiguity | most specific wins | exact, else `default_server` (= load order) |
|
|
|
|
The `:80` is not decoration: without it Caddy 2 defaults each site to `:443` with auto-HTTPS,
|
|
which on `*.local.ar` means cert provisioning that fails and breaks the listener. The
|
|
variable upstream is not decoration either: naming the upstream in a variable forces runtime
|
|
DNS resolution, so nginx **starts when the upstream container is absent** — which is what
|
|
lets one nginx front a dozen independent compose stacks.
|
|
|
|
Because the two disambiguate by **opposite** rules, a name set that is unambiguous locally
|
|
can be ambiguous on the box. `make check` asserts against the projection, not the source.
|
|
|
|
### Installing generated vhosts — an order that is not optional
|
|
|
|
1. Generated config lands in `conf.d/generated/`, **not** `conf.d/`. `ppl/ctrl/deploy.sh`
|
|
rsyncs with `--delete`; sharing a directory means one set gets erased.
|
|
2. `nginx.conf` needs a **third** include line — its `conf.d/*.conf` glob does not recurse,
|
|
which is why `conf.d/soleprint/*.conf` already needs its own.
|
|
3. That include changes **load order**, and load order decides which `:443` block catches
|
|
unmatched names. So `default.conf`'s commented-out `:443 default_server` must be restored
|
|
**first**. `make check` fails on it deliberately: it is a gate, not a warning.
|
|
|
|
---
|
|
|
|
## What the checks assert, and why
|
|
|
|
The scripts are deliberately thin on comment — the reasoning lives here. Every check below
|
|
corresponds to something that is wrong, or was wrong, in a real estate.
|
|
|
|
### `make check` — the estate
|
|
|
|
| assertion | the failure it catches |
|
|
| --- | --- |
|
|
| every service name is covered by an issued cert SAN | **a wildcard matches exactly one label.** `*.d.com` covers `a.d.com` but not `a.b.d.com`, which needs its own SAN. Surfaces otherwise as a browser TLS warning, far from its cause |
|
|
| a `:443 default_server` exists | DNS and the cert are wildcard but nginx matches `server_name` exactly, so without one the fallback for any unknown name is whichever vhost loads first — alphabetically, by accident |
|
|
| firewall rules and listeners agree | a rule allowing a port nothing listens on is **dead config**; a service no compose file declares is **undocumented state**. Neither is visible from one side alone, which is why the inventory has two halves |
|
|
| `HOST` is an ssh alias, never a hostname | there is no `Host <domain>` block, so a bare hostname falls through to the global defaults, ssh offers every key in the agent in turn, and `MaxAuthTries` (6) trips with *"Too many authentication failures"* before reaching the right one. The aliases set `IdentitiesOnly yes` |
|
|
|
|
### `make vpn check` — the overlay
|
|
|
|
| assertion | the failure it catches |
|
|
| --- | --- |
|
|
| peer addresses unique and inside the subnet | a duplicate is a silent misroute, never an error |
|
|
| AllowedIPs do not overlap | AllowedIPs is **cryptokey routing** — the route table and the ACL at once. Overlapping ranges resolve to the last match, so an overlap is both a misroute and an unintended grant |
|
|
| something carries `PersistentKeepalive` if anything roams | without it a NAT mapping expires and the tunnel works only while traffic flows outward — *"works sometimes"*, the hardest failure to read. Note it is **not** a property of the roaming peer's own entry: the roaming machine sets it on the entry for the peer it **dials** |
|
|
| the listen port is in the firewall | otherwise no peer can be dialed at all |
|
|
| no private key in the description | public keys are *also* 44-char base64, so the shape proves nothing. The real assertions are **no field named `priv*`** and **no key outside a `public_key` field** |
|
|
| overlay-reached services bind a reachable address | **a tunnel cannot reach loopback.** A service on `127.0.0.1` is unreachable over the overlay; one on `0.0.0.0` is reachable but also exposed to the whole LAN |
|
|
|
|
### Capturing the overlay
|
|
|
|
```bash
|
|
sudo wg show | make vpn capture --write
|
|
```
|
|
|
|
`wg show` has three forms and **only the first is safe**:
|
|
|
|
| form | safe | why |
|
|
| --- | --- | --- |
|
|
| `wg show` | **yes** | prints `private key: (hidden)` |
|
|
| `wg show <if> dump` | **no** | field 1 of the first line *is* the private key |
|
|
| `wg showconf <if>` | **no** | prints `PrivateKey=` outright |
|
|
|
|
`capture` refuses the latter two by shape. It matches peers by **allowed-ips address, not
|
|
public key** — the keys are exactly what is missing at that point — and **drops a roaming
|
|
peer's endpoint in the parser**, since that value is a home ISP address and a roaming peer
|
|
has no stable endpoint anyway.
|
|
|
|
## Layout
|
|
|
|
```
|
|
berth/
|
|
├── Makefile one target per ctrl/ script; the verb is an argument
|
|
├── STALE.md withdrawn assumptions, each with a check that runs
|
|
├── estate/<name>.json THE ARTIFACT — one description, many renderings
|
|
└── ctrl/
|
|
├── check.sh reports and instructs; never fixes
|
|
├── estate.sh show | list | plan | apply --yes | destroy --yes
|
|
├── services.sh list | render <aws|gcp|local> | deploy
|
|
├── ports.sh show | verify (the rig coincidence check)
|
|
├── dns.sh certs.sh host.sh registry.sh docs.sh
|
|
├── versions.env pinned toolchain (weakest layer)
|
|
├── env.d/<target>.env provider shape: aws | gcp
|
|
├── .env.example -> ctrl/.env, machine-local (gitignored)
|
|
├── lib/config.sh the four-layer load, from rig
|
|
├── lib/estate.sh reading and projecting the estate
|
|
└── render/*.tmpl nginx vhost shapes, substituted with sed
|
|
```
|
|
|
|
Config layers, weakest first: `versions.env` → `env.d/<target>.env` → `ctrl/.env` → the
|
|
caller's environment. So `make estate plan TARGET=gcp` beats everything.
|
|
|
|
**Identity is explicit — the inversion of rig.** rig derives its name from its folder so that
|
|
copies never collide. berth refuses to guess, because a deployment has exactly one production
|
|
and a wrong guess acts on the wrong estate. `ESTATE` names a file; the only convenience is
|
|
that a single `estate/*.json` is used without being asked for.
|
|
|
|
**`python3`, not `jq`.** rig ships a pinned static `jq` because its floor is "docker and
|
|
nothing else" on a machine it does not control. berth's floor is already higher, so
|
|
`python3` is a dependency it *has* rather than one it *adds* — the same reasoning by which
|
|
rig chose `sed` over `envsubst`.
|
|
|
|
---
|
|
|
|
## Status
|
|
|
|
**B0 (this) is the shape.** `estate/mcrn.json` is marked `UNVERIFIED`: it records what the
|
|
repos *claim*, because `ppl/infra/` was written and never applied — no `~/.pulumi`, no
|
|
`venv`, no stack state, files dated `mar 6`. B1's read-only inventory is what replaces those
|
|
claims with observations. Until then, `estate plan` and `estate apply` refuse: there is
|
|
nothing truthful to compare against yet.
|
|
|
|
`make check` currently fails on two real things — see `def/plans/36.0/berth.md`.
|