278 lines
11 KiB
Markdown
278 lines
11 KiB
Markdown
# From a machine with nothing on it to a project you can work in
|
||
|
||
The README says the prerequisite is Docker and nothing else. This is what that
|
||
actually looks like end to end: a bare Linux box, and a new project running under
|
||
Tilt at the end of it.
|
||
|
||
A copy of this directory is a sibling of it, named after the environment it
|
||
models (`acme-rig`). Paths below are relative to the parent checkout.
|
||
|
||
It spans three repos because the work does. **rig** prepares the machine — the
|
||
pinned toolchain, the cluster, the port arithmetic. **all** owns the shape a
|
||
project takes, in `all/projects/templates/conventions.md` and the `broad`
|
||
scaffold beside it. **ppl** owns everything after local, and is where this
|
||
document stops.
|
||
|
||
Read it once before running anything. Three of the steps below need root and one
|
||
needs a logout, so knowing about them in advance is cheaper than meeting them
|
||
halfway through.
|
||
|
||
|
||
## Docker, and the two sysctls Tilt depends on
|
||
|
||
rig installs a toolchain; it does not install Docker. That line is not modesty —
|
||
Docker is a daemon, a group membership and usually a logout, and a script that
|
||
did it would have to be trusted with root on a machine it knows nothing about.
|
||
|
||
```bash
|
||
sudo apt-get install -y docker.io && sudo usermod -aG docker "$USER"
|
||
```
|
||
|
||
Then log out and back in, and check `docker info` answers. Until it does, nothing
|
||
below works and everything below reports the same failure.
|
||
|
||
While you have root, raise the inotify limits:
|
||
|
||
```bash
|
||
echo -e 'fs.inotify.max_user_watches=524288\nfs.inotify.max_user_instances=512' \
|
||
| sudo tee /etc/sysctl.d/99-rig.conf
|
||
sudo sysctl --system
|
||
```
|
||
|
||
kind and Tilt both watch large trees, and WSL ships 8192 watches and 128
|
||
instances — far too low. The failure mode is the reason this is here at step
|
||
zero rather than mentioned later: Tilt does not error, it simply stops noticing
|
||
that files changed, and you lose an afternoon to a hot reload that silently
|
||
isn't.
|
||
|
||
|
||
## Read the docs before installing anything
|
||
|
||
```bash
|
||
cd rig
|
||
make docs
|
||
```
|
||
|
||
`ctrl/docs.sh` runs a throwaway `nginx:alpine` over a read-only bind mount of
|
||
`docs/` and prints the URL. That is deliberate: the docs are the instructions for
|
||
building everything else, so they cannot live in the cluster and cannot need
|
||
`python3 -m http.server` either — a minimal Debian has no python. What it has,
|
||
by definition, is Docker.
|
||
|
||
The port is this environment's `HTTP_PORT + 4`. Nothing is installed and nothing
|
||
persists; ctrl-c ends it.
|
||
|
||
|
||
## Ask what is wrong with this machine
|
||
|
||
```bash
|
||
make check
|
||
cp ctrl/.env.example ctrl/.env
|
||
```
|
||
|
||
`check.sh` reports and instructs, and fixes nothing. It runs bare rather than
|
||
in a container because host detection only ever reads `/proc` and `/etc` — no
|
||
dependency beyond coreutils.
|
||
|
||
Read the whole output, but the `ports` block is the one to read carefully. Every
|
||
port rig binds derives from this directory's name, so the answer is specific to
|
||
this copy, and a clash here surfaces as an opaque `failed to bind host port` in
|
||
the middle of cluster creation if you skip it.
|
||
|
||
Copy the `.env` even though the check only warns about it. It is gitignored, it is
|
||
where a machine-local override goes, and `ports.sh persist` expects it to exist.
|
||
|
||
|
||
## Install the toolchain — through the container
|
||
|
||
This is the step where "nothing installed" stops being rhetorical.
|
||
|
||
`make deps` runs `ctrl/deps.sh install` directly on the host, and the installer
|
||
fetches with `curl`. A stock `debian:trixie-slim` has no curl — detection runs
|
||
fine, then the first download dies with `curl: command not found` and an exit
|
||
code of 127. That is the bootstrap paradox `ctrl/Dockerfile.deps` exists
|
||
to kill — the installer carries its own toolchain so the host needs only Docker —
|
||
but building the image and running it are two different things, and only the
|
||
build has a Makefile target today. **On a genuinely bare machine, run it by
|
||
hand:**
|
||
|
||
```bash
|
||
make deps-image # builds rig-deps:deps
|
||
mkdir -p ~/.local/bin
|
||
docker run --rm \
|
||
-v /:/host:ro \
|
||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||
-v "$HOME/.local/bin:/out/bin" \
|
||
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
|
||
rig-deps:deps install dev
|
||
```
|
||
|
||
The image name follows the directory, like everything else here: in `rig`
|
||
it is `rig-deps`, in a copy called `acme-rig` it is `acme-rig-deps`. The
|
||
tag is `deps` (or `full`, below), not `latest`.
|
||
|
||
None of the four arguments are guessable, so:
|
||
|
||
- **`/:/host:ro`** — the installer reads the *host's* `/etc/os-release` and
|
||
`/etc/wsl.conf`, not the container's. `HOST_ROOT=/host` is already baked into
|
||
the image; this is what it points at. Read-only, and it is the only reason
|
||
detection inside a container tells you anything about the machine.
|
||
- **the docker socket** — how detection reaches the daemon it is reporting on,
|
||
and how it counts kind clusters already running.
|
||
- **`/out/bin`** — the image's `OUT_BIN`. Whatever you mount here is where the
|
||
four binaries land.
|
||
- **`HOST_UID` / `HOST_GID`** — the installer runs as root so it can reach that
|
||
socket, which means everything it writes into a mounted volume is root-owned
|
||
and useless to you. These drive the `chown` back. Omit them and the install
|
||
looks like it worked.
|
||
|
||
`dev` is kubectl, jq, kind and tilt. `core` is kubectl and jq alone — no cluster
|
||
tooling — which is the right answer on a managed or corporate-issued machine and
|
||
is why the split exists.
|
||
|
||
Then put them on PATH, which the installer will remind you about because it cannot
|
||
edit your shell for you:
|
||
|
||
```bash
|
||
export PATH="$HOME/.local/bin:$PATH" # and add the same line to ~/.bashrc
|
||
```
|
||
|
||
If something else on this machine already provides `kubectl`, the installer says so
|
||
by name rather than shadowing it quietly. `OUT_BIN=$PWD/def/bin` installs
|
||
somewhere private instead.
|
||
|
||
**Two variants worth knowing before you need them.** `make deps-image full` bakes
|
||
every pinned binary into the image at build time (`DEPS_SOURCE=baked`), so
|
||
`docker save` gives you the entire installer as one file to carry into an
|
||
air-gapped network. And `DEPS_SOURCE=artifactory` with `DEPS_ARTIFACTORY_URL`
|
||
pulls from a generic internal repo, which is usually the only thing a locked-down
|
||
client allows.
|
||
|
||
From here on this machine has curl, so **`make deps` is the short form** for
|
||
every later run and every later copy of this directory. The container path is
|
||
the first-time path.
|
||
|
||
|
||
## Prove the machine before blaming the project
|
||
|
||
```bash
|
||
make setup
|
||
make cluster up
|
||
kubectl get nodes
|
||
```
|
||
|
||
`make setup` re-runs every check as a group. It is idempotent and it deliberately
|
||
does not abort on the first failure — a setup script that dies at step two hides
|
||
the fact that steps four and five were also going to fail. Run now, it should be
|
||
`ok` and `done` all the way down, and that is the point: it is the scoreboard,
|
||
not the installer.
|
||
|
||
`make cluster up` builds the default `minimal` profile — one node, no addons,
|
||
boots fast. You do not need it to develop anything, but you do want to know that
|
||
kind, the kubeconfig context and the derived port block work *before* a new
|
||
project has any problems of its own to confuse them with. `make cluster down`
|
||
when you are done looking.
|
||
|
||
Before starting a second cluster, and it will not be long:
|
||
|
||
```bash
|
||
make cluster list
|
||
```
|
||
|
||
Available memory, per-cluster usage and each cluster's port block. On a 16 GiB
|
||
box four single-node clusters are comfortable and six push into swap, so this is
|
||
worth reading before rather than after. `make cluster free <names>` stops
|
||
clusters without deleting them; `docker start` brings them back untouched.
|
||
|
||
|
||
## Scaffold the project
|
||
|
||
The canonical layout is [`all/projects/templates/conventions.md`](../all/projects/templates/conventions.md).
|
||
Read it — it is short, opinionated, and exists precisely so nobody
|
||
reverse-engineers a layout from whichever repo they happened to open. What
|
||
follows is only the mechanical part.
|
||
|
||
```bash
|
||
SLUG=<slug> # short, lowercase, no separators
|
||
cp -r ~/wdir/semester/all/projects/templates/broad ~/wdir/semester/"$SLUG"
|
||
cd ~/wdir/semester/"$SLUG"
|
||
grep -rl '<slug>' ctrl | xargs sed -i "s/<slug>/$SLUG/g"
|
||
cp ctrl/k8s/.env.example ctrl/k8s/.env
|
||
git init && git add -A && git commit -m "scaffold $SLUG from broad"
|
||
```
|
||
|
||
`<slug>` is the only placeholder and it lives only under `ctrl/` — cluster name,
|
||
namespace, ConfigMap name, and the `NAME=` in `kind-up.sh` / `kind-down.sh`. One
|
||
sed does all of it.
|
||
|
||
The slug is the folder name, lowercase and short — `mpr`, `unt`, `nvi`. The
|
||
cluster takes that name and the context becomes `kind-<slug>`, derived by the
|
||
scaffold's Makefile from the directory, so there is nothing to edit for either.
|
||
|
||
**Pick the Tilt port deliberately.** `ctrl/k8s/.env.example` ships a value that
|
||
is already in use, so copying it unchanged puts two projects on one port:
|
||
|
||
```bash
|
||
grep -h '^TILT_PORT=' ~/wdir/semester/*/ctrl/k8s/.env 2>/dev/null | sort
|
||
```
|
||
|
||
Choose a free one in `10300–10399` — the range ALL reserves in
|
||
`projects/index.json` under `policy` — avoiding `10350`, which is Tilt's own
|
||
default. Currently taken: `nvi` 10330, `unt` 10340, `mpr` 10360, `mlv` 10370,
|
||
`eth` 10380, `lng` 10390. This is the Tilt *web UI* port, not a service port;
|
||
each project owns its own service ports separately. The scaffold ships it blank
|
||
on purpose, so there is nothing to collide with until you choose.
|
||
|
||
The scaffold's `ctrl/k8s/` is the same shape as every other project here, and it
|
||
builds as shipped:
|
||
|
||
```
|
||
kind-config.yaml one node; gateway NodePort 30080 -> hostPort 8080
|
||
base/ namespace, configmap, app (Deployment + Service)
|
||
overlays/dev/ promotes the app Service to NodePort 30080
|
||
```
|
||
|
||
Check it before `kind` spends minutes on anything — this renders the whole tree
|
||
without a cluster and catches a broken patch immediately:
|
||
|
||
```bash
|
||
kubectl kustomize ctrl/k8s/overlays/dev
|
||
```
|
||
|
||
The workload is an nginx placeholder so a fresh copy reaches something that
|
||
answers; replace it. Keep `30080` in step between the overlay patch and
|
||
`kind-config.yaml`'s `containerPort` — the hostPort is this project's to pick.
|
||
Reachability is a plain kind port mapping: no ingress controller and no MetalLB.
|
||
Caddy maps `<slug>.local.ar` onto the host port (`~/wdir/semester/ppl/local/Caddyfile`),
|
||
with `*.local.ar` resolving to 127.0.0.1 through dnsmasq. That is the whole chain.
|
||
|
||
**The one file the scaffold still does not ship is `ctrl/Tiltfile`** — `make
|
||
tilt-up` runs `cd ctrl && tilt up`, and there is nothing to run until you write
|
||
one. Copy it from a live project; `unt` and `nvi` are closest to the plain shape.
|
||
|
||
|
||
## Run it
|
||
|
||
```bash
|
||
make kind-up # idempotent create, then selects the context
|
||
make tilt-up # context + your assigned port
|
||
```
|
||
|
||
`tilt-up` passes `--context kind-<slug>` every time, which is the point of going
|
||
through `make` at all: tilt cannot deploy into whichever cluster you last looked
|
||
at.
|
||
|
||
`make tilt-down` and `make kind-down` close the loop, and `make kind-reset` is
|
||
delete-and-recreate for when a cluster wedges.
|
||
|
||
|
||
## Register it
|
||
|
||
The project exists; now it is findable. Add an entry to
|
||
`~/wdir/semester/all/projects/index.json` and write its `projects/<slug>.md` beside the
|
||
others. Structured fields in the index, prose in the markdown.
|
||
|
||
Putting it on the CI server and deploying it is `ppl`'s half, and it starts at
|
||
`~/wdir/semester/ppl/ctrl/init-repo.sh` — gitea remote, then Woodpecker. That is a
|
||
different document.
|