216 lines
8.2 KiB
Markdown
216 lines
8.2 KiB
Markdown
# From a machine with nothing on it to an environment you can work in
|
|
|
|
The README says the prerequisite is Docker and nothing else. This is what that
|
|
actually looks like end to end: a bare Linux box, and an overlay running under
|
|
Tilt at the end of it.
|
|
|
|
Read it once before running anything. Three of the steps below need root and one
|
|
needs a logout, so knowing about them in advance is cheaper than meeting them
|
|
halfway through.
|
|
|
|
|
|
## Docker, and the two sysctls Tilt depends on
|
|
|
|
rig installs a toolchain; it does not install Docker. That line is not modesty —
|
|
Docker is a daemon, a group membership and usually a logout, and a script that
|
|
did it would have to be trusted with root on a machine it knows nothing about.
|
|
|
|
```bash
|
|
sudo apt-get install -y docker.io && sudo usermod -aG docker "$USER"
|
|
```
|
|
|
|
Then log out and back in, and check `docker info` answers. Until it does, nothing
|
|
below works and everything below reports the same failure.
|
|
|
|
While you have root, raise the inotify limits:
|
|
|
|
```bash
|
|
echo -e 'fs.inotify.max_user_watches=524288\nfs.inotify.max_user_instances=512' \
|
|
| sudo tee /etc/sysctl.d/99-rig.conf
|
|
sudo sysctl --system
|
|
```
|
|
|
|
kind and Tilt both watch large trees, and WSL ships 8192 watches and 128
|
|
instances — far too low. The failure mode is the reason this is here at step
|
|
zero rather than mentioned later: Tilt does not error, it simply stops noticing
|
|
that files changed, and you lose an afternoon to a hot reload that silently
|
|
isn't.
|
|
|
|
|
|
## Read the docs before installing anything
|
|
|
|
```bash
|
|
cd rig
|
|
make docs
|
|
```
|
|
|
|
`ctrl/docs.sh` runs a throwaway `nginx:alpine` over a read-only bind mount of
|
|
`docs/` and prints the URL. That is deliberate: the docs are the instructions for
|
|
building everything else, so they cannot live in the cluster and cannot need
|
|
`python3 -m http.server` either — a minimal Debian has no python. What it has,
|
|
by definition, is Docker.
|
|
|
|
The port is this environment's `HTTP_PORT + 4`. Nothing is installed and nothing
|
|
persists; ctrl-c ends it.
|
|
|
|
|
|
## Ask what is wrong with this machine
|
|
|
|
```bash
|
|
make check
|
|
cp ctrl/.env.example ctrl/.env
|
|
```
|
|
|
|
`check.sh` reports and instructs, and fixes nothing. It runs bare rather than
|
|
in a container because host detection only ever reads `/proc` and `/etc` — no
|
|
dependency beyond coreutils.
|
|
|
|
Read the whole output, but the `ports` block is the one to read carefully. Every
|
|
port rig binds derives from this directory's name, so the answer is specific to
|
|
this copy, and a clash here surfaces as an opaque `failed to bind host port` in
|
|
the middle of cluster creation if you skip it.
|
|
|
|
Copy the `.env` even though the check only warns about it. It is gitignored, it is
|
|
where a machine-local override goes, and `ports.sh persist` expects it to exist.
|
|
|
|
|
|
## Install the toolchain — through the container
|
|
|
|
This is the step where "nothing installed" stops being rhetorical.
|
|
|
|
`make deps` runs `ctrl/deps.sh install` directly on the host, and the installer
|
|
fetches with `curl`. A stock `debian:trixie-slim` has no curl — detection runs
|
|
fine, then the first download dies with `curl: command not found` and an exit
|
|
code of 127. That is the bootstrap paradox `ctrl/Dockerfile.deps` exists
|
|
to kill — the installer carries its own toolchain so the host needs only Docker —
|
|
but building the image and running it are two different things, and only the
|
|
build has a Makefile target today. **On a genuinely bare machine, run it by
|
|
hand:**
|
|
|
|
```bash
|
|
make deps image # builds rig-deps:deps
|
|
mkdir -p ~/.local/bin
|
|
docker run --rm \
|
|
-v /:/host:ro \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-v "$HOME/.local/bin:/out/bin" \
|
|
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
|
|
rig-deps:deps install dev
|
|
```
|
|
|
|
The image name follows the directory, like everything else here: in `rig`
|
|
it is `rig-deps`, in a copy called `other-rig` it is `other-rig-deps`. The
|
|
tag is `deps` (or `full`, below), not `latest`.
|
|
|
|
None of the four arguments are guessable, so:
|
|
|
|
- **`/:/host:ro`** — the installer reads the *host's* `/etc/os-release` and
|
|
`/etc/wsl.conf`, not the container's. `HOST_ROOT=/host` is already baked into
|
|
the image; this is what it points at. Read-only, and it is the only reason
|
|
detection inside a container tells you anything about the machine.
|
|
- **the docker socket** — how detection reaches the daemon it is reporting on,
|
|
and how it counts kind clusters already running.
|
|
- **`/out/bin`** — the image's `OUT_BIN`. Whatever you mount here is where the
|
|
four binaries land.
|
|
- **`HOST_UID` / `HOST_GID`** — the installer runs as root so it can reach that
|
|
socket, which means everything it writes into a mounted volume is root-owned
|
|
and useless to you. These drive the `chown` back. Omit them and the install
|
|
looks like it worked.
|
|
|
|
`dev` is kubectl, jq, kind and tilt. `core` is kubectl and jq alone — no cluster
|
|
tooling — which is the right answer on a managed or corporate-issued machine and
|
|
is why the split exists.
|
|
|
|
Then put them on PATH, which the installer will remind you about because it cannot
|
|
edit your shell for you:
|
|
|
|
```bash
|
|
export PATH="$HOME/.local/bin:$PATH" # and add the same line to ~/.bashrc
|
|
```
|
|
|
|
If something else on this machine already provides `kubectl`, the installer says so
|
|
by name rather than shadowing it quietly. `OUT_BIN=$PWD/def/bin` installs
|
|
somewhere private instead.
|
|
|
|
**Two variants worth knowing before you need them.** `make deps image full` bakes
|
|
every pinned binary into the image at build time (`DEPS_SOURCE=baked`), so
|
|
`docker save` gives you the entire installer as one file to carry into an
|
|
air-gapped network. And `DEPS_SOURCE=artifactory` with `DEPS_ARTIFACTORY_URL`
|
|
pulls from a generic internal repo, which is usually the only thing a locked-down
|
|
client allows.
|
|
|
|
From here on this machine has curl, so **`make deps` is the short form** for
|
|
every later run and every later copy of this directory. The container path is
|
|
the first-time path.
|
|
|
|
|
|
## Prove the machine before blaming the project
|
|
|
|
```bash
|
|
make check
|
|
make cluster up
|
|
kubectl get nodes
|
|
```
|
|
|
|
`make check` re-runs every check — host, docker, toolchain, memory, ports — and
|
|
changes nothing. Run now, it should end with nothing left to do by hand, and that
|
|
is the point: it is the scoreboard, not the installer.
|
|
|
|
`make cluster up` builds rig's built-in defaults — one node, no addons,
|
|
boots fast. You do not need it to develop anything, but you do want to know that
|
|
kind, the kubeconfig context and the derived port block work *before* a new
|
|
project has any problems of its own to confuse them with. `make cluster down`
|
|
when you are done looking.
|
|
|
|
Before starting a second cluster, and it will not be long:
|
|
|
|
```bash
|
|
make cluster list
|
|
```
|
|
|
|
Available memory, per-cluster usage and each cluster's port block. On a 16 GiB
|
|
box four single-node clusters are comfortable and six push into swap, so this is
|
|
worth reading before rather than after. `make cluster free <names>` stops
|
|
clusters without deleting them; `docker start` brings them back untouched.
|
|
|
|
|
|
## Start an overlay
|
|
|
|
What runs lives outside rig, in an overlay — see
|
|
[`docs/notes/overlay.md`](docs/notes/overlay.md). Start from rig's own:
|
|
|
|
```bash
|
|
cp -r examples/starter local/myenv # local/ is gitignored by rig
|
|
echo 'OVERLAY=local/myenv' >> ctrl/.env
|
|
make check # shows the overlay, its cluster and ports
|
|
```
|
|
|
|
The cluster takes the overlay's folder name and the context becomes
|
|
`kind-<name>`, so there is nothing to edit for either. Replace the two example
|
|
components under `k8s/base/`, and add your images and resources to the
|
|
overlay's `Tiltfile`. Check the manifests before `kind` spends minutes on anything
|
|
— this renders the whole tree without a cluster and catches a broken patch
|
|
immediately:
|
|
|
|
```bash
|
|
kubectl kustomize local/myenv/k8s/overlays/dev
|
|
```
|
|
|
|
If the overlay is to be versioned, make `local/myenv` a repository of its own (rig
|
|
never tracks it), or keep it anywhere else and name it by path.
|
|
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
make kind-up # idempotent create, then selects the context
|
|
make tilt-up # context + your assigned port
|
|
```
|
|
|
|
`tilt-up` passes `--context kind-<name>` every time, which is the point of going
|
|
through `make` at all: tilt cannot deploy into whichever cluster you last looked
|
|
at.
|
|
|
|
`make tilt-down` and `make kind-down` close the loop, and `make kind-reset` is
|
|
delete-and-recreate for when a cluster wedges.
|