sanitized rig
This commit is contained in:
37
rig/standalone/README.md
Normal file
37
rig/standalone/README.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# standalone — single files for a machine the full rig is not going to
|
||||
|
||||
Each script here does one of rig's jobs without the rest of the tree. Copy one
|
||||
file onto a machine, run it, read the output. Nothing to clone, nothing to
|
||||
install first.
|
||||
|
||||
| file | does | full-rig equivalent |
|
||||
| --- | --- | --- |
|
||||
| `rigdeps.sh` | installs kind, kubectl, tilt, ctlptl and jq at rig's pins, checksum-verified, no sudo | `make deps` (`ctrl/deps.sh`) |
|
||||
| `rigmini.sh` | reports how much memory the machine *advertises* and what caps it; `push` measures what it will actually *survive* | `make mem`, and the memory section of `make check` |
|
||||
|
||||
**These are transitional.** Where the full rig is installed, use its own
|
||||
targets instead; they read `ctrl/versions.env` and the profile, which these
|
||||
cannot.
|
||||
|
||||
## Why single files
|
||||
|
||||
`rigdeps.sh` carries its pins inline, because `ctrl/versions.env` is not on the
|
||||
machine it is for. That makes two copies of the same versions and checksums.
|
||||
`make pins` compares them and fails on any difference — `ctrl/versions.env` is
|
||||
the source of truth.
|
||||
|
||||
`rigmini.sh` exists because on a container or managed workspace `/proc/meminfo`
|
||||
reports the *host's* memory while a cgroup cap kills processes at a fraction of
|
||||
it. `status` reads the caps; `push` allocates until something stops it.
|
||||
|
||||
## Use
|
||||
|
||||
```bash
|
||||
bash rigdeps.sh detect # report, change nothing
|
||||
bash rigdeps.sh install dev # install into ~/.local/bin
|
||||
bash rigmini.sh status # advertised memory and caps; safe
|
||||
bash rigmini.sh push # allocates until it stops — not on a machine you need
|
||||
```
|
||||
|
||||
`rigmini.sh push` deliberately consumes memory. Run `status` first, and only run
|
||||
`push` somewhere it is acceptable for other processes to be squeezed.
|
||||
574
rig/standalone/rigdeps.sh
Executable file
574
rig/standalone/rigdeps.sh
Executable file
@@ -0,0 +1,574 @@
|
||||
#!/usr/bin/env bash
|
||||
# Put kind, tilt and kubectl on a machine that has none of them.
|
||||
#
|
||||
# The single file companion to rigmini.sh, for the same reason: rig installs its
|
||||
# toolchain from ctrl/deps.sh reading ctrl/versions.env, and neither of those is
|
||||
# going to a fresh AWS WorkSpace. The pins live inline here instead.
|
||||
#
|
||||
# What it will not do, deliberately:
|
||||
#
|
||||
# * no sudo, no apt, no yum. It writes ONLY into $OUT_BIN (default
|
||||
# ~/.local/bin). Everything needing root — installing Docker, joining the
|
||||
# docker group, raising inotify limits — is REPORTED for you to decide on.
|
||||
# That is what makes it safe to run on a machine that already works.
|
||||
# * no unverified download. Every artifact is checked against a SHA256 taken
|
||||
# from the publisher's own release list. A mismatch aborts.
|
||||
# * no guessing at another architecture. See ARCHITECTURE below.
|
||||
#
|
||||
# Two tiers, because "install the toolchain" is not one decision:
|
||||
#
|
||||
# core kubectl, jq — talk to a cluster someone else runs. Nothing that
|
||||
# creates one. The right answer on a managed or corporate machine.
|
||||
# dev core plus kind, tilt and ctlptl — build clusters and hot-reload
|
||||
# into them. The default, and what you want on a workspace of your own.
|
||||
#
|
||||
# Usage:
|
||||
# rigdeps.sh detect report the host, change nothing
|
||||
# rigdeps.sh list the pinned versions and where they come from
|
||||
# rigdeps.sh install [core|dev] detect, download, verify, install, report
|
||||
# rigdeps.sh fetch [core|dev] [--to DIR] download + verify only
|
||||
# rigdeps.sh verify run what is installed and see if it works
|
||||
set -euo pipefail
|
||||
|
||||
OUT_BIN="${OUT_BIN:-$HOME/.local/bin}"
|
||||
|
||||
# ── the pinned toolchain ───────────────────────────────────────────────────
|
||||
#
|
||||
# ARCHITECTURE. These checksums are the upstream-published SHA256 of the
|
||||
# **linux/amd64** artifact and of nothing else. An arm64 WorkSpace bundle needs
|
||||
# a different binary with a different checksum, and this script refuses rather
|
||||
# than reusing these — a checksum that is merely plausible is worse than none,
|
||||
# because it turns a verified download into a ceremony.
|
||||
#
|
||||
# To bump a version, or to add arm64: take the checksum from the release's own
|
||||
# published list, never from a download you did.
|
||||
#
|
||||
# curl -sSL https://github.com/<org>/<repo>/releases/download/<tag>/checksums.txt
|
||||
#
|
||||
# kubectl publishes its own instead, at <KUBECTL_URL>.sha256.
|
||||
|
||||
KIND_VERSION=v0.32.0
|
||||
KIND_SHA256=50030de23cf40a18505f20426f6a8506bedf13c6e509244bd1fa9463721b0f54
|
||||
KIND_URL="https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64"
|
||||
|
||||
KUBECTL_VERSION=v1.36.3
|
||||
KUBECTL_SHA256=ebbd080e7c2e275093b55915722043257eb24004363e20acb3c4d71919f88336
|
||||
KUBECTL_URL="https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl"
|
||||
|
||||
TILT_VERSION=0.37.6
|
||||
TILT_SHA256=e9672b8a18d43501f35dcfe98465969a7db0e436b36cf0c50c7e6f8d40de5fe6
|
||||
TILT_URL="https://github.com/tilt-dev/tilt/releases/download/v${TILT_VERSION}/tilt.${TILT_VERSION}.linux.x86_64.tar.gz"
|
||||
|
||||
# ctlptl creates a kind cluster WITH a local registry wired in, which is what
|
||||
# keeps images off docker.io — an unqualified image name resolves to
|
||||
# docker.io/library/<name>, and there is nothing structural stopping a push there.
|
||||
CTLPTL_VERSION=0.9.4
|
||||
CTLPTL_SHA256=c63a1ec28e60bc3faf6becb76f53355c5cf5e0143dafdd27ad85db5584fa6b1e
|
||||
CTLPTL_URL="https://github.com/tilt-dev/ctlptl/releases/download/v${CTLPTL_VERSION}/ctlptl.${CTLPTL_VERSION}.linux.x86_64.tar.gz"
|
||||
|
||||
# Upstream's static build. Debian's jq is linked against libjq/libonig, which is
|
||||
# fine on Debian and not portable anywhere else.
|
||||
JQ_VERSION=1.8.2
|
||||
JQ_SHA256=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f
|
||||
JQ_URL="https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-amd64"
|
||||
|
||||
CORE_TOOLS="kubectl jq"
|
||||
DEV_TOOLS="kind tilt ctlptl"
|
||||
|
||||
# No helm: every rig addon installs with `kubectl apply -f`, so nothing has ever
|
||||
# invoked it. Add it the day something actually needs a chart.
|
||||
|
||||
# Collected as we go, printed by report_manual() at the very end. Anything that
|
||||
# needs root or a decision lands here instead of being done.
|
||||
MANUAL=()
|
||||
|
||||
# ── platform ───────────────────────────────────────────────────────────────
|
||||
|
||||
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
|
||||
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
|
||||
# the tooling. Detectable, so name it instead.
|
||||
require_linux() {
|
||||
case "$(uname -s)" in
|
||||
MINGW*|MSYS*|CYGWIN*)
|
||||
cat >&2 <<'EOF'
|
||||
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
|
||||
|
||||
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
|
||||
|
||||
wsl --install
|
||||
|
||||
That enables Windows features and needs a reboot, so it is not something this
|
||||
script will do for you. Afterwards, open the Linux shell it installs and run
|
||||
this from there.
|
||||
EOF
|
||||
exit 1 ;;
|
||||
Linux) ;;
|
||||
*) echo "$(uname -s) is not Linux. These are linux binaries; nothing here" >&2
|
||||
echo "would run even if it downloaded." >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
arch() {
|
||||
case "$(uname -m)" in
|
||||
x86_64|amd64) echo amd64 ;;
|
||||
aarch64|arm64) echo arm64 ;;
|
||||
*) uname -m ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# The pins above are amd64. Rather than download something that cannot execute
|
||||
# and let it fail as "cannot execute binary file: Exec format error", say so
|
||||
# here and hand over the commands that produce the right checksums.
|
||||
require_amd64() {
|
||||
local a; a=$(arch)
|
||||
[ "$a" = "amd64" ] && return 0
|
||||
cat >&2 <<EOF
|
||||
This machine is ${a} ($(uname -m)); every pin in this script is linux/amd64.
|
||||
|
||||
Nothing here would run, so it does not download. To make an ${a} version, the
|
||||
URLs need the ${a} artifact and the checksums need to come from each project's
|
||||
own published list — not from these values, and not from a download you did:
|
||||
|
||||
curl -sSL https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/checksums.txt
|
||||
curl -sSL https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${a}/kubectl.sha256
|
||||
curl -sSL https://github.com/tilt-dev/tilt/releases/download/v${TILT_VERSION}/checksums.txt
|
||||
curl -sSL https://github.com/tilt-dev/ctlptl/releases/download/v${CTLPTL_VERSION}/checksums.txt
|
||||
curl -sSL https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/sha256sum.txt
|
||||
|
||||
Edit the pinned block at the top of this file with what those print.
|
||||
EOF
|
||||
exit 1
|
||||
}
|
||||
|
||||
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
|
||||
|
||||
# ── the tools this script itself needs ─────────────────────────────────────
|
||||
|
||||
# A fresh minimal image may genuinely have neither curl nor wget. Find out once,
|
||||
# up front, rather than half way through the first download.
|
||||
DL=""
|
||||
pick_downloader() {
|
||||
if command -v curl >/dev/null 2>&1; then DL=curl
|
||||
elif command -v wget >/dev/null 2>&1; then DL=wget
|
||||
else
|
||||
echo "neither curl nor wget is installed, so nothing can be downloaded." >&2
|
||||
echo "Install one first: $(pkg_install_cmd curl)" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
download() {
|
||||
local url="$1" out="$2"
|
||||
case "$DL" in
|
||||
curl) curl -fsSL --retry 3 -o "$out" "$url" ;;
|
||||
wget) wget -q --tries=3 -O "$out" "$url" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# sha256sum is coreutils; shasum is the perl one that turns up on stripped
|
||||
# images. Verification is not optional, so if neither exists that is fatal.
|
||||
SHA=""
|
||||
pick_sha() {
|
||||
if command -v sha256sum >/dev/null 2>&1; then SHA=sha256sum
|
||||
elif command -v shasum >/dev/null 2>&1; then SHA="shasum -a 256"
|
||||
else
|
||||
echo "no sha256sum and no shasum — downloads could not be verified." >&2
|
||||
echo "Refusing to install unverified binaries." >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ── package manager, for the instructions only ─────────────────────────────
|
||||
# This never runs a package manager. It names one so the reported action is
|
||||
# something you can paste, on the distro you are actually on — an apt line on
|
||||
# Amazon Linux 2 is a wrong answer dressed up as help.
|
||||
|
||||
pkg_install_cmd() {
|
||||
local pkg="$1"
|
||||
if command -v apt-get >/dev/null 2>&1; then echo "sudo apt-get update && sudo apt-get install -y $pkg"
|
||||
elif command -v dnf >/dev/null 2>&1; then echo "sudo dnf install -y $pkg"
|
||||
elif command -v yum >/dev/null 2>&1; then echo "sudo yum install -y $pkg"
|
||||
elif command -v zypper >/dev/null 2>&1; then echo "sudo zypper install -y $pkg"
|
||||
elif command -v apk >/dev/null 2>&1; then echo "sudo apk add $pkg"
|
||||
else echo "install '$pkg' with this system's package manager"
|
||||
fi
|
||||
}
|
||||
|
||||
docker_pkg() {
|
||||
# Debian and Ubuntu call it docker.io; the RPM distros call it docker.
|
||||
if command -v apt-get >/dev/null 2>&1; then echo docker.io; else echo docker; fi
|
||||
}
|
||||
|
||||
# ── detect ─────────────────────────────────────────────────────────────────
|
||||
|
||||
detect() {
|
||||
echo "host"
|
||||
echo " kernel $(uname -r)"
|
||||
echo " arch $(arch) ($(uname -m))"
|
||||
[ -r /etc/os-release ] && \
|
||||
echo " distro $(sed -n 's/^PRETTY_NAME="\(.*\)"/\1/p' /etc/os-release)"
|
||||
if is_wsl; then echo " platform WSL"; else echo " platform native linux"; fi
|
||||
|
||||
local total_kb avail_kb
|
||||
total_kb=$(awk '/^MemTotal:/{print $2}' /proc/meminfo)
|
||||
avail_kb=$(awk '/^MemAvailable:/{print $2}' /proc/meminfo)
|
||||
printf " memory %d GB total, %d GB available\n" \
|
||||
$((total_kb / 1024 / 1024)) $((avail_kb / 1024 / 1024))
|
||||
if [ $((avail_kb / 1024 / 1024)) -lt 4 ]; then
|
||||
echo " ! under 4 GB available — a cluster will struggle here."
|
||||
echo " rigmini.sh says how much this box will actually give you."
|
||||
fi
|
||||
|
||||
echo " install to $OUT_BIN"
|
||||
detect_libc
|
||||
detect_prereqs
|
||||
detect_docker
|
||||
detect_inotify
|
||||
return 0
|
||||
}
|
||||
|
||||
# tilt is the one binary here that needs a recent glibc. MEASURED, not guessed:
|
||||
# tilt 0.37.6 on Amazon Linux 2 (glibc 2.26) fails with
|
||||
#
|
||||
# /lib64/libc.so.6: version `GLIBC_2.34' not found (required by .../tilt)
|
||||
#
|
||||
# which names a symbol rather than the problem. Amazon Linux 2 is a stock
|
||||
# WorkSpaces bundle, so this is the likely case, not an exotic one. Report the
|
||||
# version now; `verify` catches the actual failure after installing.
|
||||
detect_libc() {
|
||||
local v=""
|
||||
if command -v ldd >/dev/null 2>&1; then
|
||||
v=$(ldd --version 2>/dev/null | head -1 | grep -oE '[0-9]+\.[0-9]+$' || true)
|
||||
fi
|
||||
if [ -z "$v" ]; then
|
||||
echo " libc unknown (no ldd) — 'verify' is the real test"
|
||||
return 0
|
||||
fi
|
||||
echo " libc glibc $v"
|
||||
if [ "$(printf '%s\n2.34\n' "$v" | sort -V | head -1)" != "2.34" ]; then
|
||||
echo " ! older than glibc 2.34, which tilt needs. kubectl, kind, jq and"
|
||||
echo " ctlptl are static or libc-only and work here; tilt will not start."
|
||||
echo " Install the core tier, or run tilt from a container."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# What this script needs to do its own job. Reported here so `detect` answers
|
||||
# "will install work?" instead of leaving you to find out one download in.
|
||||
# Amazon Linux 2 ships without tar, which is exactly the surprise this catches.
|
||||
detect_prereqs() {
|
||||
local missing=""
|
||||
if command -v curl >/dev/null 2>&1; then echo " download curl"
|
||||
elif command -v wget >/dev/null 2>&1; then echo " download wget"
|
||||
else echo " ! no curl and no wget — nothing can be downloaded"; missing+=" curl"
|
||||
fi
|
||||
|
||||
if command -v sha256sum >/dev/null 2>&1 || command -v shasum >/dev/null 2>&1; then
|
||||
echo " checksums ok"
|
||||
else
|
||||
echo " ! no sha256sum or shasum — downloads could not be verified"
|
||||
missing+=" coreutils"
|
||||
fi
|
||||
|
||||
if command -v tar >/dev/null 2>&1 && command -v gzip >/dev/null 2>&1; then
|
||||
echo " archives tar + gzip"
|
||||
else
|
||||
echo " ! no tar/gzip — tilt and ctlptl ship as tarballs, so the dev tier"
|
||||
echo " cannot be unpacked. The core tier is two bare binaries and is fine."
|
||||
missing+=" tar gzip"
|
||||
fi
|
||||
|
||||
if [ -n "$missing" ]; then
|
||||
MANUAL+=("Install what this script needs to run at all:
|
||||
$(pkg_install_cmd "${missing# }")")
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
detect_docker() {
|
||||
# kind builds a cluster out of containers. Without a reachable daemon,
|
||||
# everything here installs perfectly and then does nothing.
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
if [ -S /var/run/docker.sock ]; then
|
||||
echo " docker socket present, no cli"
|
||||
return 0
|
||||
fi
|
||||
echo " ! docker not installed — kind has nothing to build a cluster in"
|
||||
MANUAL+=("Install Docker. It is the one real prerequisite, and the only
|
||||
thing here that needs root:
|
||||
$(pkg_install_cmd "$(docker_pkg)")
|
||||
sudo systemctl enable --now docker
|
||||
sudo usermod -aG docker \"\$USER\"
|
||||
then log out and back in, so the new group applies to your shell.")
|
||||
return 0
|
||||
fi
|
||||
if docker info >/dev/null 2>&1; then
|
||||
echo " docker $(docker version --format '{{.Server.Version}}' 2>/dev/null)"
|
||||
local n
|
||||
n=$(docker ps --filter "label=io.x-k8s.kind.cluster" --format '{{.Names}}' 2>/dev/null | wc -l)
|
||||
# Must be an `if`, not `[ ] && echo`: as the last statement here the
|
||||
# latter returns 1 when the count is zero, and `set -e` kills the
|
||||
# caller. That is the fresh-machine case, where it does most harm.
|
||||
if [ "$n" -gt 0 ]; then
|
||||
echo " - $n kind node container(s) already running"
|
||||
fi
|
||||
else
|
||||
echo " ! docker cli present but the daemon is unreachable"
|
||||
MANUAL+=("Start Docker, or add yourself to the docker group:
|
||||
sudo systemctl enable --now docker
|
||||
sudo usermod -aG docker \"\$USER\" # then log out and back in")
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# kind and tilt both watch large trees. Distro defaults are far too low and the
|
||||
# failure mode is silent: tilt simply stops noticing that files changed.
|
||||
detect_inotify() {
|
||||
local w i
|
||||
w=$(cat /proc/sys/fs/inotify/max_user_watches 2>/dev/null || echo 0)
|
||||
i=$(cat /proc/sys/fs/inotify/max_user_instances 2>/dev/null || echo 0)
|
||||
echo " inotify watches=$w instances=$i"
|
||||
if [ "$w" -lt 524288 ] || [ "$i" -lt 512 ]; then
|
||||
echo " ! low — tilt will silently stop seeing file changes"
|
||||
MANUAL+=("Raise the inotify limits (needs root):
|
||||
echo -e 'fs.inotify.max_user_watches=524288\\nfs.inotify.max_user_instances=512' \\
|
||||
| sudo tee /etc/sysctl.d/99-rig.conf
|
||||
sudo sysctl --system")
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── fetch ──────────────────────────────────────────────────────────────────
|
||||
|
||||
verify_sha() {
|
||||
local file="$1" want="$2" name="$3" got
|
||||
got=$($SHA "$file" | awk '{print $1}')
|
||||
if [ "$got" != "$want" ]; then
|
||||
echo >&2
|
||||
echo "CHECKSUM MISMATCH for $name — not installing it." >&2
|
||||
echo " expected $want" >&2
|
||||
echo " got $got" >&2
|
||||
echo >&2
|
||||
echo "Either the pin in this script is stale, or what arrived is not what" >&2
|
||||
echo "the publisher released. Neither is worth guessing about." >&2
|
||||
rm -f "$file"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# fetch_bin <name> <url> <sha256> <dest-dir> — a bare binary
|
||||
fetch_bin() {
|
||||
local name="$1" url="$2" sha="$3" dest="$4"
|
||||
local tmp="$dest/.$name.tmp"
|
||||
printf ' %-8s ' "$name"
|
||||
download "$url" "$tmp"
|
||||
verify_sha "$tmp" "$sha" "$name"
|
||||
mv "$tmp" "$dest/$name"
|
||||
chmod +x "$dest/$name"
|
||||
echo "ok"
|
||||
}
|
||||
|
||||
# fetch_tgz <name> <url> <sha256> <dest-dir> <path-inside> <strip>
|
||||
# Archive layouts differ, so the caller says which. tilt and ctlptl both ship
|
||||
# the binary at the archive root, hence strip=0.
|
||||
fetch_tgz() {
|
||||
local name="$1" url="$2" sha="$3" dest="$4" inner="$5" strip="$6"
|
||||
local tmp="$dest/.$name.tgz"
|
||||
printf ' %-8s ' "$name"
|
||||
download "$url" "$tmp"
|
||||
verify_sha "$tmp" "$sha" "$name"
|
||||
# --no-same-owner: some archives ship as uid 1001, and extracting as root
|
||||
# would otherwise restore an owner that is not you.
|
||||
tar -xzf "$tmp" -C "$dest" --strip-components="$strip" --no-same-owner "$inner"
|
||||
rm -f "$tmp"
|
||||
chmod +x "$dest/$name"
|
||||
echo "ok"
|
||||
}
|
||||
|
||||
fetch() {
|
||||
local dest="$OUT_BIN" tier="dev"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--to) dest="${2:?--to needs a directory}"; shift 2 ;;
|
||||
core|dev) tier="$1"; shift ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
mkdir -p "$dest"
|
||||
|
||||
if ! command -v tar >/dev/null 2>&1 && [ "$tier" = "dev" ]; then
|
||||
echo "tar is missing, and tilt and ctlptl ship as tarballs." >&2
|
||||
echo " $(pkg_install_cmd tar)" >&2
|
||||
echo "Or install the core tier, which is two bare binaries: $0 install core" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "fetching '$tier' into $dest (verifying every checksum)"
|
||||
fetch_bin kubectl "$KUBECTL_URL" "$KUBECTL_SHA256" "$dest"
|
||||
fetch_bin jq "$JQ_URL" "$JQ_SHA256" "$dest"
|
||||
if [ "$tier" = "dev" ]; then
|
||||
fetch_bin kind "$KIND_URL" "$KIND_SHA256" "$dest"
|
||||
fetch_tgz tilt "$TILT_URL" "$TILT_SHA256" "$dest" tilt 0
|
||||
fetch_tgz ctlptl "$CTLPTL_URL" "$CTLPTL_SHA256" "$dest" ctlptl 0
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── verify ─────────────────────────────────────────────────────────────────
|
||||
|
||||
tier_tools() { [ "$1" = "core" ] && echo "$CORE_TOOLS" || echo "$CORE_TOOLS $DEV_TOOLS"; }
|
||||
|
||||
# Downloading a verified binary proves it is the right file, not that this
|
||||
# machine can run it. On an old distro tilt fails here, with a linker error
|
||||
# about a missing symbol, and finding that out now beats finding out during a
|
||||
# first cluster build.
|
||||
verify_tools() {
|
||||
local tier="${1:-dev}" b bin out rc broke=0
|
||||
echo "checking that each one actually runs"
|
||||
for b in $(tier_tools "$tier"); do
|
||||
bin="$OUT_BIN/$b"
|
||||
if [ ! -x "$bin" ]; then
|
||||
printf ' %-8s not installed\n' "$b"
|
||||
continue
|
||||
fi
|
||||
rc=0
|
||||
case "$b" in
|
||||
kubectl) out=$("$bin" version --client 2>&1 | head -1) || rc=$? ;;
|
||||
jq) out=$("$bin" --version 2>&1 | head -1) || rc=$? ;;
|
||||
*) out=$("$bin" version 2>&1 | head -1) || rc=$? ;;
|
||||
esac
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
printf ' %-8s %s\n' "$b" "$out"
|
||||
else
|
||||
printf ' ! %-6s does not run here: %s\n' "$b" "$out"
|
||||
broke=1
|
||||
fi
|
||||
done
|
||||
if [ "$broke" -eq 1 ]; then
|
||||
echo
|
||||
echo " A binary that downloads and verifies but will not start is almost"
|
||||
echo " always this distro's libc being older than the release needs."
|
||||
echo " 'detect' prints the glibc version. The core tier (kubectl + jq)"
|
||||
echo " has no such dependency and will work regardless."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── install ────────────────────────────────────────────────────────────────
|
||||
|
||||
# Installing into a directory early in PATH silently replaces whatever the
|
||||
# machine was already using, which on a shared or corporate machine can break
|
||||
# unrelated work — kubectl more than one minor away from its cluster is the
|
||||
# common one. Say so; never decide it.
|
||||
warn_shadowing() {
|
||||
local b existing shadowed="" tier="${1:-dev}"
|
||||
case ":${PATH}:" in
|
||||
*":$OUT_BIN:"*) ;;
|
||||
*) return 0 ;; # not on PATH, so nothing is being shadowed yet
|
||||
esac
|
||||
for b in $(tier_tools "$tier"); do
|
||||
[ -x "$OUT_BIN/$b" ] || continue
|
||||
existing=$(PATH=$(echo "$PATH" | tr ':' '\n' | grep -vx "$OUT_BIN" | paste -sd:) \
|
||||
command -v "$b" 2>/dev/null || true)
|
||||
[ -n "$existing" ] || continue
|
||||
[ "$existing" = "$OUT_BIN/$b" ] && continue
|
||||
shadowed+=" $b $existing"$'\n'
|
||||
done
|
||||
[ -n "$shadowed" ] || return 0
|
||||
|
||||
echo
|
||||
echo " ! these were already installed elsewhere and are now shadowed:"
|
||||
printf '%s' "$shadowed"
|
||||
MANUAL+=("Decide which toolchain wins. To keep the previous one:
|
||||
rm -f $(for b in $(tier_tools "$tier"); do printf '%s ' "$OUT_BIN/$b"; done)
|
||||
Or install somewhere private instead:
|
||||
OUT_BIN=\$PWD/bin $0 install")
|
||||
return 0
|
||||
}
|
||||
|
||||
report_manual() {
|
||||
echo
|
||||
if [ ${#MANUAL[@]} -eq 0 ]; then
|
||||
echo "nothing left to do by hand."
|
||||
return 0
|
||||
fi
|
||||
echo "host actions this cannot perform (${#MANUAL[@]}):"
|
||||
echo
|
||||
local n=1 m
|
||||
for m in "${MANUAL[@]}"; do
|
||||
echo " $n. $m"
|
||||
echo
|
||||
n=$((n + 1))
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
install() {
|
||||
local tier="${1:-dev}"
|
||||
detect
|
||||
echo
|
||||
fetch "$tier"
|
||||
echo
|
||||
verify_tools "$tier"
|
||||
warn_shadowing "$tier"
|
||||
|
||||
if [ "$tier" = "core" ]; then
|
||||
echo
|
||||
echo " core tier: no kind, tilt or ctlptl. '$0 install dev' adds them."
|
||||
fi
|
||||
|
||||
case ":${PATH}:" in
|
||||
*":$OUT_BIN:"*) ;;
|
||||
*) MANUAL+=("Put the toolchain on your PATH — add to ~/.bashrc:
|
||||
export PATH=\"${OUT_BIN}:\$PATH\"
|
||||
then: source ~/.bashrc") ;;
|
||||
esac
|
||||
|
||||
report_manual
|
||||
|
||||
if [ "$tier" = "dev" ]; then
|
||||
echo "Once Docker is reachable and this is on PATH:"
|
||||
echo
|
||||
echo " kind create cluster --name scratch"
|
||||
echo " kubectl cluster-info --context kind-scratch"
|
||||
echo " kind delete cluster --name scratch"
|
||||
echo
|
||||
echo "That round trip is the real test that this machine can host a rig."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
list() {
|
||||
echo "pinned, linux/amd64 only:"
|
||||
printf ' %-8s %s\n' kubectl "$KUBECTL_VERSION"
|
||||
printf ' %-8s %s\n' jq "$JQ_VERSION"
|
||||
printf ' %-8s %s\n' kind "$KIND_VERSION"
|
||||
printf ' %-8s %s\n' tilt "$TILT_VERSION"
|
||||
printf ' %-8s %s\n' ctlptl "$CTLPTL_VERSION"
|
||||
echo
|
||||
echo " core = $CORE_TOOLS"
|
||||
echo " dev = $CORE_TOOLS $DEV_TOOLS"
|
||||
echo
|
||||
echo "Checksums are pinned in the block at the top of this file. To bump one,"
|
||||
echo "take the new checksum from the publisher's own release list — the header"
|
||||
echo "comment has the exact commands."
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
require_linux
|
||||
|
||||
case "${1:-install}" in
|
||||
detect) detect; report_manual ;;
|
||||
list) list ;;
|
||||
verify) verify_tools "${2:-dev}" ;;
|
||||
fetch) shift; require_amd64; pick_downloader; pick_sha; fetch "$@" ;;
|
||||
install) shift; require_amd64; pick_downloader; pick_sha; install "${1:-dev}" ;;
|
||||
*) echo "usage: $0 [detect|list|install|fetch|verify]" >&2
|
||||
echo " install [core|dev] (default dev)" >&2
|
||||
echo " fetch [core|dev] [--to DIR]" >&2
|
||||
echo " OUT_BIN=<dir> overrides the install directory" >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
635
rig/standalone/rigmini.sh
Executable file
635
rig/standalone/rigmini.sh
Executable file
@@ -0,0 +1,635 @@
|
||||
#!/usr/bin/env bash
|
||||
# How much memory this box will actually give you before something dies.
|
||||
#
|
||||
# rig answers this for a machine it is installed on. This is the single file
|
||||
# version, for a machine rig is not going to: paste it onto a fresh AWS
|
||||
# WorkSpace, an EC2 box or a container, run it, and get the same numbers in the
|
||||
# same order so two machines can be read side by side.
|
||||
#
|
||||
# There are two numbers and they are rarely the same. `status` reports what the
|
||||
# machine ADVERTISES and what is quietly capping it. `push` finds what it will
|
||||
# SURVIVE, by allocating until it stops.
|
||||
#
|
||||
# The gap between them is the whole reason this exists. Under WSL the cap lives
|
||||
# in .wslconfig; in a container or a managed workspace it is a cgroup limit, and
|
||||
# there /proc/meminfo reports the HOST's memory while the kernel kills you at a
|
||||
# fraction of it. A script that only read MemTotal would confidently report 32 GB
|
||||
# on a box that OOMs at 2.
|
||||
#
|
||||
# Reports and instructs. It never raises a limit, frees anything, writes a
|
||||
# config or installs a package — on a machine you are still evaluating, a probe
|
||||
# that changes what it is measuring is worse than no probe.
|
||||
#
|
||||
# Usage:
|
||||
# rigmini.sh status what it has, what caps it
|
||||
# rigmini.sh push [--to GB] [--to-oom] climb until it stops
|
||||
# rigmini.sh all [--budget GB] both, then the verdict
|
||||
set -euo pipefail
|
||||
|
||||
# ── defaults ───────────────────────────────────────────────────────────────
|
||||
|
||||
STEP_MB=0 # per allocation; 0 means scale it to the ceiling. See push().
|
||||
STEP_EXPLICIT=no # whether --step was given, which turns the scaling off.
|
||||
TO_MB="" # --to: stop here regardless. Empty means no hard cap.
|
||||
TO_OOM=no # --to-oom: opt in to running until the kernel intervenes.
|
||||
BUDGET_GB=6 # what the rig data profile is assumed to want; see all().
|
||||
BUDGET_EXPLICIT=no # whether --budget was given, which retires the guess below.
|
||||
|
||||
# ── platform ───────────────────────────────────────────────────────────────
|
||||
|
||||
# Windows outside WSL — Git Bash, MSYS, Cygwin — looks close enough to work and
|
||||
# then fails in a pile of confusing ways: no /proc, no docker socket, none of
|
||||
# the tooling. Detectable, so name it instead.
|
||||
require_linux() {
|
||||
case "$(uname -s)" in
|
||||
MINGW*|MSYS*|CYGWIN*)
|
||||
cat >&2 <<'EOF'
|
||||
This has to run inside WSL, not Git Bash / MSYS / Cygwin.
|
||||
|
||||
If WSL is not installed yet, from an elevated PowerShell or Command Prompt:
|
||||
|
||||
wsl --install
|
||||
|
||||
That enables Windows features and needs a reboot, so it is not something this
|
||||
script will do for you. Afterwards, open the Linux shell it installs and run
|
||||
this from there.
|
||||
EOF
|
||||
exit 1 ;;
|
||||
esac
|
||||
|
||||
# Everything below reads /proc. Without it there is nothing to measure, and
|
||||
# failing here beats printing a page of empty fields.
|
||||
if [ ! -r /proc/meminfo ]; then
|
||||
echo "no readable /proc/meminfo — this needs a Linux kernel." >&2
|
||||
echo "On macOS or a BSD none of the numbers below exist." >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
is_wsl() { grep -qi microsoft /proc/version 2>/dev/null; }
|
||||
|
||||
is_container() {
|
||||
[ -f /.dockerenv ] && return 0
|
||||
grep -qE '(docker|containerd|kubepods|lxc|podman)' /proc/1/cgroup 2>/dev/null
|
||||
}
|
||||
|
||||
platform() {
|
||||
if is_wsl; then echo WSL
|
||||
elif is_container; then echo container
|
||||
else echo "native linux"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── reading memory ─────────────────────────────────────────────────────────
|
||||
|
||||
mb() { echo $(( $(awk "/^$1:/{print \$2}" /proc/meminfo) / 1024 )); }
|
||||
|
||||
# MemAvailable arrived in kernel 3.14. Older kernels — and they turn up on
|
||||
# corporate images — need the estimate it replaced, which is worse but not wrong.
|
||||
avail_meminfo_mb() {
|
||||
if grep -q '^MemAvailable:' /proc/meminfo; then
|
||||
mb MemAvailable
|
||||
else
|
||||
awk '/^(MemFree|Buffers|Cached):/{t+=$2} END{print int(t/1024)}' /proc/meminfo
|
||||
fi
|
||||
}
|
||||
|
||||
# Where a cgroup records this cgroup's own limit and usage. Set once by
|
||||
# find_cgroup, because every later reading needs both and hunting for the files
|
||||
# on each call would be the slow part of the poll loop.
|
||||
CG_MAX_FILE=""
|
||||
CG_CUR_FILE=""
|
||||
CG_VERSION=""
|
||||
|
||||
find_cgroup() {
|
||||
local rel
|
||||
|
||||
# Inside a container the cgroup namespace makes the top of the tree BE the
|
||||
# container's own cgroup, so the unqualified path is already the right one.
|
||||
# On a host it is the root cgroup, which is never limited — hence the second
|
||||
# attempt via /proc/self/cgroup, which names the slice this shell is in.
|
||||
if [ -r /sys/fs/cgroup/memory.max ]; then
|
||||
CG_VERSION=v2
|
||||
CG_MAX_FILE=/sys/fs/cgroup/memory.max
|
||||
CG_CUR_FILE=/sys/fs/cgroup/memory.current
|
||||
elif [ -r /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
|
||||
CG_VERSION=v1
|
||||
CG_MAX_FILE=/sys/fs/cgroup/memory/memory.limit_in_bytes
|
||||
CG_CUR_FILE=/sys/fs/cgroup/memory/memory.usage_in_bytes
|
||||
fi
|
||||
|
||||
rel=$(awk -F: '$1=="0"{print $3; exit}' /proc/self/cgroup 2>/dev/null || true)
|
||||
if [ -n "$rel" ] && [ "$rel" != "/" ] && [ -r "/sys/fs/cgroup${rel}/memory.max" ]; then
|
||||
CG_VERSION=v2
|
||||
CG_MAX_FILE="/sys/fs/cgroup${rel}/memory.max"
|
||||
CG_CUR_FILE="/sys/fs/cgroup${rel}/memory.current"
|
||||
return 0
|
||||
fi
|
||||
|
||||
rel=$(awk -F: '$2 ~ /(^|,)memory(,|$)/{print $3; exit}' /proc/self/cgroup 2>/dev/null || true)
|
||||
if [ -n "$rel" ] && [ "$rel" != "/" ] \
|
||||
&& [ -r "/sys/fs/cgroup/memory${rel}/memory.limit_in_bytes" ]; then
|
||||
CG_VERSION=v1
|
||||
CG_MAX_FILE="/sys/fs/cgroup/memory${rel}/memory.limit_in_bytes"
|
||||
CG_CUR_FILE="/sys/fs/cgroup/memory${rel}/memory.usage_in_bytes"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# The cap in MB, or "" when there is none worth reporting. v2 spells unlimited
|
||||
# "max"; v1 spells it as a number near 2^63, which is why this compares against
|
||||
# MemTotal rather than testing for a magic value — a "limit" above the machine's
|
||||
# own memory is not a limit, however it is written.
|
||||
cgroup_cap_mb() {
|
||||
local raw cap
|
||||
[ -n "$CG_MAX_FILE" ] && [ -r "$CG_MAX_FILE" ] || { echo ""; return 0; }
|
||||
raw=$(cat "$CG_MAX_FILE" 2>/dev/null || echo max)
|
||||
[ "$raw" = "max" ] && { echo ""; return 0; }
|
||||
case "$raw" in ''|*[!0-9]*) echo ""; return 0 ;; esac
|
||||
cap=$((raw / 1024 / 1024))
|
||||
[ "$cap" -ge "$(mb MemTotal)" ] && { echo ""; return 0; }
|
||||
echo "$cap"
|
||||
}
|
||||
|
||||
cgroup_used_mb() {
|
||||
local raw
|
||||
[ -n "$CG_CUR_FILE" ] && [ -r "$CG_CUR_FILE" ] || { echo ""; return 0; }
|
||||
raw=$(cat "$CG_CUR_FILE" 2>/dev/null || echo "")
|
||||
case "$raw" in ''|*[!0-9]*) echo ""; return 0 ;; esac
|
||||
echo $((raw / 1024 / 1024))
|
||||
}
|
||||
|
||||
# ulimit -v is a per-process address-space cap. It stops YOU long before the box
|
||||
# does, and because it is inherited from a login shell it is easy to hit without
|
||||
# knowing it is set.
|
||||
ulimit_v_mb() {
|
||||
local v; v=$(ulimit -v 2>/dev/null || echo unlimited)
|
||||
[ "$v" = "unlimited" ] && { echo ""; return 0; }
|
||||
case "$v" in ''|*[!0-9]*) echo ""; return 0 ;; esac
|
||||
echo $((v / 1024))
|
||||
}
|
||||
|
||||
# The number everything else is about: the lowest of the things that can stop
|
||||
# you. Printed at the end of `status` and used as the sanity bound in `push`.
|
||||
effective_ceiling_mb() {
|
||||
local c; c=$(mb MemTotal)
|
||||
local cap; cap=$(cgroup_cap_mb)
|
||||
local ul; ul=$(ulimit_v_mb)
|
||||
[ -n "$cap" ] && [ "$cap" -lt "$c" ] && c="$cap"
|
||||
[ -n "$ul" ] && [ "$ul" -lt "$c" ] && c="$ul"
|
||||
echo "$c"
|
||||
}
|
||||
|
||||
# How much room is left RIGHT NOW, from whichever accounting actually governs.
|
||||
# In a capped container /proc/meminfo describes the host and is worse than
|
||||
# useless for this — it would report tens of gigabytes free on a box that is one
|
||||
# allocation from being killed.
|
||||
headroom_mb() {
|
||||
local cap used
|
||||
cap=$(cgroup_cap_mb)
|
||||
used=$(cgroup_used_mb)
|
||||
if [ -n "$cap" ] && [ -n "$used" ]; then
|
||||
echo $(( cap - used ))
|
||||
else
|
||||
avail_meminfo_mb
|
||||
fi
|
||||
}
|
||||
|
||||
# ── status ─────────────────────────────────────────────────────────────────
|
||||
|
||||
# /mnt/c/Users can hold several real accounts — a renamed login leaves the old
|
||||
# directory behind — so picking the first alphabetically is a coin toss. Ask
|
||||
# Windows, then fall back to whichever profile actually owns a config.
|
||||
wslconfig_path() {
|
||||
local profile winpath found
|
||||
profile=$(cmd.exe /c "echo %USERPROFILE%" 2>/dev/null | tr -d "\r\n" || true)
|
||||
case "$profile" in
|
||||
""|*%*) ;;
|
||||
*) winpath=$(wslpath -u "$profile" 2>/dev/null || true)
|
||||
if [ -n "$winpath" ] && [ -d "$winpath" ]; then
|
||||
echo "$winpath/.wslconfig"; return 0
|
||||
fi ;;
|
||||
esac
|
||||
found=$(ls -d /mnt/c/Users/*/.wslconfig 2>/dev/null | head -1 || true)
|
||||
[ -n "$found" ] && echo "$found"
|
||||
return 0
|
||||
}
|
||||
|
||||
hogs() {
|
||||
echo " holding the most:"
|
||||
ps -eo rss,comm --sort=-rss 2>/dev/null \
|
||||
| awk 'NR>1 && NR<=6 {printf " %6.0f MB %s\n", $1/1024, $2}'
|
||||
return 0
|
||||
}
|
||||
|
||||
status() {
|
||||
local total avail swap_total swap_free cap ul cur
|
||||
|
||||
echo "host"
|
||||
echo " platform $(platform)"
|
||||
echo " kernel $(uname -r)"
|
||||
[ -r /etc/os-release ] && \
|
||||
echo " distro $(sed -n 's/^PRETTY_NAME="\(.*\)"/\1/p' /etc/os-release)"
|
||||
echo " cpu $(getconf _NPROCESSORS_ONLN 2>/dev/null || echo '?') online, load $(cut -d' ' -f1-3 /proc/loadavg)"
|
||||
|
||||
# ── the caps first, because they decide what the totals below are worth ──
|
||||
echo
|
||||
echo "caps"
|
||||
cap=$(cgroup_cap_mb)
|
||||
if [ -n "$cap" ]; then
|
||||
cur=$(cgroup_used_mb)
|
||||
echo " cgroup ${cap} MB (${CG_VERSION}, ${CG_CUR_FILE##*/} says ${cur:-?} MB used)"
|
||||
echo " ! /proc/meminfo below describes the HOST, not this cgroup."
|
||||
echo " $(mb MemTotal) MB total is not yours; ${cap} MB is."
|
||||
elif [ -n "$CG_VERSION" ]; then
|
||||
echo " cgroup none (${CG_VERSION} present, no memory limit set)"
|
||||
else
|
||||
echo " cgroup no memory controller found"
|
||||
fi
|
||||
|
||||
ul=$(ulimit_v_mb)
|
||||
if [ -n "$ul" ]; then
|
||||
echo " ! ulimit -v ${ul} MB — a per-process cap, inherited from your shell"
|
||||
echo " it stops this process long before the machine runs out"
|
||||
else
|
||||
echo " ulimit -v unlimited"
|
||||
fi
|
||||
|
||||
# overcommit_memory=0 is the default heuristic: a large allocation is
|
||||
# granted on a guess, and the reckoning arrives later as an OOM kill rather
|
||||
# than as a failed malloc. It is why `push` touches every page it asks for.
|
||||
local om or_
|
||||
om=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null || echo '?')
|
||||
or_=$(cat /proc/sys/vm/overcommit_ratio 2>/dev/null || echo '?')
|
||||
case "$om" in
|
||||
0) echo " overcommit 0 heuristic — allocations are granted on a guess," ;;
|
||||
1) echo " overcommit 1 always — every allocation succeeds; the OOM killer is the only limit," ;;
|
||||
2) echo " overcommit 2 strict (ratio ${or_}%) — allocation fails honestly instead of killing later," ;;
|
||||
*) echo " overcommit ${om}" ;;
|
||||
esac
|
||||
[ "$om" != "?" ] && echo " so RSS is the number to trust, not what a process asked for"
|
||||
|
||||
# ── what it says it has ──
|
||||
total=$(mb MemTotal); avail=$(avail_meminfo_mb)
|
||||
swap_total=$(mb SwapTotal); swap_free=$(mb SwapFree)
|
||||
echo
|
||||
echo "memory"
|
||||
echo " total ${total} MB"
|
||||
echo " available ${avail} MB"
|
||||
echo " swap ${swap_total} MB ($(( swap_total - swap_free )) MB used)"
|
||||
if [ "$swap_total" -eq 0 ]; then
|
||||
echo " - no swap: this box has no cushion. It goes from fine to OOM-killed"
|
||||
echo " with nothing in between, which is the abrupt failure you get in a VM."
|
||||
fi
|
||||
|
||||
# postgres puts its shared buffers in /dev/shm. Docker's default is 64 MB,
|
||||
# and the resulting failure names neither shm nor the size.
|
||||
if [ -d /dev/shm ]; then
|
||||
local shm; shm=$(df -Pm /dev/shm 2>/dev/null | awk 'NR==2{print $2}')
|
||||
if [ -n "$shm" ]; then
|
||||
if [ "$shm" -le 64 ]; then
|
||||
echo " ! /dev/shm ${shm} MB — postgres puts shared memory here and 64 MB"
|
||||
echo " is docker's default. Raise it with --shm-size when the cabinet fails."
|
||||
else
|
||||
echo " /dev/shm ${shm} MB"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "disk"
|
||||
local d
|
||||
for d in / /tmp /var/lib/docker; do
|
||||
[ -d "$d" ] || continue
|
||||
df -Pm "$d" 2>/dev/null | awk -v p="$d" 'NR==2{printf " %-12s %s MB free of %s MB\n", p, $4, $2}'
|
||||
done
|
||||
|
||||
# kind and Tilt both watch large trees, and the failure mode is silent:
|
||||
# they simply stop noticing file changes. Cheap to report while we are here.
|
||||
local w i
|
||||
w=$(cat /proc/sys/fs/inotify/max_user_watches 2>/dev/null || echo 0)
|
||||
i=$(cat /proc/sys/fs/inotify/max_user_instances 2>/dev/null || echo 0)
|
||||
echo
|
||||
echo "tooling"
|
||||
echo " inotify watches=$w instances=$i"
|
||||
if [ "$w" -lt 524288 ] || [ "$i" -lt 512 ]; then
|
||||
echo " ! low — anything watching files will silently stop seeing changes"
|
||||
fi
|
||||
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
if [ -S /var/run/docker.sock ]; then
|
||||
echo " docker socket present, no cli"
|
||||
else
|
||||
echo " docker not installed"
|
||||
fi
|
||||
elif docker info >/dev/null 2>&1; then
|
||||
local n
|
||||
n=$(docker ps -q 2>/dev/null | wc -l)
|
||||
echo " docker $(docker version --format '{{.Server.Version}}' 2>/dev/null), ${n} container(s) running"
|
||||
else
|
||||
echo " ! docker cli present but the daemon is unreachable"
|
||||
fi
|
||||
|
||||
# WSL keeps its cap on the Windows side, in a file this shell can read but
|
||||
# not usefully apply — the change costs a full VM restart. Report it, and
|
||||
# report the commonest mistake, which is editing it and not restarting.
|
||||
if is_wsl; then
|
||||
local cfg conf
|
||||
cfg=$(wslconfig_path)
|
||||
echo
|
||||
echo "wsl"
|
||||
if [ -z "$cfg" ]; then
|
||||
echo " ! cannot tell which Windows profile owns .wslconfig"
|
||||
else
|
||||
echo " config $cfg"
|
||||
conf=$(sed -n 's/^[[:space:]]*memory[[:space:]]*=[[:space:]]*//p' "$cfg" 2>/dev/null \
|
||||
| tail -1 | tr -d '[:space:]')
|
||||
if [ -n "$conf" ]; then
|
||||
echo " configured $conf (booted ${total} MB)"
|
||||
echo " - if those disagree the edit has not been applied."
|
||||
echo " From a WINDOWS terminal: wsl --shutdown"
|
||||
else
|
||||
echo " configured no memory= set (WSL defaults to half the host RAM, or 8 GB,"
|
||||
echo " whichever is less — which is where your Airflow ceiling comes from)"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "effective ceiling $(effective_ceiling_mb) MB"
|
||||
echo " the lowest of MemTotal, the cgroup cap and ulimit -v. What the box"
|
||||
echo " claims. 'push' measures what it will actually hand over."
|
||||
|
||||
[ "$avail" -lt $(( total / 5 )) ] && { echo; hogs; }
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── push ───────────────────────────────────────────────────────────────────
|
||||
|
||||
STATE=""
|
||||
CHILD=""
|
||||
|
||||
cleanup() {
|
||||
if [ -n "$CHILD" ] && kill -0 "$CHILD" 2>/dev/null; then
|
||||
kill -KILL "$CHILD" 2>/dev/null || true
|
||||
wait "$CHILD" 2>/dev/null || true
|
||||
fi
|
||||
[ -n "$STATE" ] && rm -f "$STATE"
|
||||
return 0
|
||||
}
|
||||
|
||||
# The child allocates and stops itself; the parent only watches. That split is
|
||||
# the point: under --to-oom the allocating process is expected to be killed, and
|
||||
# something has to survive to say how far it got.
|
||||
allocator() {
|
||||
# Raise our own OOM score to the maximum so the kernel picks THIS process
|
||||
# first. Raising needs no privilege (only lowering does). Without it, the
|
||||
# kernel is free to choose your shell, your ssh session or dockerd — on a
|
||||
# box you are still using, that is not an acceptable coin toss.
|
||||
echo 1000 > "/proc/$BASHPID/oom_score_adj" 2>/dev/null || true
|
||||
|
||||
local arr=() held=0 i=0 rss swapped avail first_swap=0
|
||||
local bytes=$((STEP_MB * 1024 * 1024))
|
||||
local swap_used_start
|
||||
swap_used_start=$(( $(mb SwapTotal) - $(mb SwapFree) ))
|
||||
|
||||
while :; do
|
||||
# Written STRAIGHT INTO the array element. The obvious spelling —
|
||||
# build one chunk and `arr+=("$chunk")` — costs three copies per step,
|
||||
# not one: the template stays resident, expanding "$chunk" makes a
|
||||
# temporary word, and the append makes the element. A 128 MB step then
|
||||
# needs 384 MB transiently, and on a small box it is killed on the
|
||||
# first append while reporting a third of the true ceiling.
|
||||
#
|
||||
# printf -v into a subscript also means every page is written, so it is
|
||||
# resident rather than merely promised — the only kind of allocation
|
||||
# that measures anything under heuristic overcommit.
|
||||
printf -v "arr[$i]" '%*s' "$bytes" ''
|
||||
i=$((i + 1)); held=$((held + STEP_MB))
|
||||
|
||||
rss=$(awk '/^VmRSS:/{print int($2/1024)}' "/proc/$BASHPID/status" 2>/dev/null || echo 0)
|
||||
avail=$(headroom_mb)
|
||||
swapped=$(( $(mb SwapTotal) - $(mb SwapFree) - swap_used_start ))
|
||||
[ "$swapped" -lt 0 ] && swapped=0
|
||||
|
||||
printf '%8s MB held rss %7s MB headroom %7s MB swap +%s MB\n' \
|
||||
"$held" "$rss" "$avail" "$swapped"
|
||||
printf '%s %s %s %s\n' "$held" "$rss" "$avail" "$swapped" >> "$STATE"
|
||||
|
||||
# Worth calling out separately from the ceiling: this is where the box
|
||||
# stops being fast and starts being unusable, which for a scheduler is
|
||||
# a different and earlier problem than being killed.
|
||||
if [ "$swapped" -gt 0 ] && [ "$first_swap" -eq 0 ]; then
|
||||
first_swap=$held
|
||||
echo " - first swap page at ${held} MB — past here it works but crawls"
|
||||
echo "swapat $held" >> "$STATE"
|
||||
fi
|
||||
|
||||
if [ -n "$TO_MB" ] && [ "$held" -ge "$TO_MB" ]; then
|
||||
echo "stop reached-the-cap" >> "$STATE"; return 0
|
||||
fi
|
||||
if [ "$TO_OOM" = no ] && [ "$avail" -lt "$FLOOR_MB" ]; then
|
||||
echo "stop floor" >> "$STATE"; return 0
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
push() {
|
||||
local total ceiling rc=0 last held rss swapat stop
|
||||
total=$(mb MemTotal)
|
||||
ceiling=$(effective_ceiling_mb)
|
||||
|
||||
# A step is worth about a sixty-fourth of the ceiling: enough resolution to
|
||||
# find the edge, few enough lines to read, and small enough that the
|
||||
# transient cost of one allocation never dominates a small box. A fixed
|
||||
# size cannot do all three — 128 MB is fine on 16 GB and absurd on 512 MB.
|
||||
if [ "$STEP_EXPLICIT" = no ]; then
|
||||
STEP_MB=$(( ceiling / 64 ))
|
||||
[ "$STEP_MB" -lt 4 ] && STEP_MB=4
|
||||
[ "$STEP_MB" -gt 256 ] && STEP_MB=256
|
||||
fi
|
||||
|
||||
# Stop with a cushion rather than riding it to the kill. How big a cushion
|
||||
# depends on what it is protecting. Under a cgroup cap, running out kills
|
||||
# only this container's own processes, so it need cover no more than the
|
||||
# shell that prints the result — and a 512 MB cushion on a 1 GB box would
|
||||
# halve the answer. On a host there is everything else to protect, and the
|
||||
# OOM killer does not promise to pick the process that caused the problem.
|
||||
if [ -n "$(cgroup_cap_mb)" ]; then FLOOR_MB=64; else FLOOR_MB=512; fi
|
||||
[ $(( ceiling / 20 )) -gt "$FLOOR_MB" ] && FLOOR_MB=$(( ceiling / 20 ))
|
||||
|
||||
STATE=$(mktemp "${TMPDIR:-/tmp}/rigmini.XXXXXX")
|
||||
trap cleanup EXIT
|
||||
# INT kills the child and lets the summary below print anyway, so an
|
||||
# impatient Ctrl-C still tells you how far it got — and, more importantly,
|
||||
# still gives the memory back.
|
||||
trap 'echo; echo " interrupted"; echo "stop interrupted" >> "$STATE"; [ -n "$CHILD" ] && kill -KILL "$CHILD" 2>/dev/null || true' INT
|
||||
|
||||
echo "push"
|
||||
echo " step ${STEP_MB} MB per allocation, every page touched"
|
||||
echo " ceiling ${ceiling} MB claimed"
|
||||
if [ -n "$TO_MB" ]; then
|
||||
echo " stopping at ${TO_MB} MB (--to)"
|
||||
elif [ "$TO_OOM" = yes ]; then
|
||||
echo " ! stopping only when the kernel stops it (--to-oom)"
|
||||
echo " the allocating child is marked as the preferred OOM victim,"
|
||||
echo " but nothing about an OOM kill is entirely polite. Not on a box"
|
||||
echo " running anything you mind losing."
|
||||
else
|
||||
echo " stopping when headroom drops below ${FLOOR_MB} MB"
|
||||
fi
|
||||
echo
|
||||
|
||||
allocator &
|
||||
CHILD=$!
|
||||
wait "$CHILD" || rc=$?
|
||||
CHILD=""
|
||||
trap - INT
|
||||
|
||||
last=$(grep -E '^[0-9]' "$STATE" 2>/dev/null | tail -1 || true)
|
||||
held=$(echo "$last" | awk '{print $1}')
|
||||
rss=$(echo "$last" | awk '{print $2}')
|
||||
swapat=$(awk '/^swapat/{print $2}' "$STATE" 2>/dev/null | head -1 || true)
|
||||
stop=$(awk '/^stop/{print $2}' "$STATE" 2>/dev/null | head -1 || true)
|
||||
|
||||
echo
|
||||
if [ -z "$held" ]; then
|
||||
echo " ! nothing was allocated. Even one ${STEP_MB} MB chunk failed —"
|
||||
echo " try a smaller --step, or check ulimit -v in 'status'."
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo " reached ${rss:-$held} MB resident"
|
||||
[ -n "$swapat" ] && echo " swapping from ${swapat} MB"
|
||||
|
||||
case "$stop" in
|
||||
reached-the-cap)
|
||||
echo " outcome stopped at the --to cap, not at a limit."
|
||||
echo " The box held ${TO_MB} MB without complaint; there is more." ;;
|
||||
floor)
|
||||
echo " outcome stopped with a cushion intact, by choice."
|
||||
echo " The real ceiling is higher — --to-oom finds it, at the"
|
||||
echo " cost of an actual OOM kill." ;;
|
||||
interrupted)
|
||||
echo " outcome interrupted at ${rss:-$held} MB — where you stopped it,"
|
||||
echo " not where the box did." ;;
|
||||
*)
|
||||
# No stop line means the child did not decide to stop: it was ended.
|
||||
if [ "$rc" -ge 128 ]; then
|
||||
echo " outcome the child was killed (signal $((rc - 128))) at ${rss:-$held} MB."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
echo " outcome the allocation failed at ${rss:-$held} MB (exit ${rc})."
|
||||
echo " bash could not get the next chunk — an honest malloc"
|
||||
echo " failure rather than a kill. That is the strict-overcommit"
|
||||
echo " or ulimit path."
|
||||
else
|
||||
echo " outcome ended at ${rss:-$held} MB."
|
||||
fi
|
||||
local ev
|
||||
ev=$(dmesg 2>/dev/null | tail -80 | grep -iE 'oom-kill|killed process' | tail -1 || true)
|
||||
if [ -n "$ev" ]; then
|
||||
echo " kernel ${ev#*] }"
|
||||
else
|
||||
echo " - dmesg is unreadable here (dmesg_restrict, or no privilege),"
|
||||
echo " so the kill cannot be confirmed from this side. The number stands."
|
||||
fi ;;
|
||||
esac
|
||||
|
||||
# The gap between the claim and the measurement is the finding — but only
|
||||
# when the BOX chose where to stop. An empty $stop means the child was ended
|
||||
# rather than deciding to end; anything else (--to, the floor) is a stop we
|
||||
# asked for, and flagging those as short of the ceiling would put a warning
|
||||
# on every deliberately small run.
|
||||
local got="${rss:-$held}"
|
||||
echo
|
||||
if [ -z "$stop" ] && [ "$got" -lt $(( ceiling * 70 / 100 )) ]; then
|
||||
echo " ! claimed ${ceiling} MB, gave up ${got} MB — under 70% of it."
|
||||
echo " Something is taking the difference. 'status' names the candidates:"
|
||||
echo " a cgroup cap, ulimit -v, or memory already resident."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── all ────────────────────────────────────────────────────────────────────
|
||||
|
||||
all() {
|
||||
status
|
||||
echo
|
||||
echo "────────────────────────────────────────────────────────────"
|
||||
echo
|
||||
push
|
||||
|
||||
local got budget_mb ceiling
|
||||
budget_mb=$(( BUDGET_GB * 1024 ))
|
||||
ceiling=$(effective_ceiling_mb)
|
||||
got=$(grep -E '^[0-9]' "$STATE" 2>/dev/null | tail -1 | awk '{print $2}' || true)
|
||||
[ -n "$got" ] || got=0
|
||||
|
||||
echo
|
||||
echo "verdict"
|
||||
echo " budget ${BUDGET_GB} GB for kind + postgres + redis + airflow"
|
||||
# Only worth explaining while it is still a guess. Once --budget is given
|
||||
# the number came from somewhere better than this reasoning, and repeating
|
||||
# the derivation would describe a figure that is no longer in use.
|
||||
if [ "$BUDGET_EXPLICIT" = no ]; then
|
||||
echo " - that is 2 GB per kind node, which is rig's own figure, plus about"
|
||||
echo " 4 GB for the three cabinets. THE 4 GB IS AN ESTIMATE, not something"
|
||||
echo " measured. Re-run with --budget once you have watched the real thing."
|
||||
fi
|
||||
echo " measured ${got} MB handed over"
|
||||
|
||||
if [ "$got" -ge "$budget_mb" ]; then
|
||||
echo " fits, with $(( got - budget_mb )) MB spare."
|
||||
if [ "$got" -lt $(( budget_mb * 130 / 100 )) ]; then
|
||||
echo " - under 30% spare is thin for a scheduler. Airflow's memory use"
|
||||
echo " is spiky, and the spikes are what get killed."
|
||||
fi
|
||||
else
|
||||
echo " ! short by $(( budget_mb - got )) MB."
|
||||
if [ "$ceiling" -ge "$budget_mb" ]; then
|
||||
echo " The box CLAIMS enough (${ceiling} MB) but did not deliver it."
|
||||
echo " Free something, or read the caps section again."
|
||||
else
|
||||
echo " The box does not have it to give. A bigger bundle, or a smaller"
|
||||
echo " profile: PROFILE=minimal drops the cabinets entirely."
|
||||
fi
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
parse_flags() {
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--to) TO_MB=$(( ${2:?--to needs a value in GB} * 1024 )); shift 2 ;;
|
||||
--to-mb) TO_MB="${2:?--to-mb needs a value in MB}"; shift 2 ;;
|
||||
--step) STEP_MB="${2:?--step needs a value in MB}"; STEP_EXPLICIT=yes; shift 2 ;;
|
||||
--to-oom) TO_OOM=yes; shift ;;
|
||||
--budget) BUDGET_GB="${2:?--budget needs a value in GB}"; BUDGET_EXPLICIT=yes; shift 2 ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
if [ "$TO_OOM" = yes ] && [ -n "$TO_MB" ]; then
|
||||
echo "--to and --to-oom contradict each other: one stops early, the other" >&2
|
||||
echo "refuses to stop at all. Pick one." >&2
|
||||
exit 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
require_linux
|
||||
find_cgroup
|
||||
|
||||
cmd="${1:-status}"
|
||||
[ $# -gt 0 ] && shift
|
||||
|
||||
case "$cmd" in
|
||||
status) parse_flags "$@"; status ;;
|
||||
push) parse_flags "$@"; push ;;
|
||||
all) parse_flags "$@"; all ;;
|
||||
*) echo "usage: $0 [status|push|all]" >&2
|
||||
echo " push [--to GB] [--to-mb MB] [--step MB] [--to-oom]" >&2
|
||||
echo " all [--budget GB]" >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
Reference in New Issue
Block a user