soleprint/soleprint/station/tools/histgen/README.md

histgen

Seeds a clean, logical history into a repo — reading one directory and writing another, so the tree it reads is never touched.

histgen copy      # just the files: no .git, nothing ignored, no keys
histgen run       # scan the source, plan the commits, write the briefs
histgen list      # the commits, to confirm
histgen commands  # copy the files, hand back the git add/commit commands
histgen export    # ...or have it do the committing itself

Two directories, and the whole tool hangs off the difference:

source	the tree to read. Opened read-only, always. Not a commit, not a .git, not a state file is written into it — it can be a checkout you do not own or a read-only mount.
out	everything produced. The index, the plan, the briefs, and out/<name>/ — a copy of the source with the designed history committed into it.

That separation is what makes the history safe to argue with. It is an argument you will have more than once, and every attempt is a directory you can delete rather than a repo you have to put back.

Stdlib only, no network, no API key. Also readable in the browser at /station/tools/histgen/, which shows a plan and never writes one.

Copying it out

Copy the folder anywhere, cd into it, and use the Makefile. It derives the package name and path from where it sits, so the directory can be renamed and still work, and nothing has to be installed.

cp -r histgen ~/tools/ && cd ~/tools/histgen

make check                                  # prove it works, on its own fixture
make doctor                                 # what this machine has
make init-config SOURCE=~/work/x OUT=~/out  # set both once
make run                                    # scan + plan + brief
make list                                   # the commits, to confirm
make export                                 # write them into ~/out/x
make help                                   # every target

One target per verb, plus dry-run, against-history, verify and clean. Extra flags go in ARGS:

make plan REPO=/path/to/repo ARGS="--max-files 25"

make install drops a histgen command in ~/.local/bin pointing back at the folder, if you would rather not cd into it.

Just the files, without the repo

make copy SOURCE=~/code/myproject OUT=~/clean

Gives you ~/clean/myproject holding what the project actually is — no .git, nothing gitignored, nothing a build regenerates, and nothing that looks like a key. No history is planned and nothing is committed; this is the plain utility underneath the rest.

14 files tracked, 7 to copy, 7 left behind.

  secret — looks like a key or a credential (2):
    .env
    certs/server.key

  ignored — tracked, but the ignore rules say they should not be (1):
    data/raw/dump.sql

  derived — a build regenerates these (3):
    dist/bundle.js.map
    dist/bundle.min.js
    package-lock.json

  oversize — larger than --max-bytes (1):
    data/big.csv

Copied to /home/you/clean/myproject
  no .git — the source has one and it was not copied
  what was left behind: /home/you/clean/copied.md

Every drop is named, on screen and in copied.md. A file quietly missing from a copy is the same class of failure as a file quietly missing from a history, one directory earlier.

The same filter runs on the history path. scan drops secrets and ignored-but-tracked files before they ever reach a plan, and says so:

Scanned 5 files: 5 read, 0 reused from cache.
  3 left out of the history:
    .env  (looks like a key or a credential)
    certs/tls.key  (looks like a key or a credential)
    data/dump.sql  (tracked, but the ignore rules say they should not be)

so make commands and make export cannot commit a key that make copy would have left behind. make status keeps saying it afterwards, and --keep-secrets turns it off.

One deliberate difference between the two. copy also drops what a build regenerates — lockfiles, maps, minified output — because a snapshot is for reading. scan keeps them, because a lockfile is content in a repo somebody is going to use. copy --all keeps them too.

What gets left behind, and why

secret	.env, *.pem, *.key, id_rsa, .netrc, credentials.json, service-account*.json, .ssh/
ignored	tracked despite the repo's own ignore rules — someone ran git add -f once
derived	lockfiles, *.map, *.min.js, *.pyc, *.so, __pycache__/, node_modules/
oversize	whatever --max-bytes says

The derived list is ported from ppl/ctrl/distill.sh, including the lesson in its comments: the line is derived-vs-content, not text-vs-binary. Images, fonts, spreadsheets and PDFs are content and are kept — none of them can be regenerated from what is left, which is the only thing that makes a file safe to drop. That distinction was wrong in distill once and cost real files.

Lookalikes are kept on purpose. .env.example is the documented way to say what the real one needs, and dropping it takes the documentation with the secret. Same for server.key.pub — a public key is not a private one.

The ignored case is the one worth reading. git add -f is not always a mistake, so these are named rather than assumed either way; but a dump or a credentials file that went in once and was never noticed since looks exactly like this, and the repo is already contradicting itself about them.

Knobs

--keep-secrets, --exclude and --include work on copy, scan and run alike, and can live in histgen.json. The rest are copy's own.

make copy ARGS="--dry-run"                      # report only, write nothing
make copy ARGS="--max-bytes 100000"             # leave anything bigger
make copy ARGS="--exclude '*.csv' --exclude data/"
make copy ARGS="--include package-lock.json"    # keep it, whatever the filters say
make copy ARGS="--all --keep-secrets"           # turn the two filters off

--exclude follows distill's rule: a pattern with no / matches basenames at any depth. --include is checked first and wins outright, so one file can be rescued without turning a whole filter off.

To then plan a history from the cleaned tree, point a fresh run at it:

make run SOURCE=~/clean/myproject OUT=~/history

Recipe: a copy with a clean history

The common case. You have a repo, you want the same files somewhere new with a history that reads like the thing was built on purpose, and you do not care what the old history said.

Nothing is written to the original. The old history is simply not carried over — that is the default, and --keep-history is the opt-in for when you do want it.

OUT is the parent directory, not the repo. The copy keeps the source's own name underneath it. OUT does not have to exist yet; it is created on the first command.

cd ~/tools/histgen                       # wherever you copied the folder

make init-config SOURCE=~/code/myproject OUT=~/clean
make config                              # check both paths before anything runs

source       /home/you/code/myproject    read-only
out          /home/you/clean
exported to  /home/you/clean/myproject   <- the copy ends up here

1. Plan it

make run      # reads the source, groups the files, writes a brief per commit
make list     # the proposed commits, in order

make list is the thing to look at. Every commit is marked * until it has a message. If a commit holds two unrelated ideas, move a path between groups in ~/clean/plan.json and run make plan && make list again — the grouping is a proposal, and re-planning keeps every message whose group still holds the same files.

2. Write the messages

~/clean/briefs/ has one markdown file per commit, holding each file's opening comment. Read them and write a title and body into each group in ~/clean/plan.json.

This is the part worth doing properly: the briefs carry the reasoning already in the code, which is what makes a message worth reading. A message reconstructed from the diff just restates the diff.

make list     # again — the titles you wrote now show instead of the * marks

To see the shape end to end before writing any of them, use ARGS=--allow-untitled in the next step; the commits get their group name as a subject, which is fine for a throwaway pass and not fine for anything you keep.

3. Get the commands, and run them yourself

make commands

This copies the planned files into ~/clean/myproject and creates no repo — no git init, no .git. What comes back is the list, also saved to ~/clean/commands.sh:

cd /home/you/clean/myproject
git init

# 01  Repo skeleton: ignore rules and line-endings policy
git add -- .gitattributes .gitignore
git commit -F /home/you/clean/messages/01-skeleton.txt

# 02  Pin the toolchain in one manifest
git add -- versions.env
git commit -F /home/you/clean/messages/02-versions.txt

...

# Worth running afterwards. The first says no file was silently
# missed; the second says the result is byte-identical to the source.
git status --porcelain
git rev-parse HEAD^{tree}   # expect 3132703a817922f9f83bafa0e86e6bdf002ce8cb

git init is the first line of the list rather than something already done: a repo that appeared without you asking is exactly what someone reaching for this mode does not want. Read the list, edit it, reorder it, run it a line at a time.

Messages go in files rather than -m because bodies are multi-line, and the body is where the why lives. Edit the message files directly if you want to reword something — nothing has been committed yet.

The last two commands are worth running when you are done. git status --porcelain printing nothing means no file was silently missed, which is the failure this whole exercise exists to prevent. The tree hash matching means the result is byte-identical to the source.

Only the files in the plan are copied: not the source's .git, not anything gitignored. If ~/clean/myproject already contains a repo, this refuses rather than handing you commands that would commit into it.

Or let it do the committing

make dry-run  # optional: writes ~/clean/regen.sh, a script that does everything
make export   # copy and commit, checking both guards itself

18 commits on main in /home/you/clean/myproject. Checking:
  nothing left untracked: ok
  tree matches source (3132703a8179): ok

Three modes, and the difference is who does what:

	copies the files	makes the repo	commits
make commands	yes	no — you run git init	you
make dry-run	no — writes a script that would	in the script	in the script
make export	yes	yes	yes, and checks both guards

cd ~/clean/myproject
git log --oneline

If something goes wrong

make status   # says which of the four states out is in, and what to do next

• Interrupted partway? Run make export again — it continues from the commit after the last one recorded, rather than starting over or refusing. (This applies to make export; with make commands the repo is yours, so a half-finished run is yours to continue from the list.)
• Changed the plan after exporting? make status says so; make export ARGS=--force discards the copy and redoes it.
• Want to start completely fresh? make clean deletes the whole OUT directory. The source is not touched, so there is nothing to put back.

Handing this to someone else

Everything above needs the folder, python3 and git — nothing installed, no network, no API key. Copy the directory, then:

cd histgen && make check     # proves the whole pipeline on a fixture it builds
make help                    # every target

What is already in out

export writes, so it starts by working out what it is writing into. Four states, and they are genuinely different — histgen status prints which one:

absent	nothing there yet. Copy the tree, init, commit.
unfinished	commits this tool made, and a record of where it stopped. Something interrupted the run. Continue from the group after the last one recorded.
foreign	commits this tool did not make. That history is someone's, so nothing is rewritten, moved or deleted: the designed account goes on its own orphan branch and the existing branch is left exactly where it was.
stale	the plan changed, or the copy moved underneath us. Refuse, and say which of the two it was. --force discards and starts over.

Telling unfinished from foreign is the whole reason progress.json exists. Without it both read as "there are commits here", and the tool either destroys work it should have kept or refuses to finish work it started — which is exactly what it used to do.

$ histgen status
source    /home/mariano/wdir/rdir/adapter
out       /home/mariano/histories/adapter
census    91 files
plan      24 commits, 6 without a message
export    unfinished — 18 group(s) committed by this tool, 6 to go
          committed: 1-18
          remaining: 19-24

Run `export` again; it continues from where it stopped.

The record is written after each commit, not at the end — the point is to survive the run not reaching the end. It stores the plan's fingerprint (the groups and their paths, never the messages, so rewording commit 20 does not invalidate the nineteen already made) and each commit's sha, which must still be where the branch tip is or the copy has moved and it says so.

Keeping a history that already exists

histgen export --keep-history

Carries the source's .git into the copy and commits the designed account to an orphan branch, leaving the original branch pointing exactly where it did. Two tiers, which is what all/ctrl/handover.sh has been saying all along:

main               o-o-o-o          "updates 33.1 139"   (untouched)
designed-history   o-o-o-o-o-o-o-o  the designed account (no shared parent)

Both are present in out/<name>/; which one to publish is a decision for later and by hand. Nothing is rewritten and the source is not touched either way.

Settings

make init-config SOURCE=~/work/thing OUT=~/histories/thing
make config     # what everything resolves to, and where it came from
make run        # no arguments

Writes histgen.json beside the tool — the arrangement ppl/ctrl/distill.sh already uses, where the JSON next to the script is picked up when nothing else says otherwise.

{
  "source": "~/work/thing",
  "out":    "~/histories/thing",
  "max_files": null,
  "keep_history": false,
  "branch": null
}

Precedence, most specific first:

the command line  ->  --config FILE  ->  histgen.json beside the tool  ->  defaults

so a config sets a starting point and never wins an argument with a flag typed deliberately. A mistyped key is refused rather than ignored, because a setting plainly written in the file and silently not applied is a bad thing to debug. out inside source is refused too — the source is read-only by design, and out would end up in its own census.

histgen.json is gitignored by the folder's own .gitignore: which tree this machine points at is not a fact about the tool.

The verbs

Each writes one file under the repo's .histgen/, so the step before it is never repeated.

scan	out/index.json	what is in the source, cached by content hash
plan	out/plan.json	the order, cut into commits
brief	out/briefs/*.md	one pack per commit, for the messages
copy	out/<name>/	the files alone, no repo — needs no plan
list	—	the commits, printed to confirm
export	out/<name>/	the copy and its history, with both guards
status	—	which of the four states out is in
verify	—	the guards, on their own

plan.json is the seam. Everything above it is analysis that can be recomputed from the tree; everything below is git commands. That split is the whole design: the expensive half is a model reading code, and it should run once.

Where the messages come from

brief writes a markdown pack per commit holding each file's opening comment — not the file. An agent reads the packs and writes title and body back into plan.json.

That is deliberate. A commit message reconstructed from a diff restates the diff, and the thing worth recording was never in the diff: it was in the comment explaining why the ignore rules exist before the code they exclude, or why a port offset has to mean the same thing in two different projects. Handing over the reasoning that is already written down produces a message worth reading; handing over the diff produces Update files.

Keeping the model outside the tool is also what keeps the tool offline, keeps every message editable before a single commit exists, and keeps the cost of a 500-file repo to the comments rather than the code.

The order

Role first, references second.

skeleton (.gitignore) -> README -> version pins -> config layer -> profiles
  -> templates -> the things that source them -> front door (Makefile) LATE
  -> the bootstrap account LAST

The front door is late because it only dispatches; the bootstrap account is last because it narrates everything above it. References refine within that, so a config lands before the script that sources it.

References never override roles. A reference in code is a dependency; a reference in a comment is a footnote. .gitignore names ctrl/wizard.sh to say the opposite of "I need this", and a README names every file in the repo. Counting those as edges commits the ignore rules after the code they exclude — consistent, and unreadable. So refs are taken from non-comment lines only, and narrative files (.gitignore, README, docs, BOOTSTRAP) contribute no outgoing edges at all.

The grouping is a proposal

One coherent idea per commit, not one directory per commit. What holds a group together is that its files name each other; a hub and the directory named after it (addons.sh and addons/) always travel together, because committing a loader without the things it loads produces a commit that cannot run.

Where it cannot know — five scripts in one directory that never mention each other are five ideas or one, and nothing in the text says which — it guesses and says so. Moving a path from one group to another in plan.json is the expected way to use this, and plan re-run afterwards keeps every message whose group still holds the same files.

--max-files sets the cap. The default is about a twentieth of the tree with a floor of eight, which lands near how these repos were actually built — rig plans 18 against a real 18, spr 77 against a real 78. Raising it gives fewer, larger commits; lowering it gives more.

The two guards

export refuses a plan whose paths are missing, duplicated, or do not cover the source — before it writes anything. After the last commit it checks both:

1. nothing left untracked, with nothing exempt. The state lives in out and the copy lives inside it, so there is genuinely nothing of ours in the tree being checked. A file silently missed is the failure this whole tool exists to prevent. It is quiet at the time and surfaces much later, when something does not build on a fresh clone and the history offers no clue which commit should have carried it.
2. the exported tree still matches the source, by tree hash. Every path committed is not the same claim as the same tree: a stale index, a path in two groups, or a file edited mid-plan all pass the first check and fail this one.

--dry-run writes out/regen.sh and out/messages/ instead — ordinary git commands that copy the files and make the commits, both guards included, reviewable before anything runs.

Reporting on a history that already exists

python -m station.tools.histgen plan /path/to/repo --against-history

Maps each of the source's existing commits onto the group holding most of the files it touched, then reports what agrees and what does not:

  = 01 skeleton              a127b1d          matched one commit
  ~ 03 ctrl                  split across 2   one idea, committed piecemeal
  + 04 ctrl-lib              no commit        never landed as its own change
  ! 4 commit(s) land earlier in the proposed order than work already done
  ? 31 commit(s) carry no usable account of the change   ("updates 33.1 139")

It reads and prints. It never rewrites: published history is someone else's clone.

Verified against

rig's 18-commit history, which was built by hand and is what this reproduces. Run over the same 64 files, histgen plans 18 commits; the profiles, the cluster templates, the addons hub, the k8s manifests, the Makefile, sample-rig and BOOTSTRAP.md all land as their own commits in the same places. Replaying it produces a tree hash identical to the one rig ships.

Where it differs is where the difference is semantic: rig splits its wizard, host checks, cluster lifecycle and registry into four commits, and nothing in those four files' text says they are four things.

The CLI shape

cli.py is a shared scaffold — subcommand registration, one spelling for --source/-o/-n/--force/--dry-run, Error: … -> stderr -> exit 1 as the only exit path, deferred heavy imports so --help stays instant, and refuse_to_clobber. It exists because every tool here grew its own slightly different copy of the same three things.

histgen is its first user. Nothing else was rewritten to use it: a scaffold earns adoption by being there when the next tool is written.