Files
soleprint/soleprint/atlas2/docgen/docs/index.html
2026-09-24 00:58:19 -03:00

1613 lines
75 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>docgen</title>
<link rel="stylesheet" href="docs.css">
</head>
<body>
<div class="layout">
<nav class="sidebar">
<div class="sidebar-header">
<b>docgen</b>
<small>code &amp; data &rarr; documentation</small>
</div>
<nav>
<div class="group">Start</div>
<a href="#what">What it is</a>
<a href="#idea">The idea</a>
<a href="#quick">Five minutes</a>
<a href="#first">Your first project</a>
<a href="#models">Working with a model</a>
<div class="group">Architecture</div>
<a href="#layers">The three concerns</a>
<a href="#book">The book</a>
<a href="#ir">The IR</a>
<a href="#shape">Shape decides the drawing</a>
<a href="#pages">Pages</a>
<div class="group">Reading</div>
<a href="#extractors">Extractors</a>
<a href="#usage">Usage, not just the spec</a>
<div class="group">Narrowing</div>
<a href="#views">Views</a>
<div class="group">Writing</div>
<a href="#emitters">Emitters</a>
<a href="#explore">Explore</a>
<a href="#notebooks">Notebooks</a>
<a href="#notes">Notes</a>
<a href="#style">Style &amp; colour</a>
<div class="group">Reference</div>
<a href="#standalone">Standalone</a>
<a href="#runs">Run files</a>
<a href="#projects">Project files</a>
<a href="#commands">Commands</a>
<a href="#deps">Dependencies</a>
<a href="#testing">Testing</a>
<a href="#limits">Limits &amp; non-goals</a>
</nav>
</nav>
<main class="content">
<h1>docgen</h1>
<p class="lede">
Turn source artifacts — a codebase, a database, an API spec, a recording of
real traffic — into one canonical graph format, then render that format to
whatever the audience needs. The point is not the diagram. The point is the
format in the middle.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="what">What it is</h2>
<p>
Eight demos under <code>semester/</code> draw their architecture with
Graphviz. Every one of them hand-writes a <code>.dot</code> file with the
palette inlined and commits the <code>.svg</code> beside it. Three different
dark palettes between them, eight answers to the same question, and every one
of those diagrams is out of date the moment somebody moves a file.
</p>
<p>
docgen is the one answer. It reads the source rather than being told about it,
so a diagram cannot drift from the thing it describes, and it separates
<em>what a graph is</em> from <em>how it looks</em> so one extraction feeds
a diagram, an index, a notebook and a browsable minisite without being redone.
</p>
<div class="note">
<b>Nothing here writes a parser or a graph algorithm.</b> Parsers are adopted
(<code>ast</code>, tree-sitter, SQLAlchemy reflection via modelgen), algorithms
are networkx's. What docgen owns is the adapters, the schema, the style tables
and the emitters — all small, and all the places where the value is that
<em>we</em> made the call.
</div>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="idea">The idea</h2>
<p>
N sources and M outputs need N&times;M converters if you join them directly,
or <strong>N+M</strong> if you put a hub in the middle. The hub is an
<em>intermediate representation</em> — the compiler term, and the same bargain:
both sides depend on the IR and neither on the other.
</p>
<p>
It is lossy on purpose. It throws away every token of syntax and keeps
<em>"a class named User inherits from Base"</em>. That is the part that is
worth versioning, worth diffing, and worth drawing.
</p>
<p>
The practical consequence is the thing to judge it on: <strong>adding a source
costs one extractor and every emitter works on it unchanged; adding an output
costs one emitter and every extractor feeds it unchanged.</strong> When the
OpenAPI reader was written it emitted schemas using the same vocabulary the
database reader uses — and the ER diagram drew an API's data model without
anyone teaching it what an API was.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="quick">Five minutes</h2>
<p>
One command, if you want the whole thing — a <a href="#book">book</a>: what
went in, every step, what came out, and a page to open.
</p>
<pre><code>make book SRC=/path/to/repo BOOK=out/book/mine
make check BOOK=out/book/mine</code></pre>
<p>
More than one, rebuilt together, is a <a href="#runs">run file</a>:
<code>make run CONFIG=docgen.toml</code>.
</p>
<pre><code> larder /path/to/repo — 45 files read, 2 failed, 12 packages
book 312 nodes · 244 edges · 26 external · 8 artifacts
ok 45 file(s) read produced 45 module(s)
open out/book/mine/minisite/index.html</code></pre>
<p>
Underneath it is three commands that compose, and each still works on its own.
That is the interface, and the book does not replace it:
</p>
<pre><code><span class="c"># 1. read something</span>
python3 -m docgen extract python --root ../station/tools/histgen -o ir.json
<span class="c"># 2. narrow it to a useful view</span>
python3 -m docgen view ir.json --overview -o view.json
<span class="c"># 3. draw whatever its structure asks for</span>
python3 -m docgen emit auto view.json -o out/</code></pre>
<p>Or through the Makefile, which is a thin wrapper over exactly those:</p>
<pre><code>make ir SRC=/path/to/repo OUT=out <span class="c"># extract</span>
make explore OUT=out <span class="c"># the two-pane navigator</span>
make minisite OUT=out <span class="c"># a one-page site with a sidebar</span>
make self <span class="c"># docgen's book of soleprint, then check it</span></code></pre>
<p>
Everything is offline and self-contained. No server, no CDN, no build step —
the outputs open over <code>file://</code>.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="first">Your first project</h2>
<p>
From nothing to a notebook and a minisite for a project of yours, with your own text
in the right places. docgen lives in its own folder, and the project only gets one
config file and, later, a folder of notes.
</p>
<h3>1. Put docgen somewhere and check the machine</h3>
<pre><code>cp -r docgen ~/tools/docgen
cd ~/tools/docgen
make doctor <span class="c"># what is installed; never fails</span>
make sync <span class="c"># optional: C#/TypeScript, OpenAPI, all from uv.lock</span></code></pre>
<p>
From anywhere else, docgen runs as <code>python3 -m docgen</code> with its parent
folder on the path:
</p>
<pre><code>export PYTHONPATH=~/tools <span class="c"># the folder that holds docgen/</span>
python3 -m docgen --help</code></pre>
<h3>2. Write <code>docgen.toml</code> at the project's root</h3>
<pre><code>[project]
name = "orders-service"
[extract.code]
root = "src"
ext = [".cs"]
[emit.notebook]
output = "../docs_out/orders_service_notebook"
[emit.minisite]
output = "../docs_out/orders_service_minisite"</code></pre>
<p>
Paths are relative to this file. Keep the outputs outside the source tree. See
<a href="#projects">project files</a> for every key, and add
<code>.docgen/</code> to the project's <code>.gitignore</code>.
</p>
<h3>3. Check what it will do, then build</h3>
<pre><code>cd ~/work/orders-service
python3 -m docgen run docgen.toml --list <span class="c"># resolved paths; builds nothing</span>
python3 -m docgen run docgen.toml --check <span class="c"># build, then check every book</span></code></pre>
<pre><code>docgen.toml — 1 book(s)
ok code …/.docgen/orders-service/books/code
ok orders-service (document) …/.docgen/orders-service
1 book(s) in 1 document built and checked</code></pre>
<p>
A file docgen can't use is refused before anything is built, with every problem
listed at once and what to do about each. Open
<code>../docs_out/orders_service_site/index.html</code>: each book shows what
came in and what came out, and its pages click through from the overview.
</p>
<h3>4. Add your own text</h3>
<pre><code>python3 -m docgen scaffold docgen.toml --list <span class="c"># every spot a note can go</span>
python3 -m docgen scaffold docgen.toml <span class="c"># or: write an empty stub for each</span></code></pre>
<p>
Pick a spot and write markdown into <code>notes/&lt;book&gt;/&lt;spot&gt;.md</code>,
for example <code>notes/code/overview.md</code> above the overview page. Rebuild
and it's there, in the notebook and on the minisite. Stubs you leave empty add
nothing.
</p>
<div class="note">
<b>If a note stops appearing</b>, the page it was filed under was probably
renamed, for instance a package that moved. The rebuild says so and
<code>--check</code> fails with the note's name. Rename the file to the new spot
(<code>scaffold --list</code> shows them) or delete it.
</div>
<h3>5. Rebuild whenever the code moves</h3>
<p>
<code>python3 -m docgen run docgen.toml --check</code> again. A rebuild replaces
exactly what the last one wrote and never touches your notes or anything you
added to the output folders. <code>--only code</code> rebuilds one book while
you're writing its notes.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="models">Working with a model</h2>
<p>
A model can write the config and the notes. It shouldn't write anything else,
since every other file is generated. docgen ships the instructions to give it:
</p>
<pre><code>python3 -m docgen contract &gt; docgen-contract.md</code></pre>
<p>
That file is <code>docs/models.md</code>, printed verbatim. It lists every key
docgen accepts and the extensions it reads, has rules like "never a credential"
and "don't guess page ids", and ends with a table that maps each error docgen
can produce to the fix. docgen's own suite checks it against the code, so it
can't quietly go out of date: a key the loader accepts but the doc doesn't
name, or an example in it that doesn't load, fails <code>make check</code>.
</p>
<h3>The loop</h3>
<ol>
<li><b>Give it the project and the contract.</b> A digest of the project
(<code>distill</code>), plus <code>docgen-contract.md</code> as an
attachment rather than pasted. If replies go through <code>explode</code>,
attach its contract too: explode's says how to format the reply, docgen's
says what goes in the files.</li>
<li><b>Ask for <code>docgen.toml</code> first.</b> Unpack it, then run
<code>run docgen.toml --list</code>. If it's refused, paste the errors back
verbatim. Each one is in the contract's repair table.</li>
<li><b>Build once, then list the spots.</b> Run
<code>scaffold docgen.toml --list</code> and paste the output back. Page
names depend on the real directories, and a model that guesses them writes
notes that land nowhere.</li>
<li><b>Ask for notes for those spots.</b> Unpack, rebuild with
<code>--check</code>, and paste back any note reported as filed under a page
that doesn't exist.</li>
</ol>
<div class="note warn">
<b>What a model tends to write that docgen refuses.</b> In a dry run, a model
asked for a live database URL (with its password), a <code>seed</code> reader,
and shell, YAML and SQL files as code. All three are refused with a reason, and
the URL is never echoed in the error. The contract tells the model up front, but
the refusal is what makes it safe.
</div>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="layers">The three concerns</h2>
<p>
DOT collapses three separate questions into one file format, which is why a
hand-written <code>.dot</code> is never reusable: you cannot change the palette
without editing the structure, and you cannot change the structure without
re-deciding the layout. docgen keeps them apart.
</p>
<table>
<tr><th>concern</th><th>question</th><th>owner</th></tr>
<tr><td><b>structure</b></td><td>what the graph <em>is</em></td><td><code>ir/schema.json</code></td></tr>
<tr><td><b>meaning</b></td><td>what things <em>mean visually</em></td><td><code>style/*.json</code>, keyed on <code>kind</code></td></tr>
<tr><td><b>placement</b></td><td>where things <em>go</em></td><td>the emitter, and only there</td></tr>
</table>
<p>
An extractor has never heard of SVG, colours or layout. An emitter has never
heard of Python, <code>ast</code> or SQL. Both halves of that are checked by
parsing the source and looking at what it imports, because a rule nobody
enforces is a rule that lasts about a month.
</p>
<figure>
<a href="viewer.html?src=img/architecture.svg">
<img src="img/architecture.svg" alt="docgen's own module structure">
</a>
<figcaption>
docgen read by docgen. <code>extractors/</code> reaches only <code>ir</code>;
<code>emitters/</code> reaches <code>ir</code> and <code>style</code>;
<code>ir/</code> reaches nothing outside itself; <code>lab/</code> has no
edges at all. Click to open the viewer — then click again for actual size.
</figcaption>
</figure>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="book">The book</h2>
<p>
A <b>book</b> is one docgen operation, and it has a fixed shape: it begins by
saying what went in and ends by saying what came out. Everything in between is
an ordinary file that stands on its own.
</p>
<pre><code>larder ──► step ──► step ──► step ──► book
what each one usable what
came in by itself came out</code></pre>
<p>
The word comes from Atlas 1.0, where a book is a <em>larder</em> composed with a
pattern into something published and served. docgen had already built all three
parts under other names, so this is less an adoption than a renaming back.
</p>
<h3>Why both ends, rather than just the result</h3>
<p>
Because the two numbers are only worth having together. <em>"1,505 nodes"</em>
is not a fact about anything. <em>"225 files in, 1,505 nodes out, nothing
lost"</em> is. Before this, an extractor that read 45 of 47 files produced
exactly the same document as one that read all 47, and the diagram looked
complete either way — there was nowhere for the other two to be mentioned.
</p>
<div class="note warn">
<b>A clean diagram over an incomplete read is a lie by omission.</b> It is also
the failure mode the IR already guards against one level down: an unresolved
name becomes an <code>external</code> node rather than being dropped, because
silently losing a thing is worse than recording an unresolved one. The larder
measure is that same rule applied to the input as a whole.
</div>
<h3>The larder — what came in</h3>
<p>
Deliberately not called a bucket. A bucket is somewhere bytes sit; a larder is
<em>stocked from outside</em>, has an inventory, and goes stale. All three are
worth measuring, and they are what the measure records:
</p>
<pre><code>"larder": {
"kind": "python", <span class="c">// which extractor stocked it</span>
"identity": "../../station", <span class="c">// path, or a DSN with the password masked</span>
"unit": "file", <span class="c">// file | table | path | entry | document</span>
"seen": 47, <span class="c">// what the larder offered</span>
"read": 45, <span class="c">// seen - len(failed), derived</span>
"failed": [{"name": "a.py", "error": "syntax: line 3"}],
"extra": {"packages": 12}
}</code></pre>
<p>
<code>read</code> is derived and never stored. Stored, it invites the question
<em>"does that include the failures?"</em> and every reader answers it
differently; derived, there is nothing to get wrong — and
<code>ir/validate.py</code> fails a document whose arithmetic disagrees with
itself.
</p>
<p>
Failures are recorded <b>by name</b>, not counted. A count tells you a book is
incomplete; a name tells you which part of it to distrust.
</p>
<div class="note">
<b><code>identity</code> is the one field in docgen that could carry a secret</b>
— a database DSN has the password in it. It is masked at construction, and
<code>validate.py</code> then sweeps for the mask having worked, using its own
independent key list. A scrubber graded by its own word is not graded.
</div>
<h3>The book measure — what came out, and reconciled</h3>
<p>
Counts by kind, edges by kind, externals, and every artifact with its byte
count. On its own that is just a summary. What makes it a measure is that it is
<em>reconciled</em> against the larder:
</p>
<pre><code> larder ../../station — 45 files read, 2 failed, 12 packages
book 312 nodes · 244 edges · 26 external · 8 artifacts
ok 45 file(s) read produced 45 module(s)
ok 2 file(s) could not be read
ok 2 unreadable file(s) appear as 2 marked module(s)</code></pre>
<p>
The relation differs by source and is declared per extractor, because getting
it wrong gives a check that passes for the wrong reason. One file becomes one
module node — fewer means input was dropped. Four hundred HAR entries becoming
twelve endpoints is not a loss, it is the <em>point</em> of the capture. And a
file that failed to parse must still appear in the graph, carrying its error,
or the picture is smaller than the source and says nothing about it.
</p>
<p>
<code>python3 -m docgen book</code> exits 1 when a reconciliation fails. The
book is still written — the evidence is the point — but a build that lost input
should fail a pipeline rather than pass quietly.
</p>
<h3>The notebook is the sequence, the web is the last step</h3>
<p>
These two rule what gets generated, and each for its own reason.
</p>
<p>
The <b>notebook</b> is not one artifact; it is the sequence. Its first cell is
the larder measure and its last cell is the book measure, which is what puts
the two ends in the <em>document</em> rather than only in the tooling. Between
them, one pair of cells per step: what the step did, and a cell that loads that
step's artifact and prints one fact about it. That is the "usable by
themselves" property made executable, and the test suite runs those cells.
</p>
<p>
The <b>web output</b> is last, so nothing depends on it, so it can be replaced
wholesale without touching anything upstream. That is exactly what lets it rule
the output without being a stable contract — the book measure is the promise,
and the page displaying it is free to change drastically and often. It is also
the artifact somebody definitely opens, which is why <em>"2 of 47 files could
not be read"</em> has to appear there, above the diagram rather than below it.
</p>
<div class="note">
<b>The spine is scaffolding, not a gate.</b> Running one step alone is still a
book, just a short one — <code>make ir</code> works exactly as it did. An
operation that cannot measure something says what it could not measure and
carries on. Gating would destroy the property that makes the intermediate
artifacts useful, which is the whole reason the sequence is worth having.
</div>
<h3>What a book looks like on disk</h3>
<pre><code>book/&lt;slug&gt;/
├── book.json <span class="c">both measures, the steps, artifacts with byte counts</span>
├── steps/ <span class="c">every intermediate — ir.json, view.json, graph.svg, …</span>
├── notebook.ipynb <span class="c">the sequence; first and last cells are the measures</span>
├── overlay.json <span class="c">hand-written, optional, re-applied every build</span>
├── checks.py <span class="c">this book's own assertions — optional</span>
└── minisite/ <span class="c">the web output, both measures at the top</span></code></pre>
<pre><code>make book SRC=../station BOOK=out/book/station
make check BOOK=out/book/station</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="ir">The IR</h2>
<p>
Plain JSON. Three keys, and it has survived four domains without gaining a
fourth.
</p>
<pre><code>{
"meta": { "source": "python", "root": "app/", "schema_version": "1" },
"nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
"parent": "app.models",
"attrs": { "file": "app/models.py", "line": 12, "lines": 40 } } ],
"edges": [ { "source": "app.models.User", "target": "app.db.Base",
"kind": "inherits", "attrs": {} } ]
}</code></pre>
<table>
<tr><th>field</th><th>meaning</th></tr>
<tr><td><code>id</code></td>
<td>Fully qualified and <b>stable across runs</b>. Stability is what makes
two extractions from two commits diffable; without it a diff reports
noise and nobody trusts it.</td></tr>
<tr><td><code>kind</code></td>
<td>The hinge of the whole system, and the <b>only</b> field style and
layout may read. A small closed vocabulary per domain —
<code>module</code>/<code>class</code>/<code>function</code>,
<code>table</code>/<code>column</code>, <code>endpoint</code>,
<code>task</code>.</td></tr>
<tr><td><code>parent</code></td>
<td>Containment, and nothing else. A module contains a class. Relationships
are edges.</td></tr>
<tr><td><code>attrs</code></td>
<td>An open bag for whatever one domain cares about.
<code>file</code>/<code>line</code>/<code>lines</code> are what let a
box link to the line it came from, and what the minimap sizes by.</td></tr>
</table>
<div class="note">
<b>No visual information, ever.</b> If a field would change between a light and
a dark theme, it does not belong in the IR. <code>shape: "cylinder"</code> is
not a field — it is <code>kind: "datastore"</code> plus a style rule, and that
is exactly what lets the same IR render in a theme that has no cylinders. The
test suite sweeps every emitted document for colour-like keys.
</div>
<h3>Stdlib dataclasses, not Pydantic</h3>
<p>
The IR's whole value is being a plain document anything can open. A format that
needs a library installed to be read is an API, not a format. Validation is
therefore a function called at the boundary rather than a property of the type,
and it reads its field lists out of <code>schema.json</code> so the schema and
the dataclasses cannot drift apart.
</p>
<pre><code>python3 -m docgen validate ir.json</code></pre>
<p>
It catches what a schema cannot: an edge naming a node that does not exist, a
containment cycle, a duplicate id, and a visual field smuggled into
<code>attrs</code>.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="shape">Shape decides the drawing</h2>
<p>
A diagram that fights its layout engine is usually the wrong <em>kind</em> of
diagram. The clearest evidence: the same 24-table database rendered
<b>32034&times;136</b> through Graphviz — a 235:1 strip — and
<b>1740&times;1860</b> through the ER emitter. Not because one engine is
better, but because a schema is a set of peer entities with references, and
laying it out in dependency ranks was never its shape.
</p>
<p>
So <code>ops.classify()</code> reads the structure and names the emitter,
with the reason attached — advice without a reason gets overridden the first
time it is inconvenient.
</p>
<table>
<tr><th>kind</th><th>drawn by</th><th>when</th></tr>
<tr><td><code>erd</code></td><td>cards in columns</td><td>entities with references</td></tr>
<tr><td><code>pipeline</code></td><td>ranks, left to right</td><td>a chain with fan-out — an Airflow DAG, a build</td></tr>
<tr><td><code>layered</code> / <code>tree</code></td><td>ranks, top down</td><td>ranks genuinely suit it</td></tr>
<tr><td><code>sheet</code></td><td>the index</td><td>one level is wider than ~20 — a strip in any engine</td></tr>
<tr><td><code>flat</code></td><td>the index</td><td>most nodes have no relationships: that is a list</td></tr>
</table>
<pre><code>$ python3 -m docgen emit auto view.json -o out/
sheet -&gt; index
109 nodes sit at one level; any layered engine draws that as a
strip. Split it, scope it, or read it as an index</code></pre>
<h3>Why twenty</h3>
<p>
Measured, one diagram per subsystem: at or under 20 nodes the output lands
around 1.6:1; at 70–106 nodes about 7:1; at 261 nodes 14:1. Aspect ratio is a
property of the <em>graph</em>, not of the renderer — a layered engine puts one
dependency level in one row, so the widest level <em>is</em> the width.
</p>
<p>
Every Graphviz lever was tried before concluding this. <code>ratio=compress</code>
squashed a graph to an unreadable 1008&times;75; <code>rankdir=LR</code> merely
rotated a 14:1 into a 1:6; packing disconnected components gained nothing.
The fix was never a flag. It was to stop asking for one picture of everything —
which is what <a href="#explore">explore</a> does.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="extractors">Extractors</h2>
<p>
<strong>Deterministic parsing only. No model in the structural path.</strong>
A diagram built from an AST cannot be out of date with the code. A diagram
built from a model's reading of the code is wrong the moment the model has a
bad day — which is the problem this exists to fix.
</p>
<table>
<tr><th>reader</th><th>reads</th><th>gives</th></tr>
<tr><td><code>python</code></td><td>a tree of <code>.py</code></td>
<td>modules, classes, functions; <code>imports</code> and <code>inherits</code> edges</td></tr>
<tr><td><code>code</code> <span class="pill opt">tree-sitter</span></td><td>C#, TypeScript, TSX</td>
<td>namespaces, classes, interfaces, methods — <b>structure only</b></td></tr>
<tr><td><code>db</code></td><td>a graphgen-compatible <code>schema.json</code></td>
<td>tables, columns, foreign keys</td></tr>
<tr><td><code>openapi</code></td><td>an OpenAPI / Swagger document</td>
<td>endpoints and the shapes they carry</td></tr>
<tr><td><code>usage</code></td><td>a HAR recording</td>
<td>what was actually called, in what order</td></tr>
</table>
<h3>Two passes, because <code>ast</code> resolves nothing</h3>
<p>
Given <code>class User(Base)</code>, Python's <code>ast</code> hands over the
literal string <code>"Base"</code>. It has no idea that came from
<code>from .db import Base</code> three lines up. So pass one collects, per
module, what it defines and what it imports; pass two resolves local names to
fully qualified ids. The edge then points at <code>app.db.Base</code> — a real
node — rather than at a box called <code>Base</code> that means nothing.
</p>
<div class="note">
<b>Unresolved names become nodes, never nothing.</b> A third-party import or a
dynamically-built base becomes a node of <code>kind: "external"</code> and
<em>keeps its edge</em>. Dropping it would be the worse failure: the diagram
would look complete and have quietly lost a dependency. Gathered up, those
nodes are the project's real dependency surface.
</div>
<h3>C# and TypeScript</h3>
<p>
Handled by tree-sitter, which is why generics, nested types and a brace inside
a string are non-events rather than special cases. The test suite asserts that
last one specifically, because it is exactly where a hand-rolled scanner breaks.
</p>
<p>
This reader produces <strong>no edges</strong>. Resolving a C#
<code>using</code> to the thing it names is a different and much larger job,
and the consumer that needs this — the minimap — needs none of it. An
extractor that quietly produced half a dependency graph would be worse than one
producing none, because the half would look whole.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="usage">Usage, not just the spec</h2>
<p>
An OpenAPI document says what endpoints <em>are</em>. It does not say how to
use them — least of all when they are not RESTful, or when a GraphQL endpoint
sits alongside. So docgen also reads a <b>HAR</b>: the recording format that
browser devtools, mitmproxy, Charles and Insomnia all export.
</p>
<table>
<tr><th>what traffic knows</th><th>what a spec cannot</th></tr>
<tr><td>the <b>order</b> of calls</td><td>a spec is a set; usage is a sequence</td></tr>
<tr><td>which parameters are <b>always</b> sent</td><td>a spec lists twenty optional ones</td></tr>
<tr><td>which statuses <b>really</b> happen</td><td>the 422 everybody hits is in no document</td></tr>
<tr><td>endpoints not in the document</td><td>GraphQL operations, found by body shape and named</td></tr>
<tr><td>which id formats a route takes</td><td>numeric <em>and</em> uuid on one route</td></tr>
</table>
<div class="note warn">
<b>No credential and no payload value reaches the IR</b> — only field names and
types. A HAR is full of live bearer tokens and cookies, and a generated
document gets committed. The test suite plants a token in its fixture and fails
if it appears anywhere in the output.
</div>
<p>
Two limits, stated rather than glossed: path templating is a <em>guess</em>
(<code>attrs.observed_paths</code> keeps what was actually seen beside it), and
consecutive is not caused-by — the edge weight is what separates a habit from
an accident, and one recording will not tell you which.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="views">Views</h2>
<p>
The first real diagram out of this pipeline was a 3000px-wide strip: four
modules of actual content and sixty <code>sys</code>/<code>json</code>/<code>typing</code>
boxes as their peers. The emitter was correct and the picture was useless.
That is a <em>missing view</em>, not a broken renderer — and the fix belongs to
every consumer at once, because the index, the diagram and the diff all want
the same narrowing.
</p>
<pre><code>python3 -m docgen view ir.json --overview -o view.json
python3 -m docgen view ir.json --around docgen.ir --hops 2 -o view.json
python3 -m docgen view ir.json --split -o parts/
python3 -m docgen view ir.json --shape <span class="c"># what will this look like?</span></code></pre>
<p>
All of them are IR&rarr;IR, all composable, and each produces a document that
still validates. <code>--overview</code> is the default and dispatches on the
source: a codebase reduces to its modules and its outside dependencies, a
schema to its tables and their keys.
</p>
<div class="note">
<b>Edges are lifted when a view collapses detail, never dropped.</b> A class in
module A inheriting from a class in module B <em>is</em> a dependency of A on
B. Collapsing docgen to its packages once kept 8 of 77 edges — those pictures
were not simpler, they were <em>wrong</em>. Lifted edges carry a
<code>weight</code> saying how many they stand for.
</div>
<h3>Depth is the tempting knob and the wrong one</h3>
<p>
A directory without an <code>__init__.py</code> is not a package, so its
modules have no parent and sit at depth 0. soleprint has <b>173 such roots</b>,
and a depth-2 cut still held 566 functions and 142 classes. Selecting by
<code>kind</code> does not care how the directories happen to be arranged.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="emitters">Emitters</h2>
<table>
<tr><th>emitter</th><th>output</th><th>audience</th></tr>
<tr><td><code>index</code></td><td>markdown, sidebar JSON</td>
<td><b>anyone</b> — no graph literacy required</td></tr>
<tr><td><code>dot</code></td><td>DOT &rarr; Graphviz &rarr; SVG</td><td>dependency structure</td></tr>
<tr><td><code>erd</code></td><td>SVG, written directly</td><td>a schema, as cards</td></tr>
<tr><td><code>minimap</code></td><td>SVG, written directly</td><td>what is where, at a glance</td></tr>
<tr><td><code>notebook</code></td><td><code>.ipynb</code></td><td>a runnable walkthrough</td></tr>
<tr><td><code>minisite</code></td><td>a static docs site</td><td>reading</td></tr>
<tr><td><code>explore</code></td><td>a two-pane navigator</td><td>finding your way</td></tr>
<tr><td><code>auto</code></td><td>whichever of the above fits</td><td>not having to choose</td></tr>
</table>
<p>
<strong>The non-visual ones matter most for reach.</strong> A sorted, described
list of what exists is readable by someone who will never open a diagram, and
it also reports the dependency surface and any file that failed to parse.
It is built second, not last — it is what proves the IR is not secretly
diagram-shaped.
</p>
<h3>ERD — and where the layout came from</h3>
<p>
Not invented here. <code>station/tools/graphgen/templates/index.html</code>,
the Supabase-style schema explorer already in this repo, had solved it:
</p>
<pre><code>const cols = Math.max(2, Math.ceil(Math.sqrt(sorted.length * 1.2)));</code></pre>
<p>
<strong>Columns from the square root of the table count.</strong> The aspect
ratio is <em>chosen</em> rather than emergent, so the result stays near-square
at 4 tables or 400. That is the one thing a rank-based engine cannot offer.
Three more things it gets right: a table is a <em>card</em> with its columns;
an edge leaves the column holding the key and lands on the target's primary
key; and the geometry is computed rather than measured, so it renders
identically on any machine.
</p>
<figure>
<a href="viewer.html?src=img/erd.svg">
<img src="img/erd.svg" alt="an entity-relationship diagram">
</a>
<figcaption>A schema from the sample room. Same emitter, same style file as
every other diagram here.</figcaption>
</figure>
<h3>Minimap</h3>
<p>
Sublime's minimap shrinks the <em>characters</em>. This draws the
<em>structure</em> at full scale: one file is a column, one line is a fixed
number of pixels, every construct a block sized by its span and coloured by
what it is. No text inside a block — the shape is the message.
</p>
<p>
The claim is that the pattern comes from the colours alone, so nesting is drawn
by inset rather than by hue. On soleprint you can see that modelgen is
class-based, histgen is function-based and tester is mixed, without reading a
line.
</p>
<figure>
<a href="viewer.html?src=img/minimap.svg">
<img src="img/minimap.svg" alt="a structural minimap of docgen">
</a>
<figcaption>docgen's own files. Blue class, amber interface, green function,
dark for everything that is not a declaration — imports, constants, prose.</figcaption>
</figure>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="pages">Pages</h2>
<p>
A notebook doesn't scroll a wide image sideways: it scales it down to the cell
width until the labels can't be read. Zooming and panning is a workaround for a
picture that holds too much. So a book's diagram is split into <b>pages</b>,
each drawn small enough to fit the column it's going into.
</p>
<pre><code>overview one box per top-level group, relationships lifted
└── station one box per child of station
└── station.tools one box per child of station.tools, and so on</code></pre>
<p>
Pages follow structure, never scissors. Slicing one large image cuts boxes and
edges at the seams, and a slice means nothing on its own. A page is a subsystem:
its boxes are the scope's children, and anything outside the scope that they
touch appears as a <em>context</em> box, collapsed to the branch it sits on. A
page still says what its subsystem reaches without drawing the rest of the tree.
</p>
<h3>Fitted by measuring, not estimating</h3>
<p>
Whether a page fits depends on its labels, fonts and spacing, so an estimate from
node counts is wrong the moment the names are long. Each page is rendered and its
width read back. If a page is too wide top-to-bottom, it's drawn left-to-right,
since height is free in a notebook and width isn't. If it still doesn't fit
(a level of many peers with no smaller structure to split into), the page is
kept and flagged <em>too wide</em> in its caption and in the index. Never cut,
never dropped.
</p>
<p>
Measured on soleprint, whose module graph rendered at 14:1 as one picture: 69
pages, and at the minisite's 800px column the widest flagged page is 988px.
Schemas page differently: cards in columns capped to the width, grouped by
tables joined by foreign keys, so a schema grows downward instead of sideways.
</p>
<div class="note">
<b>Which pages exist doesn't depend on width</b>, only how each is laid out
does. A notebook column and a minisite column get the same pages, which is what lets
a <a href="#notes">note</a> filed under a page land in the same place in both.
</div>
<h3>Clicking through, on the minisite</h3>
<p>
Every page is its own HTML file with the diagram <b>inline</b>, not in an
<code>&lt;img&gt;</code>, which swallows clicks. A box standing for a subsystem
opens that subsystem's page. A module box opens its section of the book's
reference page. A context box opens its own branch. Breadcrumbs lead back up.
The size is adjustable: <b>fit</b> scales to the column, <b>1:1</b> shows the
drawing at natural size with scrollbars, the choice is remembered, and
<b>full screen</b> opens the pan-and-zoom viewer.
</p>
<p>
A notebook can't follow a link inside an image, so it gets a <b>page index</b>
instead: every page, indented by depth, each a link to its figure.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="explore">Explore</h2>
<p>
The minimap on its own shows shape and no meaning: a block says "a 30-line
class", not <em>which</em> class or what it touches. So it is not the artifact.
It is the <strong>selector</strong>.
</p>
<pre><code>make explore OUT=out <span class="c"># then open out/explore/explore.html</span></code></pre>
<div class="cols">
<div>
<h4>Left — navigate</h4>
<p>The whole thing at once. Scan by colour, click a block.</p>
</div>
<div>
<h4>Right — explore</h4>
<p>What that is, what it reaches, what reaches it, and the neighbourhood
drawn small enough to read. Every neighbour is a link, so you walk
outward from wherever you started.</p>
</div>
</div>
<div class="note">
<b>This is what retires the 14:1 sheet.</b> The whole graph is never drawn. The
overview pane carries the overview, and only the neighbourhood of a selection
is rendered — a handful of nodes, which lays out fine every time. Overview and
detail stop competing for one picture.
</div>
<p>
The same split applies to a database: every table at once with <em>no column
detail</em>, then click one to get its columns plus the tables its keys reach.
The two differ exactly where they should — a module's neighbourhood
deliberately leaves its contents out, because those are the hundred functions
that made the sheet unreadable, while a table's brings them in, because a table
without its columns is not a table.
</p>
<h3>The selection basket</h3>
<p>
Shift-click accumulates blocks. The basket is a copyable list of paths with a
line count — enough to hand to <code>distill</code>, and enough to see that the
selection got too big <em>before</em> spending the context on it. Navigating a
tree quickly in order to decide what to feed a model is a real use, and this is
the part that serves it.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="notebooks">Notebooks</h2>
<p>
A notebook is normally a source file somebody confects by hand: prose, code and
stored output braided together, diffing badly, drifting from whatever it
documents the moment either moves, with no way to tell by looking.
</p>
<p>
Here a notebook is a <strong>build artifact</strong>. The source is the OpenAPI
document — the same file the server is built from — and the notebook is
regenerated from it. Nobody edits the <code>.ipynb</code>, the same way nobody
edits a <code>.o</code>. "Is this document current" stops being a question
about somebody's diligence and becomes a question about whether the build ran.
</p>
<blockquote>
This is the disagreement with jupytext. Jupytext fixes the <em>diffing</em> —
it makes a notebook editable as text — and leaves the actual problem: you still
hand-author it, so it still rots.
</blockquote>
<h3>Generated base, hand-written overlay</h3>
<p>
Generation alone gives a document that is never stale and never says anything a
parser could not work out. Hand-authoring alone gives insight and a document
that rots. Two files is the only arrangement that gets both:
</p>
<pre><code>IR ──► spec ──(+ overlay)──► merged spec ──► .ipynb
generated hand-written merged emitted</code></pre>
<p>
The <b>spec</b> is an ordered list of steps with no Jupyter in it — a Swagger
for notebooks, readable and diffable. The <b>overlay</b> is the only file
anyone edits, and it is re-applied on every build. It can
<code>annotate</code>, <code>replace</code>, <code>insert</code>,
<code>drop</code> and <code>order</code>.
</p>
<p>
<code>replace</code> is the one that matters. It is how real usage gets into a
document that a spec could not describe — the call that is always made with
<code>status=available</code>, the GraphQL endpoint that is not in the OpenAPI
file at all — and it keeps working unchanged once a usage recording supplies
the same facts automatically.
</p>
<div class="note">
Three properties hold it together: regenerating <b>re-applies the overlay
byte-for-byte</b>; when the base moves underneath it the mismatch is
<b>reported, never silently dropped</b>; and extraction works with the overlay
<b>absent</b> — it is an addition, never a dependency.
</div>
<pre><code>python3 -m docgen emit notebook ir.json --scaffold overlay.json
python3 -m docgen emit notebook ir.json --overlay overlay.json -o walkthrough.ipynb</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="notes">Notes</h2>
<p>
Generation alone gives a document that's never stale and never says anything a
parser couldn't work out. Prose is the part only a person adds, and it has to
survive every rebuild, so it can't live in the output. It lives in a
<b>notes folder</b>, one markdown file per spot, named after the page or step it
belongs above:
</p>
<pre><code>notes/
├── _intro.md top of the document
├── _end.md bottom of the document
└── &lt;book&gt;/
├── _intro.md top of that book's part
├── larder.md above "what came in"
├── overview.md above the overview page
└── station.tools.md above the page for station.tools</code></pre>
<p>
To put text above a page, create the file named after it. No schema, no JSON,
no ids to look up. That's something a person or a model can do without reading
anything first. Local images in a note are embedded: as a cell attachment in the
notebook, as a data URI on the minisite.
</p>
<pre><code>make scaffold CONFIG=docgen.toml <span class="c"># an empty stub for every spot, never overwriting</span></code></pre>
<p>
<code>scaffold</code> lists the spots rather than making you guess them. A stub
is a single HTML comment and renders nothing, so scaffolding a hundred spots
changes the output not at all until one is filled in. Run it again after the
sources change and it only adds the new spots.
</p>
<div class="note warn">
<b>A note whose page is gone is reported, not dropped.</b> Renaming a package
renames its page, and the note filed under the old name has nowhere to land.
Discarding it quietly is how hand-written text goes stale without anyone
noticing, so it appears in the document ledger and fails the check level. It's
the same rule the notebook overlay follows.
</div>
<p>
The <a href="#notebooks">overlay</a> stays for what notes can't do: drop a
step, reorder, replace generated code. Notes are applied after the overlay, so a
step that moves takes its note with it.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="style">Style &amp; colour</h2>
<p>
A style rule names a <strong>slot</strong>, never a colour.
<code>"border": "atlas"</code> is the rule; a theme binds <code>atlas</code> to
<code>#43A047</code> in print and <code>#15803d</code> on the docs site.
</p>
<p>
That indirection is the whole point. <code>common/theme/tokens.css</code>,
<code>docs/graphs/themes/*.gvpr</code> and <code>style/lucid.json</code> use
the same slot names, so a diagram and the page around it match by construction
— which is the rule <code>docs/graphs/README.md</code> already states. The dark
theme's <code>artery</code>, <code>atlas</code> and <code>station</code> slots
are exactly the <code>--system-accent</code> values the three system pages set,
and the test suite fails if they drift apart.
</p>
<p>
Dark is the default, because a generated diagram lands in a dark docs page far
more often than in a document. <code>--theme lucid</code> gives the print
palette — and gives it to the <em>page</em> as well as the diagram, since both
are baked from the same slots.
</p>
<div class="note">
An unknown <code>kind</code> falls back to <code>default</code> rather than
crashing. That matters more than it sounds: a new extractor with a new
vocabulary renders plainly and legibly on day one, instead of requiring
somebody to write a style file before they can see anything.
</div>
<h3>Where DOT stops</h3>
<p>
The emitter writes what DOT expresses natively and stops at the boundary rather
than growing machinery. The limits are recorded in the style file itself: a
cluster has a label and a fill but not a header bar;
<code>stroke-dasharray</code> is not parameterised, so <code>4,4</code> and
<code>5,5</code> collapse; <code>rounded</code> is binary, so 4px and 6px are
identical. Those mark where a richer emitter would begin — and the style file
carries the full specification regardless, so that emitter needs no
re-authoring.
</p>
<p>
One limit <em>was</em> worth solving: DOT cannot use a cluster as an edge
endpoint, so every module-to-module import silently vanished. The native answer
is <code>compound=true</code> with <code>lhead</code>/<code>ltail</code> — draw
between a representative leaf and clip the line at the cluster border.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="standalone">Standalone</h2>
<p>
Copy the <code>docgen/</code> folder anywhere and it works. The Makefile
derives its own package name from where it sits, so it can be renamed too, and
everything else resolves inside the directory.
</p>
<pre><code>cp -r docgen /somewhere/else
cd /somewhere/else/docgen
make doctor <span class="c"># what this machine has</span>
make sync <span class="c"># optional: every group, from uv.lock</span>
make check <span class="c"># the suite, from the copy</span>
make book SRC=/path/to/any/repo BOOK=out/book/theirs</code></pre>
<h3>One seam, and it is optional</h3>
<p>
Exactly one capability needs more than the folder: reading an <b>OpenAPI</b>
document goes through <code>station/tools/modelgen</code>, which parses the
spec and resolves <code>$ref</code>. It is deliberately not reimplemented here
— a second OpenAPI reader in one repo is two things to keep correct.
</p>
<p>
So if you are using docgen standalone but keeping the repo alongside as
reference, point at it:
</p>
<pre><code>export DOCGEN_REFERENCE=/path/to/repo
make doctor
<span class="c"># reference: /path/to/repo (from $DOCGEN_REFERENCE)</span></code></pre>
<p>
Resolution is <code>$DOCGEN_REFERENCE</code> first, then walking up from the
package — so in place it needs no configuration, and an explicit path wins
when set. Without it, the four other extractors and every emitter work
unchanged; the OpenAPI reader reports what to set, and the suite
<em>skips</em> rather than fails.
</p>
<div class="note">
<b>An env var rather than a config file</b>, because there is one setting and
it is a path. A config file for one path is a file to find, parse, document and
validate, and the first question anyone asks of it is "where does it live" —
which is the same question again.
</div>
<h3>It is asserted, not asserted-in-prose</h3>
<p>
A standalone claim decays the moment somebody adds a convenient import, and it
decays <em>silently</em>, because the suite still passes inside the repo. So
the suite reads its own source:
</p>
<ul>
<li><b>Imports are the stdlib, docgen itself, and a short allow-list</b> —
tree-sitter, lxml, yaml, networkx, modelgen, each optional and each with a
reason. A new name is a new dependency in a folder meant to be copied.</li>
<li><b>Only <code>extractors/openapi.py</code> imports modelgen.</b> One seam
is a seam; two is a dependency.</li>
<li><b>Only <code>reference.py</code> knows the repo layout</b>, so pointing
docgen elsewhere is one change rather than a search.</li>
<li><b>A book writes only inside its own output directory</b> — nothing is
left in the tree being read.</li>
</ul>
<p>
The folder was also literally copied to <code>/tmp</code> and run, which is
how the one real bug here was found: a check asserting the reference repo is
reachable, correct in place and wrong the moment there was nothing above. It
now reports which case applies instead of assuming one.
</p>
<table>
<tr><th>context</th><th>checks</th><th>skipped</th></tr>
<tr><td>in the repo, synced</td><td>350</td><td>—</td></tr>
<tr><td>copied out, bare <code>python3</code></td><td>333</td><td>tree-sitter, OpenAPI, the in-place case</td></tr>
<tr><td>copied out, synced</td><td>342</td><td>OpenAPI, the in-place case</td></tr>
<tr><td>copied out, synced, <code>DOCGEN_REFERENCE</code> set</td><td>349</td>
<td>the in-place case — correctly, there is no repo above</td></tr>
</table>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="runs">Run files</h2>
<p>
A book gets rebuilt many times — after a merge, before a release, whenever the
source moves — and each one is a source, an output directory and a handful of
options. Typed out every time, those drift: one run excludes
<code>migrations</code>, the next forgets. A <b>run file</b> writes them down
once, so the rebuild is exact.
</p>
<pre><code><span class="c"># docgen.toml — kept beside the project it describes</span>
reference = "../soleprint" <span class="c"># optional; $DOCGEN_REFERENCE wins</span>
[defaults]
out = "out/book" <span class="c"># each book goes to &lt;out&gt;/&lt;name&gt;</span>
theme = "dark"
exclude = ["migrations", "tests"]
[[book]]
name = "station"
root = "../soleprint/station"
[[book]]
name = "frontend"
root = "../web/src"
reader = "code" <span class="c"># C# / TypeScript, through tree-sitter</span>
[[book]]
name = "orders-api"
openapi = "specs/orders.yaml"
overlay = "overlays/orders.json"
out = "/srv/docs/orders"</code></pre>
<pre><code>make run CONFIG=docgen.toml
make run CONFIG=docgen.toml ONLY=station,frontend CHECK=1
python3 -m docgen run docgen.toml --list <span class="c"># resolved paths, nothing built</span></code></pre>
<pre><code>docgen.toml — 3 book(s)
ok station /home/me/proj/out/book/station
ok frontend /home/me/proj/out/book/frontend
FAIL orders-api ImportError: modelgen is not reachable — …
1 of 3 book(s) did not hold — --verbose for each book's own output</code></pre>
<p>
Each book names <b>exactly one</b> source — <code>root</code>,
<code>schema</code>, <code>openapi</code> or <code>har</code> — the same choice
the <code>book</code> command makes you take. One failing book never stops the
rest, and the run exits 1 if any book failed to build, lost input between its
two ends, or failed its checks. <code>docgen.example.toml</code> ships with
docgen and runs against its own fixtures.
</p>
<h3>The rules, each one a thing that goes wrong otherwise</h3>
<ul>
<li><b>Paths are relative to the run file</b>, not to wherever the command ran.
Otherwise the same file builds different books from different
directories.</li>
<li><b>A book's value replaces the default, for every key</b> — including
<code>exclude</code>, where merging looks tempting. One rule nobody has to
remember beats a clever one somebody has to look up. A default
<code>exclude</code> simply does not apply to a schema or a spec.</li>
<li><b>Unknown keys are refused, not ignored</b>, all of them at once.
<code>exlude = [...]</code> silently doing nothing is how a rebuild quietly
starts reading <code>node_modules</code>.</li>
<li><b>A book may not be written inside the tree it reads</b> — the next run
would read the last run's output — and <b>two books may not share an output
directory</b>, where the second would clear the first every time.</li>
<li><b>The environment beats the file.</b> A <code>reference</code> in the run
file is used only when <code>$DOCGEN_REFERENCE</code> is not set, which is
rig's precedence for every setting it has.</li>
</ul>
<div class="note">
<b>A rebuild replaces the previous build, and nothing else.</b> Before writing,
a book removes exactly what a build writes — <code>book.json</code>,
<code>notebook.ipynb</code>, <code>steps/</code>, <code>minisite/</code>,
<code>explore/</code> — and only from a directory that already holds a
<code>book.json</code>. Without it a graph that is now <code>graph.md</code> kept
its old <code>graph.svg</code>, unlisted in the ledger and still on the page.
A hand-written <code>checks.py</code> or <code>overlay.json</code> is never
touched, and neither is a directory docgen did not write.
</div>
<p>
TOML because a run file is written by people: <code>tomllib</code> is the
stdlib, it takes comments — the reason a book excludes something belongs next
to the exclusion — and it is already the format of the
<code>pyproject.toml</code> beside it. The IR and the style files stay JSON,
because those are data that tools write.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="projects">Project files</h2>
<p>
A run file lists independent books. A <b>project file</b> describes one
<em>document</em> built from several sources: an API contract and the code
behind it, a schema and the service that uses it. It's the shape a model wrote
unprompted in a dry run, so docgen reads it as written:
</p>
<pre><code><span class="c"># &lt;project&gt;/docgen.toml</span>
[project]
name = "orders-api"
title = "Orders API"
notes = "notes" <span class="c"># default</span>
build = ".docgen" <span class="c"># book artifacts, default; gitignore it</span>
[extract.openapi]
spec = "doc/contracts/openapi.yaml"
[extract.code]
root = "src"
ext = [".cs"]
[ops]
drop_stdlib = true
drop_builtins = true
[emit.notebook]
output = "../docs_out/orders_api_notebook"
theme = "lucid"
[emit.minisite]
output = "../docs_out/orders_api_minisite"
width = 800</code></pre>
<p>
Each <code>[extract.*]</code> table is a <b>book</b>, and a book is a part of
every output rather than an output of its own. Each <code>[emit.*]</code> table
is an output made of all of them: one notebook and one minisite, each with a part
per book. Several sources of one kind use <code>[[extract.code]]</code> with a
<code>name</code> each. <code>[ops]</code> applies to every book's view before
paging.
</p>
<p>
Every part still opens with what came in and closes with what came out, and the
document opens with every larder and closes with every reconciliation. A book
that can't be built doesn't cost the others: the outputs are assembled from
what was built, and both say <em>this document is incomplete</em> and name what's
missing.
</p>
<h3>Refused, with what to do instead</h3>
<table>
<tr><th>written</th><th>why not</th></tr>
<tr><td><code>[extract.db] url = …</code></td>
<td>docgen reads a schema dump and never connects, and a URL in a committed
file is a credential in a committed file. Write the schema with
<code>modelgen from-db</code>, then <code>schema = …</code>. The URL is never
repeated in the error.</td></tr>
<tr><td><code>ext = [".sh", ".yaml", ".sql"]</code></td>
<td>No grammar for those yet. The error names the extensions that are read.</td></tr>
<tr><td><code>[extract.seed]</code></td>
<td>No such reader. The error names the ones that exist.</td></tr>
<tr><td><code>[project]</code> and <code>[[book]]</code> in one file</td>
<td>Which one it meant is exactly what nobody should have to guess.</td></tr>
</table>
<p>
<code>docgen.project.example.toml</code> ships beside the run file example and
builds docgen's own fixtures into one document, with a sample note.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="commands">Commands</h2>
<h3>Make</h3>
<table>
<tr><th>target</th><th>does</th></tr>
<tr><td><code>make sync</code></td><td><code>.venv</code> with every optional group, from <code>uv.lock</code></td></tr>
<tr><td><code>make book SRC=…</code></td><td><b>one whole operation</b>, measured at both ends</td></tr>
<tr><td><code>make run CONFIG=…</code></td><td>every book a <a href="#runs">run file</a> lists, or a <a href="#projects">project</a>'s document</td></tr>
<tr><td><code>make scaffold CONFIG=…</code></td><td>an empty <a href="#notes">note</a> for every page and step</td></tr>
<tr><td><code>python3 -m docgen contract</code></td><td>the <a href="#models">instructions for a model</a> that writes the config and notes</td></tr>
<tr><td><code>make check</code></td><td>docgen's own suite, offline, nothing installed</td></tr>
<tr><td><code>make check BOOK=…</code></td><td>one book's own level — generated and custom</td></tr>
<tr><td><code>make doctor</code></td><td>what this machine has and what it is missing</td></tr>
<tr><td><code>make ir SRC=…</code></td><td>extract Python into <code>OUT/ir.json</code></td></tr>
<tr><td><code>make code SRC=…</code></td><td>extract C#/TypeScript <span class="pill opt">tree-sitter</span></td></tr>
<tr><td><code>make db SCHEMA=…</code></td><td>extract a database schema</td></tr>
<tr><td><code>make view</code></td><td>the default view for that source type</td></tr>
<tr><td><code>make graph</code></td><td>draw whatever the structure asks for</td></tr>
<tr><td><code>make index</code></td><td>markdown index and sidebar JSON</td></tr>
<tr><td><code>make minimap</code></td><td>what is where, read from the colours</td></tr>
<tr><td><code>make explore</code></td><td>the two-pane navigator</td></tr>
<tr><td><code>make minisite</code></td><td>a self-contained one-page site</td></tr>
<tr><td><code>make self</code></td><td>docgen's book of soleprint, then check it</td></tr>
</table>
<p>
Variables: <code>SRC</code>, <code>OUT</code>, <code>SCHEMA</code>,
<code>OPENAPI</code>, <code>HAR</code>, <code>BOOK</code>, <code>SLUG</code>,
<code>READER</code>, <code>OVERLAY</code>, <code>CONFIG</code>,
<code>ONLY</code>, <code>CHECK</code>, <code>STYLE</code>, <code>THEME</code>,
<code>SCALE</code>, <code>PY</code>. <code>PY</code> is the synced
<code>.venv</code> when there is one and <code>python3</code> when there is not.
The Makefile derives its own package name from where it sits, so the folder can
be copied anywhere and renamed and still work.
</p>
<p>
<code>READER</code> rather than <code>LANG</code> because <code>LANG</code> is
the shell's locale variable, so <code>?=</code> inherits
<code>en_US.UTF-8</code> from the environment and the argument is rejected.
Every target above is one step of a book and still works alone — that is the
property the spine exists to preserve, not to replace.
</p>
<h3>The command line</h3>
<p>
One entry point, <code>python3 -m docgen &lt;command&gt;</code>, run from the
directory <em>above</em> the folder (or with it on <code>PYTHONPATH</code>, which
is what the Makefile does). Each command is plain argparse and takes
<code>--help</code>.
</p>
<pre><code>python3 -m docgen book --root SRC -o out/book/slug <span class="c"># the whole operation</span>
python3 -m docgen run docgen.toml <span class="c"># every book a run file lists</span>
python3 -m docgen check <span class="c"># docgen's own suite</span>
python3 -m docgen check out/book/slug <span class="c"># that book's level</span>
python3 -m docgen check .docgen/&lt;project&gt; <span class="c"># a project document's level</span>
python3 -m docgen scaffold docgen.toml <span class="c"># stub notes for every spot</span>
python3 -m docgen scaffold docgen.toml --list <span class="c"># the spots, written nowhere</span>
python3 -m docgen contract <span class="c"># the instructions for a model</span>
python3 -m docgen extract python --root SRC -o ir.json
python3 -m docgen extract code --root SRC -o ir.json
python3 -m docgen extract db --schema schema.json -o ir.json
python3 -m docgen extract openapi --spec spec.yaml -o ir.json
python3 -m docgen extract usage --har session.har -o ir.json
python3 -m docgen validate ir.json
python3 -m docgen view ir.json --overview -o view.json
python3 -m docgen emit auto view.json -o out/
python3 -m docgen emit index ir.json -o index.md
python3 -m docgen emit dot view.json -o graph.svg --theme lucid
python3 -m docgen emit erd ir.json -o schema.svg
python3 -m docgen emit minimap ir.json -o map.svg --scale 0.5
python3 -m docgen emit notebook ir.json -o book.ipynb --overlay overlay.json
python3 -m docgen emit minisite view.json -o minisite/
python3 -m docgen emit explore ir.json -o explore/</code></pre>
<h3>Where the command line lives</h3>
<p>
All of it in <code>cli/</code>, and nowhere else. <code>extractors/</code>,
<code>ops/</code>, <code>emitters/</code> and <code>book/</code> hold library
code only — no argparse, no <code>__main__.py</code>, no
<code>cli_dot.py</code> sitting next to <code>dot.py</code>. Two kinds of file
in one folder make every folder answer two questions, and the command-line half
is the half that grows copies: eight <code>cli_*</code> files each
re-implemented "read the IR, validate it, load the style" by hand, and had begun
to word their errors differently.
</p>
<pre><code>cli/
├── __init__.py <span class="c">the command table and the one exit path</span>
├── common.py <span class="c">read and validate an IR, load a style, write or print</span>
├── book.py run.py check.py
└── extract.py validate.py view.py emit.py</code></pre>
<p>
A command reports an expected failure by raising, and <code>cli.main</code>
turns it into the only exit path: one <code>Error:</code> line on stderr, exit 1,
never a traceback. Doing that split also surfaced a third copy of "which drawing
does this graph want" — in the <code>auto</code> command, the <code>minisite</code>
command and the book — which is now <code>emitters.auto.draw()</code>, returning
content and leaving the file name to the caller.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="deps">Dependencies</h2>
<p>
The core is <strong>standard library only</strong>, Python 3.11 or later —
<code>pyproject.toml</code> declares no required dependency at all. Everything
else is an optional <b>dependency group</b>, pinned in a committed
<code>uv.lock</code>, and reported by <code>make doctor</code>; when something is
missing you lose exactly one capability and get told what to install. See
<a href="#standalone">standalone</a> for the one seam out of the folder.
</p>
<pre><code>make sync <span class="c"># uv sync --all-groups → .venv/</span>
uv sync --group code <span class="c"># or just the one you need</span>
make lock <span class="c"># after editing pyproject.toml</span></code></pre>
<p>
<code>[tool.uv] package = false</code>, the same choice
<code>station/tools/dataconvert</code> makes: this is a folder of modules run in
place, not something to install. The lock pins what the optional groups resolve
to; the folder still runs on a bare <code>python3</code> without it.
</p>
<table>
<tr><th>needs</th><th>for</th><th>without it</th></tr>
<tr><td><code>graphviz</code> (binary)</td><td>rendering DOT to SVG</td>
<td>ERD, minimap, index and notebooks still work</td></tr>
<tr><td>group <code>code</code> — tree-sitter + grammars</td><td>C#, TypeScript, TSX</td>
<td>Python only</td></tr>
<tr><td>group <code>openapi</code> — pyyaml</td><td>reading OpenAPI (with the reference repo)</td>
<td>the other four sources</td></tr>
<tr><td>group <code>harvest</code> — lxml</td><td>a theme from exported diagrams</td>
<td>the shipped themes</td></tr>
<tr><td>group <code>lab</code> — networkx</td><td><code>lab/</code> experiments</td>
<td>nothing — nothing depends on it yet</td></tr>
<tr><td><code>node</code></td><td>testing the browser pages</td>
<td>those checks skip</td></tr>
<tr><td><code>psql</code></td><td>the <code>lab/</code> schema probe</td>
<td>use modelgen's <code>from-db</code> instead</td></tr>
</table>
<div class="note">
<b><code>lab/</code> is where a dependency gets tried before anything depends
on it.</b> Nothing in <code>ir/</code>, <code>extractors/</code>,
<code>ops/</code> or <code>emitters/</code> may import from it. When an
experiment earns its place it graduates into <code>ops/</code> behind an
IR&rarr;IR signature, and <em>then</em> the dependency is declared.
</div>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="testing">Testing</h2>
<p>
Three levels, and they differ by <em>what they assert about</em>. The
distinction decides what a failure means, which is why it is worth keeping:
</p>
<table>
<tr><th>command</th><th>asks about</th><th>fails?</th></tr>
<tr><td><code>make doctor</code></td><td>the machine — what is installed</td>
<td>never; it reports</td></tr>
<tr><td><code>make check</code></td><td>docgen — 350 checks</td>
<td>exit 1</td></tr>
<tr><td><code>make check BOOK=&lt;dir&gt;</code></td><td>that one book, or a project's document</td>
<td>exit 1</td></tr>
</table>
<p>
All 350 after <code>make sync</code>, 335 with nothing optional installed — the
suite skips rather than fails when tree-sitter, lxml or the OpenAPI reader is
absent. See <a href="#standalone">standalone</a> for the counts outside the repo.
Among them, two that keep this page's structure true: nothing outside
<code>cli/</code> imports argparse, and every import the suite allows is a group
in <code>pyproject.toml</code> and pinned in <code>uv.lock</code>.
</p>
<h3>The book level, where custom checks live</h3>
<p>
The third level is the one that reaches a project docgen has never seen, and it
is where <b>framework and hand-written checks live together</b>. The generated
half is the spine's own assertions, identical for every book: both measures
present, the reconciliation holding, every artifact where the ledger says it
is, the notebook still executing. The custom half goes in the book's own
<code>checks.py</code> and uses the same helpers, so a project's line and a
framework line read identically and fail identically.
</p>
<pre><code><span class="c"># out/book/station/checks.py</span>
<span class="k">def</span> checks(book, check, note, skip):
note("what this project will not give up")
<span class="c"># Payments moved once already and the move broke three dashboards.</span>
<span class="c"># If it is not here, something renamed it again.</span>
check("payments is still a module", True,
any(n["id"] == "app.payments" <span class="k">for</span> n <span class="k">in</span> book.ir["nodes"]))</code></pre>
<p>
Same split as the notebook's base and overlay, for the same reason: generation
alone cannot know what <em>this</em> project cares about, and hand-authoring
alone rots.
</p>
<div class="note">
<b>Each check is one decision that has already been made, with the reason above
it.</b> Not coverage, and deliberately not an exhaustive sweep. A rule without
its reason gets overridden the first time it is inconvenient, so failing a
check should read as <em>"you are about to undo this"</em> rather than
<em>"something broke"</em>. The idiom is carried from rig's
<code>ctrl/selftest.sh</code>, which is where the three-level split comes from.
</div>
<h3>The four that are the design</h3>
<p>
Of docgen's own checks, four assert the <em>architecture</em> rather than guard
a regression, and they are the ones to keep if anything is ever cut:
</p>
<ul>
<li><b>No visual field reaches the IR</b> — extractors cannot decide appearance.</li>
<li><b>No emitter reads a source file</b> — the layering, checked from the other side by parsing imports.</li>
<li><b>Style names slots, not colours</b> — one colour language rather than three.</li>
<li><b>Ids are stable across runs</b> — without it, diffing is noise.</li>
</ul>
<div class="note">
<b>Golden tests go on the IR, never on the SVG.</b> Graphviz measures label
text with the host's fonts to size nodes, so identical input produces different
geometry on a machine with different fontconfig. The IR is deterministic; the
SVG is not. Pinning the wrong one gives a suite that fails on somebody else's
laptop for no reason anyone can act on.
</div>
<p>
The browser pages are JavaScript, so they are tested as JavaScript: a stub DOM
under <code>node</code> drives the viewer's zoom and 1:1 toggle, and the
explorer's select-and-walk. Both skip cleanly where node is absent.
</p>
<p>
Self-hosting is the honest end-to-end check, and it is where the real bugs came
from — two name-resolution faults and a duplicate-id crash that no fixture had
reached. <code>make self</code> builds docgen's book <em>of soleprint</em> and
then runs that book's own level against it, so the two ends have to reconcile
on 225 real files. If the index does not read like the system, something is
wrong.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="limits">Limits &amp; non-goals</h2>
<p>Things deliberately not done, with the reason, so they are not re-litigated:</p>
<ul>
<li><b>No layout engine.</b> No positioning, no <code>neato -n2</code>, no ELK.
Aspect ratio was solved by choosing the right emitter and by not drawing
everything at once.</li>
<li><b>No <code>calls</code> edges.</b> Resolving <code>self.foo()</code> needs
type inference, and a call graph that is quietly 60% right is worse than
none because it reads as authoritative.</li>
<li><b>No edges from the C#/TypeScript reader.</b> Structure only — half a
dependency graph would look whole.</li>
<li><b>No model in the structural path.</b> Annotation — summarising a module,
naming a cluster — is a later layer, cached to its own file keyed by node
id, merged at emit time, and extraction must work with it absent.</li>
<li><b>No configuration knobs</b> until two real consumers disagree.</li>
</ul>
<p>Known gaps, stated plainly:</p>
<ul>
<li>The C# reader is verified against a written fixture, <b>not a real
repository</b>. That is the next check that matters.</li>
<li>The diff emitter is not built. Stable ids exist to make it possible, and
two IRs from two commits is the cheapest useful thing left.</li>
<li>Minimap blocks carry no names. Rendering them legible only at 1:1, through
the viewer, is the other half of reading a file without reading every line.</li>
</ul>
</main>
</div>
<script>
// Highlight the section being read. No dependency, no build step.
var links = [].slice.call(document.querySelectorAll('.sidebar a[href^="#"]'));
var byId = {};
links.forEach(function (a) { byId[a.getAttribute('href').slice(1)] = a; });
var observer = new IntersectionObserver(function (entries) {
entries.forEach(function (entry) {
var a = byId[entry.target.id];
if (a && entry.isIntersecting) {
links.forEach(function (l) { l.classList.remove('active'); });
a.classList.add('active');
}
});
}, { rootMargin: '-8% 0px -82% 0px' });
document.querySelectorAll('h2[id]').forEach(function (h) { observer.observe(h); });
</script>
</body>
</html>