docgen: drop the superseded first pass, port the style harvester

The IR supersedes both intermediate designs (requirements D6, D7):
station/tools/docgen and the graph model that briefly lived in graphgen.
Shipping them beside atlas2/docgen would mean two graph models, which is the
thing the architecture argues against.

Carried across rather than lost:
  - style/extract.py and tokens.py, the offline theme harvester (R31). Rewritten
    to emit a *theme* — a slot-to-hex binding — rather than a whole style file,
    since what harvesting recovers is which colour a slot should be, not what a
    kind should look like.
  - graphgen/README.md, rewritten for what graphgen actually is now: the
    schema explorer. It fixes the blank station-index entry at run.py:304.

Two bugs found while doing it:
  - Canvas and ink are the two lightness extremes, not the two most common
    values. In a Graphviz SVG every label carries a fill, so the ink outnumbers
    the canvas 87 to 43 and the old rule produced a theme whose text was
    invisible against its own background.
  - A name defined in both branches of an if/else produced a duplicate id, which
    failed validation on docgen's own source. Disambiguated by line.
This commit is contained in:
2026-09-13 21:41:29 -03:00
parent 358b98f826
commit 160ee31b8c
51 changed files with 4504 additions and 2906 deletions

View File

@@ -0,0 +1,793 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>docgen</title>
<link rel="stylesheet" href="docs.css">
</head>
<body>
<div class="layout">
<nav class="sidebar">
<div class="sidebar-header">
<b>docgen</b>
<small>code &amp; data &rarr; documentation</small>
</div>
<nav>
<div class="group">Start</div>
<a href="#what">What it is</a>
<a href="#idea">The idea</a>
<a href="#quick">Five minutes</a>
<div class="group">Architecture</div>
<a href="#layers">The three concerns</a>
<a href="#ir">The IR</a>
<a href="#shape">Shape decides the drawing</a>
<div class="group">Reading</div>
<a href="#extractors">Extractors</a>
<a href="#usage">Usage, not just the spec</a>
<div class="group">Narrowing</div>
<a href="#views">Views</a>
<div class="group">Writing</div>
<a href="#emitters">Emitters</a>
<a href="#explore">Explore</a>
<a href="#notebooks">Notebooks</a>
<a href="#style">Style &amp; colour</a>
<div class="group">Reference</div>
<a href="#commands">Commands</a>
<a href="#deps">Dependencies</a>
<a href="#testing">Testing</a>
<a href="#limits">Limits &amp; non-goals</a>
</nav>
</nav>
<main class="content">
<h1>docgen</h1>
<p class="lede">
Turn source artifacts — a codebase, a database, an API spec, a recording of
real traffic — into one canonical graph format, then render that format to
whatever the audience needs. The point is not the diagram. The point is the
format in the middle.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="what">What it is</h2>
<p>
Eight demos under <code>semester/</code> draw their architecture with
Graphviz. Every one of them hand-writes a <code>.dot</code> file with the
palette inlined and commits the <code>.svg</code> beside it. Three different
dark palettes between them, eight answers to the same question, and every one
of those diagrams is out of date the moment somebody moves a file.
</p>
<p>
docgen is the one answer. It reads the source rather than being told about it,
so a diagram cannot drift from the thing it describes, and it separates
<em>what a graph is</em> from <em>how it looks</em> so one extraction feeds
a diagram, an index, a notebook and a browsable site without being redone.
</p>
<div class="note">
<b>Nothing here writes a parser or a graph algorithm.</b> Parsers are adopted
(<code>ast</code>, tree-sitter, SQLAlchemy reflection via modelgen), algorithms
are networkx's. What docgen owns is the adapters, the schema, the style tables
and the emitters — all small, and all the places where the value is that
<em>we</em> made the call.
</div>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="idea">The idea</h2>
<p>
N sources and M outputs need N&times;M converters if you join them directly,
or <strong>N+M</strong> if you put a hub in the middle. The hub is an
<em>intermediate representation</em> — the compiler term, and the same bargain:
both sides depend on the IR and neither on the other.
</p>
<p>
It is lossy on purpose. It throws away every token of syntax and keeps
<em>"a class named User inherits from Base"</em>. That is the part that is
worth versioning, worth diffing, and worth drawing.
</p>
<p>
The practical consequence is the thing to judge it on: <strong>adding a source
costs one extractor and every emitter works on it unchanged; adding an output
costs one emitter and every extractor feeds it unchanged.</strong> When the
OpenAPI reader was written it emitted schemas using the same vocabulary the
database reader uses — and the ER diagram drew an API's data model without
anyone teaching it what an API was.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="quick">Five minutes</h2>
<p>Three commands, and they compose. That is the whole interface.</p>
<pre><code><span class="c"># 1. read something</span>
python3 -m docgen.extractors.python --root ../station/tools/histgen -o ir.json
<span class="c"># 2. narrow it to a useful view</span>
python3 -m docgen.ops ir.json --overview -o view.json
<span class="c"># 3. draw whatever its structure asks for</span>
python3 -m docgen.emitters auto view.json -o out/</code></pre>
<p>Or through the Makefile, which is a thin wrapper over exactly those:</p>
<pre><code>make ir SRC=/path/to/repo OUT=out <span class="c"># extract</span>
make explore OUT=out <span class="c"># the two-pane navigator</span>
make site OUT=out <span class="c"># a docs site with a sidebar</span>
make self <span class="c"># run the whole thing over soleprint</span></code></pre>
<p>
Everything is offline and self-contained. No server, no CDN, no build step —
the outputs open over <code>file://</code>.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="layers">The three concerns</h2>
<p>
DOT collapses three separate questions into one file format, which is why a
hand-written <code>.dot</code> is never reusable: you cannot change the palette
without editing the structure, and you cannot change the structure without
re-deciding the layout. docgen keeps them apart.
</p>
<table>
<tr><th>concern</th><th>question</th><th>owner</th></tr>
<tr><td><b>structure</b></td><td>what the graph <em>is</em></td><td><code>ir/schema.json</code></td></tr>
<tr><td><b>meaning</b></td><td>what things <em>mean visually</em></td><td><code>style/*.json</code>, keyed on <code>kind</code></td></tr>
<tr><td><b>placement</b></td><td>where things <em>go</em></td><td>the emitter, and only there</td></tr>
</table>
<p>
An extractor has never heard of SVG, colours or layout. An emitter has never
heard of Python, <code>ast</code> or SQL. Both halves of that are checked by
parsing the source and looking at what it imports, because a rule nobody
enforces is a rule that lasts about a month.
</p>
<figure>
<a href="viewer.html?src=img/architecture.svg">
<img src="img/architecture.svg" alt="docgen's own module structure">
</a>
<figcaption>
docgen read by docgen. <code>extractors/</code> reaches only <code>ir</code>;
<code>emitters/</code> reaches <code>ir</code> and <code>style</code>;
<code>ir/</code> reaches nothing outside itself; <code>lab/</code> has no
edges at all. Click to open the viewer — then click again for actual size.
</figcaption>
</figure>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="ir">The IR</h2>
<p>
Plain JSON. Three keys, and it has survived four domains without gaining a
fourth.
</p>
<pre><code>{
"meta": { "source": "python", "root": "app/", "schema_version": "1" },
"nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
"parent": "app.models",
"attrs": { "file": "app/models.py", "line": 12, "lines": 40 } } ],
"edges": [ { "source": "app.models.User", "target": "app.db.Base",
"kind": "inherits", "attrs": {} } ]
}</code></pre>
<table>
<tr><th>field</th><th>meaning</th></tr>
<tr><td><code>id</code></td>
<td>Fully qualified and <b>stable across runs</b>. Stability is what makes
two extractions from two commits diffable; without it a diff reports
noise and nobody trusts it.</td></tr>
<tr><td><code>kind</code></td>
<td>The hinge of the whole system, and the <b>only</b> field style and
layout may read. A small closed vocabulary per domain —
<code>module</code>/<code>class</code>/<code>function</code>,
<code>table</code>/<code>column</code>, <code>endpoint</code>,
<code>task</code>.</td></tr>
<tr><td><code>parent</code></td>
<td>Containment, and nothing else. A module contains a class. Relationships
are edges.</td></tr>
<tr><td><code>attrs</code></td>
<td>An open bag for whatever one domain cares about.
<code>file</code>/<code>line</code>/<code>lines</code> are what let a
box link to the line it came from, and what the minimap sizes by.</td></tr>
</table>
<div class="note">
<b>No visual information, ever.</b> If a field would change between a light and
a dark theme, it does not belong in the IR. <code>shape: "cylinder"</code> is
not a field — it is <code>kind: "datastore"</code> plus a style rule, and that
is exactly what lets the same IR render in a theme that has no cylinders. The
test suite sweeps every emitted document for colour-like keys.
</div>
<h3>Stdlib dataclasses, not Pydantic</h3>
<p>
The IR's whole value is being a plain document anything can open. A format that
needs a library installed to be read is an API, not a format. Validation is
therefore a function called at the boundary rather than a property of the type,
and it reads its field lists out of <code>schema.json</code> so the schema and
the dataclasses cannot drift apart.
</p>
<pre><code>python3 -m docgen.ir ir.json</code></pre>
<p>
It catches what a schema cannot: an edge naming a node that does not exist, a
containment cycle, a duplicate id, and a visual field smuggled into
<code>attrs</code>.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="shape">Shape decides the drawing</h2>
<p>
A diagram that fights its layout engine is usually the wrong <em>kind</em> of
diagram. The clearest evidence: the same 24-table database rendered
<b>32034&times;136</b> through Graphviz — a 235:1 strip — and
<b>1740&times;1860</b> through the ER emitter. Not because one engine is
better, but because a schema is a set of peer entities with references, and
laying it out in dependency ranks was never its shape.
</p>
<p>
So <code>ops.classify()</code> reads the structure and names the emitter,
with the reason attached — advice without a reason gets overridden the first
time it is inconvenient.
</p>
<table>
<tr><th>kind</th><th>drawn by</th><th>when</th></tr>
<tr><td><code>erd</code></td><td>cards in columns</td><td>entities with references</td></tr>
<tr><td><code>pipeline</code></td><td>ranks, left to right</td><td>a chain with fan-out — an Airflow DAG, a build</td></tr>
<tr><td><code>layered</code> / <code>tree</code></td><td>ranks, top down</td><td>ranks genuinely suit it</td></tr>
<tr><td><code>sheet</code></td><td>the index</td><td>one level is wider than ~20 — a strip in any engine</td></tr>
<tr><td><code>flat</code></td><td>the index</td><td>most nodes have no relationships: that is a list</td></tr>
</table>
<pre><code>$ python3 -m docgen.emitters auto view.json -o out/
sheet -&gt; index
109 nodes sit at one level; any layered engine draws that as a
strip. Split it, scope it, or read it as an index</code></pre>
<h3>Why twenty</h3>
<p>
Measured, one diagram per subsystem: at or under 20 nodes the output lands
around 1.6:1; at 70–106 nodes about 7:1; at 261 nodes 14:1. Aspect ratio is a
property of the <em>graph</em>, not of the renderer — a layered engine puts one
dependency level in one row, so the widest level <em>is</em> the width.
</p>
<p>
Every Graphviz lever was tried before concluding this. <code>ratio=compress</code>
squashed a graph to an unreadable 1008&times;75; <code>rankdir=LR</code> merely
rotated a 14:1 into a 1:6; packing disconnected components gained nothing.
The fix was never a flag. It was to stop asking for one picture of everything —
which is what <a href="#explore">explore</a> does.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="extractors">Extractors</h2>
<p>
<strong>Deterministic parsing only. No model in the structural path.</strong>
A diagram built from an AST cannot be out of date with the code. A diagram
built from a model's reading of the code is wrong the moment the model has a
bad day — which is the problem this exists to fix.
</p>
<table>
<tr><th>reader</th><th>reads</th><th>gives</th></tr>
<tr><td><code>python</code></td><td>a tree of <code>.py</code></td>
<td>modules, classes, functions; <code>imports</code> and <code>inherits</code> edges</td></tr>
<tr><td><code>code</code> <span class="pill opt">tree-sitter</span></td><td>C#, TypeScript, TSX</td>
<td>namespaces, classes, interfaces, methods — <b>structure only</b></td></tr>
<tr><td><code>db</code></td><td>a graphgen-compatible <code>schema.json</code></td>
<td>tables, columns, foreign keys</td></tr>
<tr><td><code>openapi</code></td><td>an OpenAPI / Swagger document</td>
<td>endpoints and the shapes they carry</td></tr>
<tr><td><code>usage</code></td><td>a HAR recording</td>
<td>what was actually called, in what order</td></tr>
</table>
<h3>Two passes, because <code>ast</code> resolves nothing</h3>
<p>
Given <code>class User(Base)</code>, Python's <code>ast</code> hands over the
literal string <code>"Base"</code>. It has no idea that came from
<code>from .db import Base</code> three lines up. So pass one collects, per
module, what it defines and what it imports; pass two resolves local names to
fully qualified ids. The edge then points at <code>app.db.Base</code> — a real
node — rather than at a box called <code>Base</code> that means nothing.
</p>
<div class="note">
<b>Unresolved names become nodes, never nothing.</b> A third-party import or a
dynamically-built base becomes a node of <code>kind: "external"</code> and
<em>keeps its edge</em>. Dropping it would be the worse failure: the diagram
would look complete and have quietly lost a dependency. Gathered up, those
nodes are the project's real dependency surface.
</div>
<h3>C# and TypeScript</h3>
<p>
Handled by tree-sitter, which is why generics, nested types and a brace inside
a string are non-events rather than special cases. The test suite asserts that
last one specifically, because it is exactly where a hand-rolled scanner breaks.
</p>
<p>
This reader produces <strong>no edges</strong>. Resolving a C#
<code>using</code> to the thing it names is a different and much larger job,
and the consumer that needs this — the minimap — needs none of it. An
extractor that quietly produced half a dependency graph would be worse than one
producing none, because the half would look whole.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="usage">Usage, not just the spec</h2>
<p>
An OpenAPI document says what endpoints <em>are</em>. It does not say how to
use them — least of all when they are not RESTful, or when a GraphQL endpoint
sits alongside. So docgen also reads a <b>HAR</b>: the recording format that
browser devtools, mitmproxy, Charles and Insomnia all export.
</p>
<table>
<tr><th>what traffic knows</th><th>what a spec cannot</th></tr>
<tr><td>the <b>order</b> of calls</td><td>a spec is a set; usage is a sequence</td></tr>
<tr><td>which parameters are <b>always</b> sent</td><td>a spec lists twenty optional ones</td></tr>
<tr><td>which statuses <b>really</b> happen</td><td>the 422 everybody hits is in no document</td></tr>
<tr><td>endpoints not in the document</td><td>GraphQL operations, found by body shape and named</td></tr>
<tr><td>which id formats a route takes</td><td>numeric <em>and</em> uuid on one route</td></tr>
</table>
<div class="note warn">
<b>No credential and no payload value reaches the IR</b> — only field names and
types. A HAR is full of live bearer tokens and cookies, and a generated
document gets committed. The test suite plants a token in its fixture and fails
if it appears anywhere in the output.
</div>
<p>
Two limits, stated rather than glossed: path templating is a <em>guess</em>
(<code>attrs.observed_paths</code> keeps what was actually seen beside it), and
consecutive is not caused-by — the edge weight is what separates a habit from
an accident, and one recording will not tell you which.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="views">Views</h2>
<p>
The first real diagram out of this pipeline was a 3000px-wide strip: four
modules of actual content and sixty <code>sys</code>/<code>json</code>/<code>typing</code>
boxes as their peers. The emitter was correct and the picture was useless.
That is a <em>missing view</em>, not a broken renderer — and the fix belongs to
every consumer at once, because the index, the diagram and the diff all want
the same narrowing.
</p>
<pre><code>python3 -m docgen.ops ir.json --overview -o view.json
python3 -m docgen.ops ir.json --around docgen.ir --hops 2 -o view.json
python3 -m docgen.ops ir.json --split -o parts/
python3 -m docgen.ops ir.json --shape <span class="c"># what will this look like?</span></code></pre>
<p>
All of them are IR&rarr;IR, all composable, and each produces a document that
still validates. <code>--overview</code> is the default and dispatches on the
source: a codebase reduces to its modules and its outside dependencies, a
schema to its tables and their keys.
</p>
<div class="note">
<b>Edges are lifted when a view collapses detail, never dropped.</b> A class in
module A inheriting from a class in module B <em>is</em> a dependency of A on
B. Collapsing docgen to its packages once kept 8 of 77 edges — those pictures
were not simpler, they were <em>wrong</em>. Lifted edges carry a
<code>weight</code> saying how many they stand for.
</div>
<h3>Depth is the tempting knob and the wrong one</h3>
<p>
A directory without an <code>__init__.py</code> is not a package, so its
modules have no parent and sit at depth 0. soleprint has <b>173 such roots</b>,
and a depth-2 cut still held 566 functions and 142 classes. Selecting by
<code>kind</code> does not care how the directories happen to be arranged.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="emitters">Emitters</h2>
<table>
<tr><th>emitter</th><th>output</th><th>audience</th></tr>
<tr><td><code>index</code></td><td>markdown, sidebar JSON</td>
<td><b>anyone</b> — no graph literacy required</td></tr>
<tr><td><code>dot</code></td><td>DOT &rarr; Graphviz &rarr; SVG</td><td>dependency structure</td></tr>
<tr><td><code>erd</code></td><td>SVG, written directly</td><td>a schema, as cards</td></tr>
<tr><td><code>minimap</code></td><td>SVG, written directly</td><td>what is where, at a glance</td></tr>
<tr><td><code>notebook</code></td><td><code>.ipynb</code></td><td>a runnable walkthrough</td></tr>
<tr><td><code>site</code></td><td>a static docs site</td><td>reading</td></tr>
<tr><td><code>explore</code></td><td>a two-pane navigator</td><td>finding your way</td></tr>
<tr><td><code>auto</code></td><td>whichever of the above fits</td><td>not having to choose</td></tr>
</table>
<p>
<strong>The non-visual ones matter most for reach.</strong> A sorted, described
list of what exists is readable by someone who will never open a diagram, and
it also reports the dependency surface and any file that failed to parse.
It is built second, not last — it is what proves the IR is not secretly
diagram-shaped.
</p>
<h3>ERD — and where the layout came from</h3>
<p>
Not invented here. <code>station/tools/graphgen/templates/index.html</code>,
the Supabase-style schema explorer already in this repo, had solved it:
</p>
<pre><code>const cols = Math.max(2, Math.ceil(Math.sqrt(sorted.length * 1.2)));</code></pre>
<p>
<strong>Columns from the square root of the table count.</strong> The aspect
ratio is <em>chosen</em> rather than emergent, so the result stays near-square
at 4 tables or 400. That is the one thing a rank-based engine cannot offer.
Three more things it gets right: a table is a <em>card</em> with its columns;
an edge leaves the column holding the key and lands on the target's primary
key; and the geometry is computed rather than measured, so it renders
identically on any machine.
</p>
<figure>
<a href="viewer.html?src=img/erd.svg">
<img src="img/erd.svg" alt="an entity-relationship diagram">
</a>
<figcaption>A schema from the sample room. Same emitter, same style file as
every other diagram here.</figcaption>
</figure>
<h3>Minimap</h3>
<p>
Sublime's minimap shrinks the <em>characters</em>. This draws the
<em>structure</em> at full scale: one file is a column, one line is a fixed
number of pixels, every construct a block sized by its span and coloured by
what it is. No text inside a block — the shape is the message.
</p>
<p>
The claim is that the pattern comes from the colours alone, so nesting is drawn
by inset rather than by hue. On soleprint you can see that modelgen is
class-based, histgen is function-based and tester is mixed, without reading a
line.
</p>
<figure>
<a href="viewer.html?src=img/minimap.svg">
<img src="img/minimap.svg" alt="a structural minimap of docgen">
</a>
<figcaption>docgen's own files. Blue class, amber interface, green function,
dark for everything that is not a declaration — imports, constants, prose.</figcaption>
</figure>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="explore">Explore</h2>
<p>
The minimap on its own shows shape and no meaning: a block says "a 30-line
class", not <em>which</em> class or what it touches. So it is not the artifact.
It is the <strong>selector</strong>.
</p>
<pre><code>make explore OUT=out <span class="c"># then open out/explore/explore.html</span></code></pre>
<div class="cols">
<div>
<h4>Left — navigate</h4>
<p>The whole thing at once. Scan by colour, click a block.</p>
</div>
<div>
<h4>Right — explore</h4>
<p>What that is, what it reaches, what reaches it, and the neighbourhood
drawn small enough to read. Every neighbour is a link, so you walk
outward from wherever you started.</p>
</div>
</div>
<div class="note">
<b>This is what retires the 14:1 sheet.</b> The whole graph is never drawn. The
overview pane carries the overview, and only the neighbourhood of a selection
is rendered — a handful of nodes, which lays out fine every time. Overview and
detail stop competing for one picture.
</div>
<p>
The same split applies to a database: every table at once with <em>no column
detail</em>, then click one to get its columns plus the tables its keys reach.
The two differ exactly where they should — a module's neighbourhood
deliberately leaves its contents out, because those are the hundred functions
that made the sheet unreadable, while a table's brings them in, because a table
without its columns is not a table.
</p>
<h3>The selection basket</h3>
<p>
Shift-click accumulates blocks. The basket is a copyable list of paths with a
line count — enough to hand to <code>distill</code>, and enough to see that the
selection got too big <em>before</em> spending the context on it. Navigating a
tree quickly in order to decide what to feed a model is a real use, and this is
the part that serves it.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="notebooks">Notebooks</h2>
<p>
A notebook is normally a source file somebody confects by hand: prose, code and
stored output braided together, diffing badly, drifting from whatever it
documents the moment either moves, with no way to tell by looking.
</p>
<p>
Here a notebook is a <strong>build artifact</strong>. The source is the OpenAPI
document — the same file the server is built from — and the notebook is
regenerated from it. Nobody edits the <code>.ipynb</code>, the same way nobody
edits a <code>.o</code>. "Is this document current" stops being a question
about somebody's diligence and becomes a question about whether the build ran.
</p>
<blockquote>
This is the disagreement with jupytext. Jupytext fixes the <em>diffing</em> —
it makes a notebook editable as text — and leaves the actual problem: you still
hand-author it, so it still rots.
</blockquote>
<h3>Generated base, hand-written overlay</h3>
<p>
Generation alone gives a document that is never stale and never says anything a
parser could not work out. Hand-authoring alone gives insight and a document
that rots. Two files is the only arrangement that gets both:
</p>
<pre><code>IR ──► spec ──(+ overlay)──► merged spec ──► .ipynb
generated hand-written merged emitted</code></pre>
<p>
The <b>spec</b> is an ordered list of steps with no Jupyter in it — a Swagger
for notebooks, readable and diffable. The <b>overlay</b> is the only file
anyone edits, and it is re-applied on every build. It can
<code>annotate</code>, <code>replace</code>, <code>insert</code>,
<code>drop</code> and <code>order</code>.
</p>
<p>
<code>replace</code> is the one that matters. It is how real usage gets into a
document that a spec could not describe — the call that is always made with
<code>status=available</code>, the GraphQL endpoint that is not in the OpenAPI
file at all — and it keeps working unchanged once a usage recording supplies
the same facts automatically.
</p>
<div class="note">
Three properties hold it together: regenerating <b>re-applies the overlay
byte-for-byte</b>; when the base moves underneath it the mismatch is
<b>reported, never silently dropped</b>; and extraction works with the overlay
<b>absent</b> — it is an addition, never a dependency.
</div>
<pre><code>python3 -m docgen.emitters notebook ir.json --scaffold overlay.json
python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthrough.ipynb</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="style">Style &amp; colour</h2>
<p>
A style rule names a <strong>slot</strong>, never a colour.
<code>"border": "atlas"</code> is the rule; a theme binds <code>atlas</code> to
<code>#43A047</code> in print and <code>#15803d</code> on the docs site.
</p>
<p>
That indirection is the whole point. <code>common/theme/tokens.css</code>,
<code>docs/graphs/themes/*.gvpr</code> and <code>style/lucid.json</code> use
the same slot names, so a diagram and the page around it match by construction
— which is the rule <code>docs/graphs/README.md</code> already states. The dark
theme's <code>artery</code>, <code>atlas</code> and <code>station</code> slots
are exactly the <code>--system-accent</code> values the three system pages set,
and the test suite fails if they drift apart.
</p>
<p>
Dark is the default, because a generated diagram lands in a dark docs page far
more often than in a document. <code>--theme lucid</code> gives the print
palette — and gives it to the <em>page</em> as well as the diagram, since both
are baked from the same slots.
</p>
<div class="note">
An unknown <code>kind</code> falls back to <code>default</code> rather than
crashing. That matters more than it sounds: a new extractor with a new
vocabulary renders plainly and legibly on day one, instead of requiring
somebody to write a style file before they can see anything.
</div>
<h3>Where DOT stops</h3>
<p>
The emitter writes what DOT expresses natively and stops at the boundary rather
than growing machinery. The limits are recorded in the style file itself: a
cluster has a label and a fill but not a header bar;
<code>stroke-dasharray</code> is not parameterised, so <code>4,4</code> and
<code>5,5</code> collapse; <code>rounded</code> is binary, so 4px and 6px are
identical. Those mark where a richer emitter would begin — and the style file
carries the full specification regardless, so that emitter needs no
re-authoring.
</p>
<p>
One limit <em>was</em> worth solving: DOT cannot use a cluster as an edge
endpoint, so every module-to-module import silently vanished. The native answer
is <code>compound=true</code> with <code>lhead</code>/<code>ltail</code> — draw
between a representative leaf and clip the line at the cluster border.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="commands">Commands</h2>
<h3>Make</h3>
<table>
<tr><th>target</th><th>does</th></tr>
<tr><td><code>make check</code></td><td>the whole test suite, offline, nothing installed</td></tr>
<tr><td><code>make doctor</code></td><td>what this machine has and what it is missing</td></tr>
<tr><td><code>make ir SRC=…</code></td><td>extract Python into <code>OUT/ir.json</code></td></tr>
<tr><td><code>make code SRC=…</code></td><td>extract C#/TypeScript <span class="pill opt">tree-sitter</span></td></tr>
<tr><td><code>make db SCHEMA=…</code></td><td>extract a database schema</td></tr>
<tr><td><code>make view</code></td><td>the default view for that source type</td></tr>
<tr><td><code>make graph</code></td><td>draw whatever the structure asks for</td></tr>
<tr><td><code>make index</code></td><td>markdown index and sidebar JSON</td></tr>
<tr><td><code>make minimap</code></td><td>what is where, read from the colours</td></tr>
<tr><td><code>make explore</code></td><td>the two-pane navigator</td></tr>
<tr><td><code>make site</code></td><td>a self-contained docs site</td></tr>
<tr><td><code>make self</code></td><td>the whole pipeline over soleprint itself</td></tr>
</table>
<p>
Variables: <code>SRC</code>, <code>OUT</code>, <code>SCHEMA</code>,
<code>STYLE</code>, <code>THEME</code>, <code>SCALE</code>, <code>DEPTH</code>,
<code>PY</code>. The Makefile derives its own package name from where it sits,
so the folder can be copied anywhere and renamed and still work.
</p>
<h3>Modules</h3>
<pre><code>python3 -m docgen.extractors.python --root SRC -o ir.json
python3 -m docgen.extractors code --root SRC -o ir.json
python3 -m docgen.extractors db --schema schema.json -o ir.json
python3 -m docgen.extractors openapi --spec spec.yaml -o ir.json
python3 -m docgen.extractors usage --har session.har -o ir.json
python3 -m docgen.ir ir.json <span class="c"># validate</span>
python3 -m docgen.ops ir.json --overview -o view.json
python3 -m docgen.emitters auto view.json -o out/
python3 -m docgen.emitters index ir.json -o index.md
python3 -m docgen.emitters dot view.json -o graph.svg --theme lucid
python3 -m docgen.emitters erd ir.json -o schema.svg
python3 -m docgen.emitters minimap ir.json -o map.svg --scale 0.5
python3 -m docgen.emitters notebook ir.json -o book.ipynb --overlay overlay.json
python3 -m docgen.emitters site view.json -o site/
python3 -m docgen.emitters explore ir.json -o explore/</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="deps">Dependencies</h2>
<p>
The core is <strong>standard library only</strong>. Everything else is optional
and reported by <code>make doctor</code>; when something is missing you lose
exactly one capability and get told what to install.
</p>
<table>
<tr><th>needs</th><th>for</th><th>without it</th></tr>
<tr><td><code>graphviz</code> (binary)</td><td>rendering DOT to SVG</td>
<td>ERD, minimap, index and notebooks still work</td></tr>
<tr><td><code>tree_sitter</code> + grammars</td><td>C#, TypeScript, TSX</td>
<td>Python only</td></tr>
<tr><td><code>networkx</code></td><td><code>lab/</code> experiments</td>
<td>nothing — nothing depends on it yet</td></tr>
<tr><td><code>node</code></td><td>testing the browser pages</td>
<td>those checks skip</td></tr>
<tr><td><code>psql</code></td><td>the <code>lab/</code> schema probe</td>
<td>use modelgen's <code>from-db</code> instead</td></tr>
</table>
<div class="note">
<b><code>lab/</code> is where a dependency gets tried before anything depends
on it.</b> Nothing in <code>ir/</code>, <code>extractors/</code>,
<code>ops/</code> or <code>emitters/</code> may import from it. When an
experiment earns its place it graduates into <code>ops/</code> behind an
IR&rarr;IR signature, and <em>then</em> the dependency is declared.
</div>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="testing">Testing</h2>
<pre><code>make check <span class="c"># 191 checks, offline, no network</span></code></pre>
<p>
Four of those are the <em>design</em> rather than regressions, and they are the
ones to keep if anything is ever cut:
</p>
<ul>
<li><b>No visual field reaches the IR</b> — extractors cannot decide appearance.</li>
<li><b>No emitter reads a source file</b> — the layering, checked from the other side by parsing imports.</li>
<li><b>Style names slots, not colours</b> — one colour language rather than three.</li>
<li><b>Ids are stable across runs</b> — without it, diffing is noise.</li>
</ul>
<div class="note">
<b>Golden tests go on the IR, never on the SVG.</b> Graphviz measures label
text with the host's fonts to size nodes, so identical input produces different
geometry on a machine with different fontconfig. The IR is deterministic; the
SVG is not. Pinning the wrong one gives a suite that fails on somebody else's
laptop for no reason anyone can act on.
</div>
<p>
The browser pages are JavaScript, so they are tested as JavaScript: a stub DOM
under <code>node</code> drives the viewer's zoom and 1:1 toggle, and the
explorer's select-and-walk. Both skip cleanly where node is absent.
</p>
<p>
Self-hosting is the honest end-to-end check, and it is where the real bugs came
from — two name-resolution faults and a duplicate-id crash that no fixture had
reached. <code>make self</code> runs the whole pipeline over soleprint; if the
index does not read like the system, something is wrong.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="limits">Limits &amp; non-goals</h2>
<p>Things deliberately not done, with the reason, so they are not re-litigated:</p>
<ul>
<li><b>No layout engine.</b> No positioning, no <code>neato -n2</code>, no ELK.
Aspect ratio was solved by choosing the right emitter and by not drawing
everything at once.</li>
<li><b>No <code>calls</code> edges.</b> Resolving <code>self.foo()</code> needs
type inference, and a call graph that is quietly 60% right is worse than
none because it reads as authoritative.</li>
<li><b>No edges from the C#/TypeScript reader.</b> Structure only — half a
dependency graph would look whole.</li>
<li><b>No model in the structural path.</b> Annotation — summarising a module,
naming a cluster — is a later layer, cached to its own file keyed by node
id, merged at emit time, and extraction must work with it absent.</li>
<li><b>No configuration knobs</b> until two real consumers disagree.</li>
</ul>
<p>Known gaps, stated plainly:</p>
<ul>
<li>The C# reader is verified against a written fixture, <b>not a real
repository</b>. That is the next check that matters.</li>
<li>The diff emitter is not built. Stable ids exist to make it possible, and
two IRs from two commits is the cheapest useful thing left.</li>
<li>Minimap blocks carry no names. Rendering them legible only at 1:1, through
the viewer, is the other half of reading a file without reading every line.</li>
</ul>
</main>
</div>
<script>
// Highlight the section being read. No dependency, no build step.
var links = [].slice.call(document.querySelectorAll('.sidebar a[href^="#"]'));
var byId = {};
links.forEach(function (a) { byId[a.getAttribute('href').slice(1)] = a; });
var observer = new IntersectionObserver(function (entries) {
entries.forEach(function (entry) {
var a = byId[entry.target.id];
if (a && entry.isIntersecting) {
links.forEach(function (l) { l.classList.remove('active'); });
a.classList.add('active');
}
});
}, { rootMargin: '-8% 0px -82% 0px' });
document.querySelectorAll('h2[id]').forEach(function (h) { observer.observe(h); });
</script>
</body>
</html>