docgen: drop the superseded first pass, port the style harvester
The IR supersedes both intermediate designs (requirements D6, D7):
station/tools/docgen and the graph model that briefly lived in graphgen.
Shipping them beside atlas2/docgen would mean two graph models, which is the
thing the architecture argues against.
Carried across rather than lost:
- style/extract.py and tokens.py, the offline theme harvester (R31). Rewritten
to emit a *theme* — a slot-to-hex binding — rather than a whole style file,
since what harvesting recovers is which colour a slot should be, not what a
kind should look like.
- graphgen/README.md, rewritten for what graphgen actually is now: the
schema explorer. It fixes the blank station-index entry at run.py:304.
Two bugs found while doing it:
- Canvas and ink are the two lightness extremes, not the two most common
values. In a Graphviz SVG every label carries a fill, so the ink outnumbers
the canvas 87 to 43 and the old rule produced a theme whose text was
invisible against its own background.
- A name defined in both branches of an if/else produced a duplicate id, which
failed validation on docgen's own source. Disambiguated by line.
This commit is contained in:
793
soleprint/atlas2/docgen/docs/index.html
Normal file
793
soleprint/atlas2/docgen/docs/index.html
Normal file
@@ -0,0 +1,793 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>docgen</title>
|
||||
<link rel="stylesheet" href="docs.css">
|
||||
</head>
|
||||
<body>
|
||||
<div class="layout">
|
||||
|
||||
<nav class="sidebar">
|
||||
<div class="sidebar-header">
|
||||
<b>docgen</b>
|
||||
<small>code & data → documentation</small>
|
||||
</div>
|
||||
<nav>
|
||||
<div class="group">Start</div>
|
||||
<a href="#what">What it is</a>
|
||||
<a href="#idea">The idea</a>
|
||||
<a href="#quick">Five minutes</a>
|
||||
|
||||
<div class="group">Architecture</div>
|
||||
<a href="#layers">The three concerns</a>
|
||||
<a href="#ir">The IR</a>
|
||||
<a href="#shape">Shape decides the drawing</a>
|
||||
|
||||
<div class="group">Reading</div>
|
||||
<a href="#extractors">Extractors</a>
|
||||
<a href="#usage">Usage, not just the spec</a>
|
||||
|
||||
<div class="group">Narrowing</div>
|
||||
<a href="#views">Views</a>
|
||||
|
||||
<div class="group">Writing</div>
|
||||
<a href="#emitters">Emitters</a>
|
||||
<a href="#explore">Explore</a>
|
||||
<a href="#notebooks">Notebooks</a>
|
||||
<a href="#style">Style & colour</a>
|
||||
|
||||
<div class="group">Reference</div>
|
||||
<a href="#commands">Commands</a>
|
||||
<a href="#deps">Dependencies</a>
|
||||
<a href="#testing">Testing</a>
|
||||
<a href="#limits">Limits & non-goals</a>
|
||||
</nav>
|
||||
</nav>
|
||||
|
||||
<main class="content">
|
||||
|
||||
<h1>docgen</h1>
|
||||
<p class="lede">
|
||||
Turn source artifacts — a codebase, a database, an API spec, a recording of
|
||||
real traffic — into one canonical graph format, then render that format to
|
||||
whatever the audience needs. The point is not the diagram. The point is the
|
||||
format in the middle.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="what">What it is</h2>
|
||||
|
||||
<p>
|
||||
Eight demos under <code>semester/</code> draw their architecture with
|
||||
Graphviz. Every one of them hand-writes a <code>.dot</code> file with the
|
||||
palette inlined and commits the <code>.svg</code> beside it. Three different
|
||||
dark palettes between them, eight answers to the same question, and every one
|
||||
of those diagrams is out of date the moment somebody moves a file.
|
||||
</p>
|
||||
<p>
|
||||
docgen is the one answer. It reads the source rather than being told about it,
|
||||
so a diagram cannot drift from the thing it describes, and it separates
|
||||
<em>what a graph is</em> from <em>how it looks</em> so one extraction feeds
|
||||
a diagram, an index, a notebook and a browsable site without being redone.
|
||||
</p>
|
||||
|
||||
<div class="note">
|
||||
<b>Nothing here writes a parser or a graph algorithm.</b> Parsers are adopted
|
||||
(<code>ast</code>, tree-sitter, SQLAlchemy reflection via modelgen), algorithms
|
||||
are networkx's. What docgen owns is the adapters, the schema, the style tables
|
||||
and the emitters — all small, and all the places where the value is that
|
||||
<em>we</em> made the call.
|
||||
</div>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="idea">The idea</h2>
|
||||
|
||||
<p>
|
||||
N sources and M outputs need N×M converters if you join them directly,
|
||||
or <strong>N+M</strong> if you put a hub in the middle. The hub is an
|
||||
<em>intermediate representation</em> — the compiler term, and the same bargain:
|
||||
both sides depend on the IR and neither on the other.
|
||||
</p>
|
||||
<p>
|
||||
It is lossy on purpose. It throws away every token of syntax and keeps
|
||||
<em>"a class named User inherits from Base"</em>. That is the part that is
|
||||
worth versioning, worth diffing, and worth drawing.
|
||||
</p>
|
||||
<p>
|
||||
The practical consequence is the thing to judge it on: <strong>adding a source
|
||||
costs one extractor and every emitter works on it unchanged; adding an output
|
||||
costs one emitter and every extractor feeds it unchanged.</strong> When the
|
||||
OpenAPI reader was written it emitted schemas using the same vocabulary the
|
||||
database reader uses — and the ER diagram drew an API's data model without
|
||||
anyone teaching it what an API was.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="quick">Five minutes</h2>
|
||||
|
||||
<p>Three commands, and they compose. That is the whole interface.</p>
|
||||
|
||||
<pre><code><span class="c"># 1. read something</span>
|
||||
python3 -m docgen.extractors.python --root ../station/tools/histgen -o ir.json
|
||||
|
||||
<span class="c"># 2. narrow it to a useful view</span>
|
||||
python3 -m docgen.ops ir.json --overview -o view.json
|
||||
|
||||
<span class="c"># 3. draw whatever its structure asks for</span>
|
||||
python3 -m docgen.emitters auto view.json -o out/</code></pre>
|
||||
|
||||
<p>Or through the Makefile, which is a thin wrapper over exactly those:</p>
|
||||
|
||||
<pre><code>make ir SRC=/path/to/repo OUT=out <span class="c"># extract</span>
|
||||
make explore OUT=out <span class="c"># the two-pane navigator</span>
|
||||
make site OUT=out <span class="c"># a docs site with a sidebar</span>
|
||||
make self <span class="c"># run the whole thing over soleprint</span></code></pre>
|
||||
|
||||
<p>
|
||||
Everything is offline and self-contained. No server, no CDN, no build step —
|
||||
the outputs open over <code>file://</code>.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="layers">The three concerns</h2>
|
||||
|
||||
<p>
|
||||
DOT collapses three separate questions into one file format, which is why a
|
||||
hand-written <code>.dot</code> is never reusable: you cannot change the palette
|
||||
without editing the structure, and you cannot change the structure without
|
||||
re-deciding the layout. docgen keeps them apart.
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<tr><th>concern</th><th>question</th><th>owner</th></tr>
|
||||
<tr><td><b>structure</b></td><td>what the graph <em>is</em></td><td><code>ir/schema.json</code></td></tr>
|
||||
<tr><td><b>meaning</b></td><td>what things <em>mean visually</em></td><td><code>style/*.json</code>, keyed on <code>kind</code></td></tr>
|
||||
<tr><td><b>placement</b></td><td>where things <em>go</em></td><td>the emitter, and only there</td></tr>
|
||||
</table>
|
||||
|
||||
<p>
|
||||
An extractor has never heard of SVG, colours or layout. An emitter has never
|
||||
heard of Python, <code>ast</code> or SQL. Both halves of that are checked by
|
||||
parsing the source and looking at what it imports, because a rule nobody
|
||||
enforces is a rule that lasts about a month.
|
||||
</p>
|
||||
|
||||
<figure>
|
||||
<a href="viewer.html?src=img/architecture.svg">
|
||||
<img src="img/architecture.svg" alt="docgen's own module structure">
|
||||
</a>
|
||||
<figcaption>
|
||||
docgen read by docgen. <code>extractors/</code> reaches only <code>ir</code>;
|
||||
<code>emitters/</code> reaches <code>ir</code> and <code>style</code>;
|
||||
<code>ir/</code> reaches nothing outside itself; <code>lab/</code> has no
|
||||
edges at all. Click to open the viewer — then click again for actual size.
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="ir">The IR</h2>
|
||||
|
||||
<p>
|
||||
Plain JSON. Three keys, and it has survived four domains without gaining a
|
||||
fourth.
|
||||
</p>
|
||||
|
||||
<pre><code>{
|
||||
"meta": { "source": "python", "root": "app/", "schema_version": "1" },
|
||||
"nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
|
||||
"parent": "app.models",
|
||||
"attrs": { "file": "app/models.py", "line": 12, "lines": 40 } } ],
|
||||
"edges": [ { "source": "app.models.User", "target": "app.db.Base",
|
||||
"kind": "inherits", "attrs": {} } ]
|
||||
}</code></pre>
|
||||
|
||||
<table>
|
||||
<tr><th>field</th><th>meaning</th></tr>
|
||||
<tr><td><code>id</code></td>
|
||||
<td>Fully qualified and <b>stable across runs</b>. Stability is what makes
|
||||
two extractions from two commits diffable; without it a diff reports
|
||||
noise and nobody trusts it.</td></tr>
|
||||
<tr><td><code>kind</code></td>
|
||||
<td>The hinge of the whole system, and the <b>only</b> field style and
|
||||
layout may read. A small closed vocabulary per domain —
|
||||
<code>module</code>/<code>class</code>/<code>function</code>,
|
||||
<code>table</code>/<code>column</code>, <code>endpoint</code>,
|
||||
<code>task</code>.</td></tr>
|
||||
<tr><td><code>parent</code></td>
|
||||
<td>Containment, and nothing else. A module contains a class. Relationships
|
||||
are edges.</td></tr>
|
||||
<tr><td><code>attrs</code></td>
|
||||
<td>An open bag for whatever one domain cares about.
|
||||
<code>file</code>/<code>line</code>/<code>lines</code> are what let a
|
||||
box link to the line it came from, and what the minimap sizes by.</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="note">
|
||||
<b>No visual information, ever.</b> If a field would change between a light and
|
||||
a dark theme, it does not belong in the IR. <code>shape: "cylinder"</code> is
|
||||
not a field — it is <code>kind: "datastore"</code> plus a style rule, and that
|
||||
is exactly what lets the same IR render in a theme that has no cylinders. The
|
||||
test suite sweeps every emitted document for colour-like keys.
|
||||
</div>
|
||||
|
||||
<h3>Stdlib dataclasses, not Pydantic</h3>
|
||||
<p>
|
||||
The IR's whole value is being a plain document anything can open. A format that
|
||||
needs a library installed to be read is an API, not a format. Validation is
|
||||
therefore a function called at the boundary rather than a property of the type,
|
||||
and it reads its field lists out of <code>schema.json</code> so the schema and
|
||||
the dataclasses cannot drift apart.
|
||||
</p>
|
||||
<pre><code>python3 -m docgen.ir ir.json</code></pre>
|
||||
<p>
|
||||
It catches what a schema cannot: an edge naming a node that does not exist, a
|
||||
containment cycle, a duplicate id, and a visual field smuggled into
|
||||
<code>attrs</code>.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="shape">Shape decides the drawing</h2>
|
||||
|
||||
<p>
|
||||
A diagram that fights its layout engine is usually the wrong <em>kind</em> of
|
||||
diagram. The clearest evidence: the same 24-table database rendered
|
||||
<b>32034×136</b> through Graphviz — a 235:1 strip — and
|
||||
<b>1740×1860</b> through the ER emitter. Not because one engine is
|
||||
better, but because a schema is a set of peer entities with references, and
|
||||
laying it out in dependency ranks was never its shape.
|
||||
</p>
|
||||
<p>
|
||||
So <code>ops.classify()</code> reads the structure and names the emitter,
|
||||
with the reason attached — advice without a reason gets overridden the first
|
||||
time it is inconvenient.
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<tr><th>kind</th><th>drawn by</th><th>when</th></tr>
|
||||
<tr><td><code>erd</code></td><td>cards in columns</td><td>entities with references</td></tr>
|
||||
<tr><td><code>pipeline</code></td><td>ranks, left to right</td><td>a chain with fan-out — an Airflow DAG, a build</td></tr>
|
||||
<tr><td><code>layered</code> / <code>tree</code></td><td>ranks, top down</td><td>ranks genuinely suit it</td></tr>
|
||||
<tr><td><code>sheet</code></td><td>the index</td><td>one level is wider than ~20 — a strip in any engine</td></tr>
|
||||
<tr><td><code>flat</code></td><td>the index</td><td>most nodes have no relationships: that is a list</td></tr>
|
||||
</table>
|
||||
|
||||
<pre><code>$ python3 -m docgen.emitters auto view.json -o out/
|
||||
sheet -> index
|
||||
109 nodes sit at one level; any layered engine draws that as a
|
||||
strip. Split it, scope it, or read it as an index</code></pre>
|
||||
|
||||
<h3>Why twenty</h3>
|
||||
<p>
|
||||
Measured, one diagram per subsystem: at or under 20 nodes the output lands
|
||||
around 1.6:1; at 70–106 nodes about 7:1; at 261 nodes 14:1. Aspect ratio is a
|
||||
property of the <em>graph</em>, not of the renderer — a layered engine puts one
|
||||
dependency level in one row, so the widest level <em>is</em> the width.
|
||||
</p>
|
||||
<p>
|
||||
Every Graphviz lever was tried before concluding this. <code>ratio=compress</code>
|
||||
squashed a graph to an unreadable 1008×75; <code>rankdir=LR</code> merely
|
||||
rotated a 14:1 into a 1:6; packing disconnected components gained nothing.
|
||||
The fix was never a flag. It was to stop asking for one picture of everything —
|
||||
which is what <a href="#explore">explore</a> does.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="extractors">Extractors</h2>
|
||||
|
||||
<p>
|
||||
<strong>Deterministic parsing only. No model in the structural path.</strong>
|
||||
A diagram built from an AST cannot be out of date with the code. A diagram
|
||||
built from a model's reading of the code is wrong the moment the model has a
|
||||
bad day — which is the problem this exists to fix.
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<tr><th>reader</th><th>reads</th><th>gives</th></tr>
|
||||
<tr><td><code>python</code></td><td>a tree of <code>.py</code></td>
|
||||
<td>modules, classes, functions; <code>imports</code> and <code>inherits</code> edges</td></tr>
|
||||
<tr><td><code>code</code> <span class="pill opt">tree-sitter</span></td><td>C#, TypeScript, TSX</td>
|
||||
<td>namespaces, classes, interfaces, methods — <b>structure only</b></td></tr>
|
||||
<tr><td><code>db</code></td><td>a graphgen-compatible <code>schema.json</code></td>
|
||||
<td>tables, columns, foreign keys</td></tr>
|
||||
<tr><td><code>openapi</code></td><td>an OpenAPI / Swagger document</td>
|
||||
<td>endpoints and the shapes they carry</td></tr>
|
||||
<tr><td><code>usage</code></td><td>a HAR recording</td>
|
||||
<td>what was actually called, in what order</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>Two passes, because <code>ast</code> resolves nothing</h3>
|
||||
<p>
|
||||
Given <code>class User(Base)</code>, Python's <code>ast</code> hands over the
|
||||
literal string <code>"Base"</code>. It has no idea that came from
|
||||
<code>from .db import Base</code> three lines up. So pass one collects, per
|
||||
module, what it defines and what it imports; pass two resolves local names to
|
||||
fully qualified ids. The edge then points at <code>app.db.Base</code> — a real
|
||||
node — rather than at a box called <code>Base</code> that means nothing.
|
||||
</p>
|
||||
|
||||
<div class="note">
|
||||
<b>Unresolved names become nodes, never nothing.</b> A third-party import or a
|
||||
dynamically-built base becomes a node of <code>kind: "external"</code> and
|
||||
<em>keeps its edge</em>. Dropping it would be the worse failure: the diagram
|
||||
would look complete and have quietly lost a dependency. Gathered up, those
|
||||
nodes are the project's real dependency surface.
|
||||
</div>
|
||||
|
||||
<h3>C# and TypeScript</h3>
|
||||
<p>
|
||||
Handled by tree-sitter, which is why generics, nested types and a brace inside
|
||||
a string are non-events rather than special cases. The test suite asserts that
|
||||
last one specifically, because it is exactly where a hand-rolled scanner breaks.
|
||||
</p>
|
||||
<p>
|
||||
This reader produces <strong>no edges</strong>. Resolving a C#
|
||||
<code>using</code> to the thing it names is a different and much larger job,
|
||||
and the consumer that needs this — the minimap — needs none of it. An
|
||||
extractor that quietly produced half a dependency graph would be worse than one
|
||||
producing none, because the half would look whole.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="usage">Usage, not just the spec</h2>
|
||||
|
||||
<p>
|
||||
An OpenAPI document says what endpoints <em>are</em>. It does not say how to
|
||||
use them — least of all when they are not RESTful, or when a GraphQL endpoint
|
||||
sits alongside. So docgen also reads a <b>HAR</b>: the recording format that
|
||||
browser devtools, mitmproxy, Charles and Insomnia all export.
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<tr><th>what traffic knows</th><th>what a spec cannot</th></tr>
|
||||
<tr><td>the <b>order</b> of calls</td><td>a spec is a set; usage is a sequence</td></tr>
|
||||
<tr><td>which parameters are <b>always</b> sent</td><td>a spec lists twenty optional ones</td></tr>
|
||||
<tr><td>which statuses <b>really</b> happen</td><td>the 422 everybody hits is in no document</td></tr>
|
||||
<tr><td>endpoints not in the document</td><td>GraphQL operations, found by body shape and named</td></tr>
|
||||
<tr><td>which id formats a route takes</td><td>numeric <em>and</em> uuid on one route</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="note warn">
|
||||
<b>No credential and no payload value reaches the IR</b> — only field names and
|
||||
types. A HAR is full of live bearer tokens and cookies, and a generated
|
||||
document gets committed. The test suite plants a token in its fixture and fails
|
||||
if it appears anywhere in the output.
|
||||
</div>
|
||||
|
||||
<p>
|
||||
Two limits, stated rather than glossed: path templating is a <em>guess</em>
|
||||
(<code>attrs.observed_paths</code> keeps what was actually seen beside it), and
|
||||
consecutive is not caused-by — the edge weight is what separates a habit from
|
||||
an accident, and one recording will not tell you which.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="views">Views</h2>
|
||||
|
||||
<p>
|
||||
The first real diagram out of this pipeline was a 3000px-wide strip: four
|
||||
modules of actual content and sixty <code>sys</code>/<code>json</code>/<code>typing</code>
|
||||
boxes as their peers. The emitter was correct and the picture was useless.
|
||||
That is a <em>missing view</em>, not a broken renderer — and the fix belongs to
|
||||
every consumer at once, because the index, the diagram and the diff all want
|
||||
the same narrowing.
|
||||
</p>
|
||||
|
||||
<pre><code>python3 -m docgen.ops ir.json --overview -o view.json
|
||||
python3 -m docgen.ops ir.json --around docgen.ir --hops 2 -o view.json
|
||||
python3 -m docgen.ops ir.json --split -o parts/
|
||||
python3 -m docgen.ops ir.json --shape <span class="c"># what will this look like?</span></code></pre>
|
||||
|
||||
<p>
|
||||
All of them are IR→IR, all composable, and each produces a document that
|
||||
still validates. <code>--overview</code> is the default and dispatches on the
|
||||
source: a codebase reduces to its modules and its outside dependencies, a
|
||||
schema to its tables and their keys.
|
||||
</p>
|
||||
|
||||
<div class="note">
|
||||
<b>Edges are lifted when a view collapses detail, never dropped.</b> A class in
|
||||
module A inheriting from a class in module B <em>is</em> a dependency of A on
|
||||
B. Collapsing docgen to its packages once kept 8 of 77 edges — those pictures
|
||||
were not simpler, they were <em>wrong</em>. Lifted edges carry a
|
||||
<code>weight</code> saying how many they stand for.
|
||||
</div>
|
||||
|
||||
<h3>Depth is the tempting knob and the wrong one</h3>
|
||||
<p>
|
||||
A directory without an <code>__init__.py</code> is not a package, so its
|
||||
modules have no parent and sit at depth 0. soleprint has <b>173 such roots</b>,
|
||||
and a depth-2 cut still held 566 functions and 142 classes. Selecting by
|
||||
<code>kind</code> does not care how the directories happen to be arranged.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="emitters">Emitters</h2>
|
||||
|
||||
<table>
|
||||
<tr><th>emitter</th><th>output</th><th>audience</th></tr>
|
||||
<tr><td><code>index</code></td><td>markdown, sidebar JSON</td>
|
||||
<td><b>anyone</b> — no graph literacy required</td></tr>
|
||||
<tr><td><code>dot</code></td><td>DOT → Graphviz → SVG</td><td>dependency structure</td></tr>
|
||||
<tr><td><code>erd</code></td><td>SVG, written directly</td><td>a schema, as cards</td></tr>
|
||||
<tr><td><code>minimap</code></td><td>SVG, written directly</td><td>what is where, at a glance</td></tr>
|
||||
<tr><td><code>notebook</code></td><td><code>.ipynb</code></td><td>a runnable walkthrough</td></tr>
|
||||
<tr><td><code>site</code></td><td>a static docs site</td><td>reading</td></tr>
|
||||
<tr><td><code>explore</code></td><td>a two-pane navigator</td><td>finding your way</td></tr>
|
||||
<tr><td><code>auto</code></td><td>whichever of the above fits</td><td>not having to choose</td></tr>
|
||||
</table>
|
||||
|
||||
<p>
|
||||
<strong>The non-visual ones matter most for reach.</strong> A sorted, described
|
||||
list of what exists is readable by someone who will never open a diagram, and
|
||||
it also reports the dependency surface and any file that failed to parse.
|
||||
It is built second, not last — it is what proves the IR is not secretly
|
||||
diagram-shaped.
|
||||
</p>
|
||||
|
||||
<h3>ERD — and where the layout came from</h3>
|
||||
<p>
|
||||
Not invented here. <code>station/tools/graphgen/templates/index.html</code>,
|
||||
the Supabase-style schema explorer already in this repo, had solved it:
|
||||
</p>
|
||||
<pre><code>const cols = Math.max(2, Math.ceil(Math.sqrt(sorted.length * 1.2)));</code></pre>
|
||||
<p>
|
||||
<strong>Columns from the square root of the table count.</strong> The aspect
|
||||
ratio is <em>chosen</em> rather than emergent, so the result stays near-square
|
||||
at 4 tables or 400. That is the one thing a rank-based engine cannot offer.
|
||||
Three more things it gets right: a table is a <em>card</em> with its columns;
|
||||
an edge leaves the column holding the key and lands on the target's primary
|
||||
key; and the geometry is computed rather than measured, so it renders
|
||||
identically on any machine.
|
||||
</p>
|
||||
|
||||
<figure>
|
||||
<a href="viewer.html?src=img/erd.svg">
|
||||
<img src="img/erd.svg" alt="an entity-relationship diagram">
|
||||
</a>
|
||||
<figcaption>A schema from the sample room. Same emitter, same style file as
|
||||
every other diagram here.</figcaption>
|
||||
</figure>
|
||||
|
||||
<h3>Minimap</h3>
|
||||
<p>
|
||||
Sublime's minimap shrinks the <em>characters</em>. This draws the
|
||||
<em>structure</em> at full scale: one file is a column, one line is a fixed
|
||||
number of pixels, every construct a block sized by its span and coloured by
|
||||
what it is. No text inside a block — the shape is the message.
|
||||
</p>
|
||||
<p>
|
||||
The claim is that the pattern comes from the colours alone, so nesting is drawn
|
||||
by inset rather than by hue. On soleprint you can see that modelgen is
|
||||
class-based, histgen is function-based and tester is mixed, without reading a
|
||||
line.
|
||||
</p>
|
||||
|
||||
<figure>
|
||||
<a href="viewer.html?src=img/minimap.svg">
|
||||
<img src="img/minimap.svg" alt="a structural minimap of docgen">
|
||||
</a>
|
||||
<figcaption>docgen's own files. Blue class, amber interface, green function,
|
||||
dark for everything that is not a declaration — imports, constants, prose.</figcaption>
|
||||
</figure>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="explore">Explore</h2>
|
||||
|
||||
<p>
|
||||
The minimap on its own shows shape and no meaning: a block says "a 30-line
|
||||
class", not <em>which</em> class or what it touches. So it is not the artifact.
|
||||
It is the <strong>selector</strong>.
|
||||
</p>
|
||||
|
||||
<pre><code>make explore OUT=out <span class="c"># then open out/explore/explore.html</span></code></pre>
|
||||
|
||||
<div class="cols">
|
||||
<div>
|
||||
<h4>Left — navigate</h4>
|
||||
<p>The whole thing at once. Scan by colour, click a block.</p>
|
||||
</div>
|
||||
<div>
|
||||
<h4>Right — explore</h4>
|
||||
<p>What that is, what it reaches, what reaches it, and the neighbourhood
|
||||
drawn small enough to read. Every neighbour is a link, so you walk
|
||||
outward from wherever you started.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="note">
|
||||
<b>This is what retires the 14:1 sheet.</b> The whole graph is never drawn. The
|
||||
overview pane carries the overview, and only the neighbourhood of a selection
|
||||
is rendered — a handful of nodes, which lays out fine every time. Overview and
|
||||
detail stop competing for one picture.
|
||||
</div>
|
||||
|
||||
<p>
|
||||
The same split applies to a database: every table at once with <em>no column
|
||||
detail</em>, then click one to get its columns plus the tables its keys reach.
|
||||
The two differ exactly where they should — a module's neighbourhood
|
||||
deliberately leaves its contents out, because those are the hundred functions
|
||||
that made the sheet unreadable, while a table's brings them in, because a table
|
||||
without its columns is not a table.
|
||||
</p>
|
||||
|
||||
<h3>The selection basket</h3>
|
||||
<p>
|
||||
Shift-click accumulates blocks. The basket is a copyable list of paths with a
|
||||
line count — enough to hand to <code>distill</code>, and enough to see that the
|
||||
selection got too big <em>before</em> spending the context on it. Navigating a
|
||||
tree quickly in order to decide what to feed a model is a real use, and this is
|
||||
the part that serves it.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="notebooks">Notebooks</h2>
|
||||
|
||||
<p>
|
||||
A notebook is normally a source file somebody confects by hand: prose, code and
|
||||
stored output braided together, diffing badly, drifting from whatever it
|
||||
documents the moment either moves, with no way to tell by looking.
|
||||
</p>
|
||||
<p>
|
||||
Here a notebook is a <strong>build artifact</strong>. The source is the OpenAPI
|
||||
document — the same file the server is built from — and the notebook is
|
||||
regenerated from it. Nobody edits the <code>.ipynb</code>, the same way nobody
|
||||
edits a <code>.o</code>. "Is this document current" stops being a question
|
||||
about somebody's diligence and becomes a question about whether the build ran.
|
||||
</p>
|
||||
|
||||
<blockquote>
|
||||
This is the disagreement with jupytext. Jupytext fixes the <em>diffing</em> —
|
||||
it makes a notebook editable as text — and leaves the actual problem: you still
|
||||
hand-author it, so it still rots.
|
||||
</blockquote>
|
||||
|
||||
<h3>Generated base, hand-written overlay</h3>
|
||||
<p>
|
||||
Generation alone gives a document that is never stale and never says anything a
|
||||
parser could not work out. Hand-authoring alone gives insight and a document
|
||||
that rots. Two files is the only arrangement that gets both:
|
||||
</p>
|
||||
|
||||
<pre><code>IR ──► spec ──(+ overlay)──► merged spec ──► .ipynb
|
||||
generated hand-written merged emitted</code></pre>
|
||||
|
||||
<p>
|
||||
The <b>spec</b> is an ordered list of steps with no Jupyter in it — a Swagger
|
||||
for notebooks, readable and diffable. The <b>overlay</b> is the only file
|
||||
anyone edits, and it is re-applied on every build. It can
|
||||
<code>annotate</code>, <code>replace</code>, <code>insert</code>,
|
||||
<code>drop</code> and <code>order</code>.
|
||||
</p>
|
||||
<p>
|
||||
<code>replace</code> is the one that matters. It is how real usage gets into a
|
||||
document that a spec could not describe — the call that is always made with
|
||||
<code>status=available</code>, the GraphQL endpoint that is not in the OpenAPI
|
||||
file at all — and it keeps working unchanged once a usage recording supplies
|
||||
the same facts automatically.
|
||||
</p>
|
||||
|
||||
<div class="note">
|
||||
Three properties hold it together: regenerating <b>re-applies the overlay
|
||||
byte-for-byte</b>; when the base moves underneath it the mismatch is
|
||||
<b>reported, never silently dropped</b>; and extraction works with the overlay
|
||||
<b>absent</b> — it is an addition, never a dependency.
|
||||
</div>
|
||||
|
||||
<pre><code>python3 -m docgen.emitters notebook ir.json --scaffold overlay.json
|
||||
python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthrough.ipynb</code></pre>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="style">Style & colour</h2>
|
||||
|
||||
<p>
|
||||
A style rule names a <strong>slot</strong>, never a colour.
|
||||
<code>"border": "atlas"</code> is the rule; a theme binds <code>atlas</code> to
|
||||
<code>#43A047</code> in print and <code>#15803d</code> on the docs site.
|
||||
</p>
|
||||
<p>
|
||||
That indirection is the whole point. <code>common/theme/tokens.css</code>,
|
||||
<code>docs/graphs/themes/*.gvpr</code> and <code>style/lucid.json</code> use
|
||||
the same slot names, so a diagram and the page around it match by construction
|
||||
— which is the rule <code>docs/graphs/README.md</code> already states. The dark
|
||||
theme's <code>artery</code>, <code>atlas</code> and <code>station</code> slots
|
||||
are exactly the <code>--system-accent</code> values the three system pages set,
|
||||
and the test suite fails if they drift apart.
|
||||
</p>
|
||||
<p>
|
||||
Dark is the default, because a generated diagram lands in a dark docs page far
|
||||
more often than in a document. <code>--theme lucid</code> gives the print
|
||||
palette — and gives it to the <em>page</em> as well as the diagram, since both
|
||||
are baked from the same slots.
|
||||
</p>
|
||||
|
||||
<div class="note">
|
||||
An unknown <code>kind</code> falls back to <code>default</code> rather than
|
||||
crashing. That matters more than it sounds: a new extractor with a new
|
||||
vocabulary renders plainly and legibly on day one, instead of requiring
|
||||
somebody to write a style file before they can see anything.
|
||||
</div>
|
||||
|
||||
<h3>Where DOT stops</h3>
|
||||
<p>
|
||||
The emitter writes what DOT expresses natively and stops at the boundary rather
|
||||
than growing machinery. The limits are recorded in the style file itself: a
|
||||
cluster has a label and a fill but not a header bar;
|
||||
<code>stroke-dasharray</code> is not parameterised, so <code>4,4</code> and
|
||||
<code>5,5</code> collapse; <code>rounded</code> is binary, so 4px and 6px are
|
||||
identical. Those mark where a richer emitter would begin — and the style file
|
||||
carries the full specification regardless, so that emitter needs no
|
||||
re-authoring.
|
||||
</p>
|
||||
<p>
|
||||
One limit <em>was</em> worth solving: DOT cannot use a cluster as an edge
|
||||
endpoint, so every module-to-module import silently vanished. The native answer
|
||||
is <code>compound=true</code> with <code>lhead</code>/<code>ltail</code> — draw
|
||||
between a representative leaf and clip the line at the cluster border.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="commands">Commands</h2>
|
||||
|
||||
<h3>Make</h3>
|
||||
<table>
|
||||
<tr><th>target</th><th>does</th></tr>
|
||||
<tr><td><code>make check</code></td><td>the whole test suite, offline, nothing installed</td></tr>
|
||||
<tr><td><code>make doctor</code></td><td>what this machine has and what it is missing</td></tr>
|
||||
<tr><td><code>make ir SRC=…</code></td><td>extract Python into <code>OUT/ir.json</code></td></tr>
|
||||
<tr><td><code>make code SRC=…</code></td><td>extract C#/TypeScript <span class="pill opt">tree-sitter</span></td></tr>
|
||||
<tr><td><code>make db SCHEMA=…</code></td><td>extract a database schema</td></tr>
|
||||
<tr><td><code>make view</code></td><td>the default view for that source type</td></tr>
|
||||
<tr><td><code>make graph</code></td><td>draw whatever the structure asks for</td></tr>
|
||||
<tr><td><code>make index</code></td><td>markdown index and sidebar JSON</td></tr>
|
||||
<tr><td><code>make minimap</code></td><td>what is where, read from the colours</td></tr>
|
||||
<tr><td><code>make explore</code></td><td>the two-pane navigator</td></tr>
|
||||
<tr><td><code>make site</code></td><td>a self-contained docs site</td></tr>
|
||||
<tr><td><code>make self</code></td><td>the whole pipeline over soleprint itself</td></tr>
|
||||
</table>
|
||||
|
||||
<p>
|
||||
Variables: <code>SRC</code>, <code>OUT</code>, <code>SCHEMA</code>,
|
||||
<code>STYLE</code>, <code>THEME</code>, <code>SCALE</code>, <code>DEPTH</code>,
|
||||
<code>PY</code>. The Makefile derives its own package name from where it sits,
|
||||
so the folder can be copied anywhere and renamed and still work.
|
||||
</p>
|
||||
|
||||
<h3>Modules</h3>
|
||||
<pre><code>python3 -m docgen.extractors.python --root SRC -o ir.json
|
||||
python3 -m docgen.extractors code --root SRC -o ir.json
|
||||
python3 -m docgen.extractors db --schema schema.json -o ir.json
|
||||
python3 -m docgen.extractors openapi --spec spec.yaml -o ir.json
|
||||
python3 -m docgen.extractors usage --har session.har -o ir.json
|
||||
|
||||
python3 -m docgen.ir ir.json <span class="c"># validate</span>
|
||||
python3 -m docgen.ops ir.json --overview -o view.json
|
||||
|
||||
python3 -m docgen.emitters auto view.json -o out/
|
||||
python3 -m docgen.emitters index ir.json -o index.md
|
||||
python3 -m docgen.emitters dot view.json -o graph.svg --theme lucid
|
||||
python3 -m docgen.emitters erd ir.json -o schema.svg
|
||||
python3 -m docgen.emitters minimap ir.json -o map.svg --scale 0.5
|
||||
python3 -m docgen.emitters notebook ir.json -o book.ipynb --overlay overlay.json
|
||||
python3 -m docgen.emitters site view.json -o site/
|
||||
python3 -m docgen.emitters explore ir.json -o explore/</code></pre>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="deps">Dependencies</h2>
|
||||
|
||||
<p>
|
||||
The core is <strong>standard library only</strong>. Everything else is optional
|
||||
and reported by <code>make doctor</code>; when something is missing you lose
|
||||
exactly one capability and get told what to install.
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<tr><th>needs</th><th>for</th><th>without it</th></tr>
|
||||
<tr><td><code>graphviz</code> (binary)</td><td>rendering DOT to SVG</td>
|
||||
<td>ERD, minimap, index and notebooks still work</td></tr>
|
||||
<tr><td><code>tree_sitter</code> + grammars</td><td>C#, TypeScript, TSX</td>
|
||||
<td>Python only</td></tr>
|
||||
<tr><td><code>networkx</code></td><td><code>lab/</code> experiments</td>
|
||||
<td>nothing — nothing depends on it yet</td></tr>
|
||||
<tr><td><code>node</code></td><td>testing the browser pages</td>
|
||||
<td>those checks skip</td></tr>
|
||||
<tr><td><code>psql</code></td><td>the <code>lab/</code> schema probe</td>
|
||||
<td>use modelgen's <code>from-db</code> instead</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="note">
|
||||
<b><code>lab/</code> is where a dependency gets tried before anything depends
|
||||
on it.</b> Nothing in <code>ir/</code>, <code>extractors/</code>,
|
||||
<code>ops/</code> or <code>emitters/</code> may import from it. When an
|
||||
experiment earns its place it graduates into <code>ops/</code> behind an
|
||||
IR→IR signature, and <em>then</em> the dependency is declared.
|
||||
</div>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="testing">Testing</h2>
|
||||
|
||||
<pre><code>make check <span class="c"># 191 checks, offline, no network</span></code></pre>
|
||||
|
||||
<p>
|
||||
Four of those are the <em>design</em> rather than regressions, and they are the
|
||||
ones to keep if anything is ever cut:
|
||||
</p>
|
||||
<ul>
|
||||
<li><b>No visual field reaches the IR</b> — extractors cannot decide appearance.</li>
|
||||
<li><b>No emitter reads a source file</b> — the layering, checked from the other side by parsing imports.</li>
|
||||
<li><b>Style names slots, not colours</b> — one colour language rather than three.</li>
|
||||
<li><b>Ids are stable across runs</b> — without it, diffing is noise.</li>
|
||||
</ul>
|
||||
|
||||
<div class="note">
|
||||
<b>Golden tests go on the IR, never on the SVG.</b> Graphviz measures label
|
||||
text with the host's fonts to size nodes, so identical input produces different
|
||||
geometry on a machine with different fontconfig. The IR is deterministic; the
|
||||
SVG is not. Pinning the wrong one gives a suite that fails on somebody else's
|
||||
laptop for no reason anyone can act on.
|
||||
</div>
|
||||
|
||||
<p>
|
||||
The browser pages are JavaScript, so they are tested as JavaScript: a stub DOM
|
||||
under <code>node</code> drives the viewer's zoom and 1:1 toggle, and the
|
||||
explorer's select-and-walk. Both skip cleanly where node is absent.
|
||||
</p>
|
||||
<p>
|
||||
Self-hosting is the honest end-to-end check, and it is where the real bugs came
|
||||
from — two name-resolution faults and a duplicate-id crash that no fixture had
|
||||
reached. <code>make self</code> runs the whole pipeline over soleprint; if the
|
||||
index does not read like the system, something is wrong.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="limits">Limits & non-goals</h2>
|
||||
|
||||
<p>Things deliberately not done, with the reason, so they are not re-litigated:</p>
|
||||
|
||||
<ul>
|
||||
<li><b>No layout engine.</b> No positioning, no <code>neato -n2</code>, no ELK.
|
||||
Aspect ratio was solved by choosing the right emitter and by not drawing
|
||||
everything at once.</li>
|
||||
<li><b>No <code>calls</code> edges.</b> Resolving <code>self.foo()</code> needs
|
||||
type inference, and a call graph that is quietly 60% right is worse than
|
||||
none because it reads as authoritative.</li>
|
||||
<li><b>No edges from the C#/TypeScript reader.</b> Structure only — half a
|
||||
dependency graph would look whole.</li>
|
||||
<li><b>No model in the structural path.</b> Annotation — summarising a module,
|
||||
naming a cluster — is a later layer, cached to its own file keyed by node
|
||||
id, merged at emit time, and extraction must work with it absent.</li>
|
||||
<li><b>No configuration knobs</b> until two real consumers disagree.</li>
|
||||
</ul>
|
||||
|
||||
<p>Known gaps, stated plainly:</p>
|
||||
<ul>
|
||||
<li>The C# reader is verified against a written fixture, <b>not a real
|
||||
repository</b>. That is the next check that matters.</li>
|
||||
<li>The diff emitter is not built. Stable ids exist to make it possible, and
|
||||
two IRs from two commits is the cheapest useful thing left.</li>
|
||||
<li>Minimap blocks carry no names. Rendering them legible only at 1:1, through
|
||||
the viewer, is the other half of reading a file without reading every line.</li>
|
||||
</ul>
|
||||
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// Highlight the section being read. No dependency, no build step.
|
||||
var links = [].slice.call(document.querySelectorAll('.sidebar a[href^="#"]'));
|
||||
var byId = {};
|
||||
links.forEach(function (a) { byId[a.getAttribute('href').slice(1)] = a; });
|
||||
var observer = new IntersectionObserver(function (entries) {
|
||||
entries.forEach(function (entry) {
|
||||
var a = byId[entry.target.id];
|
||||
if (a && entry.isIntersecting) {
|
||||
links.forEach(function (l) { l.classList.remove('active'); });
|
||||
a.classList.add('active');
|
||||
}
|
||||
});
|
||||
}, { rootMargin: '-8% 0px -82% 0px' });
|
||||
document.querySelectorAll('h2[id]').forEach(function (h) { observer.observe(h); });
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user