docgen

Turn source artifacts — a codebase, a database, an API spec, a recording of real traffic — into one canonical graph format, then render that format to whatever the audience needs. The point is not the diagram. The point is the format in the middle.

What it is

Eight demos under semester/ draw their architecture with Graphviz. Every one of them hand-writes a .dot file with the palette inlined and commits the .svg beside it. Three different dark palettes between them, eight answers to the same question, and every one of those diagrams is out of date the moment somebody moves a file.

docgen is the one answer. It reads the source rather than being told about it, so a diagram cannot drift from the thing it describes, and it separates what a graph is from how it looks so one extraction feeds a diagram, an index, a notebook and a browsable site without being redone.

The idea

N sources and M outputs need N×M converters if you join them directly, or N+M if you put a hub in the middle. The hub is an intermediate representation — the compiler term, and the same bargain: both sides depend on the IR and neither on the other.

It is lossy on purpose. It throws away every token of syntax and keeps "a class named User inherits from Base". That is the part that is worth versioning, worth diffing, and worth drawing.

The practical consequence is the thing to judge it on: adding a source costs one extractor and every emitter works on it unchanged; adding an output costs one emitter and every extractor feeds it unchanged. When the OpenAPI reader was written it emitted schemas using the same vocabulary the database reader uses — and the ER diagram drew an API's data model without anyone teaching it what an API was.

Five minutes

Three commands, and they compose. That is the whole interface.

# 1. read something
python3 -m docgen.extractors.python --root ../station/tools/histgen -o ir.json

# 2. narrow it to a useful view
python3 -m docgen.ops ir.json --overview -o view.json

# 3. draw whatever its structure asks for
python3 -m docgen.emitters auto view.json -o out/

Or through the Makefile, which is a thin wrapper over exactly those:

make ir SRC=/path/to/repo OUT=out   # extract
make explore OUT=out                # the two-pane navigator
make site OUT=out                   # a docs site with a sidebar
make self                           # run the whole thing over soleprint

Everything is offline and self-contained. No server, no CDN, no build step — the outputs open over file://.

The three concerns

DOT collapses three separate questions into one file format, which is why a hand-written .dot is never reusable: you cannot change the palette without editing the structure, and you cannot change the structure without re-deciding the layout. docgen keeps them apart.

An extractor has never heard of SVG, colours or layout. An emitter has never heard of Python, ast or SQL. Both halves of that are checked by parsing the source and looking at what it imports, because a rule nobody enforces is a rule that lasts about a month.

The IR

Plain JSON. Three keys, and it has survived four domains without gaining a fourth.

{
  "meta":  { "source": "python", "root": "app/", "schema_version": "1" },
  "nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
               "parent": "app.models",
               "attrs": { "file": "app/models.py", "line": 12, "lines": 40 } } ],
  "edges": [ { "source": "app.models.User", "target": "app.db.Base",
               "kind": "inherits", "attrs": {} } ]
}

Stdlib dataclasses, not Pydantic

The IR's whole value is being a plain document anything can open. A format that needs a library installed to be read is an API, not a format. Validation is therefore a function called at the boundary rather than a property of the type, and it reads its field lists out of schema.json so the schema and the dataclasses cannot drift apart.

python3 -m docgen.ir ir.json

It catches what a schema cannot: an edge naming a node that does not exist, a containment cycle, a duplicate id, and a visual field smuggled into attrs.

Shape decides the drawing

A diagram that fights its layout engine is usually the wrong kind of diagram. The clearest evidence: the same 24-table database rendered 32034×136 through Graphviz — a 235:1 strip — and 1740×1860 through the ER emitter. Not because one engine is better, but because a schema is a set of peer entities with references, and laying it out in dependency ranks was never its shape.

So ops.classify() reads the structure and names the emitter, with the reason attached — advice without a reason gets overridden the first time it is inconvenient.

$ python3 -m docgen.emitters auto view.json -o out/
  sheet    -> index
           109 nodes sit at one level; any layered engine draws that as a
           strip. Split it, scope it, or read it as an index

Why twenty

Measured, one diagram per subsystem: at or under 20 nodes the output lands around 1.6:1; at 70–106 nodes about 7:1; at 261 nodes 14:1. Aspect ratio is a property of the graph, not of the renderer — a layered engine puts one dependency level in one row, so the widest level is the width.

Every Graphviz lever was tried before concluding this. ratio=compress squashed a graph to an unreadable 1008×75; rankdir=LR merely rotated a 14:1 into a 1:6; packing disconnected components gained nothing. The fix was never a flag. It was to stop asking for one picture of everything — which is what explore does.

Extractors

Deterministic parsing only. No model in the structural path. A diagram built from an AST cannot be out of date with the code. A diagram built from a model's reading of the code is wrong the moment the model has a bad day — which is the problem this exists to fix.

Two passes, because ast resolves nothing

Given class User(Base), Python's ast hands over the literal string "Base". It has no idea that came from from .db import Base three lines up. So pass one collects, per module, what it defines and what it imports; pass two resolves local names to fully qualified ids. The edge then points at app.db.Base — a real node — rather than at a box called Base that means nothing.

C# and TypeScript

Handled by tree-sitter, which is why generics, nested types and a brace inside a string are non-events rather than special cases. The test suite asserts that last one specifically, because it is exactly where a hand-rolled scanner breaks.

This reader produces no edges. Resolving a C# using to the thing it names is a different and much larger job, and the consumer that needs this — the minimap — needs none of it. An extractor that quietly produced half a dependency graph would be worse than one producing none, because the half would look whole.

Usage, not just the spec

An OpenAPI document says what endpoints are. It does not say how to use them — least of all when they are not RESTful, or when a GraphQL endpoint sits alongside. So docgen also reads a HAR: the recording format that browser devtools, mitmproxy, Charles and Insomnia all export.

Two limits, stated rather than glossed: path templating is a guess (attrs.observed_paths keeps what was actually seen beside it), and consecutive is not caused-by — the edge weight is what separates a habit from an accident, and one recording will not tell you which.

Views

The first real diagram out of this pipeline was a 3000px-wide strip: four modules of actual content and sixty sys/json/typing boxes as their peers. The emitter was correct and the picture was useless. That is a missing view, not a broken renderer — and the fix belongs to every consumer at once, because the index, the diagram and the diff all want the same narrowing.

python3 -m docgen.ops ir.json --overview -o view.json
python3 -m docgen.ops ir.json --around docgen.ir --hops 2 -o view.json
python3 -m docgen.ops ir.json --split -o parts/
python3 -m docgen.ops ir.json --shape          # what will this look like?

All of them are IR→IR, all composable, and each produces a document that still validates. --overview is the default and dispatches on the source: a codebase reduces to its modules and its outside dependencies, a schema to its tables and their keys.

Depth is the tempting knob and the wrong one

A directory without an __init__.py is not a package, so its modules have no parent and sit at depth 0. soleprint has 173 such roots, and a depth-2 cut still held 566 functions and 142 classes. Selecting by kind does not care how the directories happen to be arranged.

Emitters

The non-visual ones matter most for reach. A sorted, described list of what exists is readable by someone who will never open a diagram, and it also reports the dependency surface and any file that failed to parse. It is built second, not last — it is what proves the IR is not secretly diagram-shaped.

ERD — and where the layout came from

Not invented here. station/tools/graphgen/templates/index.html, the Supabase-style schema explorer already in this repo, had solved it:

const cols = Math.max(2, Math.ceil(Math.sqrt(sorted.length * 1.2)));

Columns from the square root of the table count. The aspect ratio is chosen rather than emergent, so the result stays near-square at 4 tables or 400. That is the one thing a rank-based engine cannot offer. Three more things it gets right: a table is a card with its columns; an edge leaves the column holding the key and lands on the target's primary key; and the geometry is computed rather than measured, so it renders identically on any machine.

Minimap

Sublime's minimap shrinks the characters. This draws the structure at full scale: one file is a column, one line is a fixed number of pixels, every construct a block sized by its span and coloured by what it is. No text inside a block — the shape is the message.

The claim is that the pattern comes from the colours alone, so nesting is drawn by inset rather than by hue. On soleprint you can see that modelgen is class-based, histgen is function-based and tester is mixed, without reading a line.

Explore

The minimap on its own shows shape and no meaning: a block says "a 30-line class", not which class or what it touches. So it is not the artifact. It is the selector.

make explore OUT=out     # then open out/explore/explore.html

The same split applies to a database: every table at once with no column detail, then click one to get its columns plus the tables its keys reach. The two differ exactly where they should — a module's neighbourhood deliberately leaves its contents out, because those are the hundred functions that made the sheet unreadable, while a table's brings them in, because a table without its columns is not a table.

The selection basket

Shift-click accumulates blocks. The basket is a copyable list of paths with a line count — enough to hand to distill, and enough to see that the selection got too big before spending the context on it. Navigating a tree quickly in order to decide what to feed a model is a real use, and this is the part that serves it.

Notebooks

A notebook is normally a source file somebody confects by hand: prose, code and stored output braided together, diffing badly, drifting from whatever it documents the moment either moves, with no way to tell by looking.

Here a notebook is a build artifact. The source is the OpenAPI document — the same file the server is built from — and the notebook is regenerated from it. Nobody edits the .ipynb, the same way nobody edits a .o. "Is this document current" stops being a question about somebody's diligence and becomes a question about whether the build ran.

This is the disagreement with jupytext. Jupytext fixes the diffing — it makes a notebook editable as text — and leaves the actual problem: you still hand-author it, so it still rots.

Generated base, hand-written overlay

Generation alone gives a document that is never stale and never says anything a parser could not work out. Hand-authoring alone gives insight and a document that rots. Two files is the only arrangement that gets both:

IR ──► spec ──(+ overlay)──► merged spec ──► .ipynb
   generated   hand-written       merged       emitted

The spec is an ordered list of steps with no Jupyter in it — a Swagger for notebooks, readable and diffable. The overlay is the only file anyone edits, and it is re-applied on every build. It can annotate, replace, insert, drop and order.

replace is the one that matters. It is how real usage gets into a document that a spec could not describe — the call that is always made with status=available, the GraphQL endpoint that is not in the OpenAPI file at all — and it keeps working unchanged once a usage recording supplies the same facts automatically.

python3 -m docgen.emitters notebook ir.json --scaffold overlay.json
python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthrough.ipynb

Style & colour

A style rule names a slot, never a colour. "border": "atlas" is the rule; a theme binds atlas to #43A047 in print and #15803d on the docs site.

That indirection is the whole point. common/theme/tokens.css, docs/graphs/themes/*.gvpr and style/lucid.json use the same slot names, so a diagram and the page around it match by construction — which is the rule docs/graphs/README.md already states. The dark theme's artery, atlas and station slots are exactly the --system-accent values the three system pages set, and the test suite fails if they drift apart.

Dark is the default, because a generated diagram lands in a dark docs page far more often than in a document. --theme lucid gives the print palette — and gives it to the page as well as the diagram, since both are baked from the same slots.

Where DOT stops

The emitter writes what DOT expresses natively and stops at the boundary rather than growing machinery. The limits are recorded in the style file itself: a cluster has a label and a fill but not a header bar; stroke-dasharray is not parameterised, so 4,4 and 5,5 collapse; rounded is binary, so 4px and 6px are identical. Those mark where a richer emitter would begin — and the style file carries the full specification regardless, so that emitter needs no re-authoring.

One limit was worth solving: DOT cannot use a cluster as an edge endpoint, so every module-to-module import silently vanished. The native answer is compound=true with lhead/ltail — draw between a representative leaf and clip the line at the cluster border.

Commands

Make

Variables: SRC, OUT, SCHEMA, STYLE, THEME, SCALE, DEPTH, PY. The Makefile derives its own package name from where it sits, so the folder can be copied anywhere and renamed and still work.

Modules

python3 -m docgen.extractors.python --root SRC -o ir.json
python3 -m docgen.extractors code    --root SRC -o ir.json
python3 -m docgen.extractors db      --schema schema.json -o ir.json
python3 -m docgen.extractors openapi --spec spec.yaml -o ir.json
python3 -m docgen.extractors usage   --har session.har -o ir.json

python3 -m docgen.ir ir.json                      # validate
python3 -m docgen.ops ir.json --overview -o view.json

python3 -m docgen.emitters auto     view.json -o out/
python3 -m docgen.emitters index    ir.json -o index.md
python3 -m docgen.emitters dot      view.json -o graph.svg --theme lucid
python3 -m docgen.emitters erd      ir.json -o schema.svg
python3 -m docgen.emitters minimap  ir.json -o map.svg --scale 0.5
python3 -m docgen.emitters notebook ir.json -o book.ipynb --overlay overlay.json
python3 -m docgen.emitters site     view.json -o site/
python3 -m docgen.emitters explore  ir.json -o explore/

Dependencies

The core is standard library only. Everything else is optional and reported by make doctor; when something is missing you lose exactly one capability and get told what to install.

Testing

make check      # 191 checks, offline, no network

Four of those are the design rather than regressions, and they are the ones to keep if anything is ever cut:

• No visual field reaches the IR — extractors cannot decide appearance.
• No emitter reads a source file — the layering, checked from the other side by parsing imports.
• Style names slots, not colours — one colour language rather than three.
• Ids are stable across runs — without it, diffing is noise.

The browser pages are JavaScript, so they are tested as JavaScript: a stub DOM under node drives the viewer's zoom and 1:1 toggle, and the explorer's select-and-walk. Both skip cleanly where node is absent.

Self-hosting is the honest end-to-end check, and it is where the real bugs came from — two name-resolution faults and a duplicate-id crash that no fixture had reached. make self runs the whole pipeline over soleprint; if the index does not read like the system, something is wrong.

Limits & non-goals

Things deliberately not done, with the reason, so they are not re-litigated:

• No layout engine. No positioning, no neato -n2, no ELK. Aspect ratio was solved by choosing the right emitter and by not drawing everything at once.
• No calls edges. Resolving self.foo() needs type inference, and a call graph that is quietly 60% right is worse than none because it reads as authoritative.
• No edges from the C#/TypeScript reader. Structure only — half a dependency graph would look whole.
• No model in the structural path. Annotation — summarising a module, naming a cluster — is a later layer, cached to its own file keyed by node id, merged at emit time, and extraction must work with it absent.
• No configuration knobs until two real consumers disagree.

Known gaps, stated plainly:

• The C# reader is verified against a written fixture, not a real repository. That is the next check that matters.
• The diff emitter is not built. Stable ids exist to make it possible, and two IRs from two commits is the cheapest useful thing left.
• Minimap blocks carry no names. Rendering them legible only at 1:1, through the viewer, is the other half of reading a file without reading every line.