docgen

Turn source artifacts — a codebase, a database, an API spec, a recording of real traffic — into one canonical graph format, then render that format to whatever the audience needs. The point is not the diagram. The point is the format in the middle.

What it is

Eight demos under semester/ draw their architecture with Graphviz. Every one of them hand-writes a .dot file with the palette inlined and commits the .svg beside it. Three different dark palettes between them, eight answers to the same question, and every one of those diagrams is out of date the moment somebody moves a file.

docgen is the one answer. It reads the source rather than being told about it, so a diagram cannot drift from the thing it describes, and it separates what a graph is from how it looks so one extraction feeds a diagram, an index, a notebook and a browsable site without being redone.

The idea

N sources and M outputs need N×M converters if you join them directly, or N+M if you put a hub in the middle. The hub is an intermediate representation — the compiler term, and the same bargain: both sides depend on the IR and neither on the other.

It is lossy on purpose. It throws away every token of syntax and keeps "a class named User inherits from Base". That is the part that is worth versioning, worth diffing, and worth drawing.

The practical consequence is the thing to judge it on: adding a source costs one extractor and every emitter works on it unchanged; adding an output costs one emitter and every extractor feeds it unchanged. When the OpenAPI reader was written it emitted schemas using the same vocabulary the database reader uses — and the ER diagram drew an API's data model without anyone teaching it what an API was.

Five minutes

One command, if you want the whole thing — a book: what went in, every step, what came out, and a page to open.

make book SRC=/path/to/repo BOOK=out/book/mine
make check BOOK=out/book/mine

More than one, rebuilt together, is a run file: make run CONFIG=docgen.toml.

  larder   /path/to/repo — 45 files read, 2 failed, 12 packages
  book     312 nodes · 244 edges · 26 external · 8 artifacts
  ok       45 file(s) read produced 45 module(s)
  open     out/book/mine/site/index.html

Underneath it is three commands that compose, and each still works on its own. That is the interface, and the book does not replace it:

# 1. read something
python3 -m docgen extract python --root ../station/tools/histgen -o ir.json

# 2. narrow it to a useful view
python3 -m docgen view ir.json --overview -o view.json

# 3. draw whatever its structure asks for
python3 -m docgen emit auto view.json -o out/

Or through the Makefile, which is a thin wrapper over exactly those:

make ir SRC=/path/to/repo OUT=out   # extract
make explore OUT=out                # the two-pane navigator
make site OUT=out                   # a docs site with a sidebar
make self                           # docgen's book of soleprint, then check it

Everything is offline and self-contained. No server, no CDN, no build step — the outputs open over file://.

Your first project

From nothing to a notebook and a site for a project of yours, with your own text in the right places. docgen lives in its own folder, and the project only gets one config file and, later, a folder of notes.

1. Put docgen somewhere and check the machine

cp -r docgen ~/tools/docgen
cd ~/tools/docgen
make doctor          # what is installed; never fails
make sync            # optional: C#/TypeScript, OpenAPI, all from uv.lock

From anywhere else, docgen runs as python3 -m docgen with its parent folder on the path:

export PYTHONPATH=~/tools             # the folder that holds docgen/
python3 -m docgen --help

2. Write docgen.toml at the project's root

[project]
name = "orders-service"

[extract.code]
root = "src"
ext  = [".cs"]

[emit.notebook]
output = "../docs_out/orders_service_notebook"

[emit.site]
output = "../docs_out/orders_service_site"

Paths are relative to this file. Keep the outputs outside the source tree. See project files for every key, and add .docgen/ to the project's .gitignore.

3. Check what it will do, then build

cd ~/work/orders-service
python3 -m docgen run docgen.toml --list      # resolved paths; builds nothing
python3 -m docgen run docgen.toml --check     # build, then check every book

docgen.toml  —  1 book(s)
  ok    code                        …/.docgen/orders-service/books/code
  ok    orders-service (document)   …/.docgen/orders-service

1 book(s) in 1 document built and checked

A file docgen can't use is refused before anything is built, with every problem listed at once and what to do about each. Open ../docs_out/orders_service_site/index.html: each book shows what came in and what came out, and its pages click through from the overview.

4. Add your own text

python3 -m docgen scaffold docgen.toml --list   # every spot a note can go
python3 -m docgen scaffold docgen.toml          # or: write an empty stub for each

Pick a spot and write markdown into notes/<book>/<spot>.md, for example notes/code/overview.md above the overview page. Rebuild and it's there, in the notebook and on the site. Stubs you leave empty add nothing.

5. Rebuild whenever the code moves

python3 -m docgen run docgen.toml --check again. A rebuild replaces exactly what the last one wrote and never touches your notes or anything you added to the output folders. --only code rebuilds one book while you're writing its notes.

Working with a model

A model can write the config and the notes. It shouldn't write anything else, since every other file is generated. docgen ships the instructions to give it:

python3 -m docgen contract > docgen-contract.md

That file is docs/models.md, printed verbatim. It lists every key docgen accepts and the extensions it reads, has rules like "never a credential" and "don't guess page ids", and ends with a table that maps each error docgen can produce to the fix. docgen's own suite checks it against the code, so it can't quietly go out of date: a key the loader accepts but the doc doesn't name, or an example in it that doesn't load, fails make check.

The loop

1. Give it the project and the contract. A digest of the project (distill), plus docgen-contract.md as an attachment rather than pasted. If replies go through explode, attach its contract too: explode's says how to format the reply, docgen's says what goes in the files.
2. Ask for docgen.toml first. Unpack it, then run run docgen.toml --list. If it's refused, paste the errors back verbatim. Each one is in the contract's repair table.
3. Build once, then list the spots. Run scaffold docgen.toml --list and paste the output back. Page names depend on the real directories, and a model that guesses them writes notes that land nowhere.
4. Ask for notes for those spots. Unpack, rebuild with --check, and paste back any note reported as filed under a page that doesn't exist.

The three concerns

DOT collapses three separate questions into one file format, which is why a hand-written .dot is never reusable: you cannot change the palette without editing the structure, and you cannot change the structure without re-deciding the layout. docgen keeps them apart.

An extractor has never heard of SVG, colours or layout. An emitter has never heard of Python, ast or SQL. Both halves of that are checked by parsing the source and looking at what it imports, because a rule nobody enforces is a rule that lasts about a month.

The book

A book is one docgen operation, and it has a fixed shape: it begins by saying what went in and ends by saying what came out. Everything in between is an ordinary file that stands on its own.

larder ──► step ──► step ──► step ──► book
  what        each one usable          what
  came in     by itself                came out

The word comes from Atlas 1.0, where a book is a larder composed with a pattern into something published and served. docgen had already built all three parts under other names, so this is less an adoption than a renaming back.

Why both ends, rather than just the result

Because the two numbers are only worth having together. "1,505 nodes" is not a fact about anything. "225 files in, 1,505 nodes out, nothing lost" is. Before this, an extractor that read 45 of 47 files produced exactly the same document as one that read all 47, and the diagram looked complete either way — there was nowhere for the other two to be mentioned.

The larder — what came in

Deliberately not called a bucket. A bucket is somewhere bytes sit; a larder is stocked from outside, has an inventory, and goes stale. All three are worth measuring, and they are what the measure records:

"larder": {
  "kind":     "python",           // which extractor stocked it
  "identity": "../../station",    // path, or a DSN with the password masked
  "unit":     "file",             // file | table | path | entry | document
  "seen":     47,                 // what the larder offered
  "read":     45,                 // seen - len(failed), derived
  "failed":   [{"name": "a.py", "error": "syntax: line 3"}],
  "extra":    {"packages": 12}
}

read is derived and never stored. Stored, it invites the question "does that include the failures?" and every reader answers it differently; derived, there is nothing to get wrong — and ir/validate.py fails a document whose arithmetic disagrees with itself.

Failures are recorded by name, not counted. A count tells you a book is incomplete; a name tells you which part of it to distrust.

The book measure — what came out, and reconciled

Counts by kind, edges by kind, externals, and every artifact with its byte count. On its own that is just a summary. What makes it a measure is that it is reconciled against the larder:

  larder   ../../station — 45 files read, 2 failed, 12 packages
  book     312 nodes · 244 edges · 26 external · 8 artifacts
  ok       45 file(s) read produced 45 module(s)
  ok       2 file(s) could not be read
  ok       2 unreadable file(s) appear as 2 marked module(s)

The relation differs by source and is declared per extractor, because getting it wrong gives a check that passes for the wrong reason. One file becomes one module node — fewer means input was dropped. Four hundred HAR entries becoming twelve endpoints is not a loss, it is the point of the capture. And a file that failed to parse must still appear in the graph, carrying its error, or the picture is smaller than the source and says nothing about it.

python3 -m docgen book exits 1 when a reconciliation fails. The book is still written — the evidence is the point — but a build that lost input should fail a pipeline rather than pass quietly.

The notebook is the sequence, the web is the last step

These two rule what gets generated, and each for its own reason.

The notebook is not one artifact; it is the sequence. Its first cell is the larder measure and its last cell is the book measure, which is what puts the two ends in the document rather than only in the tooling. Between them, one pair of cells per step: what the step did, and a cell that loads that step's artifact and prints one fact about it. That is the "usable by themselves" property made executable, and the test suite runs those cells.

The web output is last, so nothing depends on it, so it can be replaced wholesale without touching anything upstream. That is exactly what lets it rule the output without being a stable contract — the book measure is the promise, and the page displaying it is free to change drastically and often. It is also the artifact somebody definitely opens, which is why "2 of 47 files could not be read" has to appear there, above the diagram rather than below it.

What a book looks like on disk

book/<slug>/
├── book.json        both measures, the steps, artifacts with byte counts
├── steps/           every intermediate — ir.json, view.json, graph.svg, …
├── notebook.ipynb   the sequence; first and last cells are the measures
├── overlay.json     hand-written, optional, re-applied every build
├── checks.py        this book's own assertions — optional
└── site/            the web output, both measures at the top

make book SRC=../station BOOK=out/book/station
make check BOOK=out/book/station

The IR

Plain JSON. Three keys, and it has survived four domains without gaining a fourth.

{
  "meta":  { "source": "python", "root": "app/", "schema_version": "1" },
  "nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
               "parent": "app.models",
               "attrs": { "file": "app/models.py", "line": 12, "lines": 40 } } ],
  "edges": [ { "source": "app.models.User", "target": "app.db.Base",
               "kind": "inherits", "attrs": {} } ]
}

Stdlib dataclasses, not Pydantic

The IR's whole value is being a plain document anything can open. A format that needs a library installed to be read is an API, not a format. Validation is therefore a function called at the boundary rather than a property of the type, and it reads its field lists out of schema.json so the schema and the dataclasses cannot drift apart.

python3 -m docgen validate ir.json

It catches what a schema cannot: an edge naming a node that does not exist, a containment cycle, a duplicate id, and a visual field smuggled into attrs.

Shape decides the drawing

A diagram that fights its layout engine is usually the wrong kind of diagram. The clearest evidence: the same 24-table database rendered 32034×136 through Graphviz — a 235:1 strip — and 1740×1860 through the ER emitter. Not because one engine is better, but because a schema is a set of peer entities with references, and laying it out in dependency ranks was never its shape.

So ops.classify() reads the structure and names the emitter, with the reason attached — advice without a reason gets overridden the first time it is inconvenient.

$ python3 -m docgen emit auto view.json -o out/
  sheet    -> index
           109 nodes sit at one level; any layered engine draws that as a
           strip. Split it, scope it, or read it as an index

Why twenty

Measured, one diagram per subsystem: at or under 20 nodes the output lands around 1.6:1; at 70–106 nodes about 7:1; at 261 nodes 14:1. Aspect ratio is a property of the graph, not of the renderer — a layered engine puts one dependency level in one row, so the widest level is the width.

Every Graphviz lever was tried before concluding this. ratio=compress squashed a graph to an unreadable 1008×75; rankdir=LR merely rotated a 14:1 into a 1:6; packing disconnected components gained nothing. The fix was never a flag. It was to stop asking for one picture of everything — which is what explore does.

Extractors

Deterministic parsing only. No model in the structural path. A diagram built from an AST cannot be out of date with the code. A diagram built from a model's reading of the code is wrong the moment the model has a bad day — which is the problem this exists to fix.

Two passes, because ast resolves nothing

Given class User(Base), Python's ast hands over the literal string "Base". It has no idea that came from from .db import Base three lines up. So pass one collects, per module, what it defines and what it imports; pass two resolves local names to fully qualified ids. The edge then points at app.db.Base — a real node — rather than at a box called Base that means nothing.

C# and TypeScript

Handled by tree-sitter, which is why generics, nested types and a brace inside a string are non-events rather than special cases. The test suite asserts that last one specifically, because it is exactly where a hand-rolled scanner breaks.

This reader produces no edges. Resolving a C# using to the thing it names is a different and much larger job, and the consumer that needs this — the minimap — needs none of it. An extractor that quietly produced half a dependency graph would be worse than one producing none, because the half would look whole.

Usage, not just the spec

An OpenAPI document says what endpoints are. It does not say how to use them — least of all when they are not RESTful, or when a GraphQL endpoint sits alongside. So docgen also reads a HAR: the recording format that browser devtools, mitmproxy, Charles and Insomnia all export.

Two limits, stated rather than glossed: path templating is a guess (attrs.observed_paths keeps what was actually seen beside it), and consecutive is not caused-by — the edge weight is what separates a habit from an accident, and one recording will not tell you which.

Views

The first real diagram out of this pipeline was a 3000px-wide strip: four modules of actual content and sixty sys/json/typing boxes as their peers. The emitter was correct and the picture was useless. That is a missing view, not a broken renderer — and the fix belongs to every consumer at once, because the index, the diagram and the diff all want the same narrowing.

python3 -m docgen view ir.json --overview -o view.json
python3 -m docgen view ir.json --around docgen.ir --hops 2 -o view.json
python3 -m docgen view ir.json --split -o parts/
python3 -m docgen view ir.json --shape          # what will this look like?

All of them are IR→IR, all composable, and each produces a document that still validates. --overview is the default and dispatches on the source: a codebase reduces to its modules and its outside dependencies, a schema to its tables and their keys.

Depth is the tempting knob and the wrong one

A directory without an __init__.py is not a package, so its modules have no parent and sit at depth 0. soleprint has 173 such roots, and a depth-2 cut still held 566 functions and 142 classes. Selecting by kind does not care how the directories happen to be arranged.

Emitters

The non-visual ones matter most for reach. A sorted, described list of what exists is readable by someone who will never open a diagram, and it also reports the dependency surface and any file that failed to parse. It is built second, not last — it is what proves the IR is not secretly diagram-shaped.

ERD — and where the layout came from

Not invented here. station/tools/graphgen/templates/index.html, the Supabase-style schema explorer already in this repo, had solved it:

const cols = Math.max(2, Math.ceil(Math.sqrt(sorted.length * 1.2)));

Columns from the square root of the table count. The aspect ratio is chosen rather than emergent, so the result stays near-square at 4 tables or 400. That is the one thing a rank-based engine cannot offer. Three more things it gets right: a table is a card with its columns; an edge leaves the column holding the key and lands on the target's primary key; and the geometry is computed rather than measured, so it renders identically on any machine.

Minimap

Sublime's minimap shrinks the characters. This draws the structure at full scale: one file is a column, one line is a fixed number of pixels, every construct a block sized by its span and coloured by what it is. No text inside a block — the shape is the message.

The claim is that the pattern comes from the colours alone, so nesting is drawn by inset rather than by hue. On soleprint you can see that modelgen is class-based, histgen is function-based and tester is mixed, without reading a line.

Pages

A notebook doesn't scroll a wide image sideways: it scales it down to the cell width until the labels can't be read. Zooming and panning is a workaround for a picture that holds too much. So a book's diagram is split into pages, each drawn small enough to fit the column it's going into.

overview                 one box per top-level group, relationships lifted
└── station              one box per child of station
    └── station.tools    one box per child of station.tools, and so on

Pages follow structure, never scissors. Slicing one large image cuts boxes and edges at the seams, and a slice means nothing on its own. A page is a subsystem: its boxes are the scope's children, and anything outside the scope that they touch appears as a context box, collapsed to the branch it sits on. A page still says what its subsystem reaches without drawing the rest of the tree.

Fitted by measuring, not estimating

Whether a page fits depends on its labels, fonts and spacing, so an estimate from node counts is wrong the moment the names are long. Each page is rendered and its width read back. If a page is too wide top-to-bottom, it's drawn left-to-right, since height is free in a notebook and width isn't. If it still doesn't fit (a level of many peers with no smaller structure to split into), the page is kept and flagged too wide in its caption and in the index. Never cut, never dropped.

Measured on soleprint, whose module graph rendered at 14:1 as one picture: 69 pages, and at the site's 800px column the widest flagged page is 988px. Schemas page differently: cards in columns capped to the width, grouped by tables joined by foreign keys, so a schema grows downward instead of sideways.

Clicking through, on the site

Every page is its own HTML file with the diagram inline, not in an <img>, which swallows clicks. A box standing for a subsystem opens that subsystem's page. A module box opens its section of the book's reference page. A context box opens its own branch. Breadcrumbs lead back up. The size is adjustable: fit scales to the column, 1:1 shows the drawing at natural size with scrollbars, the choice is remembered, and full screen opens the pan-and-zoom viewer.

A notebook can't follow a link inside an image, so it gets a page index instead: every page, indented by depth, each a link to its figure.

Explore

The minimap on its own shows shape and no meaning: a block says "a 30-line class", not which class or what it touches. So it is not the artifact. It is the selector.

make explore OUT=out     # then open out/explore/explore.html

The same split applies to a database: every table at once with no column detail, then click one to get its columns plus the tables its keys reach. The two differ exactly where they should — a module's neighbourhood deliberately leaves its contents out, because those are the hundred functions that made the sheet unreadable, while a table's brings them in, because a table without its columns is not a table.

The selection basket

Shift-click accumulates blocks. The basket is a copyable list of paths with a line count — enough to hand to distill, and enough to see that the selection got too big before spending the context on it. Navigating a tree quickly in order to decide what to feed a model is a real use, and this is the part that serves it.

Notebooks

A notebook is normally a source file somebody confects by hand: prose, code and stored output braided together, diffing badly, drifting from whatever it documents the moment either moves, with no way to tell by looking.

Here a notebook is a build artifact. The source is the OpenAPI document — the same file the server is built from — and the notebook is regenerated from it. Nobody edits the .ipynb, the same way nobody edits a .o. "Is this document current" stops being a question about somebody's diligence and becomes a question about whether the build ran.

This is the disagreement with jupytext. Jupytext fixes the diffing — it makes a notebook editable as text — and leaves the actual problem: you still hand-author it, so it still rots.

Generated base, hand-written overlay

Generation alone gives a document that is never stale and never says anything a parser could not work out. Hand-authoring alone gives insight and a document that rots. Two files is the only arrangement that gets both:

IR ──► spec ──(+ overlay)──► merged spec ──► .ipynb
   generated   hand-written       merged       emitted

The spec is an ordered list of steps with no Jupyter in it — a Swagger for notebooks, readable and diffable. The overlay is the only file anyone edits, and it is re-applied on every build. It can annotate, replace, insert, drop and order.

replace is the one that matters. It is how real usage gets into a document that a spec could not describe — the call that is always made with status=available, the GraphQL endpoint that is not in the OpenAPI file at all — and it keeps working unchanged once a usage recording supplies the same facts automatically.

python3 -m docgen emit notebook ir.json --scaffold overlay.json
python3 -m docgen emit notebook ir.json --overlay overlay.json -o walkthrough.ipynb

Notes

Generation alone gives a document that's never stale and never says anything a parser couldn't work out. Prose is the part only a person adds, and it has to survive every rebuild, so it can't live in the output. It lives in a notes folder, one markdown file per spot, named after the page or step it belongs above:

notes/
├── _intro.md                   top of the document
├── _end.md                     bottom of the document
└── <book>/
    ├── _intro.md               top of that book's part
    ├── larder.md               above "what came in"
    ├── overview.md             above the overview page
    └── station.tools.md        above the page for station.tools

To put text above a page, create the file named after it. No schema, no JSON, no ids to look up. That's something a person or a model can do without reading anything first. Local images in a note are embedded: as a cell attachment in the notebook, as a data URI on the site.

make scaffold CONFIG=docgen.toml      # an empty stub for every spot, never overwriting

scaffold lists the spots rather than making you guess them. A stub is a single HTML comment and renders nothing, so scaffolding a hundred spots changes the output not at all until one is filled in. Run it again after the sources change and it only adds the new spots.

The overlay stays for what notes can't do: drop a step, reorder, replace generated code. Notes are applied after the overlay, so a step that moves takes its note with it.

Style & colour

A style rule names a slot, never a colour. "border": "atlas" is the rule; a theme binds atlas to #43A047 in print and #15803d on the docs site.

That indirection is the whole point. common/theme/tokens.css, docs/graphs/themes/*.gvpr and style/lucid.json use the same slot names, so a diagram and the page around it match by construction — which is the rule docs/graphs/README.md already states. The dark theme's artery, atlas and station slots are exactly the --system-accent values the three system pages set, and the test suite fails if they drift apart.

Dark is the default, because a generated diagram lands in a dark docs page far more often than in a document. --theme lucid gives the print palette — and gives it to the page as well as the diagram, since both are baked from the same slots.

Where DOT stops

The emitter writes what DOT expresses natively and stops at the boundary rather than growing machinery. The limits are recorded in the style file itself: a cluster has a label and a fill but not a header bar; stroke-dasharray is not parameterised, so 4,4 and 5,5 collapse; rounded is binary, so 4px and 6px are identical. Those mark where a richer emitter would begin — and the style file carries the full specification regardless, so that emitter needs no re-authoring.

One limit was worth solving: DOT cannot use a cluster as an edge endpoint, so every module-to-module import silently vanished. The native answer is compound=true with lhead/ltail — draw between a representative leaf and clip the line at the cluster border.

Standalone

Copy the docgen/ folder anywhere and it works. The Makefile derives its own package name from where it sits, so it can be renamed too, and everything else resolves inside the directory.

cp -r docgen /somewhere/else
cd /somewhere/else/docgen
make doctor          # what this machine has
make sync            # optional: every group, from uv.lock
make check           # the suite, from the copy
make book SRC=/path/to/any/repo BOOK=out/book/theirs

One seam, and it is optional

Exactly one capability needs more than the folder: reading an OpenAPI document goes through station/tools/modelgen, which parses the spec and resolves $ref. It is deliberately not reimplemented here — a second OpenAPI reader in one repo is two things to keep correct.

So if you are using docgen standalone but keeping the repo alongside as reference, point at it:

export DOCGEN_REFERENCE=/path/to/repo
make doctor
# reference: /path/to/repo (from $DOCGEN_REFERENCE)

Resolution is $DOCGEN_REFERENCE first, then walking up from the package — so in place it needs no configuration, and an explicit path wins when set. Without it, the four other extractors and every emitter work unchanged; the OpenAPI reader reports what to set, and the suite skips rather than fails.

It is asserted, not asserted-in-prose

A standalone claim decays the moment somebody adds a convenient import, and it decays silently, because the suite still passes inside the repo. So the suite reads its own source:

• Imports are the stdlib, docgen itself, and a short allow-list — tree-sitter, lxml, yaml, networkx, modelgen, each optional and each with a reason. A new name is a new dependency in a folder meant to be copied.
• Only extractors/openapi.py imports modelgen. One seam is a seam; two is a dependency.
• Only reference.py knows the repo layout, so pointing docgen elsewhere is one change rather than a search.
• A book writes only inside its own output directory — nothing is left in the tree being read.

The folder was also literally copied to /tmp and run, which is how the one real bug here was found: a check asserting the reference repo is reachable, correct in place and wrong the moment there was nothing above. It now reports which case applies instead of assuming one.

Run files

A book gets rebuilt many times — after a merge, before a release, whenever the source moves — and each one is a source, an output directory and a handful of options. Typed out every time, those drift: one run excludes migrations, the next forgets. A run file writes them down once, so the rebuild is exact.

# docgen.toml — kept beside the project it describes
reference = "../soleprint"          # optional; $DOCGEN_REFERENCE wins

[defaults]
out     = "out/book"                # each book goes to <out>/<name>
theme   = "dark"
exclude = ["migrations", "tests"]

[[book]]
name = "station"
root = "../soleprint/station"

[[book]]
name   = "frontend"
root   = "../web/src"
reader = "code"                     # C# / TypeScript, through tree-sitter

[[book]]
name    = "orders-api"
openapi = "specs/orders.yaml"
overlay = "overlays/orders.json"
out     = "/srv/docs/orders"

make run CONFIG=docgen.toml
make run CONFIG=docgen.toml ONLY=station,frontend CHECK=1
python3 -m docgen run docgen.toml --list        # resolved paths, nothing built

docgen.toml  —  3 book(s)
  ok    station     /home/me/proj/out/book/station
  ok    frontend    /home/me/proj/out/book/frontend
  FAIL  orders-api  ImportError: modelgen is not reachable — …

1 of 3 book(s) did not hold — --verbose for each book's own output

Each book names exactly one source — root, schema, openapi or har — the same choice the book command makes you take. One failing book never stops the rest, and the run exits 1 if any book failed to build, lost input between its two ends, or failed its checks. docgen.example.toml ships with docgen and runs against its own fixtures.

The rules, each one a thing that goes wrong otherwise

• Paths are relative to the run file, not to wherever the command ran. Otherwise the same file builds different books from different directories.
• A book's value replaces the default, for every key — including exclude, where merging looks tempting. One rule nobody has to remember beats a clever one somebody has to look up. A default exclude simply does not apply to a schema or a spec.
• Unknown keys are refused, not ignored, all of them at once. exlude = [...] silently doing nothing is how a rebuild quietly starts reading node_modules.
• A book may not be written inside the tree it reads — the next run would read the last run's output — and two books may not share an output directory, where the second would clear the first every time.
• The environment beats the file. A reference in the run file is used only when $DOCGEN_REFERENCE is not set, which is rig's precedence for every setting it has.

TOML because a run file is written by people: tomllib is the stdlib, it takes comments — the reason a book excludes something belongs next to the exclusion — and it is already the format of the pyproject.toml beside it. The IR and the style files stay JSON, because those are data that tools write.

Project files

A run file lists independent books. A project file describes one document built from several sources: an API contract and the code behind it, a schema and the service that uses it. It's the shape a model wrote unprompted in a dry run, so docgen reads it as written:

# <project>/docgen.toml
[project]
name  = "orders-api"
title = "Orders API"
notes = "notes"                     # default
build = ".docgen"                   # book artifacts, default; gitignore it

[extract.openapi]
spec = "doc/contracts/openapi.yaml"

[extract.code]
root = "src"
ext  = [".cs"]

[ops]
drop_stdlib   = true
drop_builtins = true

[emit.notebook]
output = "../docs_out/orders_api_notebook"
theme  = "lucid"

[emit.site]
output = "../docs_out/orders_api_site"
width  = 800

Each [extract.*] table is a book, and a book is a part of every output rather than an output of its own. Each [emit.*] table is an output made of all of them: one notebook and one site, each with a part per book. Several sources of one kind use [[extract.code]] with a name each. [ops] applies to every book's view before paging.

Every part still opens with what came in and closes with what came out, and the document opens with every larder and closes with every reconciliation. A book that can't be built doesn't cost the others: the outputs are assembled from what was built, and both say this document is incomplete and name what's missing.

Refused, with what to do instead

docgen.project.example.toml ships beside the run file example and builds docgen's own fixtures into one document, with a sample note.

Commands

Make

Variables: SRC, OUT, SCHEMA, OPENAPI, HAR, BOOK, SLUG, READER, OVERLAY, CONFIG, ONLY, CHECK, STYLE, THEME, SCALE, PY. PY is the synced .venv when there is one and python3 when there is not. The Makefile derives its own package name from where it sits, so the folder can be copied anywhere and renamed and still work.

READER rather than LANG because LANG is the shell's locale variable, so ?= inherits en_US.UTF-8 from the environment and the argument is rejected. Every target above is one step of a book and still works alone — that is the property the spine exists to preserve, not to replace.

The command line

One entry point, python3 -m docgen <command>, run from the directory above the folder (or with it on PYTHONPATH, which is what the Makefile does). Each command is plain argparse and takes --help.

python3 -m docgen book --root SRC -o out/book/slug   # the whole operation
python3 -m docgen run docgen.toml                    # every book a run file lists
python3 -m docgen check                              # docgen's own suite
python3 -m docgen check out/book/slug                # that book's level
python3 -m docgen check .docgen/<project>            # a project document's level
python3 -m docgen scaffold docgen.toml               # stub notes for every spot
python3 -m docgen scaffold docgen.toml --list        # the spots, written nowhere
python3 -m docgen contract                           # the instructions for a model

python3 -m docgen extract python --root SRC -o ir.json
python3 -m docgen extract code    --root SRC -o ir.json
python3 -m docgen extract db      --schema schema.json -o ir.json
python3 -m docgen extract openapi --spec spec.yaml -o ir.json
python3 -m docgen extract usage   --har session.har -o ir.json

python3 -m docgen validate ir.json
python3 -m docgen view ir.json --overview -o view.json

python3 -m docgen emit auto     view.json -o out/
python3 -m docgen emit index    ir.json -o index.md
python3 -m docgen emit dot      view.json -o graph.svg --theme lucid
python3 -m docgen emit erd      ir.json -o schema.svg
python3 -m docgen emit minimap  ir.json -o map.svg --scale 0.5
python3 -m docgen emit notebook ir.json -o book.ipynb --overlay overlay.json
python3 -m docgen emit site     view.json -o site/
python3 -m docgen emit explore  ir.json -o explore/

Where the command line lives

All of it in cli/, and nowhere else. extractors/, ops/, emitters/ and book/ hold library code only — no argparse, no __main__.py, no cli_dot.py sitting next to dot.py. Two kinds of file in one folder make every folder answer two questions, and the command-line half is the half that grows copies: eight cli_* files each re-implemented "read the IR, validate it, load the style" by hand, and had begun to word their errors differently.

cli/
├── __init__.py    the command table and the one exit path
├── common.py      read and validate an IR, load a style, write or print
├── book.py  run.py  check.py
└── extract.py  validate.py  view.py  emit.py

A command reports an expected failure by raising, and cli.main turns it into the only exit path: one Error: line on stderr, exit 1, never a traceback. Doing that split also surfaced a third copy of "which drawing does this graph want" — in the auto command, the site command and the book — which is now emitters.auto.draw(), returning content and leaving the file name to the caller.

Dependencies

The core is standard library only, Python 3.11 or later — pyproject.toml declares no required dependency at all. Everything else is an optional dependency group, pinned in a committed uv.lock, and reported by make doctor; when something is missing you lose exactly one capability and get told what to install. See standalone for the one seam out of the folder.

make sync                              # uv sync --all-groups  →  .venv/
uv sync --group code                   # or just the one you need
make lock                              # after editing pyproject.toml

[tool.uv] package = false, the same choice station/tools/dataconvert makes: this is a folder of modules run in place, not something to install. The lock pins what the optional groups resolve to; the folder still runs on a bare python3 without it.

Testing

Three levels, and they differ by what they assert about. The distinction decides what a failure means, which is why it is worth keeping:

All 346 after make sync, 331 with nothing optional installed — the suite skips rather than fails when tree-sitter, lxml or the OpenAPI reader is absent. See standalone for the counts outside the repo. Among them, two that keep this page's structure true: nothing outside cli/ imports argparse, and every import the suite allows is a group in pyproject.toml and pinned in uv.lock.

The book level, where custom checks live

The third level is the one that reaches a project docgen has never seen, and it is where framework and hand-written checks live together. The generated half is the spine's own assertions, identical for every book: both measures present, the reconciliation holding, every artifact where the ledger says it is, the notebook still executing. The custom half goes in the book's own checks.py and uses the same helpers, so a project's line and a framework line read identically and fail identically.

# out/book/station/checks.py
def checks(book, check, note, skip):
    note("what this project will not give up")

    # Payments moved once already and the move broke three dashboards.
    # If it is not here, something renamed it again.
    check("payments is still a module", True,
          any(n["id"] == "app.payments" for n in book.ir["nodes"]))

Same split as the notebook's base and overlay, for the same reason: generation alone cannot know what this project cares about, and hand-authoring alone rots.

The four that are the design

Of docgen's own checks, four assert the architecture rather than guard a regression, and they are the ones to keep if anything is ever cut:

• No visual field reaches the IR — extractors cannot decide appearance.
• No emitter reads a source file — the layering, checked from the other side by parsing imports.
• Style names slots, not colours — one colour language rather than three.
• Ids are stable across runs — without it, diffing is noise.

The browser pages are JavaScript, so they are tested as JavaScript: a stub DOM under node drives the viewer's zoom and 1:1 toggle, and the explorer's select-and-walk. Both skip cleanly where node is absent.

Self-hosting is the honest end-to-end check, and it is where the real bugs came from — two name-resolution faults and a duplicate-id crash that no fixture had reached. make self builds docgen's book of soleprint and then runs that book's own level against it, so the two ends have to reconcile on 225 real files. If the index does not read like the system, something is wrong.

Limits & non-goals

Things deliberately not done, with the reason, so they are not re-litigated:

• No layout engine. No positioning, no neato -n2, no ELK. Aspect ratio was solved by choosing the right emitter and by not drawing everything at once.
• No calls edges. Resolving self.foo() needs type inference, and a call graph that is quietly 60% right is worse than none because it reads as authoritative.
• No edges from the C#/TypeScript reader. Structure only — half a dependency graph would look whole.
• No model in the structural path. Annotation — summarising a module, naming a cluster — is a later layer, cached to its own file keyed by node id, merged at emit time, and extraction must work with it absent.
• No configuration knobs until two real consumers disagree.

Known gaps, stated plainly:

• The C# reader is verified against a written fixture, not a real repository. That is the next check that matters.
• The diff emitter is not built. Stable ids exist to make it possible, and two IRs from two commits is the cheapest useful thing left.
• Minimap blocks carry no names. Rendering them legible only at 1:1, through the viewer, is the other half of reading a file without reading every line.