docgen
++ Turn source artifacts — a codebase, a database, an API spec, a recording of + real traffic — into one canonical graph format, then render that format to + whatever the audience needs. The point is not the diagram. The point is the + format in the middle. +
+ + +What it is
+ +
+ Eight demos under semester/ draw their architecture with
+ Graphviz. Every one of them hand-writes a .dot file with the
+ palette inlined and commits the .svg beside it. Three different
+ dark palettes between them, eight answers to the same question, and every one
+ of those diagrams is out of date the moment somebody moves a file.
+
+ docgen is the one answer. It reads the source rather than being told about it, + so a diagram cannot drift from the thing it describes, and it separates + what a graph is from how it looks so one extraction feeds + a diagram, an index, a notebook and a browsable site without being redone. +
+ +ast, tree-sitter, SQLAlchemy reflection via modelgen), algorithms
+ are networkx's. What docgen owns is the adapters, the schema, the style tables
+ and the emitters — all small, and all the places where the value is that
+ we made the call.
+The idea
+ ++ N sources and M outputs need N×M converters if you join them directly, + or N+M if you put a hub in the middle. The hub is an + intermediate representation — the compiler term, and the same bargain: + both sides depend on the IR and neither on the other. +
++ It is lossy on purpose. It throws away every token of syntax and keeps + "a class named User inherits from Base". That is the part that is + worth versioning, worth diffing, and worth drawing. +
++ The practical consequence is the thing to judge it on: adding a source + costs one extractor and every emitter works on it unchanged; adding an output + costs one emitter and every extractor feeds it unchanged. When the + OpenAPI reader was written it emitted schemas using the same vocabulary the + database reader uses — and the ER diagram drew an API's data model without + anyone teaching it what an API was. +
+ + +Five minutes
+ +Three commands, and they compose. That is the whole interface.
+ +# 1. read something
+python3 -m docgen.extractors.python --root ../station/tools/histgen -o ir.json
+
+# 2. narrow it to a useful view
+python3 -m docgen.ops ir.json --overview -o view.json
+
+# 3. draw whatever its structure asks for
+python3 -m docgen.emitters auto view.json -o out/Or through the Makefile, which is a thin wrapper over exactly those:
+ +make ir SRC=/path/to/repo OUT=out # extract
+make explore OUT=out # the two-pane navigator
+make site OUT=out # a docs site with a sidebar
+make self # run the whole thing over soleprint
+ Everything is offline and self-contained. No server, no CDN, no build step —
+ the outputs open over file://.
+
The three concerns
+ +
+ DOT collapses three separate questions into one file format, which is why a
+ hand-written .dot is never reusable: you cannot change the palette
+ without editing the structure, and you cannot change the structure without
+ re-deciding the layout. docgen keeps them apart.
+
| concern | question | owner |
|---|---|---|
| structure | what the graph is | ir/schema.json |
| meaning | what things mean visually | style/*.json, keyed on kind |
| placement | where things go | the emitter, and only there |
+ An extractor has never heard of SVG, colours or layout. An emitter has never
+ heard of Python, ast or SQL. Both halves of that are checked by
+ parsing the source and looking at what it imports, because a rule nobody
+ enforces is a rule that lasts about a month.
+
+
+ extractors/ reaches only ir;
+ emitters/ reaches ir and style;
+ ir/ reaches nothing outside itself; lab/ has no
+ edges at all. Click to open the viewer — then click again for actual size.
+ The IR
+ ++ Plain JSON. Three keys, and it has survived four domains without gaining a + fourth. +
+ +{
+ "meta": { "source": "python", "root": "app/", "schema_version": "1" },
+ "nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
+ "parent": "app.models",
+ "attrs": { "file": "app/models.py", "line": 12, "lines": 40 } } ],
+ "edges": [ { "source": "app.models.User", "target": "app.db.Base",
+ "kind": "inherits", "attrs": {} } ]
+}| field | meaning |
|---|---|
id |
+ Fully qualified and stable across runs. Stability is what makes + two extractions from two commits diffable; without it a diff reports + noise and nobody trusts it. |
kind |
+ The hinge of the whole system, and the only field style and
+ layout may read. A small closed vocabulary per domain —
+ module/class/function,
+ table/column, endpoint,
+ task. |
parent |
+ Containment, and nothing else. A module contains a class. Relationships + are edges. |
attrs |
+ An open bag for whatever one domain cares about.
+ file/line/lines are what let a
+ box link to the line it came from, and what the minimap sizes by. |
shape: "cylinder" is
+ not a field — it is kind: "datastore" plus a style rule, and that
+ is exactly what lets the same IR render in a theme that has no cylinders. The
+ test suite sweeps every emitted document for colour-like keys.
+Stdlib dataclasses, not Pydantic
+
+ The IR's whole value is being a plain document anything can open. A format that
+ needs a library installed to be read is an API, not a format. Validation is
+ therefore a function called at the boundary rather than a property of the type,
+ and it reads its field lists out of schema.json so the schema and
+ the dataclasses cannot drift apart.
+
python3 -m docgen.ir ir.json
+ It catches what a schema cannot: an edge naming a node that does not exist, a
+ containment cycle, a duplicate id, and a visual field smuggled into
+ attrs.
+
Shape decides the drawing
+ ++ A diagram that fights its layout engine is usually the wrong kind of + diagram. The clearest evidence: the same 24-table database rendered + 32034×136 through Graphviz — a 235:1 strip — and + 1740×1860 through the ER emitter. Not because one engine is + better, but because a schema is a set of peer entities with references, and + laying it out in dependency ranks was never its shape. +
+
+ So ops.classify() reads the structure and names the emitter,
+ with the reason attached — advice without a reason gets overridden the first
+ time it is inconvenient.
+
| kind | drawn by | when |
|---|---|---|
erd | cards in columns | entities with references |
pipeline | ranks, left to right | a chain with fan-out — an Airflow DAG, a build |
layered / tree | ranks, top down | ranks genuinely suit it |
sheet | the index | one level is wider than ~20 — a strip in any engine |
flat | the index | most nodes have no relationships: that is a list |
$ python3 -m docgen.emitters auto view.json -o out/
+ sheet -> index
+ 109 nodes sit at one level; any layered engine draws that as a
+ strip. Split it, scope it, or read it as an indexWhy twenty
++ Measured, one diagram per subsystem: at or under 20 nodes the output lands + around 1.6:1; at 70–106 nodes about 7:1; at 261 nodes 14:1. Aspect ratio is a + property of the graph, not of the renderer — a layered engine puts one + dependency level in one row, so the widest level is the width. +
+
+ Every Graphviz lever was tried before concluding this. ratio=compress
+ squashed a graph to an unreadable 1008×75; rankdir=LR merely
+ rotated a 14:1 into a 1:6; packing disconnected components gained nothing.
+ The fix was never a flag. It was to stop asking for one picture of everything —
+ which is what explore does.
+
Extractors
+ ++ Deterministic parsing only. No model in the structural path. + A diagram built from an AST cannot be out of date with the code. A diagram + built from a model's reading of the code is wrong the moment the model has a + bad day — which is the problem this exists to fix. +
+ +| reader | reads | gives |
|---|---|---|
python | a tree of .py |
+ modules, classes, functions; imports and inherits edges |
code tree-sitter | C#, TypeScript, TSX | +namespaces, classes, interfaces, methods — structure only |
db | a graphgen-compatible schema.json |
+ tables, columns, foreign keys |
openapi | an OpenAPI / Swagger document | +endpoints and the shapes they carry |
usage | a HAR recording | +what was actually called, in what order |
Two passes, because ast resolves nothing
+
+ Given class User(Base), Python's ast hands over the
+ literal string "Base". It has no idea that came from
+ from .db import Base three lines up. So pass one collects, per
+ module, what it defines and what it imports; pass two resolves local names to
+ fully qualified ids. The edge then points at app.db.Base — a real
+ node — rather than at a box called Base that means nothing.
+
kind: "external" and
+ keeps its edge. Dropping it would be the worse failure: the diagram
+ would look complete and have quietly lost a dependency. Gathered up, those
+ nodes are the project's real dependency surface.
+C# and TypeScript
++ Handled by tree-sitter, which is why generics, nested types and a brace inside + a string are non-events rather than special cases. The test suite asserts that + last one specifically, because it is exactly where a hand-rolled scanner breaks. +
+
+ This reader produces no edges. Resolving a C#
+ using to the thing it names is a different and much larger job,
+ and the consumer that needs this — the minimap — needs none of it. An
+ extractor that quietly produced half a dependency graph would be worse than one
+ producing none, because the half would look whole.
+
Usage, not just the spec
+ ++ An OpenAPI document says what endpoints are. It does not say how to + use them — least of all when they are not RESTful, or when a GraphQL endpoint + sits alongside. So docgen also reads a HAR: the recording format that + browser devtools, mitmproxy, Charles and Insomnia all export. +
+ +| what traffic knows | what a spec cannot |
|---|---|
| the order of calls | a spec is a set; usage is a sequence |
| which parameters are always sent | a spec lists twenty optional ones |
| which statuses really happen | the 422 everybody hits is in no document |
| endpoints not in the document | GraphQL operations, found by body shape and named |
| which id formats a route takes | numeric and uuid on one route |
+ Two limits, stated rather than glossed: path templating is a guess
+ (attrs.observed_paths keeps what was actually seen beside it), and
+ consecutive is not caused-by — the edge weight is what separates a habit from
+ an accident, and one recording will not tell you which.
+
Views
+ +
+ The first real diagram out of this pipeline was a 3000px-wide strip: four
+ modules of actual content and sixty sys/json/typing
+ boxes as their peers. The emitter was correct and the picture was useless.
+ That is a missing view, not a broken renderer — and the fix belongs to
+ every consumer at once, because the index, the diagram and the diff all want
+ the same narrowing.
+
python3 -m docgen.ops ir.json --overview -o view.json
+python3 -m docgen.ops ir.json --around docgen.ir --hops 2 -o view.json
+python3 -m docgen.ops ir.json --split -o parts/
+python3 -m docgen.ops ir.json --shape # what will this look like?
+ All of them are IR→IR, all composable, and each produces a document that
+ still validates. --overview is the default and dispatches on the
+ source: a codebase reduces to its modules and its outside dependencies, a
+ schema to its tables and their keys.
+
weight saying how many they stand for.
+Depth is the tempting knob and the wrong one
+
+ A directory without an __init__.py is not a package, so its
+ modules have no parent and sit at depth 0. soleprint has 173 such roots,
+ and a depth-2 cut still held 566 functions and 142 classes. Selecting by
+ kind does not care how the directories happen to be arranged.
+
Emitters
+ +| emitter | output | audience |
|---|---|---|
index | markdown, sidebar JSON | +anyone — no graph literacy required |
dot | DOT → Graphviz → SVG | dependency structure |
erd | SVG, written directly | a schema, as cards |
minimap | SVG, written directly | what is where, at a glance |
notebook | .ipynb | a runnable walkthrough |
site | a static docs site | reading |
explore | a two-pane navigator | finding your way |
auto | whichever of the above fits | not having to choose |
+ The non-visual ones matter most for reach. A sorted, described + list of what exists is readable by someone who will never open a diagram, and + it also reports the dependency surface and any file that failed to parse. + It is built second, not last — it is what proves the IR is not secretly + diagram-shaped. +
+ +ERD — and where the layout came from
+
+ Not invented here. station/tools/graphgen/templates/index.html,
+ the Supabase-style schema explorer already in this repo, had solved it:
+
const cols = Math.max(2, Math.ceil(Math.sqrt(sorted.length * 1.2)));+ Columns from the square root of the table count. The aspect + ratio is chosen rather than emergent, so the result stays near-square + at 4 tables or 400. That is the one thing a rank-based engine cannot offer. + Three more things it gets right: a table is a card with its columns; + an edge leaves the column holding the key and lands on the target's primary + key; and the geometry is computed rather than measured, so it renders + identically on any machine. +
+ +
+
+ Minimap
++ Sublime's minimap shrinks the characters. This draws the + structure at full scale: one file is a column, one line is a fixed + number of pixels, every construct a block sized by its span and coloured by + what it is. No text inside a block — the shape is the message. +
++ The claim is that the pattern comes from the colours alone, so nesting is drawn + by inset rather than by hue. On soleprint you can see that modelgen is + class-based, histgen is function-based and tester is mixed, without reading a + line. +
+ +
+
+ Explore
+ ++ The minimap on its own shows shape and no meaning: a block says "a 30-line + class", not which class or what it touches. So it is not the artifact. + It is the selector. +
+ +make explore OUT=out # then open out/explore/explore.htmlLeft — navigate
+The whole thing at once. Scan by colour, click a block.
+Right — explore
+What that is, what it reaches, what reaches it, and the neighbourhood + drawn small enough to read. Every neighbour is a link, so you walk + outward from wherever you started.
++ The same split applies to a database: every table at once with no column + detail, then click one to get its columns plus the tables its keys reach. + The two differ exactly where they should — a module's neighbourhood + deliberately leaves its contents out, because those are the hundred functions + that made the sheet unreadable, while a table's brings them in, because a table + without its columns is not a table. +
+ +The selection basket
+
+ Shift-click accumulates blocks. The basket is a copyable list of paths with a
+ line count — enough to hand to distill, and enough to see that the
+ selection got too big before spending the context on it. Navigating a
+ tree quickly in order to decide what to feed a model is a real use, and this is
+ the part that serves it.
+
Notebooks
+ ++ A notebook is normally a source file somebody confects by hand: prose, code and + stored output braided together, diffing badly, drifting from whatever it + documents the moment either moves, with no way to tell by looking. +
+
+ Here a notebook is a build artifact. The source is the OpenAPI
+ document — the same file the server is built from — and the notebook is
+ regenerated from it. Nobody edits the .ipynb, the same way nobody
+ edits a .o. "Is this document current" stops being a question
+ about somebody's diligence and becomes a question about whether the build ran.
+
+ This is the disagreement with jupytext. Jupytext fixes the diffing — + it makes a notebook editable as text — and leaves the actual problem: you still + hand-author it, so it still rots. ++ +
Generated base, hand-written overlay
++ Generation alone gives a document that is never stale and never says anything a + parser could not work out. Hand-authoring alone gives insight and a document + that rots. Two files is the only arrangement that gets both: +
+ +IR ──► spec ──(+ overlay)──► merged spec ──► .ipynb
+ generated hand-written merged emitted
+ The spec is an ordered list of steps with no Jupyter in it — a Swagger
+ for notebooks, readable and diffable. The overlay is the only file
+ anyone edits, and it is re-applied on every build. It can
+ annotate, replace, insert,
+ drop and order.
+
+ replace is the one that matters. It is how real usage gets into a
+ document that a spec could not describe — the call that is always made with
+ status=available, the GraphQL endpoint that is not in the OpenAPI
+ file at all — and it keeps working unchanged once a usage recording supplies
+ the same facts automatically.
+
python3 -m docgen.emitters notebook ir.json --scaffold overlay.json
+python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthrough.ipynbStyle & colour
+ +
+ A style rule names a slot, never a colour.
+ "border": "atlas" is the rule; a theme binds atlas to
+ #43A047 in print and #15803d on the docs site.
+
+ That indirection is the whole point. common/theme/tokens.css,
+ docs/graphs/themes/*.gvpr and style/lucid.json use
+ the same slot names, so a diagram and the page around it match by construction
+ — which is the rule docs/graphs/README.md already states. The dark
+ theme's artery, atlas and station slots
+ are exactly the --system-accent values the three system pages set,
+ and the test suite fails if they drift apart.
+
+ Dark is the default, because a generated diagram lands in a dark docs page far
+ more often than in a document. --theme lucid gives the print
+ palette — and gives it to the page as well as the diagram, since both
+ are baked from the same slots.
+
kind falls back to default rather than
+ crashing. That matters more than it sounds: a new extractor with a new
+ vocabulary renders plainly and legibly on day one, instead of requiring
+ somebody to write a style file before they can see anything.
+Where DOT stops
+
+ The emitter writes what DOT expresses natively and stops at the boundary rather
+ than growing machinery. The limits are recorded in the style file itself: a
+ cluster has a label and a fill but not a header bar;
+ stroke-dasharray is not parameterised, so 4,4 and
+ 5,5 collapse; rounded is binary, so 4px and 6px are
+ identical. Those mark where a richer emitter would begin — and the style file
+ carries the full specification regardless, so that emitter needs no
+ re-authoring.
+
+ One limit was worth solving: DOT cannot use a cluster as an edge
+ endpoint, so every module-to-module import silently vanished. The native answer
+ is compound=true with lhead/ltail — draw
+ between a representative leaf and clip the line at the cluster border.
+
Commands
+ +Make
+| target | does |
|---|---|
make check | the whole test suite, offline, nothing installed |
make doctor | what this machine has and what it is missing |
make ir SRC=… | extract Python into OUT/ir.json |
make code SRC=… | extract C#/TypeScript tree-sitter |
make db SCHEMA=… | extract a database schema |
make view | the default view for that source type |
make graph | draw whatever the structure asks for |
make index | markdown index and sidebar JSON |
make minimap | what is where, read from the colours |
make explore | the two-pane navigator |
make site | a self-contained docs site |
make self | the whole pipeline over soleprint itself |
+ Variables: SRC, OUT, SCHEMA,
+ STYLE, THEME, SCALE, DEPTH,
+ PY. The Makefile derives its own package name from where it sits,
+ so the folder can be copied anywhere and renamed and still work.
+
Modules
+python3 -m docgen.extractors.python --root SRC -o ir.json
+python3 -m docgen.extractors code --root SRC -o ir.json
+python3 -m docgen.extractors db --schema schema.json -o ir.json
+python3 -m docgen.extractors openapi --spec spec.yaml -o ir.json
+python3 -m docgen.extractors usage --har session.har -o ir.json
+
+python3 -m docgen.ir ir.json # validate
+python3 -m docgen.ops ir.json --overview -o view.json
+
+python3 -m docgen.emitters auto view.json -o out/
+python3 -m docgen.emitters index ir.json -o index.md
+python3 -m docgen.emitters dot view.json -o graph.svg --theme lucid
+python3 -m docgen.emitters erd ir.json -o schema.svg
+python3 -m docgen.emitters minimap ir.json -o map.svg --scale 0.5
+python3 -m docgen.emitters notebook ir.json -o book.ipynb --overlay overlay.json
+python3 -m docgen.emitters site view.json -o site/
+python3 -m docgen.emitters explore ir.json -o explore/Dependencies
+ +
+ The core is standard library only. Everything else is optional
+ and reported by make doctor; when something is missing you lose
+ exactly one capability and get told what to install.
+
| needs | for | without it |
|---|---|---|
graphviz (binary) | rendering DOT to SVG | +ERD, minimap, index and notebooks still work |
tree_sitter + grammars | C#, TypeScript, TSX | +Python only |
networkx | lab/ experiments |
+ nothing — nothing depends on it yet |
node | testing the browser pages | +those checks skip |
psql | the lab/ schema probe |
+ use modelgen's from-db instead |
lab/ is where a dependency gets tried before anything depends
+ on it. Nothing in ir/, extractors/,
+ ops/ or emitters/ may import from it. When an
+ experiment earns its place it graduates into ops/ behind an
+ IR→IR signature, and then the dependency is declared.
+Testing
+ +make check # 191 checks, offline, no network+ Four of those are the design rather than regressions, and they are the + ones to keep if anything is ever cut: +
+-
+
- No visual field reaches the IR — extractors cannot decide appearance. +
- No emitter reads a source file — the layering, checked from the other side by parsing imports. +
- Style names slots, not colours — one colour language rather than three. +
- Ids are stable across runs — without it, diffing is noise. +
+ The browser pages are JavaScript, so they are tested as JavaScript: a stub DOM
+ under node drives the viewer's zoom and 1:1 toggle, and the
+ explorer's select-and-walk. Both skip cleanly where node is absent.
+
+ Self-hosting is the honest end-to-end check, and it is where the real bugs came
+ from — two name-resolution faults and a duplicate-id crash that no fixture had
+ reached. make self runs the whole pipeline over soleprint; if the
+ index does not read like the system, something is wrong.
+
Limits & non-goals
+ +Things deliberately not done, with the reason, so they are not re-litigated:
+ +-
+
- No layout engine. No positioning, no
neato -n2, no ELK. + Aspect ratio was solved by choosing the right emitter and by not drawing + everything at once.
+ - No
callsedges. Resolvingself.foo()needs + type inference, and a call graph that is quietly 60% right is worse than + none because it reads as authoritative.
+ - No edges from the C#/TypeScript reader. Structure only — half a + dependency graph would look whole. +
- No model in the structural path. Annotation — summarising a module, + naming a cluster — is a later layer, cached to its own file keyed by node + id, merged at emit time, and extraction must work with it absent. +
- No configuration knobs until two real consumers disagree. +
Known gaps, stated plainly:
+-
+
- The C# reader is verified against a written fixture, not a real + repository. That is the next check that matters. +
- The diff emitter is not built. Stable ids exist to make it possible, and + two IRs from two commits is the cheapest useful thing left. +
- Minimap blocks carry no names. Rendering them legible only at 1:1, through + the viewer, is the other half of reading a file without reading every line. +