soleprint/soleprint/atlas2/docgen/README.md

# docgen

Static analysis of a tree, and the artifacts that fall out of it.

The point is not the diagram. The point is the format in the middle — diagrams
are one consumer of it, and not the one that reaches the most people.

```
extractors/        →   graph IR (JSON)   →   emitters/
(per source type)      (one schema)          (per output target)
                              ↑
                         style/*.json
                    (consumed by emitters only)
```

```bash
make sync                       # optional: .venv with every group, from uv.lock
make check                      # prove it, on a tree it builds itself
make book SRC=/path/to/repo     # one whole operation, measured at both ends
make run CONFIG=docgen.toml     # every book a run file lists
make help
```

The full documentation is `docs/index.html` — open it in a browser.

## Layout

```
cli/            the command line — every command, and nothing else
book/           an operation: larder measure, steps, web output, run files
extractors/     source -> IR, one per source type
ir/             the contract: schema.json, the dataclasses, the validator
ops/            IR -> a smaller IR
emitters/       IR -> an artifact
notebook/       the notebook spec, before it is an .ipynb
style/          slots, themes, and harvesting a theme from real diagrams
fixtures/       docgen's own test inputs
lab/            sanctioned experiments; nothing imports it
reference.py    the one seam to the repo above, for reading OpenAPI
```

Library packages hold **no command-line code** — no argparse, no `__main__.py`,
no `cli_dot.py` beside `dot.py`. It all lives in `cli/`, behind one entry point,
and the selftest asserts it stays there.

`pyproject.toml` declares no required dependency: the structural path is the
stdlib, Python 3.11+. The optional groups (`code`, `openapi`, `harvest`, `lab`)
are pinned in `uv.lock`; `[tool.uv] package = false`, as dataconvert does, since
this is a folder run in place rather than something to install.

## Run files

```toml
# docgen.toml — beside the project it describes; paths relative to this file
[defaults]
out = "out/book"

[[book]]
name = "station"
root = "../soleprint/station"

[[book]]
name   = "shop"
schema = "schemas/shop.json"
```

`make run CONFIG=docgen.toml [ONLY=station] [CHECK=1]`. One failing book never
stops the others; unknown keys are refused; a rebuild replaces the previous
build's outputs and never touches a hand-written `checks.py`.
`docgen.example.toml` runs against the shipped fixtures.

Or as three composable commands, which is what the Makefile is wrapping — run
from the directory above this one:

```bash
python3 -m docgen extract python --root SRC -o ir.json
python3 -m docgen view ir.json --drop-stdlib -o view.json
python3 -m docgen emit dot view.json -o graph.svg --theme dark
```

## DOT collapses three concerns; this separates them

| concern | question | owner |
|---|---|---|
| **structure** | what the graph *is* | `ir/schema.json` — versioned, golden-tested |
| **meaning** | what things *mean visually* | `style/*.json`, keyed on `kind` |
| **placement** | where things *go* | Graphviz defaults. Phase two |

An extractor has never heard of SVG, colours or layout. An emitter has never
heard of Python, `ast` or SQL. **The IR carries no visual information** — if a
field would change between light and dark theme, it does not belong in it.
`shape="cylinder"` is not a field; it is `kind="datastore"` plus a style rule,
which is what lets the same IR render in a theme that has no cylinders.

The selftest asserts all three of those, because they are the design rather than
a nicety and they are exactly what erodes first.

## The IR

```json
{
  "meta":  { "source": "python", "root": "app/", "schema_version": "1" },
  "nodes": [ { "id": "app.models.User", "kind": "class", "label": "User",
               "parent": "app.models",
               "attrs": { "file": "app/models.py", "line": 12 } } ],
  "edges": [ { "source": "app.models.User", "target": "app.db.Base",
               "kind": "inherits", "attrs": {} } ]
}
```

- **`id`** is fully qualified and **stable across runs**. That is what makes two
  graphs from two commits diffable.
- **`kind`** is the hinge, and the only field style and layout may key on.
- **`parent`** is containment. Relationships are edges.
- **`attrs`** is an open bag; `file`/`line` let a UI link a box to a line.

Stdlib dataclasses, not Pydantic. A format that needs a library installed to be
opened is not a format, it is an API. `ir/validate.py` is the check at the
boundary, and it reads the field lists out of `schema.json` so the two cannot
drift.

```bash
python3 -m docgen validate ir.json
```

It catches what a schema cannot: an edge naming a node that does not exist, a
containment cycle, duplicate ids, and a visual field smuggled into `attrs`.

## Extraction is deterministic

**No LLM in the structural path.** A diagram from an AST cannot be out of date
with the code; one from a model's reading of the code is wrong the moment the
model has a bad day, which is the problem this exists to fix.

`ast` resolves nothing on its own — `class User(Base)` yields the literal string
`"Base"`. So there are two passes: one collects each module's definitions and
imports, the other resolves names against those tables.

```
from .db import Base ;  class User(Base)
    →  app.models.User --inherits--> app.db.Base        not "Base"
```

**Unresolved names become `kind: "external"` nodes and keep their edges.**
Dropping them is the worse failure: the diagram looks complete and has quietly
lost a dependency. Gathered by the index emitter, they *are* the project's
dependency surface.

An unparseable file is recorded as a node with an `error` attr, not a crash —
one bad file must not cost you the other four hundred.

`calls` edges are deliberately **not** attempted. Resolving `self.foo()` needs
type inference, and a call graph that is quietly 60% right is worse than none
because it reads as authoritative.

### A second source

`extractors/db.py` reads the published `{models, relationships, source}`
contract that `modelgen` already emits and `graphgen` already consumes. Tables
become nodes, columns become contained nodes, foreign keys become edges — with
no new top-level field, which was the checkpoint on whether the schema was right.

Connecting to a live database is not here. `modelgen from-db --url ...` does
that and writes the schema this reads; the two-step also keeps credentials out
of this pipeline entirely.

## Views are not an emitter concern

The first real diagram out of this pipeline was a 3000px strip: four modules of
content and sixty `sys`/`json`/`typing` boxes, all peers. The emitter was
correct and the picture was useless. That is a **missing view**, and the fix
belongs to every consumer at once — the index, the diagram and the diff all want
"just this subsystem, two hops out, without the stdlib".

```bash
python3 -m docgen view ir.json --drop-stdlib --around docgen.ir --hops 2 -o view.json
```

`drop_stdlib`, `drop_external`, `only_kinds`, `drop_kinds`, `subtree`,
`neighbourhood`, `collapse_to_depth`. All IR→IR, all composable, each producing
a document that still validates.

Graph *algorithms* are not here. Transitive reduction, cycle detection and
dominators are `networkx`'s, and reimplementing them is the classic way to
acquire a quiet bug. `lab/` is where that dependency gets tried against real IRs
before anything depends on it — the aim being to learn which part of it is
actually attractive, rather than adopting all of it on faith.

## One colour language

A style rule names a **slot**, never a colour. `"border": "atlas"` is the rule;
the theme binds `atlas` to `#43A047` in print and `#15803d` on the docs site.

That indirection is the whole point. `common/theme/tokens.css`,
`docs/graphs/themes/*.gvpr` and `style/lucid.json` use the same slot names, so a
diagram and the page around it match by construction — which is the rule
`docs/graphs/README.md` already states. The `dark` theme's `artery`, `atlas` and
`station` slots are exactly the `--system-accent` values set in
`artery/index.html:30`, `atlas/index.html:25` and `station/index.html:29`, and
the selftest fails if they drift apart.

An unknown `kind` falls back to `default` rather than crashing, so a new
extractor renders plainly and legibly on day one instead of needing a style file
written first.

**How a container picks its colour without the IR naming one:** it does not. The
IR says which spr model a group belongs to (`attrs.domain` — semantic), and
`domain_slots` maps that to a slot. Same mechanism as `--system-accent`. With no
domain, the emitter assigns by sorted id, so two runs agree.

## Use DOT until it hits its limits

The emitter writes what DOT expresses natively and stops at the boundary rather
than growing machinery. The limits are recorded in `style/lucid.json` under
`limits` and reachable as `Style.limits()`:

| | |
|---|---|
| header bars | a cluster has a label and a fill, not a 100%-width header rectangle |
| `stroke-dasharray` | not parameterised — `4,4` and `5,5` collapse to one dash |
| corner radius | `rounded` is binary, so 4px and 6px are identical |
| icon above label | needs an HTML-like label table |
| sequence badges | `xlabel` carries the number; the circle does not exist |

Those mark where a richer emitter would begin. The style file carries the full
spec regardless, so that emitter needs no re-authoring.

One limit that *was* worth solving: DOT cannot use a cluster as an edge
endpoint, so every module-to-module import silently vanished. The native answer
is `compound=true` with `lhead`/`ltail` — draw between a representative leaf and
clip at the cluster border.

## The output is addressable

`id` and `kind` pass through to the SVG as the element's `id` and `class`, and
`attrs.file`/`attrs.line` become an `href`. A front end can bind behaviour to a
box and a box can link to the line it came from, without the emitter knowing
about either.

## Testing

```bash
make check                  # docgen's own suite: 287 after make sync, 272 with nothing
make check BOOK=out/book/x  # one book's own level — generated and hand-written checks
make doctor                 # the machine; never fails
```

**Golden tests go on the IR, never on the SVG.** Graphviz measures label text
with the host's fonts to size nodes, so identical input gives different geometry
on a machine with different fontconfig. The IR is deterministic; the SVG is not.

Self-hosting is the honest end-to-end check, and it is where the real bugs came
from — two name-resolution faults that no fixture had reached:

```bash
make self      # docgen's book of the widest tree it can see, then its checks
```

## Where this sits

`docgen` belongs to Atlas — documentation is whose concern it is. It is **not** a
station tool and is not under `station/tools/`; it *may depend on* station tools,
which is the permitted direction.

Atlas 2 is a successor, not a replacement. `soleprint/atlas/` is untouched: it
carries client information and an idea still worth extracting — deriving frontend
and backend tests from one source, which is the same shape as this pointed the
other way.

## Not here

No layout system, no positioning, no ELK. No HTML-like labels, no SVG post-pass.
No LLM in the structural path — annotation (summarising a module, naming a
cluster) is a later layer, cached to its own file keyed by node `id`, merged into
`attrs` at emit time, and extraction must work with it absent. No configuration
knobs until two real consumers disagree.