mariano/soleprint
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init docgen

Browse Source
This commit is contained in:
buenosairesam
2026-09-22 04:54:49 -03:00
parent 12ddd01940
commit b693e324ef
31 changed files with 4521 additions and 816 deletions
Download Patch File Download Diff File Expand all files Collapse all files

353
soleprint/atlas2/docgen/docs/index.html
View File

@@ -19,12 +19,15 @@
<a href="#what">What it is</a>
<a href="#idea">The idea</a>
<a href="#quick">Five minutes</a>
<a href="#first">Your first project</a>
<a href="#models">Working with a model</a>
<div class="group">Architecture</div>
<a href="#layers">The three concerns</a>
<a href="#book">The book</a>
<a href="#ir">The IR</a>
<a href="#shape">Shape decides the drawing</a>
<a href="#pages">Pages</a>
<div class="group">Reading</div>
<a href="#extractors">Extractors</a>
@@ -37,11 +40,13 @@
<a href="#emitters">Emitters</a>
<a href="#explore">Explore</a>
<a href="#notebooks">Notebooks</a>
<a href="#notes">Notes</a>
<a href="#style">Style &amp; colour</a>
<div class="group">Reference</div>
<a href="#standalone">Standalone</a>
<a href="#runs">Run files</a>
<a href="#projects">Project files</a>
<a href="#commands">Commands</a>
<a href="#deps">Dependencies</a>
<a href="#testing">Testing</a>
@@ -154,6 +159,145 @@ make self <span class="c"># docgen's book of soleprint
the outputs open over <code>file://</code>.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="first">Your first project</h2>
<p>
From nothing to a notebook and a site for a project of yours, with your own text
in the right places. docgen lives in its own folder, and the project only gets one
config file and, later, a folder of notes.
</p>
<h3>1. Put docgen somewhere and check the machine</h3>
<pre><code>cp -r docgen ~/tools/docgen
cd ~/tools/docgen
make doctor <span class="c"># what is installed; never fails</span>
make sync <span class="c"># optional: C#/TypeScript, OpenAPI, all from uv.lock</span></code></pre>
<p>
From anywhere else, docgen runs as <code>python3 -m docgen</code> with its parent
folder on the path:
</p>
<pre><code>export PYTHONPATH=~/tools <span class="c"># the folder that holds docgen/</span>
python3 -m docgen --help</code></pre>
<h3>2. Write <code>docgen.toml</code> at the project's root</h3>
<pre><code>[project]
name = "orders-service"
[extract.code]
root = "src"
ext = [".cs"]
[emit.notebook]
output = "../docs_out/orders_service_notebook"
[emit.site]
output = "../docs_out/orders_service_site"</code></pre>
<p>
Paths are relative to this file. Keep the outputs outside the source tree. See
<a href="#projects">project files</a> for every key, and add
<code>.docgen/</code> to the project's <code>.gitignore</code>.
</p>
<h3>3. Check what it will do, then build</h3>
<pre><code>cd ~/work/orders-service
python3 -m docgen run docgen.toml --list <span class="c"># resolved paths; builds nothing</span>
python3 -m docgen run docgen.toml --check <span class="c"># build, then check every book</span></code></pre>
<pre><code>docgen.toml — 1 book(s)
ok code …/.docgen/orders-service/books/code
ok orders-service (document) …/.docgen/orders-service
1 book(s) in 1 document built and checked</code></pre>
<p>
A file docgen can't use is refused before anything is built, with every problem
listed at once and what to do about each. Open
<code>../docs_out/orders_service_site/index.html</code>: each book shows what
came in and what came out, and its pages click through from the overview.
</p>
<h3>4. Add your own text</h3>
<pre><code>python3 -m docgen scaffold docgen.toml --list <span class="c"># every spot a note can go</span>
python3 -m docgen scaffold docgen.toml <span class="c"># or: write an empty stub for each</span></code></pre>
<p>
Pick a spot and write markdown into <code>notes/&lt;book&gt;/&lt;spot&gt;.md</code>,
for example <code>notes/code/overview.md</code> above the overview page. Rebuild
and it's there, in the notebook and on the site. Stubs you leave empty add
nothing.
</p>
<div class="note">
<b>If a note stops appearing</b>, the page it was filed under was probably
renamed, for instance a package that moved. The rebuild says so and
<code>--check</code> fails with the note's name. Rename the file to the new spot
(<code>scaffold --list</code> shows them) or delete it.
</div>
<h3>5. Rebuild whenever the code moves</h3>
<p>
<code>python3 -m docgen run docgen.toml --check</code> again. A rebuild replaces
exactly what the last one wrote and never touches your notes or anything you
added to the output folders. <code>--only code</code> rebuilds one book while
you're writing its notes.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="models">Working with a model</h2>
<p>
A model can write the config and the notes. It shouldn't write anything else,
since every other file is generated. docgen ships the instructions to give it:
</p>
<pre><code>python3 -m docgen contract &gt; docgen-contract.md</code></pre>
<p>
That file is <code>docs/models.md</code>, printed verbatim. It lists every key
docgen accepts and the extensions it reads, has rules like "never a credential"
and "don't guess page ids", and ends with a table that maps each error docgen
can produce to the fix. docgen's own suite checks it against the code, so it
can't quietly go out of date: a key the loader accepts but the doc doesn't
name, or an example in it that doesn't load, fails <code>make check</code>.
</p>
<h3>The loop</h3>
<ol>
<li><b>Give it the project and the contract.</b> A digest of the project
(<code>distill</code>), plus <code>docgen-contract.md</code> as an
attachment rather than pasted. If replies go through <code>explode</code>,
attach its contract too: explode's says how to format the reply, docgen's
says what goes in the files.</li>
<li><b>Ask for <code>docgen.toml</code> first.</b> Unpack it, then run
<code>run docgen.toml --list</code>. If it's refused, paste the errors back
verbatim. Each one is in the contract's repair table.</li>
<li><b>Build once, then list the spots.</b> Run
<code>scaffold docgen.toml --list</code> and paste the output back. Page
names depend on the real directories, and a model that guesses them writes
notes that land nowhere.</li>
<li><b>Ask for notes for those spots.</b> Unpack, rebuild with
<code>--check</code>, and paste back any note reported as filed under a page
that doesn't exist.</li>
</ol>
<div class="note warn">
<b>What a model tends to write that docgen refuses.</b> In a dry run, a model
asked for a live database URL (with its password), a <code>seed</code> reader,
and shell, YAML and SQL files as code. All three are refused with a reason, and
the URL is never echoed in the error. The contract tells the model up front, but
the refusal is what makes it safe.
</div>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="layers">The three concerns</h2>
@@ -641,6 +785,69 @@ python3 -m docgen view ir.json --shape <span class="c"># what will this
dark for everything that is not a declaration — imports, constants, prose.</figcaption>
</figure>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="pages">Pages</h2>
<p>
A notebook doesn't scroll a wide image sideways: it scales it down to the cell
width until the labels can't be read. Zooming and panning is a workaround for a
picture that holds too much. So a book's diagram is split into <b>pages</b>,
each drawn small enough to fit the column it's going into.
</p>
<pre><code>overview one box per top-level group, relationships lifted
└── station one box per child of station
└── station.tools one box per child of station.tools, and so on</code></pre>
<p>
Pages follow structure, never scissors. Slicing one large image cuts boxes and
edges at the seams, and a slice means nothing on its own. A page is a subsystem:
its boxes are the scope's children, and anything outside the scope that they
touch appears as a <em>context</em> box, collapsed to the branch it sits on. A
page still says what its subsystem reaches without drawing the rest of the tree.
</p>
<h3>Fitted by measuring, not estimating</h3>
<p>
Whether a page fits depends on its labels, fonts and spacing, so an estimate from
node counts is wrong the moment the names are long. Each page is rendered and its
width read back. If a page is too wide top-to-bottom, it's drawn left-to-right,
since height is free in a notebook and width isn't. If it still doesn't fit
(a level of many peers with no smaller structure to split into), the page is
kept and flagged <em>too wide</em> in its caption and in the index. Never cut,
never dropped.
</p>
<p>
Measured on soleprint, whose module graph rendered at 14:1 as one picture: 69
pages, and at the site's 800px column the widest flagged page is 988px.
Schemas page differently: cards in columns capped to the width, grouped by
tables joined by foreign keys, so a schema grows downward instead of sideways.
</p>
<div class="note">
<b>Which pages exist doesn't depend on width</b>, only how each is laid out
does. A notebook column and a site column get the same pages, which is what lets
a <a href="#notes">note</a> filed under a page land in the same place in both.
</div>
<h3>Clicking through, on the site</h3>
<p>
Every page is its own HTML file with the diagram <b>inline</b>, not in an
<code>&lt;img&gt;</code>, which swallows clicks. A box standing for a subsystem
opens that subsystem's page. A module box opens its section of the book's
reference page. A context box opens its own branch. Breadcrumbs lead back up.
The size is adjustable: <b>fit</b> scales to the column, <b>1:1</b> shows the
drawing at natural size with scrollbars, the choice is remembered, and
<b>full screen</b> opens the pan-and-zoom viewer.
</p>
<p>
A notebook can't follow a link inside an image, so it gets a <b>page index</b>
instead: every page, indented by depth, each a link to its figure.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="explore">Explore</h2>
@@ -747,6 +954,56 @@ python3 -m docgen view ir.json --shape <span class="c"># what will this
<pre><code>python3 -m docgen emit notebook ir.json --scaffold overlay.json
python3 -m docgen emit notebook ir.json --overlay overlay.json -o walkthrough.ipynb</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="notes">Notes</h2>
<p>
Generation alone gives a document that's never stale and never says anything a
parser couldn't work out. Prose is the part only a person adds, and it has to
survive every rebuild, so it can't live in the output. It lives in a
<b>notes folder</b>, one markdown file per spot, named after the page or step it
belongs above:
</p>
<pre><code>notes/
├── _intro.md top of the document
├── _end.md bottom of the document
└── &lt;book&gt;/
├── _intro.md top of that book's part
├── larder.md above "what came in"
├── overview.md above the overview page
└── station.tools.md above the page for station.tools</code></pre>
<p>
To put text above a page, create the file named after it. No schema, no JSON,
no ids to look up. That's something a person or a model can do without reading
anything first. Local images in a note are embedded: as a cell attachment in the
notebook, as a data URI on the site.
</p>
<pre><code>make scaffold CONFIG=docgen.toml <span class="c"># an empty stub for every spot, never overwriting</span></code></pre>
<p>
<code>scaffold</code> lists the spots rather than making you guess them. A stub
is a single HTML comment and renders nothing, so scaffolding a hundred spots
changes the output not at all until one is filled in. Run it again after the
sources change and it only adds the new spots.
</p>
<div class="note warn">
<b>A note whose page is gone is reported, not dropped.</b> Renaming a package
renames its page, and the note filed under the old name has nowhere to land.
Discarding it quietly is how hand-written text goes stale without anyone
noticing, so it appears in the document ledger and fails the check level. It's
the same rule the notebook overlay follows.
</div>
<p>
The <a href="#notebooks">overlay</a> stays for what notes can't do: drop a
step, reorder, replace generated code. Notes are applied after the overlay, so a
step that moves takes its note with it.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="style">Style &amp; colour</h2>
@@ -872,10 +1129,10 @@ make doctor
<table>
<tr><th>context</th><th>checks</th><th>skipped</th></tr>
<tr><td>in the repo, synced</td><td>287</td><td>—</td></tr>
<tr><td>copied out, bare <code>python3</code></td><td>270</td><td>tree-sitter, OpenAPI, the in-place case</td></tr>
<tr><td>copied out, synced</td><td>279</td><td>OpenAPI, the in-place case</td></tr>
<tr><td>copied out, synced, <code>DOCGEN_REFERENCE</code> set</td><td>286</td>
<tr><td>in the repo, synced</td><td>346</td><td>—</td></tr>
<tr><td>copied out, bare <code>python3</code></td><td>329</td><td>tree-sitter, OpenAPI, the in-place case</td></tr>
<tr><td>copied out, synced</td><td>338</td><td>OpenAPI, the in-place case</td></tr>
<tr><td>copied out, synced, <code>DOCGEN_REFERENCE</code> set</td><td>345</td>
<td>the in-place case — correctly, there is no repo above</td></tr>
</table>
@@ -973,6 +1230,80 @@ python3 -m docgen run docgen.toml --list <span class="c"># resolved paths
because those are data that tools write.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="projects">Project files</h2>
<p>
A run file lists independent books. A <b>project file</b> describes one
<em>document</em> built from several sources: an API contract and the code
behind it, a schema and the service that uses it. It's the shape a model wrote
unprompted in a dry run, so docgen reads it as written:
</p>
<pre><code><span class="c"># &lt;project&gt;/docgen.toml</span>
[project]
name = "orders-api"
title = "Orders API"
notes = "notes" <span class="c"># default</span>
build = ".docgen" <span class="c"># book artifacts, default; gitignore it</span>
[extract.openapi]
spec = "doc/contracts/openapi.yaml"
[extract.code]
root = "src"
ext = [".cs"]
[ops]
drop_stdlib = true
drop_builtins = true
[emit.notebook]
output = "../docs_out/orders_api_notebook"
theme = "lucid"
[emit.site]
output = "../docs_out/orders_api_site"
width = 800</code></pre>
<p>
Each <code>[extract.*]</code> table is a <b>book</b>, and a book is a part of
every output rather than an output of its own. Each <code>[emit.*]</code> table
is an output made of all of them: one notebook and one site, each with a part
per book. Several sources of one kind use <code>[[extract.code]]</code> with a
<code>name</code> each. <code>[ops]</code> applies to every book's view before
paging.
</p>
<p>
Every part still opens with what came in and closes with what came out, and the
document opens with every larder and closes with every reconciliation. A book
that can't be built doesn't cost the others: the outputs are assembled from
what was built, and both say <em>this document is incomplete</em> and name what's
missing.
</p>
<h3>Refused, with what to do instead</h3>
<table>
<tr><th>written</th><th>why not</th></tr>
<tr><td><code>[extract.db] url = …</code></td>
<td>docgen reads a schema dump and never connects, and a URL in a committed
file is a credential in a committed file. Write the schema with
<code>modelgen from-db</code>, then <code>schema = …</code>. The URL is never
repeated in the error.</td></tr>
<tr><td><code>ext = [".sh", ".yaml", ".sql"]</code></td>
<td>No grammar for those yet. The error names the extensions that are read.</td></tr>
<tr><td><code>[extract.seed]</code></td>
<td>No such reader. The error names the ones that exist.</td></tr>
<tr><td><code>[project]</code> and <code>[[book]]</code> in one file</td>
<td>Which one it meant is exactly what nobody should have to guess.</td></tr>
</table>
<p>
<code>docgen.project.example.toml</code> ships beside the run file example and
builds docgen's own fixtures into one document, with a sample note.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="commands">Commands</h2>
@@ -981,7 +1312,9 @@ python3 -m docgen run docgen.toml --list <span class="c"># resolved paths
<tr><th>target</th><th>does</th></tr>
<tr><td><code>make sync</code></td><td><code>.venv</code> with every optional group, from <code>uv.lock</code></td></tr>
<tr><td><code>make book SRC=…</code></td><td><b>one whole operation</b>, measured at both ends</td></tr>
<tr><td><code>make run CONFIG=…</code></td><td>every book a <a href="#runs">run file</a> lists</td></tr>
<tr><td><code>make run CONFIG=…</code></td><td>every book a <a href="#runs">run file</a> lists, or a <a href="#projects">project</a>'s document</td></tr>
<tr><td><code>make scaffold CONFIG=…</code></td><td>an empty <a href="#notes">note</a> for every page and step</td></tr>
<tr><td><code>python3 -m docgen contract</code></td><td>the <a href="#models">instructions for a model</a> that writes the config and notes</td></tr>
<tr><td><code>make check</code></td><td>docgen's own suite, offline, nothing installed</td></tr>
<tr><td><code>make check BOOK=…</code></td><td>one book's own level — generated and custom</td></tr>
<tr><td><code>make doctor</code></td><td>what this machine has and what it is missing</td></tr>
@@ -1028,6 +1361,10 @@ python3 -m docgen run docgen.toml --list <span class="c"># resolved paths
python3 -m docgen run docgen.toml <span class="c"># every book a run file lists</span>
python3 -m docgen check <span class="c"># docgen's own suite</span>
python3 -m docgen check out/book/slug <span class="c"># that book's level</span>
python3 -m docgen check .docgen/&lt;project&gt; <span class="c"># a project document's level</span>
python3 -m docgen scaffold docgen.toml <span class="c"># stub notes for every spot</span>
python3 -m docgen scaffold docgen.toml --list <span class="c"># the spots, written nowhere</span>
python3 -m docgen contract <span class="c"># the instructions for a model</span>
python3 -m docgen extract python --root SRC -o ir.json
python3 -m docgen extract code --root SRC -o ir.json
@@ -1136,14 +1473,14 @@ make lock <span class="c"># after editing pyproject
<tr><th>command</th><th>asks about</th><th>fails?</th></tr>
<tr><td><code>make doctor</code></td><td>the machine — what is installed</td>
<td>never; it reports</td></tr>
<tr><td><code>make check</code></td><td>docgen — 287 checks</td>
<tr><td><code>make check</code></td><td>docgen — 346 checks</td>
<td>exit 1</td></tr>
<tr><td><code>make check BOOK=&lt;dir&gt;</code></td><td>that one book</td>
<tr><td><code>make check BOOK=&lt;dir&gt;</code></td><td>that one book, or a project's document</td>
<td>exit 1</td></tr>
</table>
<p>
All 287 after <code>make sync</code>, 272 with nothing optional installed — the
All 346 after <code>make sync</code>, 331 with nothing optional installed — the
suite skips rather than fails when tree-sitter, lxml or the OpenAPI reader is
absent. See <a href="#standalone">standalone</a> for the counts outside the repo.
Among them, two that keep this page's structure true: nothing outside