mariano/soleprint
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new conventions

Browse Source
This commit is contained in:
buenosairesam
2026-09-14 06:13:22 -03:00
parent 37c4d588ea
commit a29e0708e8
22 changed files with 3924 additions and 682 deletions
Download Patch File Download Diff File Expand all files Collapse all files

348
soleprint/atlas2/docgen/docs/index.html
View File

@@ -22,6 +22,7 @@
<div class="group">Architecture</div>
<a href="#layers">The three concerns</a>
<a href="#book">The book</a>
<a href="#ir">The IR</a>
<a href="#shape">Shape decides the drawing</a>
@@ -39,6 +40,7 @@
<a href="#style">Style &amp; colour</a>
<div class="group">Reference</div>
<a href="#standalone">Standalone</a>
<a href="#commands">Commands</a>
<a href="#deps">Dependencies</a>
<a href="#testing">Testing</a>
@@ -107,7 +109,23 @@
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="quick">Five minutes</h2>
<p>Three commands, and they compose. That is the whole interface.</p>
<p>
One command, if you want the whole thing — a <a href="#book">book</a>: what
went in, every step, what came out, and a page to open.
</p>
<pre><code>make book SRC=/path/to/repo BOOK=out/book/mine
make check BOOK=out/book/mine</code></pre>
<pre><code> larder /path/to/repo — 45 files read, 2 failed, 12 packages
book 312 nodes · 244 edges · 26 external · 8 artifacts
ok 45 file(s) read produced 45 module(s)
open out/book/mine/site/index.html</code></pre>
<p>
Underneath it is three commands that compose, and each still works on its own.
That is the interface, and the book does not replace it:
</p>
<pre><code><span class="c"># 1. read something</span>
python3 -m docgen.extractors.python --root ../station/tools/histgen -o ir.json
@@ -123,7 +141,7 @@ python3 -m docgen.emitters auto view.json -o out/</code></pre>
<pre><code>make ir SRC=/path/to/repo OUT=out <span class="c"># extract</span>
make explore OUT=out <span class="c"># the two-pane navigator</span>
make site OUT=out <span class="c"># a docs site with a sidebar</span>
make self <span class="c"># run the whole thing over soleprint</span></code></pre>
make self <span class="c"># docgen's book of soleprint, then check it</span></code></pre>
<p>
Everything is offline and self-contained. No server, no CDN, no build step —
@@ -166,6 +184,151 @@ make self <span class="c"># run the whole thing over s
</figcaption>
</figure>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="book">The book</h2>
<p>
A <b>book</b> is one docgen operation, and it has a fixed shape: it begins by
saying what went in and ends by saying what came out. Everything in between is
an ordinary file that stands on its own.
</p>
<pre><code>larder ──► step ──► step ──► step ──► book
what each one usable what
came in by itself came out</code></pre>
<p>
The word comes from Atlas 1.0, where a book is a <em>larder</em> composed with a
pattern into something published and served. docgen had already built all three
parts under other names, so this is less an adoption than a renaming back.
</p>
<h3>Why both ends, rather than just the result</h3>
<p>
Because the two numbers are only worth having together. <em>"1,505 nodes"</em>
is not a fact about anything. <em>"225 files in, 1,505 nodes out, nothing
lost"</em> is. Before this, an extractor that read 45 of 47 files produced
exactly the same document as one that read all 47, and the diagram looked
complete either way — there was nowhere for the other two to be mentioned.
</p>
<div class="note warn">
<b>A clean diagram over an incomplete read is a lie by omission.</b> It is also
the failure mode the IR already guards against one level down: an unresolved
name becomes an <code>external</code> node rather than being dropped, because
silently losing a thing is worse than recording an unresolved one. The larder
measure is that same rule applied to the input as a whole.
</div>
<h3>The larder — what came in</h3>
<p>
Deliberately not called a bucket. A bucket is somewhere bytes sit; a larder is
<em>stocked from outside</em>, has an inventory, and goes stale. All three are
worth measuring, and they are what the measure records:
</p>
<pre><code>"larder": {
"kind": "python", <span class="c">// which extractor stocked it</span>
"identity": "../../station", <span class="c">// path, or a DSN with the password masked</span>
"unit": "file", <span class="c">// file | table | path | entry | document</span>
"seen": 47, <span class="c">// what the larder offered</span>
"read": 45, <span class="c">// seen - len(failed), derived</span>
"failed": [{"name": "a.py", "error": "syntax: line 3"}],
"extra": {"packages": 12}
}</code></pre>
<p>
<code>read</code> is derived and never stored. Stored, it invites the question
<em>"does that include the failures?"</em> and every reader answers it
differently; derived, there is nothing to get wrong — and
<code>ir/validate.py</code> fails a document whose arithmetic disagrees with
itself.
</p>
<p>
Failures are recorded <b>by name</b>, not counted. A count tells you a book is
incomplete; a name tells you which part of it to distrust.
</p>
<div class="note">
<b><code>identity</code> is the one field in docgen that could carry a secret</b>
— a database DSN has the password in it. It is masked at construction, and
<code>validate.py</code> then sweeps for the mask having worked, using its own
independent key list. A scrubber graded by its own word is not graded.
</div>
<h3>The book measure — what came out, and reconciled</h3>
<p>
Counts by kind, edges by kind, externals, and every artifact with its byte
count. On its own that is just a summary. What makes it a measure is that it is
<em>reconciled</em> against the larder:
</p>
<pre><code> larder ../../station — 45 files read, 2 failed, 12 packages
book 312 nodes · 244 edges · 26 external · 8 artifacts
ok 45 file(s) read produced 45 module(s)
ok 2 file(s) could not be read
ok 2 unreadable file(s) appear as 2 marked module(s)</code></pre>
<p>
The relation differs by source and is declared per extractor, because getting
it wrong gives a check that passes for the wrong reason. One file becomes one
module node — fewer means input was dropped. Four hundred HAR entries becoming
twelve endpoints is not a loss, it is the <em>point</em> of the capture. And a
file that failed to parse must still appear in the graph, carrying its error,
or the picture is smaller than the source and says nothing about it.
</p>
<p>
<code>python3 -m docgen.book</code> exits 1 when a reconciliation fails. The
book is still written — the evidence is the point — but a build that lost input
should fail a pipeline rather than pass quietly.
</p>
<h3>The notebook is the sequence, the web is the last step</h3>
<p>
These two rule what gets generated, and each for its own reason.
</p>
<p>
The <b>notebook</b> is not one artifact; it is the sequence. Its first cell is
the larder measure and its last cell is the book measure, which is what puts
the two ends in the <em>document</em> rather than only in the tooling. Between
them, one pair of cells per step: what the step did, and a cell that loads that
step's artifact and prints one fact about it. That is the "usable by
themselves" property made executable, and the test suite runs those cells.
</p>
<p>
The <b>web output</b> is last, so nothing depends on it, so it can be replaced
wholesale without touching anything upstream. That is exactly what lets it rule
the output without being a stable contract — the book measure is the promise,
and the page displaying it is free to change drastically and often. It is also
the artifact somebody definitely opens, which is why <em>"2 of 47 files could
not be read"</em> has to appear there, above the diagram rather than below it.
</p>
<div class="note">
<b>The spine is scaffolding, not a gate.</b> Running one step alone is still a
book, just a short one — <code>make ir</code> works exactly as it did. An
operation that cannot measure something says what it could not measure and
carries on. Gating would destroy the property that makes the intermediate
artifacts useful, which is the whole reason the sequence is worth having.
</div>
<h3>What a book looks like on disk</h3>
<pre><code>book/&lt;slug&gt;/
├── book.json <span class="c">both measures, the steps, artifacts with byte counts</span>
├── steps/ <span class="c">every intermediate — ir.json, view.json, graph.svg, …</span>
├── notebook.ipynb <span class="c">the sequence; first and last cells are the measures</span>
├── overlay.json <span class="c">hand-written, optional, re-applied every build</span>
├── checks.py <span class="c">this book's own assertions — optional</span>
└── site/ <span class="c">the web output, both measures at the top</span></code></pre>
<pre><code>make book SRC=../station BOOK=out/book/station
make check BOOK=out/book/station</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="ir">The IR</h2>
@@ -627,13 +790,96 @@ python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthroug
between a representative leaf and clip the line at the cluster border.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="standalone">Standalone</h2>
<p>
Copy the <code>docgen/</code> folder anywhere and it works. The Makefile
derives its own package name from where it sits, so it can be renamed too, and
everything else resolves inside the directory.
</p>
<pre><code>cp -r docgen /somewhere/else
cd /somewhere/else/docgen
make doctor <span class="c"># what this machine has</span>
make check <span class="c"># the suite, from the copy</span>
make book SRC=/path/to/any/repo BOOK=out/book/theirs</code></pre>
<h3>One seam, and it is optional</h3>
<p>
Exactly one capability needs more than the folder: reading an <b>OpenAPI</b>
document goes through <code>station/tools/modelgen</code>, which parses the
spec and resolves <code>$ref</code>. It is deliberately not reimplemented here
— a second OpenAPI reader in one repo is two things to keep correct.
</p>
<p>
So if you are using docgen standalone but keeping the repo alongside as
reference, point at it:
</p>
<pre><code>export DOCGEN_REFERENCE=/path/to/repo
make doctor
<span class="c"># reference: /path/to/repo (from $DOCGEN_REFERENCE)</span></code></pre>
<p>
Resolution is <code>$DOCGEN_REFERENCE</code> first, then walking up from the
package — so in place it needs no configuration, and an explicit path wins
when set. Without it, the four other extractors and every emitter work
unchanged; the OpenAPI reader reports what to set, and the suite
<em>skips</em> rather than fails.
</p>
<div class="note">
<b>An env var rather than a config file</b>, because there is one setting and
it is a path. A config file for one path is a file to find, parse, document and
validate, and the first question anyone asks of it is "where does it live" —
which is the same question again.
</div>
<h3>It is asserted, not asserted-in-prose</h3>
<p>
A standalone claim decays the moment somebody adds a convenient import, and it
decays <em>silently</em>, because the suite still passes inside the repo. So
the suite reads its own source:
</p>
<ul>
<li><b>Imports are the stdlib, docgen itself, and a short allow-list</b> —
tree-sitter, lxml, yaml, networkx, modelgen, each optional and each with a
reason. A new name is a new dependency in a folder meant to be copied.</li>
<li><b>Only <code>extractors/openapi.py</code> imports modelgen.</b> One seam
is a seam; two is a dependency.</li>
<li><b>Only <code>reference.py</code> knows the repo layout</b>, so pointing
docgen elsewhere is one change rather than a search.</li>
<li><b>A book writes only inside its own output directory</b> — nothing is
left in the tree being read.</li>
</ul>
<p>
The folder was also literally copied to <code>/tmp</code> and run, which is
how the one real bug here was found: a check asserting the reference repo is
reachable, correct in place and wrong the moment there was nothing above. It
now reports which case applies instead of assuming one.
</p>
<table>
<tr><th>context</th><th>checks</th><th>skipped</th></tr>
<tr><td>in the repo</td><td>250</td><td>tree-sitter (259 with it)</td></tr>
<tr><td>copied out</td><td>242</td><td>tree-sitter, OpenAPI, the in-place case</td></tr>
<tr><td>copied out, <code>DOCGEN_REFERENCE</code> set</td><td>249</td>
<td>tree-sitter, the in-place case</td></tr>
</table>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="commands">Commands</h2>
<h3>Make</h3>
<table>
<tr><th>target</th><th>does</th></tr>
<tr><td><code>make check</code></td><td>the whole test suite, offline, nothing installed</td></tr>
<tr><td><code>make book SRC=…</code></td><td><b>one whole operation</b>, measured at both ends</td></tr>
<tr><td><code>make check</code></td><td>docgen's own suite, offline, nothing installed</td></tr>
<tr><td><code>make check BOOK=…</code></td><td>one book's own level — generated and custom</td></tr>
<tr><td><code>make doctor</code></td><td>what this machine has and what it is missing</td></tr>
<tr><td><code>make ir SRC=…</code></td><td>extract Python into <code>OUT/ir.json</code></td></tr>
<tr><td><code>make code SRC=…</code></td><td>extract C#/TypeScript <span class="pill opt">tree-sitter</span></td></tr>
@@ -644,18 +890,30 @@ python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthroug
<tr><td><code>make minimap</code></td><td>what is where, read from the colours</td></tr>
<tr><td><code>make explore</code></td><td>the two-pane navigator</td></tr>
<tr><td><code>make site</code></td><td>a self-contained docs site</td></tr>
<tr><td><code>make self</code></td><td>the whole pipeline over soleprint itself</td></tr>
<tr><td><code>make self</code></td><td>docgen's book of soleprint, then check it</td></tr>
</table>
<p>
Variables: <code>SRC</code>, <code>OUT</code>, <code>SCHEMA</code>,
<code>STYLE</code>, <code>THEME</code>, <code>SCALE</code>, <code>DEPTH</code>,
<code>PY</code>. The Makefile derives its own package name from where it sits,
so the folder can be copied anywhere and renamed and still work.
<code>OPENAPI</code>, <code>HAR</code>, <code>BOOK</code>, <code>SLUG</code>,
<code>READER</code>, <code>OVERLAY</code>, <code>STYLE</code>,
<code>THEME</code>, <code>SCALE</code>, <code>DEPTH</code>, <code>PY</code>.
The Makefile derives its own package name from where it sits, so the folder can
be copied anywhere and renamed and still work.
</p>
<p>
<code>READER</code> rather than <code>LANG</code> because <code>LANG</code> is
the shell's locale variable, so <code>?=</code> inherits
<code>en_US.UTF-8</code> from the environment and the argument is rejected.
Every target above is one step of a book and still works alone — that is the
property the spine exists to preserve, not to replace.
</p>
<h3>Modules</h3>
<pre><code>python3 -m docgen.extractors.python --root SRC -o ir.json
<pre><code>python3 -m docgen.book --root SRC -o out/book/slug <span class="c"># the whole operation</span>
python3 -m docgen.book.checks out/book/slug <span class="c"># that book's level</span>
python3 -m docgen.extractors.python --root SRC -o ir.json
python3 -m docgen.extractors code --root SRC -o ir.json
python3 -m docgen.extractors db --schema schema.json -o ir.json
python3 -m docgen.extractors openapi --spec spec.yaml -o ir.json
@@ -676,6 +934,11 @@ python3 -m docgen.emitters explore ir.json -o explore/</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="deps">Dependencies</h2>
<p>
Everything below is optional. The stdlib covers the whole structural path — see <a href="#standalone">standalone</a>
for the one seam out of the folder.
</p>
<p>
The core is <strong>standard library only</strong>. Everything else is optional
and reported by <code>make doctor</code>; when something is missing you lose
@@ -707,11 +970,68 @@ python3 -m docgen.emitters explore ir.json -o explore/</code></pre>
<!-- ─────────────────────────────────────────────────────────────── -->
<h2 id="testing">Testing</h2>
<pre><code>make check <span class="c"># 191 checks, offline, no network</span></code></pre>
<p>
Three levels, and they differ by <em>what they assert about</em>. The
distinction decides what a failure means, which is why it is worth keeping:
</p>
<table>
<tr><th>command</th><th>asks about</th><th>fails?</th></tr>
<tr><td><code>make doctor</code></td><td>the machine — what is installed</td>
<td>never; it reports</td></tr>
<tr><td><code>make check</code></td><td>docgen — 259 checks</td>
<td>exit 1</td></tr>
<tr><td><code>make check BOOK=&lt;dir&gt;</code></td><td>that one book</td>
<td>exit 1</td></tr>
</table>
<p>
All 259 with both optional dependencies installed, 244 with neither — the suite
skips rather than fails when tree-sitter, lxml or the OpenAPI reader is absent.
See <a href="#standalone">standalone</a> for the counts outside the repo.
</p>
<h3>The book level, where custom checks live</h3>
<p>
The third level is the one that reaches a project docgen has never seen, and it
is where <b>framework and hand-written checks live together</b>. The generated
half is the spine's own assertions, identical for every book: both measures
present, the reconciliation holding, every artifact where the ledger says it
is, the notebook still executing. The custom half goes in the book's own
<code>checks.py</code> and uses the same helpers, so a project's line and a
framework line read identically and fail identically.
</p>
<pre><code><span class="c"># out/book/station/checks.py</span>
<span class="k">def</span> checks(book, check, note, skip):
note("what this project will not give up")
<span class="c"># Payments moved once already and the move broke three dashboards.</span>
<span class="c"># If it is not here, something renamed it again.</span>
check("payments is still a module", True,
any(n["id"] == "app.payments" <span class="k">for</span> n <span class="k">in</span> book.ir["nodes"]))</code></pre>
<p>
Same split as the notebook's base and overlay, for the same reason: generation
alone cannot know what <em>this</em> project cares about, and hand-authoring
alone rots.
</p>
<div class="note">
<b>Each check is one decision that has already been made, with the reason above
it.</b> Not coverage, and deliberately not an exhaustive sweep. A rule without
its reason gets overridden the first time it is inconvenient, so failing a
check should read as <em>"you are about to undo this"</em> rather than
<em>"something broke"</em>. The idiom is carried from rig's
<code>ctrl/selftest.sh</code>, which is where the three-level split comes from.
</div>
<h3>The four that are the design</h3>
<p>
Four of those are the <em>design</em> rather than regressions, and they are the
ones to keep if anything is ever cut:
Of docgen's own checks, four assert the <em>architecture</em> rather than guard
a regression, and they are the ones to keep if anything is ever cut:
</p>
<ul>
<li><b>No visual field reaches the IR</b> — extractors cannot decide appearance.</li>
@@ -736,8 +1056,10 @@ python3 -m docgen.emitters explore ir.json -o explore/</code></pre>
<p>
Self-hosting is the honest end-to-end check, and it is where the real bugs came
from — two name-resolution faults and a duplicate-id crash that no fixture had
reached. <code>make self</code> runs the whole pipeline over soleprint; if the
index does not read like the system, something is wrong.
reached. <code>make self</code> builds docgen's book <em>of soleprint</em> and
then runs that book's own level against it, so the two ends have to reconcile
on 225 real files. If the index does not read like the system, something is
wrong.
</p>
<!-- ─────────────────────────────────────────────────────────────── -->