add project, change cli file names, use config
This commit is contained in:
@@ -41,6 +41,7 @@
|
||||
|
||||
<div class="group">Reference</div>
|
||||
<a href="#standalone">Standalone</a>
|
||||
<a href="#runs">Run files</a>
|
||||
<a href="#commands">Commands</a>
|
||||
<a href="#deps">Dependencies</a>
|
||||
<a href="#testing">Testing</a>
|
||||
@@ -117,6 +118,11 @@
|
||||
<pre><code>make book SRC=/path/to/repo BOOK=out/book/mine
|
||||
make check BOOK=out/book/mine</code></pre>
|
||||
|
||||
<p>
|
||||
More than one, rebuilt together, is a <a href="#runs">run file</a>:
|
||||
<code>make run CONFIG=docgen.toml</code>.
|
||||
</p>
|
||||
|
||||
<pre><code> larder /path/to/repo — 45 files read, 2 failed, 12 packages
|
||||
book 312 nodes · 244 edges · 26 external · 8 artifacts
|
||||
ok 45 file(s) read produced 45 module(s)
|
||||
@@ -128,13 +134,13 @@ make check BOOK=out/book/mine</code></pre>
|
||||
</p>
|
||||
|
||||
<pre><code><span class="c"># 1. read something</span>
|
||||
python3 -m docgen.extractors.python --root ../station/tools/histgen -o ir.json
|
||||
python3 -m docgen extract python --root ../station/tools/histgen -o ir.json
|
||||
|
||||
<span class="c"># 2. narrow it to a useful view</span>
|
||||
python3 -m docgen.ops ir.json --overview -o view.json
|
||||
python3 -m docgen view ir.json --overview -o view.json
|
||||
|
||||
<span class="c"># 3. draw whatever its structure asks for</span>
|
||||
python3 -m docgen.emitters auto view.json -o out/</code></pre>
|
||||
python3 -m docgen emit auto view.json -o out/</code></pre>
|
||||
|
||||
<p>Or through the Makefile, which is a thin wrapper over exactly those:</p>
|
||||
|
||||
@@ -281,7 +287,7 @@ make self <span class="c"># docgen's book of soleprint
|
||||
or the picture is smaller than the source and says nothing about it.
|
||||
</p>
|
||||
<p>
|
||||
<code>python3 -m docgen.book</code> exits 1 when a reconciliation fails. The
|
||||
<code>python3 -m docgen book</code> exits 1 when a reconciliation fails. The
|
||||
book is still written — the evidence is the point — but a build that lost input
|
||||
should fail a pipeline rather than pass quietly.
|
||||
</p>
|
||||
@@ -383,7 +389,7 @@ make check BOOK=out/book/station</code></pre>
|
||||
and it reads its field lists out of <code>schema.json</code> so the schema and
|
||||
the dataclasses cannot drift apart.
|
||||
</p>
|
||||
<pre><code>python3 -m docgen.ir ir.json</code></pre>
|
||||
<pre><code>python3 -m docgen validate ir.json</code></pre>
|
||||
<p>
|
||||
It catches what a schema cannot: an edge naming a node that does not exist, a
|
||||
containment cycle, a duplicate id, and a visual field smuggled into
|
||||
@@ -416,7 +422,7 @@ make check BOOK=out/book/station</code></pre>
|
||||
<tr><td><code>flat</code></td><td>the index</td><td>most nodes have no relationships: that is a list</td></tr>
|
||||
</table>
|
||||
|
||||
<pre><code>$ python3 -m docgen.emitters auto view.json -o out/
|
||||
<pre><code>$ python3 -m docgen emit auto view.json -o out/
|
||||
sheet -> index
|
||||
109 nodes sit at one level; any layered engine draws that as a
|
||||
strip. Split it, scope it, or read it as an index</code></pre>
|
||||
@@ -537,10 +543,10 @@ make check BOOK=out/book/station</code></pre>
|
||||
the same narrowing.
|
||||
</p>
|
||||
|
||||
<pre><code>python3 -m docgen.ops ir.json --overview -o view.json
|
||||
python3 -m docgen.ops ir.json --around docgen.ir --hops 2 -o view.json
|
||||
python3 -m docgen.ops ir.json --split -o parts/
|
||||
python3 -m docgen.ops ir.json --shape <span class="c"># what will this look like?</span></code></pre>
|
||||
<pre><code>python3 -m docgen view ir.json --overview -o view.json
|
||||
python3 -m docgen view ir.json --around docgen.ir --hops 2 -o view.json
|
||||
python3 -m docgen view ir.json --split -o parts/
|
||||
python3 -m docgen view ir.json --shape <span class="c"># what will this look like?</span></code></pre>
|
||||
|
||||
<p>
|
||||
All of them are IR→IR, all composable, and each produces a document that
|
||||
@@ -738,8 +744,8 @@ python3 -m docgen.ops ir.json --shape <span class="c"># what will this
|
||||
<b>absent</b> — it is an addition, never a dependency.
|
||||
</div>
|
||||
|
||||
<pre><code>python3 -m docgen.emitters notebook ir.json --scaffold overlay.json
|
||||
python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthrough.ipynb</code></pre>
|
||||
<pre><code>python3 -m docgen emit notebook ir.json --scaffold overlay.json
|
||||
python3 -m docgen emit notebook ir.json --overlay overlay.json -o walkthrough.ipynb</code></pre>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="style">Style & colour</h2>
|
||||
@@ -802,6 +808,7 @@ python3 -m docgen.emitters notebook ir.json --overlay overlay.json -o walkthroug
|
||||
<pre><code>cp -r docgen /somewhere/else
|
||||
cd /somewhere/else/docgen
|
||||
make doctor <span class="c"># what this machine has</span>
|
||||
make sync <span class="c"># optional: every group, from uv.lock</span>
|
||||
make check <span class="c"># the suite, from the copy</span>
|
||||
make book SRC=/path/to/any/repo BOOK=out/book/theirs</code></pre>
|
||||
|
||||
@@ -865,19 +872,116 @@ make doctor
|
||||
|
||||
<table>
|
||||
<tr><th>context</th><th>checks</th><th>skipped</th></tr>
|
||||
<tr><td>in the repo</td><td>250</td><td>tree-sitter (259 with it)</td></tr>
|
||||
<tr><td>copied out</td><td>242</td><td>tree-sitter, OpenAPI, the in-place case</td></tr>
|
||||
<tr><td>copied out, <code>DOCGEN_REFERENCE</code> set</td><td>249</td>
|
||||
<td>tree-sitter, the in-place case</td></tr>
|
||||
<tr><td>in the repo, synced</td><td>287</td><td>—</td></tr>
|
||||
<tr><td>copied out, bare <code>python3</code></td><td>270</td><td>tree-sitter, OpenAPI, the in-place case</td></tr>
|
||||
<tr><td>copied out, synced</td><td>279</td><td>OpenAPI, the in-place case</td></tr>
|
||||
<tr><td>copied out, synced, <code>DOCGEN_REFERENCE</code> set</td><td>286</td>
|
||||
<td>the in-place case — correctly, there is no repo above</td></tr>
|
||||
</table>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="runs">Run files</h2>
|
||||
|
||||
<p>
|
||||
A book gets rebuilt many times — after a merge, before a release, whenever the
|
||||
source moves — and each one is a source, an output directory and a handful of
|
||||
options. Typed out every time, those drift: one run excludes
|
||||
<code>migrations</code>, the next forgets. A <b>run file</b> writes them down
|
||||
once, so the rebuild is exact.
|
||||
</p>
|
||||
|
||||
<pre><code><span class="c"># docgen.toml — kept beside the project it describes</span>
|
||||
reference = "../soleprint" <span class="c"># optional; $DOCGEN_REFERENCE wins</span>
|
||||
|
||||
[defaults]
|
||||
out = "out/book" <span class="c"># each book goes to <out>/<name></span>
|
||||
theme = "dark"
|
||||
exclude = ["migrations", "tests"]
|
||||
|
||||
[[book]]
|
||||
name = "station"
|
||||
root = "../soleprint/station"
|
||||
|
||||
[[book]]
|
||||
name = "frontend"
|
||||
root = "../web/src"
|
||||
reader = "code" <span class="c"># C# / TypeScript, through tree-sitter</span>
|
||||
|
||||
[[book]]
|
||||
name = "orders-api"
|
||||
openapi = "specs/orders.yaml"
|
||||
overlay = "overlays/orders.json"
|
||||
out = "/srv/docs/orders"</code></pre>
|
||||
|
||||
<pre><code>make run CONFIG=docgen.toml
|
||||
make run CONFIG=docgen.toml ONLY=station,frontend CHECK=1
|
||||
python3 -m docgen run docgen.toml --list <span class="c"># resolved paths, nothing built</span></code></pre>
|
||||
|
||||
<pre><code>docgen.toml — 3 book(s)
|
||||
ok station /home/me/proj/out/book/station
|
||||
ok frontend /home/me/proj/out/book/frontend
|
||||
FAIL orders-api ImportError: modelgen is not reachable — …
|
||||
|
||||
1 of 3 book(s) did not hold — --verbose for each book's own output</code></pre>
|
||||
|
||||
<p>
|
||||
Each book names <b>exactly one</b> source — <code>root</code>,
|
||||
<code>schema</code>, <code>openapi</code> or <code>har</code> — the same choice
|
||||
the <code>book</code> command makes you take. One failing book never stops the
|
||||
rest, and the run exits 1 if any book failed to build, lost input between its
|
||||
two ends, or failed its checks. <code>docgen.example.toml</code> ships with
|
||||
docgen and runs against its own fixtures.
|
||||
</p>
|
||||
|
||||
<h3>The rules, each one a thing that goes wrong otherwise</h3>
|
||||
|
||||
<ul>
|
||||
<li><b>Paths are relative to the run file</b>, not to wherever the command ran.
|
||||
Otherwise the same file builds different books from different
|
||||
directories.</li>
|
||||
<li><b>A book's value replaces the default, for every key</b> — including
|
||||
<code>exclude</code>, where merging looks tempting. One rule nobody has to
|
||||
remember beats a clever one somebody has to look up. A default
|
||||
<code>exclude</code> simply does not apply to a schema or a spec.</li>
|
||||
<li><b>Unknown keys are refused, not ignored</b>, all of them at once.
|
||||
<code>exlude = [...]</code> silently doing nothing is how a rebuild quietly
|
||||
starts reading <code>node_modules</code>.</li>
|
||||
<li><b>A book may not be written inside the tree it reads</b> — the next run
|
||||
would read the last run's output — and <b>two books may not share an output
|
||||
directory</b>, where the second would clear the first every time.</li>
|
||||
<li><b>The environment beats the file.</b> A <code>reference</code> in the run
|
||||
file is used only when <code>$DOCGEN_REFERENCE</code> is not set, which is
|
||||
rig's precedence for every setting it has.</li>
|
||||
</ul>
|
||||
|
||||
<div class="note">
|
||||
<b>A rebuild replaces the previous build, and nothing else.</b> Before writing,
|
||||
a book removes exactly what a build writes — <code>book.json</code>,
|
||||
<code>notebook.ipynb</code>, <code>steps/</code>, <code>site/</code>,
|
||||
<code>explore/</code> — and only from a directory that already holds a
|
||||
<code>book.json</code>. Without it a graph that is now <code>graph.md</code> kept
|
||||
its old <code>graph.svg</code>, unlisted in the ledger and still on the page.
|
||||
A hand-written <code>checks.py</code> or <code>overlay.json</code> is never
|
||||
touched, and neither is a directory docgen did not write.
|
||||
</div>
|
||||
|
||||
<p>
|
||||
TOML because a run file is written by people: <code>tomllib</code> is the
|
||||
stdlib, it takes comments — the reason a book excludes something belongs next
|
||||
to the exclusion — and it is already the format of the
|
||||
<code>pyproject.toml</code> beside it. The IR and the style files stay JSON,
|
||||
because those are data that tools write.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="commands">Commands</h2>
|
||||
|
||||
<h3>Make</h3>
|
||||
<table>
|
||||
<tr><th>target</th><th>does</th></tr>
|
||||
<tr><td><code>make sync</code></td><td><code>.venv</code> with every optional group, from <code>uv.lock</code></td></tr>
|
||||
<tr><td><code>make book SRC=…</code></td><td><b>one whole operation</b>, measured at both ends</td></tr>
|
||||
<tr><td><code>make run CONFIG=…</code></td><td>every book a <a href="#runs">run file</a> lists</td></tr>
|
||||
<tr><td><code>make check</code></td><td>docgen's own suite, offline, nothing installed</td></tr>
|
||||
<tr><td><code>make check BOOK=…</code></td><td>one book's own level — generated and custom</td></tr>
|
||||
<tr><td><code>make doctor</code></td><td>what this machine has and what it is missing</td></tr>
|
||||
@@ -896,8 +1000,10 @@ make doctor
|
||||
<p>
|
||||
Variables: <code>SRC</code>, <code>OUT</code>, <code>SCHEMA</code>,
|
||||
<code>OPENAPI</code>, <code>HAR</code>, <code>BOOK</code>, <code>SLUG</code>,
|
||||
<code>READER</code>, <code>OVERLAY</code>, <code>STYLE</code>,
|
||||
<code>THEME</code>, <code>SCALE</code>, <code>DEPTH</code>, <code>PY</code>.
|
||||
<code>READER</code>, <code>OVERLAY</code>, <code>CONFIG</code>,
|
||||
<code>ONLY</code>, <code>CHECK</code>, <code>STYLE</code>, <code>THEME</code>,
|
||||
<code>SCALE</code>, <code>PY</code>. <code>PY</code> is the synced
|
||||
<code>.venv</code> when there is one and <code>python3</code> when there is not.
|
||||
The Makefile derives its own package name from where it sits, so the folder can
|
||||
be copied anywhere and renamed and still work.
|
||||
</p>
|
||||
@@ -909,49 +1015,100 @@ make doctor
|
||||
property the spine exists to preserve, not to replace.
|
||||
</p>
|
||||
|
||||
<h3>Modules</h3>
|
||||
<pre><code>python3 -m docgen.book --root SRC -o out/book/slug <span class="c"># the whole operation</span>
|
||||
python3 -m docgen.book.checks out/book/slug <span class="c"># that book's level</span>
|
||||
<h3>The command line</h3>
|
||||
|
||||
<p>
|
||||
One entry point, <code>python3 -m docgen <command></code>, run from the
|
||||
directory <em>above</em> the folder (or with it on <code>PYTHONPATH</code>, which
|
||||
is what the Makefile does). Each command is plain argparse and takes
|
||||
<code>--help</code>.
|
||||
</p>
|
||||
|
||||
python3 -m docgen.extractors.python --root SRC -o ir.json
|
||||
python3 -m docgen.extractors code --root SRC -o ir.json
|
||||
python3 -m docgen.extractors db --schema schema.json -o ir.json
|
||||
python3 -m docgen.extractors openapi --spec spec.yaml -o ir.json
|
||||
python3 -m docgen.extractors usage --har session.har -o ir.json
|
||||
<pre><code>python3 -m docgen book --root SRC -o out/book/slug <span class="c"># the whole operation</span>
|
||||
python3 -m docgen run docgen.toml <span class="c"># every book a run file lists</span>
|
||||
python3 -m docgen check <span class="c"># docgen's own suite</span>
|
||||
python3 -m docgen check out/book/slug <span class="c"># that book's level</span>
|
||||
|
||||
python3 -m docgen.ir ir.json <span class="c"># validate</span>
|
||||
python3 -m docgen.ops ir.json --overview -o view.json
|
||||
python3 -m docgen extract python --root SRC -o ir.json
|
||||
python3 -m docgen extract code --root SRC -o ir.json
|
||||
python3 -m docgen extract db --schema schema.json -o ir.json
|
||||
python3 -m docgen extract openapi --spec spec.yaml -o ir.json
|
||||
python3 -m docgen extract usage --har session.har -o ir.json
|
||||
|
||||
python3 -m docgen.emitters auto view.json -o out/
|
||||
python3 -m docgen.emitters index ir.json -o index.md
|
||||
python3 -m docgen.emitters dot view.json -o graph.svg --theme lucid
|
||||
python3 -m docgen.emitters erd ir.json -o schema.svg
|
||||
python3 -m docgen.emitters minimap ir.json -o map.svg --scale 0.5
|
||||
python3 -m docgen.emitters notebook ir.json -o book.ipynb --overlay overlay.json
|
||||
python3 -m docgen.emitters site view.json -o site/
|
||||
python3 -m docgen.emitters explore ir.json -o explore/</code></pre>
|
||||
python3 -m docgen validate ir.json
|
||||
python3 -m docgen view ir.json --overview -o view.json
|
||||
|
||||
python3 -m docgen emit auto view.json -o out/
|
||||
python3 -m docgen emit index ir.json -o index.md
|
||||
python3 -m docgen emit dot view.json -o graph.svg --theme lucid
|
||||
python3 -m docgen emit erd ir.json -o schema.svg
|
||||
python3 -m docgen emit minimap ir.json -o map.svg --scale 0.5
|
||||
python3 -m docgen emit notebook ir.json -o book.ipynb --overlay overlay.json
|
||||
python3 -m docgen emit site view.json -o site/
|
||||
python3 -m docgen emit explore ir.json -o explore/</code></pre>
|
||||
|
||||
<h3>Where the command line lives</h3>
|
||||
|
||||
<p>
|
||||
All of it in <code>cli/</code>, and nowhere else. <code>extractors/</code>,
|
||||
<code>ops/</code>, <code>emitters/</code> and <code>book/</code> hold library
|
||||
code only — no argparse, no <code>__main__.py</code>, no
|
||||
<code>cli_dot.py</code> sitting next to <code>dot.py</code>. Two kinds of file
|
||||
in one folder make every folder answer two questions, and the command-line half
|
||||
is the half that grows copies: eight <code>cli_*</code> files each
|
||||
re-implemented "read the IR, validate it, load the style" by hand, and had begun
|
||||
to word their errors differently.
|
||||
</p>
|
||||
|
||||
<pre><code>cli/
|
||||
├── __init__.py <span class="c">the command table and the one exit path</span>
|
||||
├── common.py <span class="c">read and validate an IR, load a style, write or print</span>
|
||||
├── book.py run.py check.py
|
||||
└── extract.py validate.py view.py emit.py</code></pre>
|
||||
|
||||
<p>
|
||||
A command reports an expected failure by raising, and <code>cli.main</code>
|
||||
turns it into the only exit path: one <code>Error:</code> line on stderr, exit 1,
|
||||
never a traceback. Doing that split also surfaced a third copy of "which drawing
|
||||
does this graph want" — in the <code>auto</code> command, the <code>site</code>
|
||||
command and the book — which is now <code>emitters.auto.draw()</code>, returning
|
||||
content and leaving the file name to the caller.
|
||||
</p>
|
||||
|
||||
<!-- ─────────────────────────────────────────────────────────────── -->
|
||||
<h2 id="deps">Dependencies</h2>
|
||||
|
||||
<p>
|
||||
Everything below is optional. The stdlib covers the whole structural path — see <a href="#standalone">standalone</a>
|
||||
for the one seam out of the folder.
|
||||
The core is <strong>standard library only</strong>, Python 3.11 or later —
|
||||
<code>pyproject.toml</code> declares no required dependency at all. Everything
|
||||
else is an optional <b>dependency group</b>, pinned in a committed
|
||||
<code>uv.lock</code>, and reported by <code>make doctor</code>; when something is
|
||||
missing you lose exactly one capability and get told what to install. See
|
||||
<a href="#standalone">standalone</a> for the one seam out of the folder.
|
||||
</p>
|
||||
|
||||
<pre><code>make sync <span class="c"># uv sync --all-groups → .venv/</span>
|
||||
uv sync --group code <span class="c"># or just the one you need</span>
|
||||
make lock <span class="c"># after editing pyproject.toml</span></code></pre>
|
||||
|
||||
<p>
|
||||
The core is <strong>standard library only</strong>. Everything else is optional
|
||||
and reported by <code>make doctor</code>; when something is missing you lose
|
||||
exactly one capability and get told what to install.
|
||||
<code>[tool.uv] package = false</code>, the same choice
|
||||
<code>station/tools/dataconvert</code> makes: this is a folder of modules run in
|
||||
place, not something to install. The lock pins what the optional groups resolve
|
||||
to; the folder still runs on a bare <code>python3</code> without it.
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<tr><th>needs</th><th>for</th><th>without it</th></tr>
|
||||
<tr><td><code>graphviz</code> (binary)</td><td>rendering DOT to SVG</td>
|
||||
<td>ERD, minimap, index and notebooks still work</td></tr>
|
||||
<tr><td><code>tree_sitter</code> + grammars</td><td>C#, TypeScript, TSX</td>
|
||||
<tr><td>group <code>code</code> — tree-sitter + grammars</td><td>C#, TypeScript, TSX</td>
|
||||
<td>Python only</td></tr>
|
||||
<tr><td><code>networkx</code></td><td><code>lab/</code> experiments</td>
|
||||
<tr><td>group <code>openapi</code> — pyyaml</td><td>reading OpenAPI (with the reference repo)</td>
|
||||
<td>the other four sources</td></tr>
|
||||
<tr><td>group <code>harvest</code> — lxml</td><td>a theme from exported diagrams</td>
|
||||
<td>the shipped themes</td></tr>
|
||||
<tr><td>group <code>lab</code> — networkx</td><td><code>lab/</code> experiments</td>
|
||||
<td>nothing — nothing depends on it yet</td></tr>
|
||||
<tr><td><code>node</code></td><td>testing the browser pages</td>
|
||||
<td>those checks skip</td></tr>
|
||||
@@ -979,16 +1136,19 @@ python3 -m docgen.emitters explore ir.json -o explore/</code></pre>
|
||||
<tr><th>command</th><th>asks about</th><th>fails?</th></tr>
|
||||
<tr><td><code>make doctor</code></td><td>the machine — what is installed</td>
|
||||
<td>never; it reports</td></tr>
|
||||
<tr><td><code>make check</code></td><td>docgen — 259 checks</td>
|
||||
<tr><td><code>make check</code></td><td>docgen — 287 checks</td>
|
||||
<td>exit 1</td></tr>
|
||||
<tr><td><code>make check BOOK=<dir></code></td><td>that one book</td>
|
||||
<td>exit 1</td></tr>
|
||||
</table>
|
||||
|
||||
<p>
|
||||
All 259 with both optional dependencies installed, 244 with neither — the suite
|
||||
skips rather than fails when tree-sitter, lxml or the OpenAPI reader is absent.
|
||||
See <a href="#standalone">standalone</a> for the counts outside the repo.
|
||||
All 287 after <code>make sync</code>, 272 with nothing optional installed — the
|
||||
suite skips rather than fails when tree-sitter, lxml or the OpenAPI reader is
|
||||
absent. See <a href="#standalone">standalone</a> for the counts outside the repo.
|
||||
Among them, two that keep this page's structure true: nothing outside
|
||||
<code>cli/</code> imports argparse, and every import the suite allows is a group
|
||||
in <code>pyproject.toml</code> and pinned in <code>uv.lock</code>.
|
||||
</p>
|
||||
|
||||
<h3>The book level, where custom checks live</h3>
|
||||
|
||||
Reference in New Issue
Block a user