mariano/soleprint
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates 33.1 84

Browse Source
This commit is contained in:
buenosairesam
2026-08-10 05:36:32 -03:00
parent 0b04516cbb
commit 9a6337e493
55 changed files with 6387 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
docs/data/en/export.md Normal file
View File

@@ -0,0 +1,124 @@
# Export / Compile
Soleprint's source tree is not what runs. `build.py` compiles the framework plus
a room's configuration into a self-contained instance under `gen/<room>/`, and
that directory is what a container boots, what `deploy.sh` rsyncs, and what the
cluster manifests point at.
Everything below is a `make` target, and every target is one script in `ctrl/`.
The logic lives in the scripts, never in the Makefile.
```bash
make build # cfg/standalone -> gen/standalone
make build sample # cfg/sample -> gen/sample
make start # run it
```
## The targets
| Command | Runs | Does |
| --- | --- | --- |
| `make build [<room>\|all\|models]` | `ctrl/build.sh` | compile a room into `gen/` |
| `make start [<room>] [-d]` | `ctrl/start.sh` | run a built room's compose stack |
| `make stop [<room>]` | `ctrl/stop.sh` | stop it |
| `make cluster [up\|down\|status]` | `ctrl/cluster.sh` | the shared kind cluster |
| `make component [list\|publish\|sync\|watch\|diff]` | `ctrl/spr.py` | publish a distributable component |
| `make deploy` | `ctrl/deploy.sh` | rsync `gen/standalone` to the server and restart |
Bare words pass straight through, so `make build sample` becomes
`ctrl/build.sh sample`. Anything starting with a dash would be eaten by make
itself, so those go through `ARGS`:
```bash
make deploy ARGS="--build"
make component ARGS="publish soleprint-ui /tmp/out --dist"
```
## What a build does
`python build.py --cfg <room>` runs these in order:
1. **Clean** `gen/<room>/`. A build is not incremental — a stale file left
behind is worse than a slow build.
2. **Copy the framework.** `main.py`, `run.py`, `index.html`, `Dockerfile`,
`requirements.txt`, `dataloader/`, `common/`, and the three systems
(`artery/`, `atlas/`, `station/`).
3. **Merge the room** (`copy_cfg`). `cfg/<room>/config.json` lands in `cfg/`,
`data/*.json` in `data/`, and anything under `cfg/<room>/soleprint/artery|atlas|station/`
is merged *over* the framework copy — which is how a room adds its own vein,
shunt, tool or generator without forking the tree.
4. **Compose cabinets.** The dependency containers the room declared in
`data/cabinets.json` are merged into its `docker-compose.yml`. See
[Cabinets](#station-cabinets).
5. **Generate models.** modelgen reads the room's `config.json` and writes
`models/pydantic/__init__.py`.
6. **Render k8s** (optional). When the room's config enables it,
`soleprint/ctrl/k8s/` writes manifests and lifecycle scripts.
## What comes out
A standalone room is a flat instance:
```
gen/standalone/
run.py main.py Dockerfile docker-compose.yml
artery/ atlas/ station/ common/
cfg/config.json
data/*.json
models/pydantic/
```
A **managed** room — one that wraps an existing application — is three folders
instead, because soleprint sits beside the app rather than containing it:
```
gen/<room>/
<app>/ the application's repos, plus its ctrl scripts
link/ bridge code between the two
soleprint/ the instance, exactly as above
```
`build.py` picks between them on whether the room's `config.json` has a
`managed` block. `gen/` is gitignored in full: it is an artifact, and the way to
change it is to change `cfg/<room>/` and rebuild.
## Distributing components
Rooms are compiled; *components* are published. `registry.json` lists what can
be shipped out of this repo on its own:
```bash
make component # list
make component ARGS="publish soleprint-ui /tmp/out --dist"
make component ARGS="watch soleprint-ui ../unt/ui/framework"
```
`--dist` copies only the built bundle — `dist/**` plus `package.json`,
`README.md` and `LICENSE` — rather than the source. It refuses to publish an
empty `dist/`, because a component whose bundle was never built is the failure
that shows up later as a container that starts and renders nothing:
```
no build at soleprint/common/ui
build it first: cd soleprint/common/ui && pnpm build
```
Each publish leaves a `.spr` stamp in the destination recording name, version,
type, source and mode, so a copy can say where it came from.
## Deploying
```bash
make deploy ARGS="--build" # rebuild, sync, restart
make deploy ARGS="--sync-only" # sync, leave it running
```
`deploy.sh` rsyncs `gen/standalone/` and runs `docker compose up -d --build` on
the far side. `.env` is excluded, so server secrets stay on the server.
## Running without building
`python run.py` from `soleprint/` serves every subsystem on one port (12000 by
default) straight from the source tree. It is for developing the framework
itself; a room's `cfg/config.json` does not exist there, so the landing pages
fall back to their defaults. Rooms use docker.

107
docs/data/en/station-cabinets.md Normal file
View File

@@ -0,0 +1,107 @@
# Cabinets
A cabinet is a **dependency container** a room can switch on: postgres, redis,
airflow. The vocabulary already had the word — `execution.container` in every
room's `config.json` is *"Cabinet — tool container"* — and until now nothing
stood behind it.
The problem it solves: a generated artifact knows what it needs and had no way
to say so. A shunt built from a client's spreadsheets holds its rows in memory
happily, but the moment you want them to survive a restart you need postgres,
and wiring postgres in meant hand-editing a room's `docker-compose.yml` and then
hand-editing the cluster too. A cabinet is that declaration, made once and read
by both paths.
## Switching one on
Add `cfg/<room>/data/cabinets.json`, the same shape as its sibling `data/*.json`
files:
```json
[
{ "name": "postgres" },
{ "name": "redis" },
{ "name": "airflow", "env": { "AIRFLOW_ADMIN_PASSWORD": "change-me" } }
]
```
Then build. The compose merge is a step in [Export / Compile](#export):
```bash
python build.py --cfg sample
cd gen/sample && docker compose up -d
```
`build.py` merges each cabinet's compose fragment into the room's
`docker-compose.yml`, declares its named volumes, and appends its settings to
`.env.example` — never to `.env`.
**A service the room already declares wins.** `cfg/amar/docker-compose.yml`
ships its own `db`; switching the postgres cabinet on will not replace it. The
build says so when it skips one:
```
Composing cabinets...
cabinets: redis, airflow
cabinets already declared by the room, left alone: postgres
```
Dependencies come along automatically. Airflow without a metadata database is a
container that exits on boot, so asking for `airflow` brings `postgres` and
`redis` with it, ordered so compose reads them before the thing that needs them.
## On a cluster
Every cabinet names a `rig_addon`. Where a room runs on kind rather than
compose, the same dependency installs as a rig addon of that name:
```bash
cd rig
PROFILE=data make cluster up
PROFILE=data make addons install
kubectl -n data port-forward svc/postgres 5432:5432
kubectl -n data port-forward svc/airflow 8080:8080
```
The two paths are deliberately separate — compose for a laptop, manifests for a
cluster — and `rig_addon` is the thread between them, so the room declares the
dependency once either way. The addons generate their own passwords on first
install and keep them across re-runs, so re-running never rotates a credential
out from under something already connected.
## What ships
| Cabinet | Image | Notes |
| --- | --- | --- |
| `postgres` | `postgres:16-alpine` | healthcheck wired, so `depends_on: service_healthy` works |
| `redis` | `redis:7-alpine` | cache, and the broker for anything queue-shaped |
| `airflow` | `apache/airflow:2.10.4` | one container on `standalone`; needs postgres and redis |
## Writing one
```
soleprint/station/cabinets/<name>/
cabinet.json what it is, what it needs, what it exports
service.yml the compose service, verbatim
```
`cabinet.json`:
| Key | Purpose |
| --- | --- |
| `name` | must match the directory |
| `title`, `description` | shown on the station index |
| `service` | the key to merge under in `services:` (defaults to `name`) |
| `env` | settings and defaults, written to `.env.example` |
| `volumes` | named volumes to declare at the top level |
| `depends_on` | other cabinets that must come with it |
| `rig_addon` | the matching `rig/ctrl/addons/<name>.sh`, if there is one |
| `notes` | lines written into `.env.example` as comments |
`service.yml` is a plain compose fragment with one top-level key — the service
name. It stays YAML rather than being generated from JSON so it reads like the
file it becomes, and so anything compose supports is available without this tool
learning about it first.
Adding a cabinet is adding a directory. Nothing dispatches on the name.

103
docs/data/en/station-datagen.md
View File

@@ -1,6 +1,8 @@
# Datagen
Test data generator using faker. Produces realistic, domain-specific data for testing and development.
Test data generator. Produces realistic, domain-specific records for testing and
development, from generators a room writes or that
[modelgen](#station-modelgen) writes for it.
**Status:** live
@@ -8,50 +10,95 @@ Test data generator using faker. Produces realistic, domain-specific data for te
## What It Does
Datagen generates fake but realistic data. Names, emails, addresses, transactions -- whatever your domain needs. It uses Python's faker library under the hood.
Datagen hands out instances of a room's models. Core ships the base class, the
discovery, the HTTP API and the browser UI; the generators themselves belong to
a room, because what counts as realistic is a property of the domain.
Core datagen is a placeholder. The real work happens in room-specific generators.
Generation is stdlib `random`, `uuid` and `datetime` — **not** faker, which is
not a dependency of this repo.
## Structure
```
soleprint/station/tools/datagen/ # Core (base classes, placeholder)
cfg/<room>/soleprint/station/tools/datagen/ # Room-specific generators
soleprint/station/tools/datagen/ # base class, api, UI
cfg/<room>/soleprint/station/tools/datagen/ # the room's generators
```
After build, both merge into `gen/<room>/station/tools/datagen/`.
## Pattern
## The contract
Rooms subclass a base generator and provide domain-specific data factories:
A generator subclasses `BaseDataGenerator` and defines **one method per model,
named after it**. There is no registration step: the method name *is* the model
name.
```python
from station.tools.datagen.base import BaseGenerator
from station.tools.datagen.base import BaseDataGenerator
class RoomDataGenerator(BaseGenerator):
def generate_customers(self, count=10):
return [self.fake_customer() for _ in range(count)]
class RoomDataGenerator(BaseDataGenerator):
def customer(self, **kwargs):
return {"id": str(uuid4()), "name": ..., "email": ..., **kwargs}
def fake_customer(self):
return {
"name": self.faker.name(),
"email": self.faker.email(),
"phone": self.faker.phone_number(),
}
def invoice(self, customer_id=None, **kwargs):
return {"id": str(uuid4()), "customer_id": customer_id, **kwargs}
```
The base class provides:
| Method | Does |
| --- | --- |
| `generate(model, count=1, **kwargs)` | calls the matching method `count` times; `kwargs` pass through to every call |
| `available_models()` | the method names, which are the model names |
| `schema()` | override to return a graphgen-compatible schema |
Discovery is by convention too: any `*.py` in the datagen directory whose first
class ends in `Generator` is loaded and keyed by its filename.
## Generating a generator
Writing one by hand is optional. modelgen's `datagen` target emits the whole
class from a schema — and when the schema came from spreadsheets, the generated
class **samples the real rows** rather than inventing values:
```bash
python -m station.tools.modelgen from-tabular -s ./sheets -o out/ -t datagen
python -m station.tools.modelgen from-openapi -s api.yaml -o out/ -t datagen
```
Each room defines what data it needs. Core provides the faker instance and base class. Rooms provide the factories.
This is how [shuntgen](#station-shuntgen) fills a generated shunt.
## HTTP API
Mounted under `/station/tools/datagen/`:
| Route | Returns |
| --- | --- |
| `GET /` | the browser UI |
| `GET /api/generators` | loaded generator files and their models |
| `GET /api/models?generator=` | model names |
| `POST /api/generate` | `{model, count, generator?, kwargs?}` → the records |
| `GET /api/schema?generator=` | the generator's schema, if it exposes one |
## Room Configuration
## Feeding graphgen
Room generators live in `cfg/<room>/soleprint/station/tools/datagen/`. They are fully self-contained -- they define their own models, factories, and output formats.
A generator that overrides `schema()` is surfaced at `/api/schema` in the format
[graphgen](#station-graphgen) reads, so the same definition draws the diagram:
The core module provides:
- Base generator class with faker instance
- CLI entry point
- Output formatting (JSON, CSV)
```python
def schema(self):
return {
"models": {
"Invoice": {
"doc": "A billed order.",
"fields": {
"id": {"type": "UUID", "pk": True},
"customer_id": {"type": "FK:Customer"},
"total": {"type": "float"},
},
}
}
}
```
Rooms provide:
- Domain-specific generator subclasses
- Field definitions and relationships
- Volume and distribution configuration
`FK:<Model>` and `M2M:<Model>` are how relations are written. modelgen's
`datagen` target emits this method for you.

117
docs/data/en/station-modelgen.md
View File

@@ -1,54 +1,111 @@
# Modelgen
Generates platform-specific models from JSON Schema. Reads schema once, writes models for multiple targets.
Multi-source, multi-target model generator. Reads a schema from wherever it
already lives, and writes it out for every stack that needs it.
**Status:** dev
**Status:** live
---
## What It Does
Modelgen takes a JSON Schema definition and produces model code for different platforms:
Everything passes through one intermediate representation — `ModelDefinition`,
`FieldDefinition`, `EnumDefinition`. **Loaders** fill it, **generators** emit
from it, and the two sides do not know about each other. Adding an input means
one extractor and every output comes with it; adding an output means one
generator and every input already feeds it.
- **Pydantic** -- Python data validation models
- **Django ORM** -- Django model classes
- **Prisma** -- Prisma schema definitions
```
dataclasses ─┐ ┌─ pydantic
Django │ ├─ django
SQLAlchemy ├──▶ ModelDefinition ──▶├─ sqlmodel
a live DB │ FieldDefinition ├─ typescript
OpenAPI │ EnumDefinition ├─ protobuf
CSV/ODS ─┘ ├─ prisma
├─ strawberry
├─ schema (graphgen)
└─ datagen
```
One schema, multiple outputs.
Core is **pure standard library**. It is published as `soleprint-modelgen` and
installs with no dependencies; live-database extraction is an extra
(`pip install "soleprint-modelgen[db]"`), and YAML specs need PyYAML.
## Extractors
## Sources
Modelgen also works in reverse. Extractors read existing codebases and produce a normalized schema representation:
| Command | Reads |
| --- | --- |
| `from-schema` | Python dataclasses in a `schema/` folder |
| `from-config` | a room's `config.json` |
| `extract` | a Django or SQLAlchemy codebase (`--framework auto` detects) |
| `from-db` | a live database, any SQLAlchemy dialect |
| `from-openapi` | an OpenAPI 3.x / Swagger 2.0 document |
| `from-tabular` | a directory of `.csv` / `.tsv` / `.ods` spreadsheets |
- **Django extractor** -- reads Django model files
- **SQLAlchemy extractor** -- reads SQLAlchemy model files
- **Prisma extractor** -- reads Prisma schema files
```bash
python -m station.tools.modelgen from-openapi -s api.yaml -o out/ -t pydantic,typescript,schema
python -m station.tools.modelgen from-tabular -s ./sheets -o out/ -t pydantic,datagen
python -m station.tools.modelgen extract -s /path/to/django -o out/ -t prisma
python -m station.tools.modelgen from-db -u postgresql://… -o out/ -t typescript
python -m station.tools.modelgen list-formats
```
Extractors feed into graphgen for visualization.
### From a spec
## Output
`components.schemas` (or Swagger's `definitions`) become models. `$ref` chains
and `allOf` are resolved, enums are materialised as real `Enum` classes so every
target names them properly, and a referenced object becomes a relation rather
than a nested type — the same call the database extractor makes, and what keeps
the generated code valid for every target.
Generated models are written to `gen/<room>/models/`.
The parse also yields the *operations*, which is what
[shuntgen](#station-shuntgen) turns into routes.
```
gen/<room>/models/
├── pydantic/
├── django/
└── prisma/
```
### From spreadsheets
## CLI
One model per CSV file, one per sheet in an ODS workbook. Column types are
inferred from the values actually present, and a blank cell makes the column
optional. Keys and relations are inferred by name and then confirmed against the
data: an `id` column that is not unique is not treated as a key, and
`customer_id` is only a foreign key if a `customers` sheet came with it.
```bash
python -m modelgen
```
The rows are kept, not just the shape — which is what lets the `datagen` target
sample real values instead of inventing them.
Reads from `schema.json` (the project source of truth) and writes to the configured output directory.
ODS is read with `zipfile` and `ElementTree`. No odfpy, no pandas: the
dependency-free promise is what makes this package publishable on its own.
## Shared Distribution
## Targets
Modelgen is also distributed as a shared component via `ctrl/spr.py`. This allows other projects to use model generation without running full soleprint.
`pydantic`, `django`, `sqlmodel`, `typescript` (`ts`), `protobuf` (`proto`),
`prisma`, `strawberry`, `schema` (`jsonschema`), `datagen`.
## Schema Source
Two are worth calling out:
- **`schema`** writes the graphgen-compatible `schema.json` — the portable
artifact [graphgen](#station-graphgen) and databrowse read directly.
Relations come out as `FK:<Model>` and `M2M:<Model>`.
- **`datagen`** writes a `BaseDataGenerator` subclass for
[datagen](#station-datagen), including its `schema()` override. Given
spreadsheet rows it samples them; otherwise it synthesises from the types.
Multiple targets in one run get one file each, named `models_<target><ext>`.
## In a build
`build.py` calls modelgen during every room build, writing
`gen/<room>/models/pydantic/__init__.py` from the room's `config.json`. See
[Export / Compile](#export).
## Tests
```bash
cd soleprint/station/tools
python -m unittest modelgen.tests.test_extractors
```
The source of truth is `schema.json` at the project root. All model generation starts from this file. Room-specific schema extensions live in `cfg/<room>/models/`.
stdlib `unittest`, no pytest, and every input is built in a temp directory — the
tests have to pass with nothing installed and nothing else in the tree. Run them
from `station/tools/`, not from inside `modelgen/`: the package ships a
`types.py`, and putting its own directory on `sys.path` shadows the standard
library module of that name.

129
docs/data/en/station-shuntgen.md Normal file
View File

@@ -0,0 +1,129 @@
# Shuntgen
Generates runnable [shunts](#artery-shunts) from the two things people actually
have: a service contract, or a folder of spreadsheets.
Writing a shunt by hand means copying `artery/shunts/example/` and filling in
`responses.json` entry by entry. That is fine for three endpoints and untenable
for eighty — and it is the wrong work anyway, because the endpoints are already
described in the spec somebody handed you.
```bash
# a spec you were handed
python -m station.tools.shuntgen from-openapi -s api.yaml -o artery/shunts/petstore
# sheets a client sent
python -m station.tools.shuntgen from-tabular -s ./sheets -o artery/shunts/books
python -m station.tools.shuntgen list
```
Run from `soleprint/`. Also in the browser at `/station/tools/shuntgen/`, where
you can upload a spec, preview the routes it would serve, and generate.
## What comes out
```
artery/shunts/<name>/
main.py builds the app from the spec
run.py uvicorn entry point (PORT, or depot/config.json)
shunt_runtime.py vendored runtime — no soleprint import
models.py pydantic, via modelgen
datagen_<name>.py BaseDataGenerator subclass, via modelgen
depot/spec.json routes, collections and schema
depot/responses.json pinned overrides — yours, never overwritten
depot/config.json delays, error rate, prefill — yours, never overwritten
depot/data.json imported rows
templates/index.html config UI
README.md
```
```bash
cd artery/shunts/books && python run.py
curl localhost:8098/customers
```
The routes are built at startup from `spec.json` rather than written out as
source. That keeps the generated code short enough to read, and puts the
behaviour in one reviewable place: fixing `runtime.py` fixes every shunt, and
regenerating is a copy.
## Where a response comes from
First hit wins:
1. `depot/responses.json` — a pinned override, keyed `"METHOD /path"`
2. the store — rows imported from sheets, plus anything POSTed since
3. the spec's `example`, if the source document carried one
4. `datagen_<name>.py`, synthesising from the schema
5. `{}`
The store is what makes it behave like a service rather than a random-value
faucet: POST something and GET it back, ask for `/pets/7` and get the pet whose
id is 7. Collections that arrived with no rows are prefilled with generated
ones, so the first call answers with something.
## Two sources, one pipeline
Both inputs are [modelgen](#station-modelgen) extractors, so the same shapes
also generate pydantic, TypeScript, prisma and a
[graphgen](#station-graphgen) schema:
```bash
python -m station.tools.modelgen from-openapi -s api.yaml -o out/ -t pydantic,typescript
python -m station.tools.modelgen from-tabular -s ./sheets -o out/ -t schema,datagen
```
| Source | Becomes | Routes |
| --- | --- | --- |
| OpenAPI 3.x / Swagger 2.0 | one model per schema; enums become real Enums, `$ref` becomes a relation | the operations the document declares |
| `.csv` / `.tsv` / `.ods` | one model per file or sheet, types inferred per column | five CRUD routes per table |
Keys and relations are inferred by name and then **checked against the data**:
an `id` column that is not unique is not treated as a key, and `customer_id` is
only a foreign key if a `customers` sheet came with it.
ODS is read with `zipfile` and `ElementTree` — no odfpy, no pandas — which is
what lets modelgen stay dependency-free and publishable on its own.
## Control endpoints
Every generated shunt serves these:
| Endpoint | Purpose |
| --- | --- |
| `GET /health` | liveness |
| `GET /mock/spec` | the routes it was built from |
| `GET /mock/stats` | call counts and row counts |
| `POST /mock/reset` | restore the imported rows, clear counters |
| `GET,POST /mock/config` | delays, error rate, `unknown_id`, page size |
| `GET,POST /mock/responses` | pin an override; set a key to `null` to drop it |
```bash
# make it slow and flaky, the way the real thing is
curl -X POST localhost:8098/mock/config \
-H 'content-type: application/json' \
-d '{"enable_random_delays": true, "error_rate": 0.2}'
# make one call answer something specific
curl -X POST localhost:8098/mock/responses \
-H 'content-type: application/json' \
-d '{"GET /customers/1": {"id": 1, "name": "PINNED"}}'
```
`unknown_id` decides what an unknown key does: `generate` (the default) invents
a record wearing the id that was asked for; `404` refuses it. Generate by
default, because a client pointed at a fresh shunt should just work — flip it
when the error path is what you are testing.
## Dependency containers
`--cabinet postgres,redis` writes a `cabinet.json` declaring what the shunt
expects. `build.py` composes those services into the room's compose file, and on
a cluster they install as rig addons of the same name. See
[Cabinets](#station-cabinets).
## Regenerating
Everything is overwritten except `depot/responses.json` and `depot/config.json`.
Those two are yours.

3
docs/data/topics.json
View File

@@ -22,8 +22,11 @@
{"id": "station-datagen", "title": {"en": "↳ Datagen"}, "sub": true},
{"id": "station-modelgen", "title": {"en": "↳ Modelgen"}, "sub": true},
{"id": "station-graphgen", "title": {"en": "↳ Graphgen"}, "sub": true},
{"id": "station-shuntgen", "title": {"en": "↳ Shuntgen"}, "sub": true},
{"id": "station-databrowse", "title": {"en": "↳ Databrowse"}, "sub": true},
{"id": "station-cabinets", "title": {"en": "↳ Cabinets"}, "sub": true},
{"id": "components", "title": {"en": "Shared Components"}},
{"id": "export", "title": {"en": "Export / Compile"}},
{"id": "deployment", "title": {"en": "Deployment"}}
]