ui and tiltfile tweaks, update docs
This commit is contained in:
200
docs/index.html
200
docs/index.html
@@ -256,7 +256,8 @@
|
||||
<a onclick="show('architecture')">Architecture</a>
|
||||
<a onclick="show('plan')">Plan</a>
|
||||
<a onclick="show('analyses')">Analyses</a>
|
||||
<a onclick="show('tools')">Tools</a>
|
||||
<a onclick="show('tools')">Atomic actions</a>
|
||||
<a onclick="show('recon')">Recon</a>
|
||||
<a onclick="show('runtime')">Runtime</a>
|
||||
<a onclick="show('run')">Run it</a>
|
||||
<a onclick="show('decisions')">Decisions</a>
|
||||
@@ -278,27 +279,53 @@
|
||||
</p>
|
||||
<p>
|
||||
The shape mirrors the JD's three building blocks
|
||||
(<i>text-to-SQL, reasoning agents, planners</i>) as three distinct
|
||||
layers rather than one omnibus prompt. That gives every part a name
|
||||
and a tight contract, and lets each Analysis pick the simplest
|
||||
reasoning pattern (CoT, ReAct, plan-and-execute) for what it does
|
||||
instead of forcing one over everything.
|
||||
(<i>text-to-SQL, reasoning agents, planners</i>) as distinct layers
|
||||
rather than one omnibus prompt — <b>Plan</b> (L3), <b>Analyses</b> (L2),
|
||||
and deterministic <b>atomic actions</b> (L1) — resting on a <b>recon</b>
|
||||
foundation (L0): the dataset's typed knowledge graph. That gives every
|
||||
part a name and a tight contract, and lets each Analysis pick the
|
||||
simplest reasoning pattern (CoT, ReAct, plan-and-execute) for what it
|
||||
does instead of forcing one over everything.
|
||||
</p>
|
||||
<p>
|
||||
The load-bearing decision: <b>the LLM never authors SQL.</b> It picks a
|
||||
typed shape from finite, recon-derived sets; the composer walks recon and
|
||||
emits the query deterministically. Column–table binding, joins, and
|
||||
quoting are graph traversals, not model guesses — so a question the system
|
||||
can't express fails cleanly instead of producing plausible-but-wrong SQL.
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="architecture" class="section">
|
||||
<h2>Architecture</h2>
|
||||
<p class="lede">Three layers — Plan composes Analyses; Analyses are mini-agents; Tools are deterministic.</p>
|
||||
<p class="lede">Four layers — Plan composes Analyses; Analyses are mini-agents; atomic actions are deterministic; recon is the foundation they all consult.</p>
|
||||
<object data="graphs/architecture.svg" type="image/svg+xml" class="diagram"></object>
|
||||
<dl>
|
||||
<dt>Plan (L3)</dt>
|
||||
<dd>One LLM call that turns a question into an ordered set of Analyses with intended interpretation. Not agentic by itself — produces a structured artifact the runtime executes.</dd>
|
||||
<dd>One LLM call that turns a question into an ordered set of Analysis invocations — each with args, an intended interpretation, and an optional fallback. It sees only brief table/metric names and the Analysis catalog, never raw DDL. Stateless: it emits a plan and never observes execution.</dd>
|
||||
<dt>Analyses (L2)</dt>
|
||||
<dd>Each Analysis is a self-contained mini-agent. It picks the simplest reasoning pattern it needs — CoT for one-shot interpretive moves, ReAct for ones that iterate over Tool results, plan-and-execute for compound ones. The Analysis owns its loop.</dd>
|
||||
<dt>Tools (L1)</dt>
|
||||
<dd>Deterministic primitives — <code>text_to_sql</code>, <code>execute_sql</code>, <code>retrieve_schema</code>, <code>retrieve_metric_definition</code>, <code>python_sandbox</code>. No LLM reasoning inside. Called from inside Analyses.</dd>
|
||||
<dd>Self-contained mini-agents. Each picks the simplest reasoning pattern it needs — CoT for a one-shot move, ReAct for ones that iterate over results, plan-and-execute for compound ones — and owns its loop behind a uniform contract, <code>run(args, question) → Finding</code>. The LLM work lives here: choosing a typed <code>Pick</code> and interpreting rows, never writing SQL.</dd>
|
||||
<dt>Atomic actions (L1)</dt>
|
||||
<dd>Deterministic, no LLM. <code>compose(pick, recon) → SQL</code> authors the query by walking recon's join graph; <code>execute_sql</code> runs it read-only with a row cap and timeout. Invalid SQL isn't expressible — the composer binds columns and joins from recon, not from model output.</dd>
|
||||
<dt>Recon (L0)</dt>
|
||||
<dd>The dataset's typed knowledge graph — tables, columns, metrics, relationships, storage encodings — merged from sparse human YAML and extracted DDL. The foundation every layer above consults. (Sometimes called L4: the setup that exists before any run, not a step inside one.)</dd>
|
||||
</dl>
|
||||
<div class="prose">
|
||||
<h3>How the layers interconnect</h3>
|
||||
<p>
|
||||
The layers read <b>general → specific</b> (Plan → Analyses → atomic
|
||||
actions), but execution is a graph, not a one-way cascade. Later steps
|
||||
depend on earlier <i>intermediate results</i>: <code>drill_down</code>
|
||||
loops, choosing its next dimension from the previous slice's rows; a
|
||||
step's <b>fallback</b> Analysis fires based on whether the primary's
|
||||
Finding came back empty or errored. And <b>L0 recon is consulted at every
|
||||
level</b> — the planner reads its brief names, the <code>Pick</code>
|
||||
validates against it, the composer walks it. So the layers are
|
||||
interconnected through both the data they share (recon) and the results
|
||||
they pass forward.
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="plan" class="section">
|
||||
@@ -306,13 +333,23 @@
|
||||
<p class="lede">L3 — one LLM call that decides which Analyses to run.</p>
|
||||
<div class="prose">
|
||||
<p>
|
||||
The planner gets the user question, the metric catalog, a schema
|
||||
overview, and the catalog of Analyses. It returns a structured plan:
|
||||
an ordered list of Analysis invocations with arguments and an intended
|
||||
interpretation. It does not call tools or query data itself — it's a
|
||||
composition step.
|
||||
The planner (<code>api/plan/planner.py</code>) gets the user question
|
||||
plus three context blocks rendered from recon: brief table names, brief
|
||||
metric names, and the catalog of Analyses with their <code>args_schema</code>.
|
||||
Deliberately no DDL — it decides <i>which Analyses to run</i>, not how to
|
||||
query. One LLM call returns a structured <code>Plan</code>.
|
||||
</p>
|
||||
<p>Each step in the plan carries:</p>
|
||||
<ul>
|
||||
<li><code>analysis</code> — which Analysis to invoke (must be in the registry).</li>
|
||||
<li><code>args</code> — the arguments dict, shaped by that Analysis's <code>args_schema</code>.</li>
|
||||
<li><code>why</code> — a one-line rationale, surfaced in the live trace.</li>
|
||||
<li><code>fallback</code> — an optional alternate Analysis the runtime runs if the primary returns empty or errors.</li>
|
||||
</ul>
|
||||
<p>
|
||||
The planner never calls a tool or touches data — it's a pure composition
|
||||
step. The runtime (below) walks the steps in order.
|
||||
</p>
|
||||
<p><i>To be expanded as the planner is built out.</i></p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -321,32 +358,86 @@
|
||||
<p class="lede">L2 — each Analysis is its own mini-agent.</p>
|
||||
<div class="prose">
|
||||
<p>
|
||||
Analyses are the agent layer. Each one wraps a small piece of analytical
|
||||
intent (<i>compare two periods</i>, <i>drill down by dimension</i>,
|
||||
<i>find outliers</i>) and decides how to satisfy it — sometimes one
|
||||
LLM call is enough, sometimes a ReAct loop over Tool results is the
|
||||
right fit. The Analysis base class only enforces a uniform external
|
||||
contract (<code>run(state) → Finding</code>); the internal pattern is
|
||||
a per-Analysis decision.
|
||||
Analyses are the agent layer. Each wraps a piece of analytical intent and
|
||||
decides how to satisfy it — sometimes one LLM call is enough, sometimes a
|
||||
ReAct loop over query results is the right fit. The base class
|
||||
(<code>api/analyses/base.py</code>) enforces only a uniform external
|
||||
contract, <code>run(args, question) → Finding</code>; the internal
|
||||
pattern is a per-Analysis decision and no framework types leak out.
|
||||
</p>
|
||||
<p><i>Per-Analysis pattern table coming as each Analysis is implemented.</i></p>
|
||||
<p>
|
||||
Within an Analysis, a free-form question becomes a typed <code>Pick</code>
|
||||
via one LLM call (<code>pick_for_question</code>) over recon-narrowed
|
||||
candidate sets — that is the only place the model makes a structural
|
||||
choice, and it chooses from finite known sets, not free text. The
|
||||
<code>Pick</code> then goes to the deterministic composer (L1).
|
||||
</p>
|
||||
<h3>Implemented Analyses</h3>
|
||||
<dl>
|
||||
<dt>direct_answer</dt>
|
||||
<dd>CoT, one shot. <code>pick → compose → execute → interpret</code>. For single-query lookups and aggregations.</dd>
|
||||
<dt>compare_periods</dt>
|
||||
<dd>CoT over two queries. One Pick, cloned with two period filters → <code>compose ×2 → execute ×2 → interpret</code> the pair.</dd>
|
||||
<dt>drill_down</dt>
|
||||
<dd>ReAct loop. <code>decide_next</code> (LLM) picks a dimension from the candidate list → build Pick → <code>compose → execute</code> → loop on the slice's results. Zero LLM in the slice path itself — only the dimension choice is a model decision, and it's guarded against the candidate list.</dd>
|
||||
<dt>find_outliers · correlate</dt>
|
||||
<dd><i>Planned.</i> Additive — new Analyses register without touching the runtime or the layers below.</dd>
|
||||
</dl>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="tools" class="section">
|
||||
<h2>Tools layer</h2>
|
||||
<p class="lede">L1 — deterministic primitives.</p>
|
||||
<h2>Atomic actions</h2>
|
||||
<p class="lede">L1 — deterministic, no LLM inside. The composer is the SQL author.</p>
|
||||
<div class="prose">
|
||||
<p>
|
||||
Tools have no LLM reasoning inside them. <code>text_to_sql</code> is
|
||||
the one exception that uses an LLM, but tightly: schema-RAG context,
|
||||
sqlglot validation, retry on parse/runtime errors. <code>execute_sql</code>
|
||||
runs against Postgres read-only with a row cap and timeout.
|
||||
<code>retrieve_schema</code> and <code>retrieve_metric_definition</code>
|
||||
power the semantic layer. <code>python_sandbox</code> handles stats
|
||||
work that's awkward in SQL.
|
||||
There is no LLM in this layer — and notably no <code>text_to_sql</code>
|
||||
tool. An earlier design had the model author SQL with schema-RAG context,
|
||||
sqlglot validation, and retries; it was removed once the composer landed
|
||||
(see <b>Decisions</b>). The two atomic actions are:
|
||||
</p>
|
||||
<dl>
|
||||
<dt>compose</dt>
|
||||
<dd><code>compose(pick, recon) → SQL</code> (<code>api/composer/</code>). Builds one sqlglot expression tree from a typed <code>Pick</code> by walking recon: resolve the metric's table, bind each group-by/filter column to its owning table, compute join paths, render filters and dates, emit quoted SQL. Split by concern — <code>aliases</code>, <code>joins</code>, <code>filters</code>, <code>metrics</code>, <code>dates</code>, <code>order</code>, <code>encodings</code>. If a <code>Pick</code> can't be expressed against recon it raises <code>PickValidationError</code>; it never fabricates SQL.</dd>
|
||||
<dt>execute_sql</dt>
|
||||
<dd><code>execute_sql(sql) → rows</code> (<code>api/tools/execute_sql.py</code>). Runs read-only against Postgres (<code>SELECT</code>/<code>WITH</code> only) with a row cap and a statement timeout, returning columns + rows + truncation metadata.</dd>
|
||||
</dl>
|
||||
<p>
|
||||
Because column–table binding and joins come from recon rather than model
|
||||
output, the composer <i>cannot</i> emit SQL that references a column that
|
||||
doesn't exist. The validator that used to catch the model's mistakes after
|
||||
the fact is demoted to a paranoid self-check on the composer.
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="recon" class="section">
|
||||
<h2>Recon</h2>
|
||||
<p class="lede">L0 — the dataset's typed knowledge graph, and the only dataset-specific surface.</p>
|
||||
<div class="prose">
|
||||
<p>
|
||||
Recon (<code>api/recon/</code>) is the foundation every other layer
|
||||
consults. It's built in two stages and cached as <code>recon.json</code>:
|
||||
</p>
|
||||
<ul>
|
||||
<li><b>DDL extraction</b> — SQLAlchemy introspects Postgres for tables, columns, types, nullability → <code>extracted_schema.json</code> (auto, gitignored).</li>
|
||||
<li><b>Augmentation merge</b> — sparse human YAML in <code>api/datasets/<name>/</code> (<code>schema_docs.yaml</code> descriptions + relationships, <code>metrics.yaml</code> SQL fragments, optional <code>encodings</code>) merges onto the extracted schema.</li>
|
||||
</ul>
|
||||
<p>
|
||||
The merged <code>Recon</code> exposes the queries downstream code needs
|
||||
without going back to the model: <code>resolve_column</code> (bare or
|
||||
<code>table.col</code> → owning table), <code>join_path</code> (shortest
|
||||
table sequence between two tables), the metric catalog, and storage
|
||||
<code>encodings</code> (e.g. how a YYMMDD-int column coerces to a date).
|
||||
Rebuild with <code>make recon</code> after changing the warehouse or the
|
||||
YAML.
|
||||
</p>
|
||||
<p>
|
||||
Everything dataset-specific lives here. The Plan, Analyses, and atomic
|
||||
actions reference no dataset name — point them at a different
|
||||
<code>api/datasets/<name>/</code> and the same machinery answers
|
||||
questions over a new warehouse.
|
||||
</p>
|
||||
<p><i>Per-Tool documentation coming as each Tool is implemented.</i></p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -368,10 +459,26 @@
|
||||
the nodes <i>are</i>, and that's where the interesting reading is.
|
||||
</p>
|
||||
<p>
|
||||
Concretely: <code>api/runtime/</code> contains the langgraph builder
|
||||
and event tap. <code>api/plan/</code>, <code>api/analyses/</code>, and
|
||||
<code>api/tools/</code> contain the domain code — and those files
|
||||
don't expose langgraph types in their public interfaces.
|
||||
Concretely the graph is small and linear:
|
||||
</p>
|
||||
<pre><code>START → plan → execute → synthesize → END</code></pre>
|
||||
<p>
|
||||
<code>plan</code> runs the planner; <code>execute</code> walks the plan
|
||||
steps, dispatching each to its Analysis (and to a fallback Analysis when
|
||||
the primary returns empty or errors); <code>synthesize</code> folds the
|
||||
Findings into one answer. The whole run streams SSE events
|
||||
(<code>run_start</code>, <code>plan_ready</code>,
|
||||
<code>analysis_start/end</code>, <code>tool_call_start/end</code>,
|
||||
<code>analysis_fallback</code>) that drive the live trace UI, and every
|
||||
LLM call opens a named Langfuse generation span.
|
||||
</p>
|
||||
<p>
|
||||
<code>api/runtime/runner.py</code> is the one place langgraph appears —
|
||||
<code>StateGraph</code> is imported there and nowhere else.
|
||||
<code>api/plan/</code>, <code>api/analyses/</code>,
|
||||
<code>api/composer/</code>, and <code>api/tools/</code> contain the
|
||||
domain code, and none of those files expose langgraph types in their
|
||||
public interfaces.
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
@@ -398,7 +505,8 @@ open http://nvi.local.ar</code></pre>
|
||||
<h2>Architecture decisions</h2>
|
||||
<div class="prose">
|
||||
<ul>
|
||||
<li><b>Three layers, not one omnibus prompt.</b> Plan composes Analyses; each Analysis picks its own reasoning pattern (CoT / ReAct / plan-and-execute); Tools are deterministic. Borrowing elements from agent frameworks without committing the whole system to one.</li>
|
||||
<li><b>The LLM never authors SQL.</b> It picks a typed shape from finite, recon-derived sets; the composer walks recon and emits SQL deterministically. Schema-in-the-prompt is a <i>suggestion</i> the model can ignore; composition makes it a <i>constraint</i>. The trade is fewer question shapes for correctness by construction. (See <code>def/schema-as-constraint.md</code>.)</li>
|
||||
<li><b>Four layers, not one omnibus prompt.</b> Plan composes Analyses; each Analysis picks its own reasoning pattern (CoT / ReAct / plan-and-execute); atomic actions are deterministic; recon is the shared foundation. Borrowing elements from agent frameworks without committing the whole system to one.</li>
|
||||
<li><b>langgraph for orchestration, nvi for domain.</b> Framework owns graph wiring and streaming; nvi's <code>plan/</code>, <code>analyses/</code>, <code>tools/</code> own the interesting code and are readable without reading langgraph.</li>
|
||||
<li><b>BIRD <code>financial</code> as the warehouse.</b> Fits Nivii's stated Finance vertical; ships with golden SQL pairs so the eval set is free and verifiable.</li>
|
||||
<li><b>Langfuse hosted independently</b> by the <code>lng</code> project; reached over WireGuard. Same env var works in dev and any future deploy.</li>
|
||||
@@ -409,9 +517,17 @@ open http://nvi.local.ar</code></pre>
|
||||
|
||||
<section id="evals" class="section">
|
||||
<h2>Evals</h2>
|
||||
<p class="lede">BIRD <code>financial</code> dev set, run by <code>evals/run_evals.py</code>, results pushed to Langfuse tagged <code>eval</code>.</p>
|
||||
<p class="lede">BIRD <code>financial</code> dev set, run by <code>api/evals/run_evals.py</code> (<code>make evals</code>), traces pushed to Langfuse.</p>
|
||||
<div class="prose">
|
||||
<p><i>To be populated as the agent stack lands.</i></p>
|
||||
<p>
|
||||
BIRD ships golden question/SQL pairs, so the eval set is free and
|
||||
verifiable. Each run replays questions through the full stack
|
||||
(plan → analyses → compose → execute) and records the trace in Langfuse
|
||||
for inspection — token usage per layer, which Analysis fired, the composed
|
||||
SQL. The composer refactor's own check is the unit suite
|
||||
(<code>uv run pytest</code>): golden SQL strings plus validation
|
||||
rejections, so structural correctness is pinned independent of the model.
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user