Files
nvi/docs/index.html

552 lines
21 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>nvi — analytics agent</title>
<style>
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap');
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: #0a0e17;
color: #e8eaf0;
font-family: 'Inter', sans-serif;
line-height: 1.6;
height: 100vh;
overflow: hidden;
display: flex;
flex-direction: column;
}
header {
padding: 16px 24px;
border-bottom: 1px solid #1e2a4a;
display: flex;
align-items: baseline;
gap: 16px;
flex-shrink: 0;
}
header h1 {
font-family: 'JetBrains Mono', monospace;
font-size: 22px;
font-weight: 600;
letter-spacing: 3px;
color: #0066ff;
}
header .subtitle {
font-size: 13px;
color: #4a5568;
letter-spacing: 1px;
text-transform: uppercase;
}
.layout {
display: flex;
flex: 1;
min-height: 0;
}
nav {
display: flex;
flex-direction: column;
width: 200px;
flex-shrink: 0;
background: #121829;
border-right: 1px solid #1e2a4a;
padding: 8px 0;
overflow-y: auto;
}
nav a {
padding: 10px 20px;
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: #8892a8;
text-decoration: none;
border-left: 2px solid transparent;
transition: all 0.15s;
cursor: pointer;
}
nav a:hover { color: #e8eaf0; background: #1a2340; }
nav a.active { color: #0066ff; border-left-color: #0066ff; background: #0d1a33; }
main {
flex: 1;
overflow: auto;
padding: 32px 48px;
}
.section {
display: none;
animation: fadeIn 0.2s ease;
}
.section.active { display: block; }
@keyframes fadeIn {
from { opacity: 0; }
to { opacity: 1; }
}
.section h2 {
font-family: 'JetBrains Mono', monospace;
font-size: 15px;
font-weight: 500;
color: #8892a8;
margin-bottom: 8px;
letter-spacing: 1px;
text-transform: uppercase;
}
.section > p.lede {
font-size: 13px;
color: #4a5568;
margin-bottom: 24px;
max-width: 800px;
}
.prose { max-width: 820px; }
.prose p {
font-size: 14px;
color: #b4bccf;
margin-bottom: 14px;
line-height: 1.7;
}
.prose p b { color: #e8eaf0; font-weight: 600; }
.prose ul {
margin: 8px 0 16px 20px;
font-size: 14px;
color: #b4bccf;
line-height: 1.7;
}
.prose ul li { margin-bottom: 8px; }
.prose h3 {
font-family: 'JetBrains Mono', monospace;
font-size: 13px;
font-weight: 500;
color: #e8eaf0;
letter-spacing: 1px;
margin: 32px 0 10px;
text-transform: uppercase;
}
.prose code, pre code {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: #7ab0ff;
background: #121829;
padding: 1px 5px;
border-radius: 3px;
}
pre {
background: #121829;
border: 1px solid #1e2a4a;
padding: 16px;
overflow: auto;
margin-bottom: 16px;
}
pre code {
background: none;
padding: 0;
}
.diagram {
display: block;
max-width: 100%;
margin: 16px 0;
background: #121829;
border: 1px solid #1e2a4a;
padding: 24px;
}
dl {
display: grid;
grid-template-columns: max-content 1fr;
gap: 10px 24px;
margin: 16px 0;
max-width: 820px;
}
dt {
font-family: 'JetBrains Mono', monospace;
color: #0066ff;
font-size: 13px;
padding-top: 2px;
}
dd {
font-size: 14px;
color: #b4bccf;
line-height: 1.6;
}
.menu-toggle {
display: none;
background: transparent;
border: 1px solid #1e2a4a;
color: #8892a8;
padding: 6px 10px;
font-size: 14px;
cursor: pointer;
line-height: 1;
margin-left: auto;
}
.menu-toggle:hover { background: #1a2340; }
.nav-backdrop {
display: none;
position: absolute;
inset: 0;
background: rgba(0, 0, 0, 0.5);
z-index: 10;
}
.layout.nav-open .nav-backdrop { display: block; }
@media (max-width: 720px) {
header { padding: 10px 12px; gap: 8px; }
header h1 { font-size: 16px; letter-spacing: 1px; }
header .subtitle { display: none; }
.menu-toggle { display: inline-block; }
.layout { position: relative; }
nav {
position: absolute;
left: 0; top: 0; bottom: 0;
width: 200px;
z-index: 20;
transform: translateX(-100%);
transition: transform 0.2s ease;
box-shadow: 2px 0 8px rgba(0, 0, 0, 0.5);
}
.layout.nav-open nav { transform: translateX(0); }
main { padding: 16px; }
.section h2 { font-size: 13px; }
.prose p, .prose ul { font-size: 13px; }
}
</style>
</head>
<body>
<header>
<h1>NVI</h1>
<span class="subtitle">analytics agent — BIRD financial</span>
<button class="menu-toggle" onclick="toggleNav()" aria-label="Toggle navigation"></button>
</header>
<div class="layout">
<div class="nav-backdrop" onclick="toggleNav()"></div>
<nav>
<a class="active" onclick="show('overview')">Overview</a>
<a onclick="show('architecture')">Architecture</a>
<a onclick="show('plan')">Plan</a>
<a onclick="show('analyses')">Analyses</a>
<a onclick="show('tools')">Atomic actions</a>
<a onclick="show('recon')">Recon</a>
<a onclick="show('runtime')">Runtime</a>
<a onclick="show('run')">Run it</a>
<a onclick="show('decisions')">Decisions</a>
<a onclick="show('evals')">Evals</a>
</nav>
<main>
<section id="overview" class="section active">
<h2>Overview</h2>
<p class="lede">A text-to-SQL analytics agent over a real warehouse — structured so the agent layer is legible, not buried in a framework.</p>
<div class="prose">
<p>
nvi answers business questions over the BIRD <code>financial</code>
warehouse (PKDD'99 Czech bank: accounts, transactions, loans, cards,
clients, districts, orders, dispositions). It does this by composing
a small set of <b>Analyses</b> — each its own mini-agent — orchestrated
through a langgraph wiring layer with full Langfuse observability.
</p>
<p>
The shape mirrors the JD's three building blocks
(<i>text-to-SQL, reasoning agents, planners</i>) as distinct layers
rather than one omnibus prompt — <b>Plan</b> (L3), <b>Analyses</b> (L2),
and deterministic <b>atomic actions</b> (L1) — resting on a <b>recon</b>
foundation (L0): the dataset's typed knowledge graph. That gives every
part a name and a tight contract, and lets each Analysis pick the
simplest reasoning pattern (CoT, ReAct, plan-and-execute) for what it
does instead of forcing one over everything.
</p>
<p>
The load-bearing decision: <b>the LLM never authors SQL.</b> It picks a
typed shape from finite, recon-derived sets; the composer walks recon and
emits the query deterministically. Columntable binding, joins, and
quoting are graph traversals, not model guesses — so a question the system
can't express fails cleanly instead of producing plausible-but-wrong SQL.
</p>
</div>
</section>
<section id="architecture" class="section">
<h2>Architecture</h2>
<p class="lede">Four layers — Plan composes Analyses; Analyses are mini-agents; atomic actions are deterministic; recon is the foundation they all consult.</p>
<object data="graphs/architecture.svg" type="image/svg+xml" class="diagram"></object>
<dl>
<dt>Plan (L3)</dt>
<dd>One LLM call that turns a question into an ordered set of Analysis invocations — each with args, an intended interpretation, and an optional fallback. It sees only brief table/metric names and the Analysis catalog, never raw DDL. Stateless: it emits a plan and never observes execution.</dd>
<dt>Analyses (L2)</dt>
<dd>Self-contained mini-agents. Each picks the simplest reasoning pattern it needs — CoT for a one-shot move, ReAct for ones that iterate over results, plan-and-execute for compound ones — and owns its loop behind a uniform contract, <code>run(args, question) → Finding</code>. The LLM work lives here: choosing a typed <code>Pick</code> and interpreting rows, never writing SQL.</dd>
<dt>Atomic actions (L1)</dt>
<dd>Deterministic, no LLM. <code>compose(pick, recon) → SQL</code> authors the query by walking recon's join graph; <code>execute_sql</code> runs it read-only with a row cap and timeout. Invalid SQL isn't expressible — the composer binds columns and joins from recon, not from model output.</dd>
<dt>Recon (L0)</dt>
<dd>The dataset's typed knowledge graph — tables, columns, metrics, relationships, storage encodings — merged from sparse human YAML and extracted DDL. The foundation every layer above consults. (Sometimes called L4: the setup that exists before any run, not a step inside one.)</dd>
</dl>
<div class="prose">
<h3>How the layers interconnect</h3>
<p>
The layers read <b>general → specific</b> (Plan → Analyses → atomic
actions), but execution is a graph, not a one-way cascade. Later steps
depend on earlier <i>intermediate results</i>: <code>drill_down</code>
loops, choosing its next dimension from the previous slice's rows; a
step's <b>fallback</b> Analysis fires based on whether the primary's
Finding came back empty or errored. And <b>L0 recon is consulted at every
level</b> — the planner reads its brief names, the <code>Pick</code>
validates against it, the composer walks it. So the layers are
interconnected through both the data they share (recon) and the results
they pass forward.
</p>
</div>
</section>
<section id="plan" class="section">
<h2>Plan layer</h2>
<p class="lede">L3 — one LLM call that decides which Analyses to run.</p>
<div class="prose">
<p>
The planner (<code>api/plan/planner.py</code>) gets the user question
plus three context blocks rendered from recon: brief table names, brief
metric names, and the catalog of Analyses with their <code>args_schema</code>.
Deliberately no DDL — it decides <i>which Analyses to run</i>, not how to
query. One LLM call returns a structured <code>Plan</code>.
</p>
<p>Each step in the plan carries:</p>
<ul>
<li><code>analysis</code> — which Analysis to invoke (must be in the registry).</li>
<li><code>args</code> — the arguments dict, shaped by that Analysis's <code>args_schema</code>.</li>
<li><code>why</code> — a one-line rationale, surfaced in the live trace.</li>
<li><code>fallback</code> — an optional alternate Analysis the runtime runs if the primary returns empty or errors.</li>
</ul>
<p>
The planner never calls a tool or touches data — it's a pure composition
step. The runtime (below) walks the steps in order.
</p>
</div>
</section>
<section id="analyses" class="section">
<h2>Analyses layer</h2>
<p class="lede">L2 — each Analysis is its own mini-agent.</p>
<div class="prose">
<p>
Analyses are the agent layer. Each wraps a piece of analytical intent and
decides how to satisfy it — sometimes one LLM call is enough, sometimes a
ReAct loop over query results is the right fit. The base class
(<code>api/analyses/base.py</code>) enforces only a uniform external
contract, <code>run(args, question) → Finding</code>; the internal
pattern is a per-Analysis decision and no framework types leak out.
</p>
<p>
Within an Analysis, a free-form question becomes a typed <code>Pick</code>
via one LLM call (<code>pick_for_question</code>) over recon-narrowed
candidate sets — that is the only place the model makes a structural
choice, and it chooses from finite known sets, not free text. The
<code>Pick</code> then goes to the deterministic composer (L1).
</p>
<h3>Implemented Analyses</h3>
<dl>
<dt>direct_answer</dt>
<dd>CoT, one shot. <code>pick → compose → execute → interpret</code>. For single-query lookups and aggregations.</dd>
<dt>compare_periods</dt>
<dd>CoT over two queries. One Pick, cloned with two period filters → <code>compose ×2 → execute ×2 → interpret</code> the pair.</dd>
<dt>drill_down</dt>
<dd>ReAct loop. <code>decide_next</code> (LLM) picks a dimension from the candidate list → build Pick → <code>compose → execute</code> → loop on the slice's results. Zero LLM in the slice path itself — only the dimension choice is a model decision, and it's guarded against the candidate list.</dd>
<dt>find_outliers · correlate</dt>
<dd><i>Planned.</i> Additive — new Analyses register without touching the runtime or the layers below.</dd>
</dl>
</div>
</section>
<section id="tools" class="section">
<h2>Atomic actions</h2>
<p class="lede">L1 — deterministic, no LLM inside. The composer is the SQL author.</p>
<div class="prose">
<p>
There is no LLM in this layer — and notably no <code>text_to_sql</code>
tool. An earlier design had the model author SQL with schema-RAG context,
sqlglot validation, and retries; it was removed once the composer landed
(see <b>Decisions</b>). The two atomic actions are:
</p>
<dl>
<dt>compose</dt>
<dd><code>compose(pick, recon) → SQL</code> (<code>api/composer/</code>). Builds one sqlglot expression tree from a typed <code>Pick</code> by walking recon: resolve the metric's table, bind each group-by/filter column to its owning table, compute join paths, render filters and dates, emit quoted SQL. Split by concern — <code>aliases</code>, <code>joins</code>, <code>filters</code>, <code>metrics</code>, <code>dates</code>, <code>order</code>, <code>encodings</code>. If a <code>Pick</code> can't be expressed against recon it raises <code>PickValidationError</code>; it never fabricates SQL.</dd>
<dt>execute_sql</dt>
<dd><code>execute_sql(sql) → rows</code> (<code>api/tools/execute_sql.py</code>). Runs read-only against Postgres (<code>SELECT</code>/<code>WITH</code> only) with a row cap and a statement timeout, returning columns + rows + truncation metadata.</dd>
</dl>
<p>
Because columntable binding and joins come from recon rather than model
output, the composer <i>cannot</i> emit SQL that references a column that
doesn't exist. The validator that used to catch the model's mistakes after
the fact is demoted to a paranoid self-check on the composer.
</p>
</div>
</section>
<section id="recon" class="section">
<h2>Recon</h2>
<p class="lede">L0 — the dataset's typed knowledge graph, and the only dataset-specific surface.</p>
<div class="prose">
<p>
Recon (<code>api/recon/</code>) is the foundation every other layer
consults. It's built in two stages and cached as <code>recon.json</code>:
</p>
<ul>
<li><b>DDL extraction</b> — SQLAlchemy introspects Postgres for tables, columns, types, nullability → <code>extracted_schema.json</code> (auto, gitignored).</li>
<li><b>Augmentation merge</b> — sparse human YAML in <code>api/datasets/&lt;name&gt;/</code> (<code>schema_docs.yaml</code> descriptions + relationships, <code>metrics.yaml</code> SQL fragments, optional <code>encodings</code>) merges onto the extracted schema.</li>
</ul>
<p>
The merged <code>Recon</code> exposes the queries downstream code needs
without going back to the model: <code>resolve_column</code> (bare or
<code>table.col</code> → owning table), <code>join_path</code> (shortest
table sequence between two tables), the metric catalog, and storage
<code>encodings</code> (e.g. how a YYMMDD-int column coerces to a date).
Rebuild with <code>make recon</code> after changing the warehouse or the
YAML.
</p>
<p>
Everything dataset-specific lives here. The Plan, Analyses, and atomic
actions reference no dataset name — point them at a different
<code>api/datasets/&lt;name&gt;/</code> and the same machinery answers
questions over a new warehouse.
</p>
</div>
</section>
<section id="runtime" class="section">
<h2>Runtime</h2>
<p class="lede">langgraph wires the graph; nvi's domain code stays legible on its own.</p>
<div class="prose">
<p>
The orchestration is built on <b>langgraph</b> — it's the right tool
for the graph wiring, parallel execution, and streaming state we need,
and reaching for a custom DAG runner here would be reinventing the
wheel.
</p>
<p>
The intent is the opposite, though: nvi's domain code (Plan, Analyses,
Tools) presents a clean surface that's readable on its own terms.
Understanding what nvi does should not require first learning
langgraph's API. The framework wires nodes; nvi's code defines what
the nodes <i>are</i>, and that's where the interesting reading is.
</p>
<p>
Concretely the graph is small and linear:
</p>
<pre><code>START → plan → execute → synthesize → END</code></pre>
<p>
<code>plan</code> runs the planner; <code>execute</code> walks the plan
steps, dispatching each to its Analysis (and to a fallback Analysis when
the primary returns empty or errors); <code>synthesize</code> folds the
Findings into one answer. The whole run streams SSE events
(<code>run_start</code>, <code>plan_ready</code>,
<code>analysis_start/end</code>, <code>tool_call_start/end</code>,
<code>analysis_fallback</code>) that drive the live trace UI, and every
LLM call opens a named Langfuse generation span.
</p>
<p>
<code>api/runtime/runner.py</code> is the one place langgraph appears —
<code>StateGraph</code> is imported there and nowhere else.
<code>api/plan/</code>, <code>api/analyses/</code>,
<code>api/composer/</code>, and <code>api/tools/</code> contain the
domain code, and none of those files expose langgraph types in their
public interfaces.
</p>
</div>
</section>
<section id="run" class="section">
<h2>Run it locally</h2>
<p class="lede">Prerequisites: lng Langfuse cluster reachable, soleprint-ui framework propagated by spr, <code>.env</code> with Anthropic + Langfuse keys.</p>
<pre><code>make install # uv sync
make kind # create kind-nvi cluster
make tilt-up # bring up postgres, api, ui, docs, gateway
make seed # load BIRD financial into postgres
open http://nvi.local.ar</code></pre>
<div class="prose">
<p>
<code>*.local.ar</code> resolves via dnsmasq; the system Caddy at
<code>~/wdir/ppl/local/Caddyfile</code> proxies <code>nvi.local.ar</code>
to the kind NodePort. Reload after first install: <code>sudo systemctl reload caddy</code>.
</p>
</div>
</section>
<section id="decisions" class="section">
<h2>Architecture decisions</h2>
<div class="prose">
<ul>
<li><b>The LLM never authors SQL.</b> It picks a typed shape from finite, recon-derived sets; the composer walks recon and emits SQL deterministically. Schema-in-the-prompt is a <i>suggestion</i> the model can ignore; composition makes it a <i>constraint</i>. The trade is fewer question shapes for correctness by construction. (See <code>def/schema-as-constraint.md</code>.)</li>
<li><b>Four layers, not one omnibus prompt.</b> Plan composes Analyses; each Analysis picks its own reasoning pattern (CoT / ReAct / plan-and-execute); atomic actions are deterministic; recon is the shared foundation. Borrowing elements from agent frameworks without committing the whole system to one.</li>
<li><b>langgraph for orchestration, nvi for domain.</b> Framework owns graph wiring and streaming; nvi's <code>plan/</code>, <code>analyses/</code>, <code>tools/</code> own the interesting code and are readable without reading langgraph.</li>
<li><b>BIRD <code>financial</code> as the warehouse.</b> Fits Nivii's stated Finance vertical; ships with golden SQL pairs so the eval set is free and verifiable.</li>
<li><b>Langfuse hosted independently</b> by the <code>lng</code> project; reached over WireGuard. Same env var works in dev and any future deploy.</li>
<li><b>Postgres dedicated to nvi.</b> Persisted via <code>.data/postgres</code> hostPath mount so the BIRD load survives cluster restarts.</li>
</ul>
</div>
</section>
<section id="evals" class="section">
<h2>Evals</h2>
<p class="lede">BIRD <code>financial</code> dev set, run by <code>api/evals/run_evals.py</code> (<code>make evals</code>), traces pushed to Langfuse.</p>
<div class="prose">
<p>
BIRD ships golden question/SQL pairs, so the eval set is free and
verifiable. Each run replays questions through the full stack
(plan → analyses → compose → execute) and records the trace in Langfuse
for inspection — token usage per layer, which Analysis fired, the composed
SQL. The composer refactor's own check is the unit suite
(<code>uv run pytest</code>): golden SQL strings plus validation
rejections, so structural correctness is pinned independent of the model.
</p>
</div>
</section>
</main>
</div>
<script>
function show(id) {
document.querySelectorAll('nav a').forEach(a => a.classList.remove('active'));
document.querySelectorAll('.section').forEach(s => s.classList.remove('active'));
event.target.classList.add('active');
document.getElementById(id).classList.add('active');
if (window.innerWidth <= 720) document.querySelector('.layout').classList.remove('nav-open');
}
function toggleNav() {
document.querySelector('.layout').classList.toggle('nav-open');
}
</script>
</body>
</html>