meetus/INDEX.md

# meetus — repo index

A standalone, local-first **command-line tool**: turn screen-share meeting/training
recordings into a rich, AI-summarizable transcript. The extraction pipeline is
deterministic and offline; the summarization step is the only LLM-facing part.

meetus stands on its own. The separate **`cht`** project does the same thing in
realtime; meetus may or may not feed it, so the two are kept formally apart (the
cht-facing bits here live under `ctrl/cht/`).

> This file maps the whole repo. [`README.md`](README.md) is the detailed manual for
> the main `process_meeting.py` CLI.

## Data flow

```
video / audio
   │  process_meeting.py                       (deterministic, offline)
   ▼
output/<run>/                                  run folder: YYYYMMDD-NNN-<stem>/
   ├── <stem>.json                             Whisper/WhisperX transcript (base)
   ├── <stem>.srt / .vtt / ...                 extra transcript formats (optional)
   ├── frames/                                 extracted scene/interval frames
   ├── <stem>_enhanced.txt                     transcript + frame refs   ◀── the product
   └── manifest.json
   │  ctrl/summarize/*.py                       (local LLM — WIP, on hold)
   ▼
<stem>_summary_simple.md / _reference.md
```

`make batch` (→ `ctrl/batch.sh`) runs the pipeline over a whole directory tree,
mirroring the input folder structure into the output.

## doocus — document extraction (alongside meetus)

The same idea for documents. `process_doc.py` turns local files (Drive exports,
PDFs, images, ...) into a textified product + metadata, deterministically and
offline. Its output is what feeds permitted services (Gemini web, NotebookLM)
and the local search index; the raw source never leaves the machine.

```
local drive tree (downloaded)
   │  process_tree.py                            (deterministic, offline, no cloud AI)
   ▼
docs-output/                                     whole tree replicated, no flattening
   ├── index.json                                every file as a node (path, type, mode, url:null)
   └── <mirrored path>/<file>.doocus/            sidecar, only for docx/pdf/pptx/xlsx:
        ├── content.md                             extracted text — SEARCH INDEX ONLY
        └── meta.json                              (never replaces the original)
```

Node modes: **extracted** (docx/pdf/pptx/xlsx → text sidecar), **meeting**
(mp4/... → belongs to meetus), **link** (everything else → original is the
artifact, not duplicated). Each node has a `url` slot for the eventual Drive link
and a `created` date. `make docs` (→ `process_tree.py`) indexes a whole tree;
`process_doc.py` still does one-off single-file extraction.

Meetings stay coherent with docs: `docs-output/meetings.txt` lists the meeting
paths, and `make batch … LIST=… OUT_FORMAT="{name}.meetus"` runs meetus into a
`<file>.meetus/` sidecar beside the docs — the exact path the indexer already
pointed each `meeting` node at (`out`/`transcript`). meetus's `--out-format`
(tokens `{date} {run} {stem} {name}`) makes the run-folder name configurable;
default is unchanged. Full doocus docs: [`doocus/README.md`](doocus/README.md).
Cross-search (docs frame + meetings frame, over cached text) is a later phase. 🚧

## Status legend

✅ active · 🚧 WIP, on hold · 🗄️ deprecated (kept for reference) · 🧰 one-off / niche · 📄 docs

## Layout

```
process_meeting.py        ✅ the CLI tool — entry point (stays at root)
process_tree.py           ✅ doocus — index a whole local drive tree → index.json
process_doc.py            ✅ doocus — one-off single-file extraction
Makefile                  ✅ batch convenience wrapper (make batch / make docs)
meetus/                   ✅ core package
  ├─ workflow.py             orchestrator (whisper → frames → merge)
  ├─ frame_extractor.py      FFmpeg scene-detection / interval frames
  ├─ transcript_merger.py    interleave transcript + frame refs → enhanced.txt
  ├─ output_manager.py       run-folder naming + manifest.json
  ├─ cache_manager.py        per-step caching (skip done work on rerun)
  └─ deprecated/          🗄️ old screen-text idea (OCR/vision/hybrid) — unwired, reference only
       ├─ ocr_processor.py       (was --ocr-engine)
       ├─ vision_processor.py    (was --use-vision)
       ├─ hybrid_processor.py    (was --use-hybrid)
       └─ prompts/               vision context prompts
doocus/                   ✅ document-extraction package (see doocus/README.md)
  ├─ tree.py                 build_index — walk drive tree → index.json (+ extract sidecars)
  ├─ registry.py             extension → extractor family dispatch (lazy, graceful)
  ├─ workflow.py             single-file DocWorkflow (process_doc.py)
  ├─ output_manager.py       single-file run-folder + content.md/meta.json writer
  ├─ naming.py               YYYYMMDD-NNN naming + sha256 (reuses meetus convention)
  └─ extractors/             text, tabular(csv), office(docx/pptx/xlsx), pdf, web(html), image, media(mp4)
ctrl/                     control plane / operational scripts
  ├─ batch.sh             ✅ recursive batch runner (mirrors tree, continues past failures)
  ├─ transcribe_oneoff.sh 🧰 high-quality re-transcription over an existing run
  ├─ summarize/           🚧 last step — local-LLM summarization (WIP, on hold)
  │    ├─ summarize_simple.py    minimal map-and-append; reads every referenced frame
  │    ├─ compile_meeting.py     REFINE-pattern technical reference; on-demand frames
  │    └─ summarize_meeting.py   map→extract(validated facts)→reduce
  └─ cht/                 🧰 bridge to the separate realtime `cht` project
       └─ interleave_cht_frames.py   whisperx JSON + cht frames/index.json → enhanced.txt
ui/                       local browser UIs (Vue 3 + Vite), shared framework
  ├─ framework/           ✅ shared component/renderer library + design tokens
  ├─ meetus-app/          ✅ review meeting runs (video · transcript · frames; @review source)
  └─ doocus-app/          ✅ browse tree · search · package; embeds the meetus review
doocus-data/ meetus-data/ ✅ managed collection roots (gitignored): <root>/<source>/
docs/graphs/              📄 architecture diagram (.dot → .svg, `make graphs`)
def/                      📄 design/decision notes, in order (the feature history)
README.md                 📄 project overview (meetus + doocus) + architecture diagram
INDEX.md                  📄 this file
MARIAN.md                 📄 genesis brainstorm (explains why the deprecated OCR path exists)
local-run.sh              📄 personal scratch invocations (gitignored)
```

## Collections & UIs

- **Managed roots** (gitignored): `doocus-data/<source>/` (docs) and
  `meetus-data/<source>/` (meetings), each `<root>/<source>/` mirroring the source.
  `process_tree.py --only {all|docs|meetings}` scopes a collection.
- **doocus-app** discovers both roots (`DOOCUS_DATA` overrides, comma-separated),
  **merges collections that share `index.json.root`** into one source, searches the
  cached text, and can **scan a folder** (runs `process_tree` via `uv`) from the UI.
- **meetus-app** is a thin shell over `@review` (`ui/meetus-app/src/components/ReviewBody.vue`),
  the review composition doocus embeds — meetus stays the single source.
- **Diagram**: `docs/graphs/architecture.dot` → `docs/graphs/architecture.svg`
  (`make graphs`; needs system graphviz). Both committed.

## Notes

- **Default flow** uses `--embed-images` (frames referenced for the LLM to read) +
  `--scene-detection --scene-threshold 10 --diarize`. `make batch` adds
  `--transcript-formats srt` and writes outputs next to the sources. See `README.md`.
- **`meetus/deprecated/`** is no longer imported or reachable from the CLI (its flags
  were removed and `workflow.py` no longer imports it). Kept for reference only; the
  realtime continuation of the idea is the separate `cht` project. Its only unique
  dependency (`ollama`) is isolated to the `deprecated` uv group in `pyproject.toml`,
  so a normal `uv sync --group meetus`/`--group doocus` never installs it.
- **Dependencies** are uv feature groups in `pyproject.toml` (no `requirements.txt`):
  `meetus`, `doocus`, `ocr`, `pdf-render`, `deprecated`. whisper/whisperx are
  external CLIs (like ffmpeg), installed separately — not uv-managed.
- **`ctrl/summarize/`** scripts are standalone; run them under an env with the `openai`
  client (e.g. `~/wdir/llm/.venv`) against a local OpenAI-compatible server.

## Loose ends

- `README.md`'s "Output Files" example still shows the old run-folder format
  `YYYYMMDD_HHMMSS-video` (actual: `YYYYMMDD-NNN-<stem>`). Minor; worth a tidy.