diff --git a/README.md b/README.md index 93b84ae..37c7a43 100644 --- a/README.md +++ b/README.md @@ -1,115 +1,14 @@ -# meetus + doocus — a local, offline context extractor +# meetus + doocus -Turn a downloaded Google Drive (meetings **and** documents) into a **searchable, -browsable, packageable** local knowledge base — then hand focused subsets to the -AI services you're allowed to use (Gemini web, NotebookLM). Every extraction step -is **deterministic and offline**; no cloud AI ever touches the raw source. +A local, offline **context extractor**: turn a downloaded Google Drive — meetings +**and** documents — into a searchable, browsable, packageable knowledge base, then +hand focused subsets to the AI services you're allowed to use (Gemini web, +NotebookLM). Every extraction step is deterministic and offline; no cloud AI ever +touches the raw source. -Two pipelines feed two local UIs over a shared component framework: +- **meetus** — meeting recordings → enhanced transcripts (WhisperX + screen frames) +- **doocus** — documents → textified content + metadata, indexed into a tree -- **meetus** — meeting recordings → enhanced transcripts (WhisperX + screen frames). -- **doocus** — documents → textified content + metadata, indexed into a tree. +📖 **Documentation:** -![Architecture](docs/graphs/architecture.svg) - -> Diagram source: [`docs/graphs/architecture.dot`](docs/graphs/architecture.dot) -> (regenerate with `make graphs`). Repo map: [`INDEX.md`](INDEX.md). - -## Install - -### System dependencies -```bash -sudo apt-get install ffmpeg graphviz # ffmpeg: frames/thumbnails · graphviz: docs diagrams -# optional: tesseract-ocr (doocus --ocr), poppler-utils (pdf page render) -# macOS: brew install ffmpeg graphviz -``` - -### Python (uv feature groups) -Dependencies live in `pyproject.toml` as [uv](https://docs.astral.sh/uv/) groups — -install only what you need: -```bash -uv sync --group meetus # meeting pipeline (opencv, ffmpeg-python) -uv sync --group doocus # document extraction (docx/pdf/pptx/xlsx/…) -uv sync --group doocus --group ocr # + OCR for images / scanned pdfs -``` -Groups: `meetus`, `doocus`, `ocr`, `pdf-render`, `deprecated`. **whisper/whisperx** -are external CLI tools (like ffmpeg) — install separately (often a GPU env): -`pip install whisperx` (diarization needs a HuggingFace token with pyannote access). - -### UIs (Node) -```bash -cd ui/meetus-app && npm install -cd ui/doocus-app && npm install -``` - -## The two pipelines - -### meetus — meetings -```bash -uv run process_meeting.py samples/meeting.mkv --embed-images --scene-detection --scene-threshold 10 --diarize -``` -Runs WhisperX (speaker diarization) + FFmpeg scene-detection frames → an enhanced -transcript with frame references, in `output//`. `--out-format` -customizes the run-folder name (e.g. `{name}.meetus` for a docs-coherent sidecar). -Batch a tree with `make batch IN=`. `process_meeting.py --help` is the flag -source of truth; caching (`--no-cache`, `--skip-cache-frames/-whisper`) lets you -iterate on scene thresholds without re-running Whisper. - -### doocus — documents -```bash -uv run process_tree.py "/path/to/drive" --output "doocus-data/" --only docs -``` -Replicates the whole tree into `index.json` (every file a node), extracting text -sidecars (`content.md` + `meta.json`) only for `docx/pdf/pptx/xlsx`; other files -are linked, not duplicated. `--only {all|docs|meetings}` scopes a collection. -Full manual: [`doocus/README.md`](doocus/README.md). - -## Outputs & collections - -Each app writes to a **managed root** (gitignored), one subfolder per source, -mirroring the source structure: - -- `doocus-data//` — document collections (`index.json` + `.doocus/`) -- `meetus-data//` — meeting collections (`index.json` + `.meetus/`) - -Collections whose `index.json.root` matches are the **same source**: the doocus UI -merges their docs + meetings into one tree, toggled as one source. A meeting's -`.meetus` must sit in the **same folder** as its collection's `index.json`. - -## The UIs - -Local Vue 3 + Vite apps over `ui/framework/` (shared components + design tokens): - -- **`ui/meetus-app`** — review a meeting: video + transcript (edit/read, speaker - merge, select mode) + frame selector, time-synced. -- **`ui/doocus-app`** — browse the merged tree, **search** cached text (transcripts, - extracted docs, raw files), per-type viewers, and **package** selected files. - Meetings **embed the meetus review** (`@review`, composed — not duplicated). -```bash -cd ui/doocus-app && DOOCUS_DATA="$PWD/../../doocus-data,$PWD/../../meetus-data" npm run dev -cd ui/meetus-app && MEETUS_OUTPUT=/abs/path/to/output npm run dev -``` - -## Packaging (the point) - -The doocus package builder zips selected **originals** + their `content.md` for a -target (Gemini web ≤10 files/≤100 MB, or NotebookLM). Originals are what the -permitted services consume; the extracted text is the local **search index** and is -never a replacement for the document. - -## Project layout - -See [`INDEX.md`](INDEX.md) for the full map. In brief: -``` -process_meeting.py meetus CLI process_tree.py / process_doc.py doocus CLIs -meetus/ meetus core doocus/ doocus core -ctrl/ batch + ops scripts ui/{meetus-app,doocus-app,framework} -Makefile batch · docs · merge · graphs docs/graphs/ architecture diagram -def/ design notes samples/ output/ (gitignored) -``` - -Deprecated OCR/vision code lives unwired in `meetus/deprecated/` (the only user of -`ollama`, kept out of every default install). See [`INDEX.md`](INDEX.md). - -## License -For personal use. +🗺️ Repo map for contributors: [`INDEX.md`](INDEX.md) diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..c2866b5 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,146 @@ + + + + + +meetus + doocus — local offline context extractor + + + +
+
+

meetus + doocus

+
A local, offline context extractor · deterministic · no cloud AI touches the source
+
+ +
+

Turn a downloaded Google Drive — meetings and documents — into a + searchable, browsable, packageable knowledge base, then hand focused subsets to the + AI services you're allowed to use (Gemini web, NotebookLM). Two offline pipelines + feed two local UIs over a shared component framework.

+

+ meetus → meeting transcripts (WhisperX + frames) + doocus → documents → text + metadata tree +

+
meetus + doocus architecture
+
+ +
+

The two pipelines

+
+
+

meetus — meetings

+

WhisperX (speaker diarization) + FFmpeg scene-detection frames → an enhanced + transcript with frame references.

+ 
uv run process_meeting.py meeting.mkv \
+  --embed-images --scene-detection \
+  --scene-threshold 10 --diarize
+

Batch a tree with make batch IN=<dir>. --out-format + controls the run-folder name (e.g. {name}.meetus).

+
+
+

doocus — documents

+

Replicates the whole tree into index.json; extracts text sidecars + (content.md + meta.json) only for docx/pdf/pptx/xlsx — + other files are linked, not duplicated.

+ 
uv run process_tree.py "/path/to/drive" \
+  --output "doocus-data/<source>" --only docs
+

--only {all|docs|meetings} scopes a collection.

+
+
+
+ +
+

Outputs & collections

+

Each app writes to a managed root (gitignored), one subfolder per + source, mirroring the source structure:

+
    +
  • doocus-data/<source>/ — document collections (index.json + .doocus/)
    • +
  • meetus-data/<source>/ — meeting collections (index.json + .meetus/)
    • +
+

Collections whose index.json.root match are the same source: + the doocus UI merges their docs + meetings into one tree, toggled as one source. When a + file exists in two collections, the copy whose sidecar resolves wins. A meeting's + .meetus must sit in the same folder as its collection's index.

+
+ +
+

The UIs

+

Local Vue 3 + Vite apps over ui/framework/ (shared components + tokens):

+
    +
  • meetus-app — review a meeting: video + transcript (edit/read, speaker + merge, select mode) + frame selector, time-synced.
    • +
  • doocus-app — browse the merged tree, search the cached + text (transcripts, extracted docs, raw files), per-type viewers, and package + selected files. Meetings embed the meetus review (@review, + composed — not duplicated).
    • +
+ 
cd ui/doocus-app && DOOCUS_DATA="/abs/doocus-data,/abs/meetus-data" npm run dev
+cd ui/meetus-app && MEETUS_OUTPUT=/abs/output npm run dev
+
+ +
+

Packaging

+

The doocus package builder zips selected originals + their + content.md for a target (Gemini web ≤10 files/≤100 MB, or NotebookLM). + Originals are what the permitted services consume; the extracted text is the local + search index — never a replacement for the document.

+
+ +
+

Install

+ 
# system
+sudo apt-get install ffmpeg graphviz          # + tesseract-ocr, poppler-utils (optional)
+
+# python (uv feature groups)
+uv sync --group meetus                         # meeting pipeline
+uv sync --group doocus --group ocr             # docs + OCR
+
+# whisper/whisperx are external CLIs (install separately, often a GPU env)
+pip install whisperx                           # diarization needs a HuggingFace token
+
+# UIs
+cd ui/meetus-app && npm install
+cd ui/doocus-app && npm install
+
+ + +
+ +