batch docs update

This commit is contained in:
Mariano Gabriel
2026-07-05 10:38:29 -03:00
parent ca8b3a784d
commit fa3a0e3d1a
5 changed files with 219 additions and 143 deletions

View File

@@ -39,19 +39,22 @@ offline. Its output is what feeds permitted services (Gemini web, NotebookLM)
and the local search index; the raw source never leaves the machine.
```
document (docx/pdf/xlsx/img/...)
│ process_doc.py (deterministic, offline, no cloud AI)
local drive tree (downloaded)
│ process_tree.py (deterministic, offline, no cloud AI)
docs-output/<run>/ run folder: YYYYMMDD-NNN-<stem>/
├── content.md textified document ◀── the product
── meta.json source/sha/dates/author/pages/... metadata
├── thumb.jpg optional single preview
└── manifest.json
docs-output/ whole tree replicated, no flattening
├── index.json every file as a node (path, type, mode, url:null)
── <mirrored path>/<file>.doocus/ sidecar, only for docx/pdf/pptx/xlsx:
├── content.md extracted text — SEARCH INDEX ONLY
└── meta.json (never replaces the original)
```
`make docs` (→ `ctrl/batch_docs.sh`) runs it over a whole tree. Full doocus docs:
[`doocus/README.md`](doocus/README.md). Cross-search (meetus transcripts + doocus
docs, textified-only) is a later phase. 🚧
Node modes: **extracted** (docx/pdf/pptx/xlsx → text sidecar), **meeting**
(mp4/... → belongs to meetus), **link** (everything else → original is the
artifact, not duplicated). Each node has a `url` slot for the eventual Drive link.
`make docs` (→ `process_tree.py`) indexes a whole tree; `process_doc.py` still does
one-off single-file extraction. Full doocus docs: [`doocus/README.md`](doocus/README.md).
Cross-search (docs frame + meetings frame, over cached text) is a later phase. 🚧
## Status legend
@@ -61,7 +64,8 @@ docs, textified-only) is a later phase. 🚧
```
process_meeting.py ✅ the CLI tool — entry point (stays at root)
process_doc.py ✅ doocus CLI — document extraction entry point
process_tree.py ✅ doocus — index a whole local drive tree → index.json
process_doc.py ✅ doocus — one-off single-file extraction
Makefile ✅ batch convenience wrapper (make batch / make docs)
meetus/ ✅ core package
├─ workflow.py orchestrator (whisper → frames → merge)
@@ -75,14 +79,14 @@ meetus/ ✅ core package
├─ hybrid_processor.py (was --use-hybrid)
└─ prompts/ vision context prompts
doocus/ ✅ document-extraction package (see doocus/README.md)
├─ workflow.py DocConfig + DocWorkflow (dispatch → extract → write → manifest)
├─ tree.py build_index — walk drive tree → index.json (+ extract sidecars)
├─ registry.py extension → extractor family dispatch (lazy, graceful)
├─ output_manager.py run-folder + content.md/meta.json/thumb writer
├─ workflow.py single-file DocWorkflow (process_doc.py)
├─ output_manager.py single-file run-folder + content.md/meta.json writer
├─ naming.py YYYYMMDD-NNN naming + sha256 (reuses meetus convention)
└─ extractors/ text, tabular(csv), office(docx/pptx/xlsx), pdf, web(html), image, media(mp4)
ctrl/ control plane / operational scripts
├─ batch.sh ✅ recursive batch runner (mirrors tree, continues past failures)
├─ batch_docs.sh ✅ doocus twin of batch.sh (make docs)
├─ transcribe_oneoff.sh 🧰 high-quality re-transcription over an existing run
├─ summarize/ 🚧 last step — local-LLM summarization (WIP, on hold)
│ ├─ summarize_simple.py minimal map-and-append; reads every referenced frame