batch docs update
This commit is contained in:
32
INDEX.md
32
INDEX.md
@@ -39,19 +39,22 @@ offline. Its output is what feeds permitted services (Gemini web, NotebookLM)
|
||||
and the local search index; the raw source never leaves the machine.
|
||||
|
||||
```
|
||||
document (docx/pdf/xlsx/img/...)
|
||||
│ process_doc.py (deterministic, offline, no cloud AI)
|
||||
local drive tree (downloaded)
|
||||
│ process_tree.py (deterministic, offline, no cloud AI)
|
||||
▼
|
||||
docs-output/<run>/ run folder: YYYYMMDD-NNN-<stem>/
|
||||
├── content.md textified document ◀── the product
|
||||
├── meta.json source/sha/dates/author/pages/... metadata
|
||||
├── thumb.jpg optional single preview
|
||||
└── manifest.json
|
||||
docs-output/ whole tree replicated, no flattening
|
||||
├── index.json every file as a node (path, type, mode, url:null)
|
||||
└── <mirrored path>/<file>.doocus/ sidecar, only for docx/pdf/pptx/xlsx:
|
||||
├── content.md extracted text — SEARCH INDEX ONLY
|
||||
└── meta.json (never replaces the original)
|
||||
```
|
||||
|
||||
`make docs` (→ `ctrl/batch_docs.sh`) runs it over a whole tree. Full doocus docs:
|
||||
[`doocus/README.md`](doocus/README.md). Cross-search (meetus transcripts + doocus
|
||||
docs, textified-only) is a later phase. 🚧
|
||||
Node modes: **extracted** (docx/pdf/pptx/xlsx → text sidecar), **meeting**
|
||||
(mp4/... → belongs to meetus), **link** (everything else → original is the
|
||||
artifact, not duplicated). Each node has a `url` slot for the eventual Drive link.
|
||||
`make docs` (→ `process_tree.py`) indexes a whole tree; `process_doc.py` still does
|
||||
one-off single-file extraction. Full doocus docs: [`doocus/README.md`](doocus/README.md).
|
||||
Cross-search (docs frame + meetings frame, over cached text) is a later phase. 🚧
|
||||
|
||||
## Status legend
|
||||
|
||||
@@ -61,7 +64,8 @@ docs, textified-only) is a later phase. 🚧
|
||||
|
||||
```
|
||||
process_meeting.py ✅ the CLI tool — entry point (stays at root)
|
||||
process_doc.py ✅ doocus CLI — document extraction entry point
|
||||
process_tree.py ✅ doocus — index a whole local drive tree → index.json
|
||||
process_doc.py ✅ doocus — one-off single-file extraction
|
||||
Makefile ✅ batch convenience wrapper (make batch / make docs)
|
||||
meetus/ ✅ core package
|
||||
├─ workflow.py orchestrator (whisper → frames → merge)
|
||||
@@ -75,14 +79,14 @@ meetus/ ✅ core package
|
||||
├─ hybrid_processor.py (was --use-hybrid)
|
||||
└─ prompts/ vision context prompts
|
||||
doocus/ ✅ document-extraction package (see doocus/README.md)
|
||||
├─ workflow.py DocConfig + DocWorkflow (dispatch → extract → write → manifest)
|
||||
├─ tree.py build_index — walk drive tree → index.json (+ extract sidecars)
|
||||
├─ registry.py extension → extractor family dispatch (lazy, graceful)
|
||||
├─ output_manager.py run-folder + content.md/meta.json/thumb writer
|
||||
├─ workflow.py single-file DocWorkflow (process_doc.py)
|
||||
├─ output_manager.py single-file run-folder + content.md/meta.json writer
|
||||
├─ naming.py YYYYMMDD-NNN naming + sha256 (reuses meetus convention)
|
||||
└─ extractors/ text, tabular(csv), office(docx/pptx/xlsx), pdf, web(html), image, media(mp4)
|
||||
ctrl/ control plane / operational scripts
|
||||
├─ batch.sh ✅ recursive batch runner (mirrors tree, continues past failures)
|
||||
├─ batch_docs.sh ✅ doocus twin of batch.sh (make docs)
|
||||
├─ transcribe_oneoff.sh 🧰 high-quality re-transcription over an existing run
|
||||
├─ summarize/ 🚧 last step — local-LLM summarization (WIP, on hold)
|
||||
│ ├─ summarize_simple.py minimal map-and-append; reads every referenced frame
|
||||
|
||||
Reference in New Issue
Block a user