doocus — local drive tree indexer

The document twin of meetus. Replicates a whole local drive tree into a single index.json, preserving the folder hierarchy (no flattening). Every file is a node; the original file is the artifact (what a QA/PM opens, what a package links to). doocus only extracts text for the heavy formats (docx/pdf/pptx/xlsx) — and that text is a search index only, never a replacement for the document.

Extraction is deterministic and offline — no cloud AI touches the raw source. The raw source stays on disk; permitted services (Gemini web, NotebookLM) get the original (as a link once we have Drive URLs, or re-uploaded).

Quick start

# index a whole downloaded drive tree  → docs-output/index.json
uv run make docs IN="/mnt/win/drive"
uv run make docs IN="..." DOC_EXTRA="--ocr"      # OCR scanned pdfs
make docs-dry IN="..."                           # counts only, no writes

# or directly
uv run process_tree.py /mnt/win/drive --output docs-output

# one-off single file (older per-file model, still handy)
uv run process_doc.py notes.docx

Output contract

docs-output/
  index.json                          whole tree; every file a node:
                                        { path, name, ext, family, mode, bytes,
                                          modified, url:null }   ◀ url = Drive link (later)
  <mirrored path>/<file>.doocus/      sidecar — ONLY for extracted formats:
     ├── content.md                     extracted text — SEARCH INDEX ONLY
     └── meta.json                      source/sha/dates/author/pages/... metadata

Node modes: extracted (docx/pdf/pptx/xlsx → text sidecar) · meeting (mp4/mkv/... → belongs to meetus, not a doc) · link (everything else → original is the artifact, nothing duplicated). Every node also carries created (birth time, falling back to mtime) so docs and meetings sort by creation date coherently.

Meetings

Videos are meeting nodes — doocus does not transcribe them; meetus does. To keep everything in one coherent, searchable tree, run meetus over only the meetings doocus found and write each into a <file>.meetus/ sidecar right next to its <file>.doocus/-style neighbours:

# 1. index the drive → docs-output/{index.json, meetings.txt}
uv run make docs IN="/mnt/drive"

# 2. meetus over ONLY the listed meetings, re-rooted at the mount, into
#    <file>.meetus/ sidecars (matches the pointer doocus already wrote):
make batch IN="/mnt/drive" OUT=docs-output \
     LIST=docs-output/meetings.txt \
     OUT_FORMAT="{name}.meetus"

meetings.txt holds relative paths only — mount the drive anywhere and the -l/--list re-roots them under -i, so this machine reads only the meetings.
OUT_FORMAT="{name}.meetus" is meetus's new --out-format (tokens {date} {run} {stem} {name}; default keeps YYYYMMDD-NNN-<stem>). Omitting {run} makes the folder deterministic, so it matches the out/transcript pointer the indexer set on each meeting node — no relink step.
Frames land in <file>.meetus/frames/ for now (unchanged).

Supported types

Family	Extensions	Notes
text	md, txt, yaml/yml, json	passthrough; json/yaml add a structure summary
tabular	csv	markdown table + row/col metadata (stdlib)
office	docx, pptx, xlsx	text + core properties (author, dates, title)
pdf	pdf	text per page + info dict; `--render` for page-1 thumb
web	html/htm	visible text + title/meta tags
image	jpg/jpeg, png	EXIF (incl. capture date) + thumbnail; `--ocr` for text
media	mp4	ffprobe metadata + thumbnail; delegates transcription to meetus

Each extractor is isolated: a missing optional dependency or a corrupt file is recorded as a warning in meta.json, never aborting a batch (same resilience as ctrl/batch.sh).

Dependencies

Core text/tabular use only the stdlib. Everything else is a uv group in the repo's pyproject.toml:

uv sync --group doocus                 # Pillow, PyYAML, python-docx, python-pptx,
                                       # openpyxl, pypdf, beautifulsoup4, lxml
uv sync --group doocus --group ocr     # + pytesseract (needs system `tesseract`)

mp4 uses system ffmpeg/ffprobe (shared with meetus). PDF page rendering (--render) additionally needs the pdf-render group (pdf2image) + system poppler.

Collections (multiple sources)

doocus-app discovers collections from managed roots (gitignored), <root>/<source>/ per source:

doocus-data/<source>/ — document collections
meetus-data/<source>/ — meeting collections (.meetus + a --only meetings index)

Override the roots with DOOCUS_DATA (comma-separated; DOOCUS_OUTPUT still works for a single dir). Collections whose index.json.root match are the same source — the UI merges their docs + meetings into one interleaved tree, toggled together from the ⧉ sources menu. When the same file exists in two collections, the copy whose sidecar resolves wins (a meetus-data meeting with its .meetus overrides a bare doocus copy).

Browser UI

ui/doocus-app/ (sibling of ui/meetus-app/, shares ui/framework/):

Tree (folders preserved, source-grouped) with a search bar — server-side search over the cached text (transcripts, extracted content.md, raw files); matches prune the tree and show a file count + size.
Detail: for extracted docs, markdown-rendered extracted text (labelled "search index, not the document") beside the native original (pdf inline; docx/xlsx rendered via mammoth/SheetJS); linked files render directly; meetings embed the meetus review (video + transcript + frames, from @review).
Scan a folder from the UI (⧉ menu → Scan) — runs process_tree via uv into doocus-data/; ⟳ re-discovers newly-scanned sources.
Package builder zips selected originals (+ content.md for extracted) for a target (Gemini web / NotebookLM); Drive links once nodes carry a url.

cd ui/doocus-app && npm install
DOOCUS_DATA="/abs/doocus-data,/abs/meetus-data" npm run dev

6.3 KiB Raw Blame History