doocus — local drive tree indexer

The document twin of meetus. Replicates a whole local drive tree into a single index.json, preserving the folder hierarchy (no flattening). Every file is a node; the original file is the artifact (what a QA/PM opens, what a package links to). doocus only extracts text for the heavy formats (docx/pdf/pptx/xlsx) — and that text is a search index only, never a replacement for the document.

Extraction is deterministic and offline — no cloud AI touches the raw source. The raw source stays on disk; permitted services (Gemini web, NotebookLM) get the original (as a link once we have Drive URLs, or re-uploaded).

Quick start

# index a whole downloaded drive tree  → docs-output/index.json
uv run make docs IN="/mnt/win/drive"
uv run make docs IN="..." DOC_EXTRA="--ocr"      # OCR scanned pdfs
make docs-dry IN="..."                           # counts only, no writes

# or directly
uv run process_tree.py /mnt/win/drive --output docs-output

# one-off single file (older per-file model, still handy)
uv run process_doc.py notes.docx

Output contract

docs-output/
  index.json                          whole tree; every file a node:
                                        { path, name, ext, family, mode, bytes,
                                          modified, url:null }   ◀ url = Drive link (later)
  <mirrored path>/<file>.doocus/      sidecar — ONLY for extracted formats:
     ├── content.md                     extracted text — SEARCH INDEX ONLY
     └── meta.json                      source/sha/dates/author/pages/... metadata

Node modes: extracted (docx/pdf/pptx/xlsx → text sidecar) · meeting (mp4/mkv/... → belongs to meetus, not a doc) · link (everything else → original is the artifact, nothing duplicated). Every node also carries created (birth time, falling back to mtime) so docs and meetings sort by creation date coherently.

Meetings

Videos are meeting nodes — doocus does not transcribe them; meetus does. To keep everything in one coherent, searchable tree, run meetus over only the meetings doocus found and write each into a <file>.meetus/ sidecar right next to its <file>.doocus/-style neighbours:

# 1. index the drive → docs-output/{index.json, meetings.txt}
uv run make docs IN="/mnt/drive"

# 2. meetus over ONLY the listed meetings, re-rooted at the mount, into
#    <file>.meetus/ sidecars (matches the pointer doocus already wrote):
make batch IN="/mnt/drive" OUT=docs-output \
     LIST=docs-output/meetings.txt \
     OUT_FORMAT="{name}.meetus"

meetings.txt holds relative paths only — mount the drive anywhere and the -l/--list re-roots them under -i, so this machine reads only the meetings.
OUT_FORMAT="{name}.meetus" is meetus's new --out-format (tokens {date} {run} {stem} {name}; default keeps YYYYMMDD-NNN-<stem>). Omitting {run} makes the folder deterministic, so it matches the out/transcript pointer the indexer set on each meeting node — no relink step.
Frames land in <file>.meetus/frames/ for now (unchanged).

Supported types

Family	Extensions	Notes
text	md, txt, yaml/yml, json	passthrough; json/yaml add a structure summary
tabular	csv	markdown table + row/col metadata (stdlib)
office	docx, pptx, xlsx	text + core properties (author, dates, title)
pdf	pdf	text per page + info dict; `--render` for page-1 thumb
web	html/htm	visible text + title/meta tags
image	jpg/jpeg, png	EXIF (incl. capture date) + thumbnail; `--ocr` for text
media	mp4	ffprobe metadata + thumbnail; delegates transcription to meetus

Each extractor is isolated: a missing optional dependency or a corrupt file is recorded as a warning in meta.json, never aborting a batch (same resilience as ctrl/batch.sh).

Dependencies

Core text/tabular use only the stdlib. Everything else is a uv group in the repo's pyproject.toml:

uv sync --group doocus                 # Pillow, PyYAML, python-docx, python-pptx,
                                       # openpyxl, pypdf, beautifulsoup4, lxml
uv sync --group doocus --group ocr     # + pytesseract (needs system `tesseract`)

mp4 uses system ffmpeg/ffprobe (shared with meetus). PDF page rendering (--render) additionally needs the pdf-render group (pdf2image) + system poppler.

Browser UI

ui/doocus-app/ (sibling of ui/meetus-app/, shares ui/framework/) reads index.json and shows the full tree (folders preserved). Select a file → detail view: for extracted docs, a split of the markdown-rendered extracted text (clearly labelled "search index, not the document") beside the native view of the original (pdf inline; docx/xlsx → download); for linked files the original renders directly (image / html / text / markdown). A package builder zips selected originals (+ the content.md for extracted ones) for a target (Gemini web / NotebookLM), and will emit Drive links once nodes carry a url.

cd ui/doocus-app && npm install
DOOCUS_OUTPUT=/abs/path/to/docs-output npm run dev

5.3 KiB Raw Blame History