# doocus — local drive tree indexer

The document twin of meetus. Replicates a **whole local drive tree** into a
single `index.json`, preserving the folder hierarchy (no flattening). Every file
is a node; the **original file is the artifact** (what a QA/PM opens, what a
package links to). doocus only *extracts text* for the heavy formats
(docx/pdf/pptx/xlsx) — and that text is a **search index only, never a
replacement** for the document.

Extraction is **deterministic and offline** — no cloud AI touches the raw source.
The raw source stays on disk; permitted services (Gemini web, NotebookLM) get the
original (as a link once we have Drive URLs, or re-uploaded).

## Quick start

```bash
# index a whole downloaded drive tree  → docs-output/index.json
uv run make docs IN="/mnt/win/drive"
uv run make docs IN="..." DOC_EXTRA="--ocr"      # OCR scanned pdfs
make docs-dry IN="..."                           # counts only, no writes

# or directly
uv run process_tree.py /mnt/win/drive --output docs-output

# one-off single file (older per-file model, still handy)
uv run process_doc.py notes.docx
```

## Output contract

```
docs-output/
  index.json                          whole tree; every file a node:
                                        { path, name, ext, family, mode, bytes,
                                          modified, url:null }   ◀ url = Drive link (later)
  <mirrored path>/<file>.doocus/      sidecar — ONLY for extracted formats:
     ├── content.md                     extracted text — SEARCH INDEX ONLY
     └── meta.json                      source/sha/dates/author/pages/... metadata
```

Node **modes**: `extracted` (docx/pdf/pptx/xlsx → text sidecar) · `meeting`
(mp4/mkv/... → belongs to meetus, not a doc) · `link` (everything else → original
is the artifact, nothing duplicated). Every node also carries `created` (birth
time, falling back to mtime) so docs and meetings sort by creation date coherently.

## Meetings

Videos are `meeting` nodes — doocus does **not** transcribe them; meetus does. To
keep everything in one coherent, searchable tree, run meetus over **only** the
meetings doocus found and write each into a `<file>.meetus/` sidecar right next to
its `<file>.doocus/`-style neighbours:

```bash
# 1. index the drive → docs-output/{index.json, meetings.txt}
uv run make docs IN="/mnt/drive"

# 2. meetus over ONLY the listed meetings, re-rooted at the mount, into
#    <file>.meetus/ sidecars (matches the pointer doocus already wrote):
make batch IN="/mnt/drive" OUT=docs-output \
     LIST=docs-output/meetings.txt \
     OUT_FORMAT="{name}.meetus"
```

- `meetings.txt` holds **relative** paths only — mount the drive anywhere and the
  `-l/--list` re-roots them under `-i`, so this machine reads only the meetings.
- `OUT_FORMAT="{name}.meetus"` is meetus's new `--out-format` (tokens `{date}
  {run} {stem} {name}`; default keeps `YYYYMMDD-NNN-<stem>`). Omitting `{run}`
  makes the folder deterministic, so it matches the `out`/`transcript` pointer the
  indexer set on each `meeting` node — no relink step.
- Frames land in `<file>.meetus/frames/` for now (unchanged).

## Supported types

| Family  | Extensions              | Notes |
|---------|-------------------------|-------|
| text    | md, txt, yaml/yml, json | passthrough; json/yaml add a structure summary |
| tabular | csv                     | markdown table + row/col metadata (stdlib) |
| office  | docx, pptx, xlsx        | text + core properties (author, dates, title) |
| pdf     | pdf                     | text per page + info dict; `--render` for page-1 thumb |
| web     | html/htm                | visible text + title/meta tags |
| image   | jpg/jpeg, png           | EXIF (incl. capture date) + thumbnail; `--ocr` for text |
| media   | mp4                     | ffprobe metadata + thumbnail; **delegates transcription to meetus** |

Each extractor is isolated: a missing optional dependency or a corrupt file is
recorded as a warning in `meta.json`, never aborting a batch (same resilience as
`ctrl/batch.sh`).

## Dependencies

Core text/tabular use only the stdlib. Everything else is a uv group in the
repo's `pyproject.toml`:

```bash
uv sync --group doocus                 # Pillow, PyYAML, python-docx, python-pptx,
                                       # openpyxl, pypdf, beautifulsoup4, lxml
uv sync --group doocus --group ocr     # + pytesseract (needs system `tesseract`)
```

`mp4` uses system `ffmpeg`/`ffprobe` (shared with meetus). PDF page rendering
(`--render`) additionally needs the `pdf-render` group (`pdf2image`) + system
`poppler`.

## Browser UI

`ui/doocus-app/` (sibling of `ui/meetus-app/`, shares `ui/framework/`) reads
`index.json` and shows the **full tree** (folders preserved). Select a file →
detail view: for extracted docs, a split of the markdown-rendered **extracted
text** (clearly labelled "search index, not the document") beside the **native
view of the original** (pdf inline; docx/xlsx → download); for linked files the
original renders directly (image / html / text / markdown). A package builder
zips selected **originals** (+ the `content.md` for extracted ones) for a target
(Gemini web / NotebookLM), and will emit Drive **links** once nodes carry a `url`.

```bash
cd ui/doocus-app && npm install
DOOCUS_OUTPUT=/abs/path/to/docs-output npm run dev
```