# doocus — local drive tree indexer

The document twin of meetus. Replicates a **whole local drive tree** into a
single `index.json`, preserving the folder hierarchy (no flattening). Every file
is a node; the **original file is the artifact** (what a QA/PM opens, what a
package links to). doocus only *extracts text* for the heavy formats
(docx/pdf/pptx/xlsx) — and that text is a **search index only, never a
replacement** for the document.

Extraction is **deterministic and offline** — no cloud AI touches the raw source.
The raw source stays on disk; permitted services (Gemini web, NotebookLM) get the
original (as a link once we have Drive URLs, or re-uploaded).

## Quick start

```bash
# index a whole downloaded drive tree  → docs-output/index.json
uv run make docs IN="/mnt/win/drive"
uv run make docs IN="..." DOC_EXTRA="--ocr"      # OCR scanned pdfs
make docs-dry IN="..."                           # counts only, no writes

# or directly
uv run process_tree.py /mnt/win/drive --output docs-output

# one-off single file (older per-file model, still handy)
uv run process_doc.py notes.docx
```

## Output contract

```
docs-output/
  index.json                          whole tree; every file a node:
                                        { path, name, ext, family, mode, bytes,
                                          modified, url:null }   ◀ url = Drive link (later)
  <mirrored path>/<file>.doocus/      sidecar — ONLY for extracted formats:
     ├── content.md                     extracted text — SEARCH INDEX ONLY
     └── meta.json                      source/sha/dates/author/pages/... metadata
```

Node **modes**: `extracted` (docx/pdf/pptx/xlsx → text sidecar) · `meeting`
(mp4/mkv/... → belongs to meetus, not a doc) · `link` (everything else → original
is the artifact, nothing duplicated). Every node also carries `created` (birth
time, falling back to mtime) so docs and meetings sort by creation date coherently.

## Meetings

Videos are `meeting` nodes — doocus does **not** transcribe them; meetus does. To
keep everything in one coherent, searchable tree, run meetus over **only** the
meetings doocus found and write each into a `<file>.meetus/` sidecar right next to
its `<file>.doocus/`-style neighbours:

```bash
# 1. index the drive → docs-output/{index.json, meetings.txt}
uv run make docs IN="/mnt/drive"

# 2. meetus over ONLY the listed meetings, re-rooted at the mount, into
#    <file>.meetus/ sidecars (matches the pointer doocus already wrote):
make batch IN="/mnt/drive" OUT=docs-output \
     LIST=docs-output/meetings.txt \
     OUT_FORMAT="{name}.meetus"
```

- `meetings.txt` holds **relative** paths only — mount the drive anywhere and the
  `-l/--list` re-roots them under `-i`, so this machine reads only the meetings.
- `OUT_FORMAT="{name}.meetus"` is meetus's new `--out-format` (tokens `{date}
  {run} {stem} {name}`; default keeps `YYYYMMDD-NNN-<stem>`). Omitting `{run}`
  makes the folder deterministic, so it matches the `out`/`transcript` pointer the
  indexer set on each `meeting` node — no relink step.
- Frames land in `<file>.meetus/frames/` for now (unchanged).

## Supported types

| Family  | Extensions              | Notes |
|---------|-------------------------|-------|
| text    | md, txt, yaml/yml, json | passthrough; json/yaml add a structure summary |
| tabular | csv                     | markdown table + row/col metadata (stdlib) |
| office  | docx, pptx, xlsx        | text + core properties (author, dates, title) |
| pdf     | pdf                     | text per page + info dict; `--render` for page-1 thumb |
| web     | html/htm                | visible text + title/meta tags |
| image   | jpg/jpeg, png           | EXIF (incl. capture date) + thumbnail; `--ocr` for text |
| media   | mp4                     | ffprobe metadata + thumbnail; **delegates transcription to meetus** |

Each extractor is isolated: a missing optional dependency or a corrupt file is
recorded as a warning in `meta.json`, never aborting a batch (same resilience as
`ctrl/batch.sh`).

## Dependencies

Core text/tabular use only the stdlib. Everything else is a uv group in the
repo's `pyproject.toml`:

```bash
uv sync --group doocus                 # Pillow, PyYAML, python-docx, python-pptx,
                                       # openpyxl, pypdf, beautifulsoup4, lxml
uv sync --group doocus --group ocr     # + pytesseract (needs system `tesseract`)
```

`mp4` uses system `ffmpeg`/`ffprobe` (shared with meetus). PDF page rendering
(`--render`) additionally needs the `pdf-render` group (`pdf2image`) + system
`poppler`.

## Collections (multiple sources)

doocus-app discovers collections from **managed roots** (gitignored),
`<root>/<source>/` per source:

- `doocus-data/<source>/` — document collections
- `meetus-data/<source>/` — meeting collections (`.meetus` + a `--only meetings` index)

Override the roots with `DOOCUS_DATA` (comma-separated; `DOOCUS_OUTPUT` still works
for a single dir). **Collections whose `index.json.root` match are the same source**
— the UI merges their docs + meetings into one interleaved tree, toggled together
from the `⧉` sources menu. When the same file exists in two collections, the copy
whose sidecar resolves wins (a `meetus-data` meeting with its `.meetus` overrides a
bare `doocus` copy).

## Browser UI

`ui/doocus-app/` (sibling of `ui/meetus-app/`, shares `ui/framework/`):

- **Tree** (folders preserved, source-grouped) with a **search bar** — server-side
  search over the cached text (transcripts, extracted `content.md`, raw files);
  matches **prune the tree** and show a file count + size.
- **Detail**: for extracted docs, markdown-rendered **extracted text** (labelled
  "search index, not the document") beside the **native original** (pdf inline;
  docx/xlsx rendered via mammoth/SheetJS); linked files render directly; **meetings
  embed the meetus review** (video + transcript + frames, from `@review`).
- **Scan** a folder from the UI (`⧉` menu → Scan) — runs `process_tree` via `uv`
  into `doocus-data/`; **⟳** re-discovers newly-scanned sources.
- **Package** builder zips selected **originals** (+ `content.md` for extracted) for
  a target (Gemini web / NotebookLM); Drive **links** once nodes carry a `url`.

```bash
cd ui/doocus-app && npm install
DOOCUS_DATA="/abs/doocus-data,/abs/meetus-data" npm run dev
```