138 lines
6.3 KiB
Markdown
138 lines
6.3 KiB
Markdown
# doocus — local drive tree indexer
|
|
|
|
The document twin of meetus. Replicates a **whole local drive tree** into a
|
|
single `index.json`, preserving the folder hierarchy (no flattening). Every file
|
|
is a node; the **original file is the artifact** (what a QA/PM opens, what a
|
|
package links to). doocus only *extracts text* for the heavy formats
|
|
(docx/pdf/pptx/xlsx) — and that text is a **search index only, never a
|
|
replacement** for the document.
|
|
|
|
Extraction is **deterministic and offline** — no cloud AI touches the raw source.
|
|
The raw source stays on disk; permitted services (Gemini web, NotebookLM) get the
|
|
original (as a link once we have Drive URLs, or re-uploaded).
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
# index a whole downloaded drive tree → docs-output/index.json
|
|
uv run make docs IN="/mnt/win/drive"
|
|
uv run make docs IN="..." DOC_EXTRA="--ocr" # OCR scanned pdfs
|
|
make docs-dry IN="..." # counts only, no writes
|
|
|
|
# or directly
|
|
uv run process_tree.py /mnt/win/drive --output docs-output
|
|
|
|
# one-off single file (older per-file model, still handy)
|
|
uv run process_doc.py notes.docx
|
|
```
|
|
|
|
## Output contract
|
|
|
|
```
|
|
docs-output/
|
|
index.json whole tree; every file a node:
|
|
{ path, name, ext, family, mode, bytes,
|
|
modified, url:null } ◀ url = Drive link (later)
|
|
<mirrored path>/<file>.doocus/ sidecar — ONLY for extracted formats:
|
|
├── content.md extracted text — SEARCH INDEX ONLY
|
|
└── meta.json source/sha/dates/author/pages/... metadata
|
|
```
|
|
|
|
Node **modes**: `extracted` (docx/pdf/pptx/xlsx → text sidecar) · `meeting`
|
|
(mp4/mkv/... → belongs to meetus, not a doc) · `link` (everything else → original
|
|
is the artifact, nothing duplicated). Every node also carries `created` (birth
|
|
time, falling back to mtime) so docs and meetings sort by creation date coherently.
|
|
|
|
## Meetings
|
|
|
|
Videos are `meeting` nodes — doocus does **not** transcribe them; meetus does. To
|
|
keep everything in one coherent, searchable tree, run meetus over **only** the
|
|
meetings doocus found and write each into a `<file>.meetus/` sidecar right next to
|
|
its `<file>.doocus/`-style neighbours:
|
|
|
|
```bash
|
|
# 1. index the drive → docs-output/{index.json, meetings.txt}
|
|
uv run make docs IN="/mnt/drive"
|
|
|
|
# 2. meetus over ONLY the listed meetings, re-rooted at the mount, into
|
|
# <file>.meetus/ sidecars (matches the pointer doocus already wrote):
|
|
make batch IN="/mnt/drive" OUT=docs-output \
|
|
LIST=docs-output/meetings.txt \
|
|
OUT_FORMAT="{name}.meetus"
|
|
```
|
|
|
|
- `meetings.txt` holds **relative** paths only — mount the drive anywhere and the
|
|
`-l/--list` re-roots them under `-i`, so this machine reads only the meetings.
|
|
- `OUT_FORMAT="{name}.meetus"` is meetus's new `--out-format` (tokens `{date}
|
|
{run} {stem} {name}`; default keeps `YYYYMMDD-NNN-<stem>`). Omitting `{run}`
|
|
makes the folder deterministic, so it matches the `out`/`transcript` pointer the
|
|
indexer set on each `meeting` node — no relink step.
|
|
- Frames land in `<file>.meetus/frames/` for now (unchanged).
|
|
|
|
## Supported types
|
|
|
|
| Family | Extensions | Notes |
|
|
|---------|-------------------------|-------|
|
|
| text | md, txt, yaml/yml, json | passthrough; json/yaml add a structure summary |
|
|
| tabular | csv | markdown table + row/col metadata (stdlib) |
|
|
| office | docx, pptx, xlsx | text + core properties (author, dates, title) |
|
|
| pdf | pdf | text per page + info dict; `--render` for page-1 thumb |
|
|
| web | html/htm | visible text + title/meta tags |
|
|
| image | jpg/jpeg, png | EXIF (incl. capture date) + thumbnail; `--ocr` for text |
|
|
| media | mp4 | ffprobe metadata + thumbnail; **delegates transcription to meetus** |
|
|
|
|
Each extractor is isolated: a missing optional dependency or a corrupt file is
|
|
recorded as a warning in `meta.json`, never aborting a batch (same resilience as
|
|
`ctrl/batch.sh`).
|
|
|
|
## Dependencies
|
|
|
|
Core text/tabular use only the stdlib. Everything else is a uv group in the
|
|
repo's `pyproject.toml`:
|
|
|
|
```bash
|
|
uv sync --group doocus # Pillow, PyYAML, python-docx, python-pptx,
|
|
# openpyxl, pypdf, beautifulsoup4, lxml
|
|
uv sync --group doocus --group ocr # + pytesseract (needs system `tesseract`)
|
|
```
|
|
|
|
`mp4` uses system `ffmpeg`/`ffprobe` (shared with meetus). PDF page rendering
|
|
(`--render`) additionally needs the `pdf-render` group (`pdf2image`) + system
|
|
`poppler`.
|
|
|
|
## Collections (multiple sources)
|
|
|
|
doocus-app discovers collections from **managed roots** (gitignored),
|
|
`<root>/<source>/` per source:
|
|
|
|
- `doocus-data/<source>/` — document collections
|
|
- `meetus-data/<source>/` — meeting collections (`.meetus` + a `--only meetings` index)
|
|
|
|
Override the roots with `DOOCUS_DATA` (comma-separated; `DOOCUS_OUTPUT` still works
|
|
for a single dir). **Collections whose `index.json.root` match are the same source**
|
|
— the UI merges their docs + meetings into one interleaved tree, toggled together
|
|
from the `⧉` sources menu. When the same file exists in two collections, the copy
|
|
whose sidecar resolves wins (a `meetus-data` meeting with its `.meetus` overrides a
|
|
bare `doocus` copy).
|
|
|
|
## Browser UI
|
|
|
|
`ui/doocus-app/` (sibling of `ui/meetus-app/`, shares `ui/framework/`):
|
|
|
|
- **Tree** (folders preserved, source-grouped) with a **search bar** — server-side
|
|
search over the cached text (transcripts, extracted `content.md`, raw files);
|
|
matches **prune the tree** and show a file count + size.
|
|
- **Detail**: for extracted docs, markdown-rendered **extracted text** (labelled
|
|
"search index, not the document") beside the **native original** (pdf inline;
|
|
docx/xlsx rendered via mammoth/SheetJS); linked files render directly; **meetings
|
|
embed the meetus review** (video + transcript + frames, from `@review`).
|
|
- **Scan** a folder from the UI (`⧉` menu → Scan) — runs `process_tree` via `uv`
|
|
into `doocus-data/`; **⟳** re-discovers newly-scanned sources.
|
|
- **Package** builder zips selected **originals** (+ `content.md` for extracted) for
|
|
a target (Gemini web / NotebookLM); Drive **links** once nodes carry a `url`.
|
|
|
|
```bash
|
|
cd ui/doocus-app && npm install
|
|
DOOCUS_DATA="/abs/doocus-data,/abs/meetus-data" npm run dev
|
|
```
|