doocus update and meeting list export
This commit is contained in:
Mariano Gabriel
@@ -1,37 +1,47 @@
|
||||
# doocus — local document extraction
|
||||
# doocus — local drive tree indexer
|
||||
|
||||
The document twin of meetus. Turns locally-downloaded files (Drive exports,
|
||||
PDFs, images, ...) into a shareable **textified** product (`content.md`) plus
|
||||
structured `meta.json`, mirroring the meetus `output/<run>/` contract.
|
||||
The document twin of meetus. Replicates a **whole local drive tree** into a
|
||||
single `index.json`, preserving the folder hierarchy (no flattening). Every file
|
||||
is a node; the **original file is the artifact** (what a QA/PM opens, what a
|
||||
package links to). doocus only *extracts text* for the heavy formats
|
||||
(docx/pdf/pptx/xlsx) — and that text is a **search index only, never a
|
||||
replacement** for the document.
|
||||
|
||||
Extraction is **deterministic and offline** — no cloud AI touches the raw source.
|
||||
The textified output is what feeds permitted services (Gemini web, NotebookLM)
|
||||
and the local search index; the raw source stays on disk.
|
||||
The raw source stays on disk; permitted services (Gemini web, NotebookLM) get the
|
||||
original (as a link once we have Drive URLs, or re-uploaded).
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# one document
|
||||
python process_doc.py notes.docx # → docs-output/<YYYYMMDD-NNN-notes>/
|
||||
python process_doc.py report.pdf --render # also render page 1 → thumb.jpg
|
||||
python process_doc.py scan.png --ocr # OCR the image into content.md
|
||||
# index a whole downloaded drive tree → docs-output/index.json
|
||||
uv run make docs IN="/mnt/win/drive"
|
||||
uv run make docs IN="..." DOC_EXTRA="--ocr" # OCR scanned pdfs
|
||||
make docs-dry IN="..." # counts only, no writes
|
||||
|
||||
# a whole downloaded folder (mirrors the input tree)
|
||||
make docs IN="/mnt/win/drive" # → docs-output/...
|
||||
make docs IN="..." DOC_EXTRA="--render --ocr"
|
||||
make docs-dry IN="..." # list what would run
|
||||
# or directly
|
||||
uv run process_tree.py /mnt/win/drive --output docs-output
|
||||
|
||||
# one-off single file (older per-file model, still handy)
|
||||
uv run process_doc.py notes.docx
|
||||
```
|
||||
|
||||
## Output contract
|
||||
|
||||
```
|
||||
docs-output/<YYYYMMDD-NNN-stem>/
|
||||
content.md textified document — the shareable product
|
||||
meta.json source info, sha256, dates, author, title, pages/sheets/slides, word_count, warnings
|
||||
thumb.jpg optional single preview (image / rendered page / video frame)
|
||||
manifest.json run config + outputs inventory + source pointer (path, not a copy)
|
||||
docs-output/
|
||||
index.json whole tree; every file a node:
|
||||
{ path, name, ext, family, mode, bytes,
|
||||
modified, url:null } ◀ url = Drive link (later)
|
||||
<mirrored path>/<file>.doocus/ sidecar — ONLY for extracted formats:
|
||||
├── content.md extracted text — SEARCH INDEX ONLY
|
||||
└── meta.json source/sha/dates/author/pages/... metadata
|
||||
```
|
||||
|
||||
Node **modes**: `extracted` (docx/pdf/pptx/xlsx → text sidecar) · `meeting`
|
||||
(mp4/mkv/... → belongs to meetus, not a doc) · `link` (everything else → original
|
||||
is the artifact, nothing duplicated).
|
||||
|
||||
## Supported types
|
||||
|
||||
| Family | Extensions | Notes |
|
||||
@@ -65,12 +75,16 @@ uv sync --group doocus --group ocr # + pytesseract (needs system `tesseract`
|
||||
|
||||
## Browser UI
|
||||
|
||||
`ui/doocus-app/` (sibling of `ui/meetus-app/`, shares `ui/framework/`) browses the
|
||||
extracted docs with per-type viewers, a metadata panel, and a package builder
|
||||
that zips the **original file + content.md + meta.json** per doc for a chosen
|
||||
target (Gemini web / NotebookLM):
|
||||
`ui/doocus-app/` (sibling of `ui/meetus-app/`, shares `ui/framework/`) reads
|
||||
`index.json` and shows the **full tree** (folders preserved). Select a file →
|
||||
detail view: for extracted docs, a split of the markdown-rendered **extracted
|
||||
text** (clearly labelled "search index, not the document") beside the **native
|
||||
view of the original** (pdf inline; docx/xlsx → download); for linked files the
|
||||
original renders directly (image / html / text / markdown). A package builder
|
||||
zips selected **originals** (+ the `content.md` for extracted ones) for a target
|
||||
(Gemini web / NotebookLM), and will emit Drive **links** once nodes carry a `url`.
|
||||
|
||||
```bash
|
||||
cd ui/doocus-app && npm install
|
||||
DOOCUS_OUTPUT=/path/to/docs-output npm run dev
|
||||
DOOCUS_OUTPUT=/abs/path/to/docs-output npm run dev
|
||||
```
|
||||
|
||||
@@ -87,6 +87,14 @@ def build_index(source_root, output_dir, options=None, dry_run=False) -> dict:
|
||||
json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
logger.info("Wrote index: %s (%d files)", out / "index.json", len(files))
|
||||
# Export ONLY the meeting paths (relative), so a meetus run elsewhere reads
|
||||
# just what it needs: mount the drive at a new root and feed this list to
|
||||
# `ctrl/batch.sh -l`. Relative paths re-root cleanly under the new mount.
|
||||
meetings = [f["path"] for f in files if f["mode"] == "meeting"]
|
||||
(out / "meetings.txt").write_text(
|
||||
"".join(m + "\n" for m in meetings), encoding="utf-8"
|
||||
)
|
||||
logger.info("Wrote meetings list: %s (%d meetings)", out / "meetings.txt", len(meetings))
|
||||
return index
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user