soleprint/soleprint/station/tools/dataconvert/README.md

dataconvert

Spreadsheets and CSV into schema-agnostic SQL seed files, one per table or sheet, plus a SCHEMA.md describing every table.

uv run dataconvert.py --input data/ --out-dir seed/                 # full seeds
uv run dataconvert.py --input data/ "*.xlsx" --out-dir seed/        # dirs, files, globs, .zip
uv run dataconvert.py --input data/ --out-dir sample/ --max-rows 20 # to understand the data
uv run dataconvert.py --input export.xlsx --header-row 2 --data-row 6
uv run dataconvert.py --input data/ --config their-exports.json

Reads .csv, .xlsx, .xls and .ods: single files, directories (recursively), ZIP archives and wildcard patterns. Sheets of a multi-sheet workbook are written as <workbook>_<sheet>.sql (see bare_sheet_prefixes below). Several sources feeding the same table accumulate in one file, and so does re-running into the same --out-dir, so point a fresh run at an empty directory.

Layouts: config, not code

By default row 1 holds the column names and the data starts on row 2. Exports that put a title above the header, or description rows between it and the data, are described in dataconvert.json beside the script (or --config FILE). It is gitignored because the marker values name whoever produced the files; start from dataconvert-example.json.

{
  "layouts": [
    { "name": "catalogue",
      "match": { "row": 2, "column": 1, "in": ["CODE", "ITEM_CODE"] },
      "header_row": 2, "data_row": 6 }
  ],
  "bare_sheet_prefixes": ["ref_"]
}

• layouts are checked against every sheet and CSV, first match wins. A layout matches when the cell at match.row/match.column (counted from 1, trimmed) is one of match.in; then the names come from header_row and the data from data_row on. The run prints which layout each sheet got. A sheet nothing matches is read normally.
• bare_sheet_prefixes: sheets whose name starts with one of these are written as <sheet>.sql instead of <workbook>_<sheet>.sql.

For a one-off, --header-row 2 --data-row 6 applies one layout to every file in the run and skips detection.

Sampling for a web LLM

Full seed files get large fast, and a model only needs to see the shape of the data. With --max-rows N:

• every table still gets its .sql file, holding only its first N rows, with a header line such as -- SAMPLE: first 20 of 184233 rows. The full file would be ~48.1M;
• SCHEMA.md lists every table with its total rows and the size the full seed file would be, then each table's columns: original header, inferred type, null count and one example value.

SCHEMA.md alone is often enough to hand to the model. Full sizes of sampled tables are estimated from the rows written, so they carry a ~. Types are inferred from what pandas read: a starting point, not DDL. --no-schema skips the file.

Layout

file	does
dataconvert.py	command line
config.py	dataconvert.json: layouts and sheet naming
readers.py	files, directories, ZIPs and globs into DataFrames
sqlgen.py	DataFrames into INSERT statements, with the row cap
output.py	file naming and writing
schema.py	SCHEMA.md

The modules import each other by name, so the folder works wherever it is copied: uv run dataconvert.py from inside it, or python3 path/to/dataconvert.py with pandas, openpyxl and odfpy installed.