soleprint/soleprint/station/tools/dataconvert

dataconvert

Spreadsheets and CSV into schema-agnostic SQL seed files, one per table or sheet, plus a SCHEMA.md describing every table.

uv run dataconvert.py --input data/ --out-dir seed/                 # full seeds
uv run dataconvert.py --input data/ "*.xlsx" --out-dir seed/        # dirs, files, globs, .zip
uv run dataconvert.py --input data/ --out-dir sample/ --max-rows 20 # to understand the data
uv run dataconvert.py --input data/ --out-dir seed/ --sql-format batch --max-file-size 100M  # a full dump
uv run dataconvert.py --input export.xlsx --header-row 2 --data-row 6
uv run dataconvert.py --input data/ --config their-exports.json

Reads .csv, .xlsx, .xls and .ods: single files, directories (recursively), ZIP archives and wildcard patterns. Sheets of a multi-sheet workbook are written as <workbook>_<sheet>.sql (see bare_sheet_prefixes below). Several sources feeding the same table accumulate in one file, and so does re-running into the same --out-dir, so point a fresh run at an empty directory.

Config: dataconvert.json in the source folder

Everything about one set of spreadsheets, and how you usually convert them, goes in a dataconvert.json kept in the source folder, next to the files it describes: the input itself when it is a directory, or the folder a file, wildcard match or ZIP sits in. Each input looks in its own folder, and the run prints every config it uses and what it set. --config FILE uses one file for all inputs instead. It is never read from beside the script, so the repo never holds one: the marker values name whoever produced the files. Start from dataconvert-example.json.

{
  "out_dir": "seed",
  "all_text": true,
  "keep_folders": true,
  "exclude": ["*.xlsx.ods"],
  "sql_format": "batch",
  "max_file_size": "100M",
  "schema": true,
  "header_row": 1,
  "layouts": [
    { "name": "catalogue",
      "match": { "row": 2, "column": 1, "in": ["CODE", "ITEM_CODE"] },
      "header_row": 2, "data_row": 6 }
  ],
  "bare_sheet_prefixes": ["ref_"]
}

Every key is optional.

• out_dir, max_rows, schema, sql_format, batch_rows, max_file_size, all_text, keep_folders, exclude are the run settings: the same as --out-dir, --max-rows, --no-schema, --sql-format, --batch-rows, --max-file-size, --all-text, --keep-folders and --exclude, and a flag on the command line wins over the file. --exclude patterns are added to the file's. There is no default output directory: without --out-dir or an out_dir in the config, the run stops before writing anything. out_dir is relative to the folder the config is in, so "sample" beside the data means <source>/sample. Inputs whose configs name different out_dirs each get their own SCHEMA.md.
• header_row, data_row at the top level say where the column names and data are for sheets that no layout matches. By default row 1 holds the names and the data starts on row 2.
• layouts are checked against every sheet and CSV, first match wins. A layout matches when the cell at match.row/match.column (counted from 1, trimmed) is one of match.in; then the names come from header_row and the data from data_row on, skipping anything between. The run prints which layout each sheet got.
• bare_sheet_prefixes: sheets whose name starts with one of these are written as <sheet>.sql instead of <workbook>_<sheet>.sql.

Keys starting with _ are comments; any other unknown key stops the run, so a typo such as max_row is not silently ignored.

For a one-off, --header-row 2 --data-row 6 applies one layout to every file in the run and skips the config's layouts.

Values as they are: all_text

By default pandas guesses a type per column, and for real exports that guess does damage: a site code 0012 becomes 12, NA becomes NULL, and a column that is numbers for the first 200,000 rows and text after that is typed differently in different rows. all_text reads every cell as the string in the file. Only an empty cell becomes NULL; NA, N/A and null stay text. Converting types is then the loader's job, where the real schema is decided.

To help decide it, SCHEMA.md describes what each text column looks like:

type column says	meaning
text, integer-like (max 6)	every value is a whole number
text, code with leading zeros (max 4)	whole numbers, some with leading zeros: keep as text
text, decimal-like / timestamp-like (YYYY-MM-DD hh:mm) / date-like (DDMONYYYY) / boolean-like	every value has that shape
text, mostly date-like (DDMONYYYY): 300 others	95% or more fit; the others are what needs a rule (partial dates, UNK)
text, 67% integer-like	at least half fit
text (max 40)	no common shape

Shapes are judged on up to 200,000 values spread through the column.

Folders and exclusions

Output is flat by default: a table's file is named after the file or sheet, and two source folders with the same table name (two studies' drm_lb.csv) write into the same file. keep_folders mirrors the source folders under out_dir instead, so each keeps its own .sql; SCHEMA.md stays one document at the top, with the folder in each table's heading.

exclude skips files by glob. A pattern with no / matches the file name at any depth ("*.xlsx.ods"); one with a / matches the path under the input folder ("Old/*"). Each skipped file is named in the run's output.

Full dumps: format and splitting

One INSERT per row repeats the table and column list on every line, so the SQL can be many times the size of the spreadsheets, which are compressed on disk to begin with. sql_format picks how rows are written for PostgreSQL:

format	shape	load with	re-runnable	size
insert (default)	one INSERT ... ON CONFLICT DO NOTHING per row	any client	yes	largest
batch	one INSERT per batch_rows rows (500), same ON CONFLICT	any client	yes	about a quarter
copy	COPY ... FROM stdin, tab-separated	psql only	no: a second load fails on duplicate keys	smallest

max_file_size ("100M", decimal k/M/G) splits a table's output into table.001.sql, table.002.sql, ... once it passes that size. Splits fall between rows (or batches), never inside one, and every part is its own transaction, so the parts load one at a time in name order:

for f in seed/*.sql; do psql -v ON_ERROR_STOP=1 -f "$f" || break; done

A table under the limit keeps its plain table.sql name. Rows are written to disk as they are rendered, so a table larger than memory converts; reading the source still needs it in memory. Samples (max_rows) are never split.

Sampling for a web LLM

Full seed files get large fast, and a model only needs to see the shape of the data. With --max-rows N:

• every table still gets its .sql file, holding only its first N rows, with a header line such as -- SAMPLE: first 20 of 184233 rows. The full file would be ~48.1M;
• SCHEMA.md lists every table with its total rows and the size the full seed file would be, then each table's columns: original header, inferred type, null count and one example value.

SCHEMA.md alone is often enough to hand to the model. Full sizes of sampled tables are estimated from the rows written, so they carry a ~. Types are inferred from what pandas read: a starting point, not DDL. --no-schema skips the file.

Layout

file	does
dataconvert.py	command line
config.py	dataconvert.json: layouts, sheet naming and run settings
readers.py	files, directories, ZIPs and globs into DataFrames
progress.py	progress lines: which file is being read, and how far a long table has got
sqlgen.py	rows as insert, batch or copy statements, and size estimates
output.py	file naming, streaming rows to disk, and splitting into parts
schema.py	SCHEMA.md

The modules import each other by name, so the folder works wherever it is copied: uv run dataconvert.py from inside it, or python3 path/to/dataconvert.py with pandas, openpyxl and odfpy installed.