mariano/soleprint
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated contract

Browse Source
This commit is contained in:
buenosairesam
2026-09-16 13:44:02 -03:00
parent 5219bd5edb
commit 99b1988504
3 changed files with 273 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
soleprint/station/tools/distill/distill.sh
View File

@@ -69,6 +69,8 @@
# foo the working tree, as it is now (uncommitted included)
# foo@main one ref, read straight out of the object store
# foo@main,topic several refs, each distilled separately
# foo@3f2a9c1 a commit: a hash works anywhere a branch does,
# and so does a tag or HEAD~3
# foo@all every local branch
# foo:src/api only that subtree
# foo:src/api,docs several subtrees, as a single output

37
soleprint/station/tools/distill/explode.md
View File

@@ -15,13 +15,46 @@ is much use without the other.
./explode.sh -o ./restored reply.md # write the tree
./explode.sh -o ./restored --force x.md # overwrite what is already there
./explode.sh --raw -o ./restored x.md # leave ⟪BT3⟫ escapes as they are
./explode.sh --contract > contract.txt # the format to hand to the model
./explode.sh --contract > contract.txt # the instructions to hand to the model
./explode.sh --check reply.md # test a reply to "check"; writes nothing
./explode.sh --selftest # check this copy against known input
```
## Test the format before the real work
A long reply that cannot be unpacked costs the whole wait. So test first, with a reply
that takes seconds:
1. Start the conversation with `contract.txt` and the digest attached.
2. Send the single word `check`. The contract tells the model to answer with a six-line
file, `_check/check.md`, built to hit what breaks: a code fence that has to be written
as `⟪BT3⟫`, a tab, straight quotes, the markers and the `~~~~~~~~` wrapper.
3. Copy the reply with the **copy response** button, save it, and run
`./explode.sh --check reply.md`.
It unpacks the reply through the real parser and compares the result byte for byte. On a
pass, ask for the real work. On a failure it names each problem (escapes skipped, tab
turned into spaces, curly quotes, missing wrapper, notes around the block) and prints a
correction to paste back. A tab that arrives as spaces while the model wrote a tab means
the copy is the problem, not the model.
The contract also sets how the model works, which is most of the waiting: no web search
(everything is in the attachments), no preamble or summary, and batches.
A reply has an output length limit, and one that runs out mid-file loses that file. So
for more than three files, or more than about 400 lines, the model first sends a plan
with the files grouped into numbered batches (at most 5 files and about 400 lines each;
a bigger file is a batch alone), waits for `go`, and then sends one batch per reply,
headed `@@ BATCH n OF m` and ending `@@ MORE` until the last. `explode.sh` reads that
line and says whether to answer `continue`. Explode each reply as it arrives; with the
project prefix on every path they all land in the same tree. A reply that was cut off
anyway is refused as unterminated, with a note to ask for smaller batches. When a real reply has literal backticks where
`⟪BT3⟫` belonged, it still unpacks, and `explode.sh` lists those files so you can tell the
model before the next one.
## Ask for `@@`, and attach it
`--contract` prints the output format to give whatever writes the reply. **Attach
`--contract` prints the instructions to give whatever writes the reply: how to work, the output format, a worked example and the format check. **Attach
that file; do not paste it into the message.** A chat box renders markdown before
the model sees it, and `===` alone under a line of text is setext syntax for a
heading — so a pasted spec gets rendered as a title and the model is told nothing.

251
soleprint/station/tools/distill/explode.sh
View File

@@ -38,7 +38,8 @@
# --force overwrite files that already exist
# --raw leave ⟪BT3⟫-style escapes as they are (see below)
# --format F fenced | marker | digest | auto (default: auto)
# --contract print the output format to hand to whatever generates the file
# --contract print the instructions to hand to whatever generates the file
# --check F test a reply to the word "check" (see below), write nothing
# --selftest check this copy of the script against known input and exit
#
# Examples:
@@ -97,6 +98,14 @@
# reply from the "copy response" button, not by selecting the rendered text —
# the button gives the markdown as written.
#
# The format check. A long reply that turns out unusable costs the whole wait.
# The contract defines a six-line check file that exercises what breaks: a code
# fence that has to be escaped, a tab, straight quotes. Send the model the single
# word "check", save its reply, and run explode.sh --check on it: it unpacks the
# reply through the real parser, compares the result byte for byte, and says what
# went wrong — escapes skipped, tabs turned into spaces by the copy, curly quotes,
# missing wrapper — with a correction to paste back. Seconds, not a lost answer.
#
# Paths come out of a text file, so they are treated as untrusted: anything
# absolute, or reaching upward with .., is refused and nothing is written. A
# file that describes /etc/cron.d/x is not a file you want to expand blindly.
@@ -114,6 +123,7 @@ FORMAT="auto"
SRC=""
SELFTEST=""
CONTRACT=""
CHECK=""
while [ $# -gt 0 ]; do
case "$1" in
@@ -123,6 +133,7 @@ while [ $# -gt 0 ]; do
--raw) RAW=1 ;;
--format) shift; FORMAT="${1:-}" ;;
--contract) CONTRACT=1 ;;
--check) CHECK=1 ;;
--selftest) SELFTEST=1 ;;
-h|--help) usage; exit 0 ;;
-*) die "unknown option: $1" ;;
@@ -137,10 +148,38 @@ done
# message body, where markdown gets a say first.
contract() {
cat <<'CONTRACT'
INSTRUCTIONS FOR THIS CONVERSATION
Read this whole file before replying, and follow it in every reply of this
conversation. Your replies are unpacked into files by a script. A reply that
does not follow the format below cannot be unpacked and is thrown away, however
good the code in it is.
HOW TO WORK
1. Everything you need is in the attached files. Do not search the web and do
not cite sources. If something you need is missing, ask for it and stop.
2. Do not restate the task, do not announce what you are about to do, and do
not summarise what you did. Files first. Notes after, three sentences at most.
3. If the request is ambiguous, ask one short question and stop. Do not write
code for a guess.
4. Replies have an output length limit, and a reply that runs out in the middle
of a file loses that file. So when the change touches more than 3 files, or
the files together are longer than about 400 lines, first reply with only a
plan: the files grouped into numbered batches, one line per file saying what
changes in it. A batch is at most 5 files and about 400 lines; a file longer
than that is a batch on its own. Then wait for "go".
5. Send one batch per reply, in order. Right after the opening ~~~~~~~~ line,
write a line that is exactly @@ BATCH n OF m. If it is not the last batch,
end the reply with a line that is exactly @@ MORE and wait for "continue".
Never start a file you cannot finish in the same reply.
OUTPUT FORMAT
Return every file you changed or created in full, one after another, using
exactly this shape and nothing else:
Return every file you changed or created in full, one after another, in exactly
this shape:
~~~~~~~~
@@ FILE: <project>/relative/path/to/file.py
@@ -162,26 +201,137 @@ Rules:
literal ⟪ as ⟪LQ⟫. Files you were given already use these escapes; copy
them through verbatim. They are turned back into the real characters when
the reply is unpacked; a real run of backticks breaks the reply.
- Keep tabs as tabs and quotes as straight quotes (" and '). Never replace them
with spaces or typographic quotes.
- Start every path with the project it belongs to, spelled exactly as the
heading of the document it came from, then the path relative to that
project's root. One reply covers every project we touched; the prefix is
the only thing that says which file goes where, so it is never optional
and never abbreviated.
project's root. The prefix is the only thing that says which file goes
where, so it is never optional and never abbreviated.
- No leading ./ or /.
- Between @@ FILE: and @@ END, emit the file verbatim apart from those
escapes. Do not wrap it in markdown fences, do not add line numbers, do
not elide anything as "unchanged" or "...". A partial file is worse than
no file.
- Anything you want to say to me goes outside the fenced block, before or
after it. Text between @@ blocks is ignored.
escapes. Do not add line numbers, and do not elide anything as "unchanged"
or "...". A partial file is worse than no file.
- Return whole files only. No diffs, no patches, no hunks.
- If a file's own content happens to contain a line starting with @@, say so
in your prose so I know to check that block by hand.
- If a file's own content has a line starting with @@, say so in your notes.
EXAMPLE
A complete reply returning one file whose content has a code block:
~~~~~~~~
@@ FILE: myproject/docs/usage.md
# Usage
⟪BT3⟫bash
make run
⟪BT3⟫
@@ END FILE: myproject/docs/usage.md
~~~~~~~~
Added the usage page.
FORMAT CHECK
When my whole message is the single word check, reply with nothing but one
file in the output format above, and no notes. Its path is _check/check.md
(no project prefix for this one), and its content is exactly these six lines:
line 1: # check
line 2: three backticks immediately followed by the word bash
line 3: echo "ok"
line 4: three backticks
line 5: one tab character, then the word indented
line 6: done
CONTRACT
}
if [ -n "$CONTRACT" ]; then contract; exit 0; fi
# ── the format check ───────────────────────────────────────────────────────
# What the contract's FORMAT CHECK asks for, as bytes once unpacked. Every line
# is there to catch one way replies break: line 2 and 4 need the escape, line 3
# straight quotes, line 5 a real tab.
CHECK_WANT='# check\n```bash\necho "ok"\n```\n\tindented\ndone\n'
check_reply() {
local src="$1" t problems="" warnings="" advice="" n
t="$(mktemp -d)"; trap 'rm -rf "$t"' RETURN
printf "$CHECK_WANT" > "$t/want"
tr -d '\r' < "$src" > "$t/reply"
fail() { problems="$problems FAIL $1"$'\n'; advice="$advice- $2"$'\n'; }
warn() { warnings="$warnings warn $1"$'\n'; advice="$advice- $2"$'\n'; }
if ! grep -qE '^@@ +FILE: +_check/check\.md[ \t]*$' "$t/reply"; then
fail "no '@@ FILE: _check/check.md' line" \
"Reply to check with the file _check/check.md in the output format, and nothing else."
else
awk '/^@@ +FILE: +_check\/check\.md[ \t]*$/ { on = 1; next }
on && /^@@ +END/ { exit }
on { print }' "$t/reply" > "$t/raw"
if ! grep -qE '^@@ +END FILE: +_check/check\.md[ \t]*$' "$t/reply"; then
fail "the block is not closed with '@@ END FILE: _check/check.md'" \
"Close every file with an @@ END FILE: line repeating its path."
fi
if grep -q '```' "$t/raw"; then
fail "real backticks inside the file" \
"Inside file contents, write three backticks as ⟪BT3⟫, never as the characters."
elif ! grep -q '⟪BT3⟫' "$t/raw"; then
fail "no ⟪BT3⟫ where the code fence goes" \
"Line 2 of the check is ⟪BT3⟫bash and line 4 is ⟪BT3⟫."
fi
if grep -q '[“”‘’]' "$t/raw"; then
fail "typographic quotes instead of straight ones" \
"Use straight quotes (\" and ') in file contents."
fi
if grep -qE '^ +indented' "$t/raw"; then
fail "the tab on line 5 arrived as spaces" \
"Keep tab characters as tabs. (If the model did write a tab, the copy converted it: use the copy-response button.)"
fi
if "$0" -o "$t/out" "$t/reply" > "$t/log" 2>&1 && [ -f "$t/out/_check/check.md" ]; then
if ! cmp -s "$t/want" "$t/out/_check/check.md"; then
fail "unpacked, but not the six expected lines:" \
"The check file is exactly six lines: # check / ⟪BT3⟫bash / echo \"ok\" / ⟪BT3⟫ / a tab then indented / done."
problems="$problems$(diff <(sed -n l "$t/want") <(sed -n l "$t/out/_check/check.md") | sed 's/^/ /' || true)"$'\n'
fi
else
fail "explode could not unpack it: $(grep -v '^format:' "$t/log" | head -1)" \
"Follow the output format exactly: markers on their own lines, nothing between the blocks."
fi
fi
n=$(grep -c '^~~~~~~~[ \t]*$' "$t/reply" || true)
if [ "$n" -lt 2 ]; then
warn "no ~~~~~~~~ lines around the block (unpacks, but the chat will render it badly)" \
"Put all the blocks between two lines of eight tildes, and use no other fence."
fi
n=$(awk '/^~~~~~~~[ \t]*$/ { inside = !inside; next }
!inside && /^@@ +FILE:/ { block = 1 }
!inside && !block && NF { c++ }
!inside && /^@@ +END/ { block = 0 }
END { print c + 0 }' "$t/reply")
if [ "$n" -gt 0 ]; then
warn "$n line(s) of notes outside the block" \
"For check, reply with the file only: no notes before or after."
fi
if [ -z "$problems" ]; then
echo " ok escapes, tab, quotes and markers all came through"
[ -n "$warnings" ] && printf '%s' "$warnings"
echo
echo "format check passed"
return 0
fi
printf '%s' "$problems" "$warnings"
echo
echo "format check FAILED. Send this back to the model:"
echo
echo "The format check failed. Re-read the instructions file, fix these, and reply to check again:"
printf '%s' "$advice"
return 1
}
# ── self-test ──────────────────────────────────────────────────────────────
# So a copy of this script on another machine can be checked without any real
# input, and without asking whether it is the version that knows a given format.
@@ -305,6 +455,35 @@ FIXTURE
"$0" -o "$t/p" "$t/p.txt" >/dev/null 2>&1 || true
check "digest: not escaped" '⟪BT3⟫' "$(cat "$t/p/x.md" 2>/dev/null)"
# The format check: a good reply passes, and each way replies break is named.
good='~~~~~~~~\n@@ FILE: _check/check.md\n# check\n⟪BT3⟫bash\necho "ok"\n⟪BT3⟫\n\tindented\ndone\n@@ END FILE: _check/check.md\n~~~~~~~~\n'
printf "$good" > "$t/q1.md"
check "check: good reply" "0" "$("$0" --check "$t/q1.md" >/dev/null 2>&1; echo $?)"
printf "$good" | sed 's/⟪BT3⟫/```/' > "$t/q2.md"
check "check: real backticks" "1" "$("$0" --check "$t/q2.md" 2>&1 | grep -c 'real backticks')"
printf "$good" | sed 's/^\tindented/ indented/' > "$t/q3.md"
check "check: tab became spaces" "1" "$("$0" --check "$t/q3.md" 2>&1 | grep -c 'arrived as spaces')"
printf "$good" | sed 's/"ok"/“ok”/' > "$t/q4.md"
check "check: curly quotes" "1" "$("$0" --check "$t/q4.md" 2>&1 | grep -c 'typographic quotes')"
{ echo "Sure! Here is the check."; printf "$good"; } > "$t/q5.md"
check "check: notes only warn" "0" "$("$0" --check "$t/q5.md" >/dev/null 2>&1; echo $?)"
printf "$good" | sed '/^done$/d' > "$t/q6.md"
check "check: wrong content" "1" "$("$0" --check "$t/q6.md" 2>&1 | grep -c 'not the six expected')"
# Real replies: raw fences unpack but are reported, and @@ MORE is noticed.
printf '@@ FILE: p/r.md\n# r\n```sh\nls\n```\ndone\n@@ END FILE: p/r.md\n@@ MORE\n' > "$t/r.txt"
"$0" -o "$t/r" "$t/r.txt" > "$t/r.log" 2>&1 || true
check "notes: raw fence reported" "1" "$(grep -c 'real ``` instead' "$t/r.log")"
check "notes: @@ MORE noticed" "1" "$(grep -c '@@ MORE' "$t/r.log")"
check "notes: still unpacked" "2" "$(grep -c '```' "$t/r/p/r.md" 2>/dev/null || echo 0)"
printf '~~~~~~~~\n@@ BATCH 2 OF 3\n@@ FILE: p/b.py\nx = 1\n@@ END FILE: p/b.py\n~~~~~~~~\n@@ MORE\n' > "$t/s.txt"
"$0" -o "$t/s" "$t/s.txt" > "$t/s.log" 2>&1 || true
check "notes: batch n of m" "1" "$(grep -c 'batch 2 of 3: say "continue" for batch 3' "$t/s.log")"
check "batch line is not a file" "x = 1" "$(cat "$t/s/p/b.py" 2>/dev/null)"
printf '~~~~~~~~\n@@ BATCH 1 OF 2\n@@ FILE: p/c.py\nx = 1\n' > "$t/u.txt"
"$0" -o "$t/u" "$t/u.txt" > "$t/u.log" 2>&1 || true
check "cut-off reply explained" "1" "$(grep -c 'length limit' "$t/u.log")"
echo
if [ "$rc" -eq 0 ]; then echo "all checks passed — this copy is current"
else echo "SOME CHECKS FAILED — this copy is out of date or broken" >&2
@@ -314,6 +493,12 @@ FIXTURE
if [ -n "$SELFTEST" ]; then selftest; exit $?; fi
if [ -n "$CHECK" ]; then
[ -n "$SRC" ] || die "--check needs the saved reply: $SELF --check reply.md"
[ -f "$SRC" ] || die "no such file: $SRC"
check_reply "$SRC"; exit $?
fi
[ -n "$SRC" ] || { usage >&2; exit 1; }
[ -f "$SRC" ] || die "no such file: $SRC"
case "$FORMAT" in fenced|marker|digest|auto) ;; *) die "--format must be fenced, marker, digest or auto" ;; esac
@@ -387,7 +572,14 @@ parse() {
first = 2; last--
} else last = n
}
if (mode == "list") printf "%s\t%d\n", path, last - first + 1
if (mode == "list") {
printf "%s\t%d\n", path, last - first + 1
# Unpacks fine, but it is what breaks the chat rendering, and
# what the model has to be told to stop doing.
if (fmt == "fenced")
for (i = first; i <= last; i++)
if (buf[i] ~ /```/) { print "RAWFENCE\t" path; break }
}
else {
out = dest "/" path
d = out; sub(/\/[^\/]*$/, "", d)
@@ -540,6 +732,7 @@ if [ -n "$mismatch" ]; then
echo "$SELF: refusing — these blocks were closed with another file's name:" >&2
printf '%s\n' "$mismatch" | awk -F'\t' '{ printf " opened %s, closed %s\n", $2, $3 }' >&2
echo "a close went missing, so one block holds more than one file" >&2
echo "tell the model: close every file with an @@ END FILE: line repeating its path, then resend those files" >&2
exit 1
fi
@@ -548,6 +741,8 @@ if [ -n "$unterminated" ]; then
echo "$SELF: refusing — this block was never closed with '@@ END' or '=== END':" >&2
printf '%s\n' "$unterminated" | sed 's/^UNTERMINATED\t/ /' >&2
echo "the file it describes would be silently truncated" >&2
echo "the reply most likely hit its length limit mid-file" >&2
echo "tell the model: that file was cut off; resend it whole, closed with @@ END FILE: and its path, and keep batches smaller" >&2
exit 1
fi
@@ -563,13 +758,38 @@ if [ -n "$clipped" ]; then
exit 1
fi
listing="$(printf '%s\n' "$scan" | grep -vE '^(UNSAFE|UNTERMINATED|WRONGFMT|CLIPPED|MISMATCH)' || true)"
listing="$(printf '%s\n' "$scan" | grep -vE '^(UNSAFE|UNTERMINATED|WRONGFMT|CLIPPED|MISMATCH|RAWFENCE)' || true)"
# Not refusals: the reply unpacks. Said anyway, because each is something to
# tell the model before the next reply rather than discover after it.
notes() {
local raw
raw="$(printf '%s\n' "$scan" | grep '^RAWFENCE' | cut -f2 || true)"
if [ -n "$raw" ]; then
echo "note: these came back with real \`\`\` instead of ⟪BT3⟫ (unpacked fine; tell the model):" >&2
printf '%s\n' "$raw" | sed 's/^/ /' >&2
fi
# The last batch line wins: a reply quoting an earlier one is still this batch.
local batch n m
batch="$(grep -oE '^@@ +BATCH +[0-9]+ +OF +[0-9]+' "$SRC" | tail -1 || true)"
if [ -n "$batch" ]; then
n="$(printf '%s' "$batch" | awk '{ print $3 }')"; m="$(printf '%s' "$batch" | awk '{ print $5 }')"
if [ "$n" -lt "$m" ]; then
echo "note: batch $n of $m: say \"continue\" for batch $((n + 1)), and explode that reply too" >&2
else
echo "note: batch $n of $m, the last one" >&2
fi
elif grep -qE '^@@ +MORE[ \t\r]*$' "$SRC"; then
echo "note: the reply ends with @@ MORE: say \"continue\" and explode the next reply too" >&2
fi
}
[ -n "$listing" ] || die "no files found in $SRC (format: $FORMAT)"
count=$(printf '%s\n' "$listing" | grep -c . )
if [ -n "$LIST" ]; then
printf '%s\n' "$listing" | awk -F'\t' '{ printf " %-60s %5d lines\n", $1, $2 }'
echo "$count files"
notes
exit 0
fi
@@ -591,3 +811,4 @@ mkdir -p "$DEST"
parse write "$UNESC" >/dev/null
printf '%s\n' "$listing" | awk -F'\t' '{ printf " %s\n", $1 }'
echo "wrote $count files to $DEST"
notes