freeze(apollo): confirmatory instrument + frozen analysis (corpus-2 replication)

Freeze — pre-specification anchor

Freezes the instrument and analysis for the Apollo 11 second-corpus test before any Apollo outcome is collected. The merge/commit timestamp is what lets us honestly report the corpus-2 analysis as pre-specified. See campaigns/gvsets/prereg-apollo-confirmatory.md (keaton).

Contents

• eval/tasks/apollo11_discovery.yaml — 8 → 24 tasks (AP1–AP24). H1 suite (guided vs unguided). GT verified vs model source.

• eval/tasks/apollo11_explanation.yaml — NEW, 16 tasks (AE1–AE16). S1 suite (cli_render vs cli_graph), each answerable in one render/stat call.

• eval/tasks/apollo11_verification_log.md — file:line / tool-command provenance for every answer key.

• scripts/apollo_confirmatory_analysis.py — frozen analysis: exact one-sided sign-flip permutation, Hedges g + BCa bootstrap CI, leave-one-task jackknife, α=0.025, fixed seed. Validated to reproduce corpus-1 O12 (g=0.709) and O4 (g=0.894) exactly. Run unmodified on Apollo results.

• docs/prereg-apollo-confirmatory.md — the pre-registration itself (v1.1), committed alongside the instrument so hypotheses + direction + α + N + success criteria share the freeze timestamp. v1.1 is a pre-outcome amendment (Appendix 4 in the doc): firewall = freeze-before-outcomes (this branch's commits); OSF deposit is optional attestation, not a gate; §E.5 aligned to actual harness failure handling; §G.1 records the run environment (NOMOGRAPH_BIN=$HOME/.cargo/bin/sysml, nomograph-sysml 0.2.0, fixture re-verified 28 files / 2,437 elements / 4,682 relationships).

• eval/tasks/apollo11_pilot.yaml — 3 throwaway plumbing-pilot tasks (APILOT1–3), disjoint from the frozen suites, run single-condition/unpaired before the confirmatory collection so pipeline faults surface while amendment is still legitimate. Results discarded.

• scripts/merge_result_shards.py — pre-outcome data handling: collection is sharded per condition (--tasks) so a crash costs one shard, not a condition; merge verified to round-trip corpus-1 data identically through the frozen analysis loader.

Run procedure (post-merge, gated on API creds + cost sign-off)

# H1 (confirmatory): discovery, guided vs unguided, N=5 both
uv run python -m eval.llm_cli --task-file eval/tasks/apollo11_discovery.yaml \
  --models claude-sonnet-4-20250514 --tool-set cli_render \
  --system-prompt-file eval/prompts/guided_discovery.txt \
  --runs 5 --max-turns 15 --temperature 0.0 \
  --fixture-dir ../sysml/tests/fixtures/apollo-11 --output results/apollo/guided.json
# (unguided = same without --system-prompt-file → default cli_render prompt)
# NOTE: prefix all runs with NOMOGRAPH_BIN=$HOME/.cargo/bin/sysml (see docs/prereg §G.1)

# S1 (secondary): explanation, render vs graph, N=10 each
#   --tool-set cli_render  → results/apollo/render.json
#   --tool-set cli_graph   → results/apollo/graph.json

python scripts/apollo_confirmatory_analysis.py --treatment results/apollo/guided.json \
  --baseline results/apollo/unguided.json --label H1 --alpha 0.025

Held for review; the discovery suite is the binding confirmatory instrument.

🤖 Generated with Claude Code