From 4cfacd47fe8f697dede17bf9657a2e753107fe9b Mon Sep 17 00:00:00 2001 From: slaguru666 <111923774+slaguru666@users.noreply.github.com> Date: Sat, 12 Sep 2026 22:22:19 +0100 Subject: [PATCH] check-lethality compares exactly, because it already seeded per creature (R-97) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I asked for this on a false premise of my own: I reported that adding a creature shifted a shared random stream and perturbed every other creature's recorded number. That is not true and the code never did it. measure() builds its own mulberry32 from the seed on every call, and check-lethality calls it once per creature, so a creature's numbers do not depend on its neighbours or its position. Measured rather than argued: inserting a creature ahead of the barghest changes 0 of 47 existing entries. The file's own claim — "the only thing that can move the number is a change to the rules or to the creature" — was accurate all along, and my last commit message says otherwise. It is wrong. The real cause, found by replaying each commit against the baseline as committed at 4b71859: 4b71859 baseline recorded 0 of 46 differ 322389b bestiary 0 of 46 differ ddc4f99 hit locations reach combat 26 of 46 differ <-- here a90c4f3 .. e5dc9b5 26 of 46 differ ddc4f99 routed every ordinary blow through the hit location table. That is the largest change the combat system has had and it moved 26 of 46 creatures, which is correct and expected. What is not correct is that nobody noticed for four commits: each creature moved by one or two points, the guard allowed six, and it reported OK while describing a game nobody was playing. So the tolerance goes. It exists for sampling noise and there is no sampling noise here — same party, same seed, same counts, and two recordings of unchanged code are byte-identical. Anything that moves is a real change, which is the entire point of the file. `rounds` is now compared too; it was recorded and then never read, so a creature could take a round longer to kill forever without a word. Because exactness only means something if the measurement is exact, the guard now proves it instead of assuming it: one creature measured twice must come back identical, and it says so plainly if a future change reaches for Math.random. Negative-tested. A 2% change to locationMaxHp now trips 8 creatures at +1.5 and +0.5 points of wipe rate — every one of which the old tolerance would have passed. Breaking determinism is caught and named. Also fixes update-readme, which advertised 8 guards while the build ran 9: check-anatomy was added without touching the list, which is precisely what the comment above that list already warned had happened once. The list is no longer trusted — it is checked against the `check` script in package.json, and refuses to write a README advertising a different set than the build runs. Co-Authored-By: Claude Opus 5 --- README.md | 3 +- system.json | 2 +- tools/check-lethality.mjs | 63 +++++++++++++++++++++++++++++---------- tools/update-readme.mjs | 27 +++++++++++++++-- 4 files changed, 75 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index ff3160e..a8f1012 100644 --- a/README.md +++ b/README.md @@ -188,6 +188,7 @@ check-templates: OK — 19 templates compile check-behaviour: OK — 75 behavioural tests check-scenarios: OK — 10 scenario files, 78 mechanic tags, every skill named resolves against 62 catalogue entries check-creatures: OK — 102 actor specs across 8 sources, every characteristic, skill, kit key, talent, species and style resolves, no duplicate keys -check-lethality: OK — 47 creatures, none drifted more than 6 points of wipe rate against the frozen party +check-anatomy: OK — 5 body plans, 36 locations, d20 complete in both modes, every location drawn, every kind handled and destructible, 20 plan changes carried 432 wounds with their severity intact, 1 flyer(s) with a survivable landing +check-lethality: OK — 47 creatures, every one fighting exactly as recorded against the frozen party ``` diff --git a/system.json b/system.json index a5252ac..61fab93 100644 --- a/system.json +++ b/system.json @@ -2,7 +2,7 @@ "id": "ringbrp", "title": "The Custodians", "description": "A Basic Roleplaying game of agents who cross the crossing to contain what should not be there — and who come back a little less from here each time.", - "version": "1.7.4", + "version": "1.7.5", "compatibility": { "minimum": 13, "verified": "14.364" diff --git a/tools/check-lethality.mjs b/tools/check-lethality.mjs index 4c46900..4f66783 100644 --- a/tools/check-lethality.mjs +++ b/tools/check-lethality.mjs @@ -47,11 +47,22 @@ const PARTY_KEYS = ["pc_holloway", "pc_okonkwo", "pc_nkemdirim", "pc_ferriby"]; const RUNS = 200; const SEED = 11; -/* How far a number may move before it is drift rather than noise. 200 runs of a coin - flip has a standard error around 3.5 points, so anything under 6 is inside the noise - and failing on it would make the guard cry wolf until somebody deleted it. */ -const WIPE_TOLERANCE = 6.0; // percentage points -const DOWN_TOLERANCE = 0.35; // agents, of four +/* NO TOLERANCE, deliberately. The reasoning it replaces sounded right — "200 runs of a + coin flip has a standard error around 3.5 points, so anything under 6 is inside the + noise" — and it is exactly backwards for this measurement. Sampling error would apply + if the runs were random; they are not. The guard used to allow six points of wipe + rate and a third of an agent before it complained, which sounds prudent and was why it + sat silent through the largest combat change the system has had: ddc4f99 routed every + ordinary blow through the hit location table, 26 of 46 creatures moved, and not one + of them moved far enough in a single number to trip the threshold. The baseline went + four commits describing a game nobody was playing. + + A tolerance is for noise, and there is none here: same party, same seed, same counts, + and measure() builds its own generator per creature, so two recordings of unchanged + code are byte-identical. Anything that moves is a real change to the rules or to the + creature, which is exactly what this file exists to notice. `rounds` is compared too; + it was recorded and then ignored, so a creature could take a round longer to kill + forever without a word. */ const party = PARTY_KEYS.map(k => { const found = ROSTER.find(r => r.key === k); @@ -100,18 +111,38 @@ if (base.runs !== RUNS || base.seed !== SEED process.exit(1); } +/* Comparing exactly is only honest if the measurement IS exact. Prove it here rather + than trust it: one creature, measured twice, must come back identical. If a future + change reaches for Math.random or a Set iteration order, this says so instead of + letting the whole guard degrade into noise-chasing. */ +{ + const probe = NPCS[0]; + const a = measure(party, [probe], { runs: RUNS, seed: SEED }); + const b = measure(party, [probe], { runs: RUNS, seed: SEED }); + const shape = r => JSON.stringify([r.wipeRate, r.downMean, r.roundsMedian, r.hurtMean]); + if (shape(a) !== shape(b)) { + console.error("check-lethality: FAILED — the simulation is not deterministic, so an exact " + + `baseline cannot mean anything. ${probe.key} measured twice gave ${shape(a)} and ${shape(b)}.`); + process.exit(1); + } +} + const problems = []; const added = [], removed = []; for (const [key, now] of Object.entries(current)) { const was = base.creatures[key]; if (!was) { added.push(key); continue; } - const dWipe = now.wipe - was.wipe; - const dDown = now.down - was.down; - if (Math.abs(dWipe) > WIPE_TOLERANCE || Math.abs(dDown) > DOWN_TOLERANCE) { - problems.push(`${key}: wiped ${was.wipe}% -> ${now.wipe}% (${dWipe >= 0 ? "+" : ""}${dWipe.toFixed(1)}), ` - + `down ${was.down} -> ${now.down} (${dDown >= 0 ? "+" : ""}${dDown.toFixed(2)} of 4)`); - } + if (now.wipe === was.wipe && now.down === was.down && now.rounds === was.rounds) continue; + const dWipe = now.wipe - was.wipe, dDown = now.down - was.down; + const bits = []; + if (now.wipe !== was.wipe) + bits.push(`wiped ${was.wipe}% -> ${now.wipe}% (${dWipe >= 0 ? "+" : ""}${dWipe.toFixed(1)})`); + if (now.down !== was.down) + bits.push(`down ${was.down} -> ${now.down} (${dDown >= 0 ? "+" : ""}${dDown.toFixed(2)} of 4)`); + if (now.rounds !== was.rounds) + bits.push(`rounds ${was.rounds} -> ${now.rounds}`); + problems.push({ key, dWipe, text: `${key}: ${bits.join(", ")}` }); } for (const key of Object.keys(base.creatures)) if (!(key in current)) removed.push(key); @@ -125,12 +156,14 @@ if (added.length) console.log(` note: ${added.length} new creature(s) not in th if (removed.length) console.log(` note: ${removed.length} creature(s) gone from content — ${removed.join(", ")}`); if (problems.length) { - console.error("check-lethality: FAILED — a creature does a different amount of damage than recorded"); - for (const p of problems) console.error(" " + p); + console.error(`check-lethality: FAILED — ${problems.length} of ${Object.keys(current).length} ` + + `creature(s) fight differently than recorded`); + problems.sort((a, b) => Math.abs(b.dWipe) - Math.abs(a.dWipe)); + for (const p of problems) console.error(" " + p.text); console.error("\n If this was deliberate, re-record with --update and say what changed in the commit."); process.exit(1); } -console.log(`check-lethality: OK — ${Object.keys(current).length} creatures, none drifted more than ` - + `${WIPE_TOLERANCE} points of wipe rate against the frozen party` +console.log(`check-lethality: OK — ${Object.keys(current).length} creatures, every one fighting ` + + `exactly as recorded against the frozen party` + `${added.length ? `, ${added.length} new` : ""}`); diff --git a/tools/update-readme.mjs b/tools/update-readme.mjs index 2c745b0..2730be8 100644 --- a/tools/update-readme.mjs +++ b/tools/update-readme.mjs @@ -9,6 +9,7 @@ */ import { readFile, writeFile } from "node:fs/promises"; import { execFileSync } from "node:child_process"; +import { readFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; import { SKILL_CATALOGUE, WEAPONS, ARMOURS, GEAR, VEHICLES, NPCS, TALENTS } from "./content.mjs"; @@ -32,9 +33,29 @@ const runGuard = name => { /* All of them, in the order `npm run check` runs them. This list was four long while the suite was seven, so the README advertised a subset and silently stopped mentioning every guard added after it was written — including the three that catch the most. */ -const guardLines = ["check-rules.mjs", "check-kits.mjs", "check-lang.mjs", - "check-templates.mjs", "check-behaviour.mjs", "check-scenarios.mjs", - "check-creatures.mjs", "check-lethality.mjs"].map(runGuard).filter(Boolean); +const GUARDS = ["check-rules.mjs", "check-kits.mjs", "check-lang.mjs", + "check-templates.mjs", "check-behaviour.mjs", "check-scenarios.mjs", + "check-creatures.mjs", "check-anatomy.mjs", "check-lethality.mjs"]; + +/* The list above went stale the moment a guard was added without touching this file — + which is what the comment above it already warned about, and which happened again + with check-anatomy. So it is no longer trusted: the suite of record is the `check` + script in package.json, and this refuses to write a README that advertises a + different set than the one the build actually runs. */ +{ + const pkg = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8")); + const actual = [...(pkg.scripts?.check ?? "").matchAll(/(check-[\w-]+\.mjs)/g)].map(m => m[1]); + const missing = actual.filter(g => !GUARDS.includes(g)); + const phantom = GUARDS.filter(g => !actual.includes(g)); + if (missing.length || phantom.length) { + console.error("update-readme: FAILED — the README's guard list does not match `npm run check`"); + if (missing.length) console.error(" run by the build, not advertised: " + missing.join(", ")); + if (phantom.length) console.error(" advertised, not run by the build: " + phantom.join(", ")); + process.exit(1); + } +} + +const guardLines = GUARDS.map(runGuard).filter(Boolean); const box = `| | | |---|---|