check-lethality compares exactly, because it already seeded per creature (R-97)

I asked for this on a false premise of my own: I reported that adding a
creature shifted a shared random stream and perturbed every other
creature's recorded number. That is not true and the code never did it.
measure() builds its own mulberry32 from the seed on every call, and
check-lethality calls it once per creature, so a creature's numbers do
not depend on its neighbours or its position. Measured rather than
argued: inserting a creature ahead of the barghest changes 0 of 47
existing entries. The file's own claim — "the only thing that can move
the number is a change to the rules or to the creature" — was accurate
all along, and my last commit message says otherwise. It is wrong.

The real cause, found by replaying each commit against the baseline as
committed at 4b71859:

  4b71859  baseline recorded            0 of 46 differ
  322389b  bestiary                     0 of 46 differ
  ddc4f99  hit locations reach combat  26 of 46 differ   <-- here
  a90c4f3 .. e5dc9b5                   26 of 46 differ

ddc4f99 routed every ordinary blow through the hit location table. That
is the largest change the combat system has had and it moved 26 of 46
creatures, which is correct and expected. What is not correct is that
nobody noticed for four commits: each creature moved by one or two
points, the guard allowed six, and it reported OK while describing a
game nobody was playing.

So the tolerance goes. It exists for sampling noise and there is no
sampling noise here — same party, same seed, same counts, and two
recordings of unchanged code are byte-identical. Anything that moves is
a real change, which is the entire point of the file. `rounds` is now
compared too; it was recorded and then never read, so a creature could
take a round longer to kill forever without a word.

Because exactness only means something if the measurement is exact, the
guard now proves it instead of assuming it: one creature measured twice
must come back identical, and it says so plainly if a future change
reaches for Math.random.

Negative-tested. A 2% change to locationMaxHp now trips 8 creatures at
+1.5 and +0.5 points of wipe rate — every one of which the old tolerance
would have passed. Breaking determinism is caught and named.

Also fixes update-readme, which advertised 8 guards while the build ran
9: check-anatomy was added without touching the list, which is precisely
what the comment above that list already warned had happened once. The
list is no longer trusted — it is checked against the `check` script in
package.json, and refuses to write a README advertising a different set
than the build runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
slaguru666
2026-09-12 22:22:19 +01:00
co-authored by Claude Opus 5
parent e5dc9b5299
commit 4cfacd47fe
4 changed files with 75 additions and 20 deletions
+2 -1
View File
@@ -188,6 +188,7 @@ check-templates: OK — 19 templates compile
check-behaviour: OK — 75 behavioural tests
check-scenarios: OK — 10 scenario files, 78 mechanic tags, every skill named resolves against 62 catalogue entries
check-creatures: OK — 102 actor specs across 8 sources, every characteristic, skill, kit key, talent, species and style resolves, no duplicate keys
check-lethality: OK — 47 creatures, none drifted more than 6 points of wipe rate against the frozen party
check-anatomy: OK — 5 body plans, 36 locations, d20 complete in both modes, every location drawn, every kind handled and destructible, 20 plan changes carried 432 wounds with their severity intact, 1 flyer(s) with a survivable landing
check-lethality: OK — 47 creatures, every one fighting exactly as recorded against the frozen party
```
<!-- guards:end -->
+1 -1
View File
@@ -2,7 +2,7 @@
"id": "ringbrp",
"title": "The Custodians",
"description": "A Basic Roleplaying game of agents who cross the crossing to contain what should not be there — and who come back a little less from here each time.",
"version": "1.7.4",
"version": "1.7.5",
"compatibility": {
"minimum": 13,
"verified": "14.364"
+48 -15
View File
@@ -47,11 +47,22 @@ const PARTY_KEYS = ["pc_holloway", "pc_okonkwo", "pc_nkemdirim", "pc_ferriby"];
const RUNS = 200;
const SEED = 11;
/* How far a number may move before it is drift rather than noise. 200 runs of a coin
flip has a standard error around 3.5 points, so anything under 6 is inside the noise
and failing on it would make the guard cry wolf until somebody deleted it. */
const WIPE_TOLERANCE = 6.0; // percentage points
const DOWN_TOLERANCE = 0.35; // agents, of four
/* NO TOLERANCE, deliberately. The reasoning it replaces sounded right — "200 runs of a
coin flip has a standard error around 3.5 points, so anything under 6 is inside the
noise" — and it is exactly backwards for this measurement. Sampling error would apply
if the runs were random; they are not. The guard used to allow six points of wipe
rate and a third of an agent before it complained, which sounds prudent and was why it
sat silent through the largest combat change the system has had: ddc4f99 routed every
ordinary blow through the hit location table, 26 of 46 creatures moved, and not one
of them moved far enough in a single number to trip the threshold. The baseline went
four commits describing a game nobody was playing.
A tolerance is for noise, and there is none here: same party, same seed, same counts,
and measure() builds its own generator per creature, so two recordings of unchanged
code are byte-identical. Anything that moves is a real change to the rules or to the
creature, which is exactly what this file exists to notice. `rounds` is compared too;
it was recorded and then ignored, so a creature could take a round longer to kill
forever without a word. */
const party = PARTY_KEYS.map(k => {
const found = ROSTER.find(r => r.key === k);
@@ -100,18 +111,38 @@ if (base.runs !== RUNS || base.seed !== SEED
process.exit(1);
}
/* Comparing exactly is only honest if the measurement IS exact. Prove it here rather
than trust it: one creature, measured twice, must come back identical. If a future
change reaches for Math.random or a Set iteration order, this says so instead of
letting the whole guard degrade into noise-chasing. */
{
const probe = NPCS[0];
const a = measure(party, [probe], { runs: RUNS, seed: SEED });
const b = measure(party, [probe], { runs: RUNS, seed: SEED });
const shape = r => JSON.stringify([r.wipeRate, r.downMean, r.roundsMedian, r.hurtMean]);
if (shape(a) !== shape(b)) {
console.error("check-lethality: FAILED — the simulation is not deterministic, so an exact "
+ `baseline cannot mean anything. ${probe.key} measured twice gave ${shape(a)} and ${shape(b)}.`);
process.exit(1);
}
}
const problems = [];
const added = [], removed = [];
for (const [key, now] of Object.entries(current)) {
const was = base.creatures[key];
if (!was) { added.push(key); continue; }
const dWipe = now.wipe - was.wipe;
const dDown = now.down - was.down;
if (Math.abs(dWipe) > WIPE_TOLERANCE || Math.abs(dDown) > DOWN_TOLERANCE) {
problems.push(`${key}: wiped ${was.wipe}% -> ${now.wipe}% (${dWipe >= 0 ? "+" : ""}${dWipe.toFixed(1)}), `
+ `down ${was.down} -> ${now.down} (${dDown >= 0 ? "+" : ""}${dDown.toFixed(2)} of 4)`);
}
if (now.wipe === was.wipe && now.down === was.down && now.rounds === was.rounds) continue;
const dWipe = now.wipe - was.wipe, dDown = now.down - was.down;
const bits = [];
if (now.wipe !== was.wipe)
bits.push(`wiped ${was.wipe}% -> ${now.wipe}% (${dWipe >= 0 ? "+" : ""}${dWipe.toFixed(1)})`);
if (now.down !== was.down)
bits.push(`down ${was.down} -> ${now.down} (${dDown >= 0 ? "+" : ""}${dDown.toFixed(2)} of 4)`);
if (now.rounds !== was.rounds)
bits.push(`rounds ${was.rounds} -> ${now.rounds}`);
problems.push({ key, dWipe, text: `${key}: ${bits.join(", ")}` });
}
for (const key of Object.keys(base.creatures)) if (!(key in current)) removed.push(key);
@@ -125,12 +156,14 @@ if (added.length) console.log(` note: ${added.length} new creature(s) not in th
if (removed.length) console.log(` note: ${removed.length} creature(s) gone from content — ${removed.join(", ")}`);
if (problems.length) {
console.error("check-lethality: FAILED — a creature does a different amount of damage than recorded");
for (const p of problems) console.error(" " + p);
console.error(`check-lethality: FAILED — ${problems.length} of ${Object.keys(current).length} `
+ `creature(s) fight differently than recorded`);
problems.sort((a, b) => Math.abs(b.dWipe) - Math.abs(a.dWipe));
for (const p of problems) console.error(" " + p.text);
console.error("\n If this was deliberate, re-record with --update and say what changed in the commit.");
process.exit(1);
}
console.log(`check-lethality: OK — ${Object.keys(current).length} creatures, none drifted more than `
+ `${WIPE_TOLERANCE} points of wipe rate against the frozen party`
console.log(`check-lethality: OK — ${Object.keys(current).length} creatures, every one fighting `
+ `exactly as recorded against the frozen party`
+ `${added.length ? `, ${added.length} new` : ""}`);
+23 -2
View File
@@ -9,6 +9,7 @@
*/
import { readFile, writeFile } from "node:fs/promises";
import { execFileSync } from "node:child_process";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { SKILL_CATALOGUE, WEAPONS, ARMOURS, GEAR, VEHICLES, NPCS, TALENTS }
from "./content.mjs";
@@ -32,9 +33,29 @@ const runGuard = name => {
/* All of them, in the order `npm run check` runs them. This list was four long while the
suite was seven, so the README advertised a subset and silently stopped mentioning
every guard added after it was written — including the three that catch the most. */
const guardLines = ["check-rules.mjs", "check-kits.mjs", "check-lang.mjs",
const GUARDS = ["check-rules.mjs", "check-kits.mjs", "check-lang.mjs",
"check-templates.mjs", "check-behaviour.mjs", "check-scenarios.mjs",
"check-creatures.mjs", "check-lethality.mjs"].map(runGuard).filter(Boolean);
"check-creatures.mjs", "check-anatomy.mjs", "check-lethality.mjs"];
/* The list above went stale the moment a guard was added without touching this file —
which is what the comment above it already warned about, and which happened again
with check-anatomy. So it is no longer trusted: the suite of record is the `check`
script in package.json, and this refuses to write a README that advertises a
different set than the one the build actually runs. */
{
const pkg = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8"));
const actual = [...(pkg.scripts?.check ?? "").matchAll(/(check-[\w-]+\.mjs)/g)].map(m => m[1]);
const missing = actual.filter(g => !GUARDS.includes(g));
const phantom = GUARDS.filter(g => !actual.includes(g));
if (missing.length || phantom.length) {
console.error("update-readme: FAILED — the README's guard list does not match `npm run check`");
if (missing.length) console.error(" run by the build, not advertised: " + missing.join(", "));
if (phantom.length) console.error(" advertised, not run by the build: " + phantom.join(", "));
process.exit(1);
}
}
const guardLines = GUARDS.map(runGuard).filter(Boolean);
const box = `| | |
|---|---|