The 100% Recall Number That Wasn't
Part of this series on grid-resilience research is being honest about the results that don't survive scrutiny. This one is the sharpest example: a state-aware screening rule that appeared to catch 43 of 43 known voltage violations — literally 100% recall — and turned out, on inspection, to have been circular by construction.
The claim
The magnitude-only selector from the previous post has an obvious blind spot: it flags events by redispatch magnitude alone, but voltage risk is dominated by background loading, which the selector never looks at. The natural fix is a state-aware rule — one that also conditions on the loading multiplier at the time of the event.
An early version of this rule used a simple loading-multiplier threshold: flag an event if the loading multiplier exceeds some cutoff. Tested against a known set of 43 violations from a recall-audit sample, it caught all 43. 100%.
That number is the kind of result you want to be true. It's also the kind of result that should make you suspicious before it makes you happy.
The check
The test that actually matters for a claim like this: is the threshold independent of the sample it's being validated against? If a threshold was chosen — even implicitly — by looking at the outcomes it's later "predicting," the validation is worthless regardless of how clean the number looks.
The check itself is a one-liner: what's the minimum loading multiplier among the 43 known violations?
The threshold in use was exactly 1.20. Not close to it — identical to the minimum value in the exact sample it was being tested against. A threshold set at the sample's own minimum will, by construction, catch every violation in that sample and nothing about how it would perform on any event outside it. It's the modeling equivalent of grading your own exam with the answer key already filled in.
Why this is worth writing about
This isn't a story about someone else's mistake caught during review — it's a mistake I made and caught by applying the same discipline I'd apply to any other result: don't trust a number that looks unusually good until you've checked whether the evaluation could have leaked into the fitting. It's an easy trap precisely because a threshold chosen this way doesn't look wrong. There's no obviously circular code path, no leaked label in an obvious sense — just a cutoff value that happens to equal the minimum of the exact population used to validate it, which is functionally the same failure mode with a less obvious signature.
The fix that stuck: real train/test discipline. Fit any state-aware rule on a 70/30 split of one population, evaluate it out-of-sample on a held-out population it never touched during fitting, and never adjust the rule after seeing how it performs on the audited set. That's the methodology behind the actual replacement result — a linear voltage-sensitivity regression, refit fresh at each operating condition, tested genuinely out-of-sample — which is the subject of a later post in this series and supersedes the 43/43 claim entirely.
The retraction, stated plainly
The 43/43 (≈100%) result is retracted. It was not a real out-of-sample test of state-aware screening; it was a threshold equal to the minimum of the population it was validated against, and any claim built on it should be discarded rather than caveated. The honest replacement — a genuinely out-of-sample, full-population-audited result across three operating-state severities — turns out to be a more interesting finding anyway, which is where this series goes next.
Reproducibility: the circularity check
import pandas as pd
from engine import GridSimulator, SyntheticLoadProvider
sample_df = pd.read_csv("final_output/recall_audit_sample.csv")
violations = sample_df[sample_df["sampled_violation"] == True].copy()
# The loading multiplier isn't a column in the sample file itself -- it's
# looked up per event from the same provider that generated the run.
simulator = GridSimulator()
provider = SyntheticLoadProvider(simulator.load_multipliers)
violations["multiplier"] = violations["event_idx"].apply(lambda i: provider.get_multiplier(int(i)))
threshold_in_use = 1.20
min_multiplier_in_sample = violations["multiplier"].min()
print(f"Known violations in sample: {len(violations)}")
print(f"Threshold in use: {threshold_in_use}")
print(f"Minimum multiplier among the {len(violations)} known violations: {min_multiplier_in_sample:.4f}")
# Known violations in sample: 43
# Threshold in use: 1.2
# Minimum multiplier among the 43 known violations: 1.2000
