Is 500 a Real Sample Size, or Just a Round Number?
Continuing this series on the research process behind a voltage-screening pipeline: after retracting a circular threshold claim, the natural next question was uncomfortable in a different way. The recall-audit code had a sample_size=500 default sitting in it, used everywhere a false-negative rate needed estimating. Ask "why 500," and there is no answer anywhere in the codebase, the comments, or the paper draft.
Where the number was doing real work
The screener only runs a full AC power-flow solve on events it flags as critical. Everything it calls "non-critical" never gets checked — so the only way to know the selector's real false-negative rate is to draw a sample from the non-critical population, solve it for real, and see how many turn out to be actual violations. That sample size directly determines the width of the resulting confidence interval on the false-negative rate, which is the number that ultimately decides how much to trust the screener at all.
500 events had been used for this since early in the project. When asked directly whether that was based on some standard guideline — a 5% rule, a power calculation, anything — the honest answer, after checking every place the number could have been justified, was no. It wasn't even a clean round percentage: 500 out of 16,531 non-critical events is 3.0%, not a tidy 5%. It was just a number that had been typed in once and never revisited.
Doing it properly
The standard approach for estimating a proportion to a target precision is the finite-population-corrected sample-size formula:
where is a pilot estimate of the rate being measured, the target margin of error, and the finite population size (16,531 non-critical events here). Using an initial small-sample estimate of the violation rate to size a proper pilot-informed sample, 500 was replaced with a larger, principled size — around 4,500 — for the conditions with a low expected violation rate.
Where even that wasn't enough
This is the part that actually forced the final decision. At one operating-state severity, the true violation rate turned out to be very low — roughly 0.4%. At that rate, even a 4,500-event pilot-informed sample produces so few observed violations that the derived statistic (the state-aware rule's catch rate among them) gets a Wilson 95% confidence interval spanning [19.7%, 57.0%] — too wide to support any real conclusion about whether the rule is working at that condition.
Increasing the sample size further chases diminishing returns against a fixed, small population. The formula above has a ceiling: once approaches , there's no more precision to buy by sampling harder — the only way to eliminate the uncertainty is to stop sampling and audit the entire population.
The full census
So that's what happened: instead of a bigger-but-still-somewhat-arbitrary sample size, all 16,531 non-critical events were solved for real, at every operating-state severity examined — not just the one where the smaller sample had failed. Equal footing across conditions mattered more than saving compute on the ones that already looked fine at a smaller sample. This removes sampling uncertainty from the false-negative rate entirely; whatever uncertainty remains in the paper's headline numbers is about model behavior, not population sampling.
The escalation, end to end: 500 → 4,500 → 16,531 (full census). Each step was forced by the data, not chosen in advance — which is exactly why writing it up honestly, including the dead ends, is worth more than presenting the final number as though it had been the plan from the start.
Reproducibility: sample-size progression on one severity condition
from engine import GridSimulator, LocalCsvIngestionLayer, ScaledLoadProvider, TimeSeriesLoadProvider, load_simbench_profile
simulator = GridSimulator()
stream = LocalCsvIngestionLayer().fetch_stream()
provider = ScaledLoadProvider(TimeSeriesLoadProvider(load_simbench_profile()), severity=1.50)
cycle_df = simulator.run_streaming_pipeline(stream, load_provider=provider)
for n in [500, 4500, 16531]:
_, summary = simulator.run_recall_audit(cycle_df, sample_size=n, load_provider=provider)
row = summary.iloc[0]
print(f"n={n}: FN rate={row['estimated_fn_rate']*100:.2f}%, "
f"95% CI=[{row['estimated_fn_rate_ci95_low']*100:.2f}%, "
f"{row['estimated_fn_rate_ci95_high']*100:.2f}%]")
