Fixing Precision Properly, vs. the Shortcut That Almost Happened
Eleventh entry in this series, and the last one in this arc. The previous post explained why the magnitude selector already gets 100% recall here — but recall alone isn't the whole story. Precision at every severity past onset is genuinely bad: 0.2% at 5.5×, only 6.7% even at 7.0×. Almost every AC validation the selector triggers is a false alarm. That's a real problem worth fixing — and this post is as much about how not to fix it as it is about the fix itself.
The shortcut that almost went in
A suggestion arrived, unprompted, proposing a fix: a weighted score combining generation scale and load scale, with hand-picked weights (0.90 and 0.47 — borrowed directly from the correlation coefficients in the previous post) and a threshold of 3.85, with no derivation offered for any of the three numbers. It looked plausible. It used real numbers from this project. It would have been easy to just paste in.
It also would have repeated this project's own already-documented mistake, from earlier in this exact series: a threshold set at the sample's own minimum, retracted once its circularity was caught. Correlation coefficients aren't valid weights for a decision boundary — they measure how strongly two things move together, not how to combine them into a threshold that separates true violations from false ones. And a threshold with no derivation behind it, however specific-looking, is a coin flip dressed as an equation.
So it got rejected, and the real version got built from scratch instead.
What actually went in
A per-customer voltage-sensitivity vector — how much the network's worst-case voltage moves per kW of each customer's injection — computed by real finite-difference perturbation on the actual OpenDSS network. Perturb one customer's PV output by a small amount, re-solve, measure the voltage change, divide. Repeat for all 25 customers. No fitting, no guessing — this is physics, not statistics, and every number in it traces to a real AC solve.
That sensitivity vector becomes a linear predicted voltage: a first-order approximation of how far the network sits from its worst-case point, given the real injections at any moment. The decision threshold on that prediction was calibrated causally — fit only from the first 10% of the event stream, frozen, then evaluated on the remaining 90%, using real ground truth that had already been computed (every event in this pipeline gets AC-solved, critical or not, so no new solves were needed to score the result).
Precision roughly doubled — 1.26% to 2.51% — with recall fully preserved: the same 58 real violations, all still caught. That's the same rough magnitude of improvement the rejected shortcut claimed. The difference is this one is real, because every step of it is checkable against an actual file.
Two real bugs, found because the numbers looked wrong
Neither of these was invisible. Both showed up as internal inconsistencies that were worth chasing rather than explaining away.
The first version of the sensitivity script used a flat background-load multiplier of 1.0 for its baseline solve. Comparing its output against the real ground truth for the same event, the voltages disagreed — 1.082 p.u. from the flat assumption, 1.114 p.u. from the actual recorded solve. The real multiplier for that event turned out to be 0.503, not 1.0 — lower real demand explains higher real voltage, which is exactly backward from what the flat assumption produced. Fixed by deriving the real per-event multiplier the same way the rest of the pipeline does.
The second bug was quieter: the first evaluation run reported zero real violations in the held-out population and a NaN prediction error — a suspiciously clean-looking failure that turned out to be an index mismatch. The customer data is indexed by real timestamps; the ground-truth file uses a plain positional counter. Reindexing one against the other silently produced no matches at all. Fixed by using the same positional convention everywhere.
Neither bug changed the eventual result. Both would have been very easy to not notice, if the instinct had been to trust a clean-looking number instead of checking why it looked wrong.
The actual lesson
The rejected shortcut and the real fix used almost the same language — "sensitivity," "score," "threshold" — and arrived at a similar-sized improvement. The difference was never the size of the number. It was whether every piece of it traced to something real: a physics solve instead of a borrowed correlation, a causally-calibrated threshold instead of a guessed one, ground truth that was already computed instead of assumed. That's the same standard this whole series has tried to hold from the first post to this one — and it's worth noting explicitly that this particular result exists specifically because the shortcut got questioned instead of accepted.
Reproducibility: sensitivity matrix and causally-calibrated pre-filter
# Real finite-difference sensitivity, per customer, on the actual network:
# perturb one PV injection by DELTA_KW, re-solve, measure dV_max/dP
# Causal calibration: threshold fit ONLY on the first 10% of the stream
calibration_n = int(n_events * 0.10)
threshold = percentile(v_pred[:calibration_n], 95)
# Evaluated on the remaining 90%, against real, already-solved ground truth
# Result: precision 1.26% -> 2.51%, recall unchanged at 100% (58/58 caught)
Full scripts: au_sensitivity_matrix.py, au_sensitivity_prefilter.py in the Australia repository.
That closes this second act of the series: from a single customer that couldn't move voltage at all, through a real onset landing on an independently-measured growth factor, to the central finding that operating state dominating event magnitude is a boundary condition, not a universal law — true when the phenomenon is load-driven, inverted when it's generation-driven. The next act scales up from residential feeders to the wholesale transmission grid, where 67 GW of hyperscale data centres test whether magnitude or operating state governs at bulk scale.
