We often talk about the clean energy transition in terms of gigawatts, policy, and infrastructure. But down in the trenches, it's a software and data engineering problem.
How do you safely dispatch a power grid when the state of that grid is changing every few milliseconds due to thousands of distributed energy resources?
For the past year, I've been wrestling with this exact question. My research has focused on building a Real-Time (RT) Grid Streaming Engine capable of handling massive telemetry ingestion and running near-instantaneous topology calculations. It culminated in an upcoming paper for ETECOM 2026 (Paris), and another study slated for 2027 based on the Australian network.
But academic papers have strict constraints. They demand polished results, formal formatting, and strict page limits. They rarely give you the space to talk about the engineering reality: the bugs that kept you up at night, the data pipelines that collapsed under load, and the architectural pivots that saved the project.
I wanted a place to share that reality.
Introducing the 12-Part Engineering Series
Starting next month, I will be publishing a 12-part technical series on the blog. This won't be a dry academic dump. It will be an honest, behind-the-scenes look at how we built the RT Grid Streaming Engine. We'll look at code, architectural decisions, and the raw data challenges.
To protect the integrity of the double-blind peer review process for my upcoming papers, I won't be publishing the final paper results or specific benchmark numbers just yet. Instead, we'll focus purely on the methodology and the software engineering.
The series will be split into two arcs:
Arc 1: The German CIGRE MV Benchmark (Starts September 2026)
- Real Data, Zero Violations: Building the Kafka/gRPC ingestion pipeline.
- The 100% Recall That Wasn't: Dealing with misleading threshold boundaries and data quality.
- Topology Streaming: Mapping the German CIGRE MV network in real-time.
- Generation vs. Load: The mathematics of correlation.
- State-Awareness at Scale: Running severity sweeps across the network.
- The Redispatch Atlas: An interactive map of our congestion management engine. (This will tie directly into our November press pitch in Electronics for Engineers).
Arc 2: The Australian Rooftop Solar Study (Starts February 2027)
- Migrating from Pandapower to OpenDSS: Why Python wasn't fast enough for time-series forecasting.
- Dijkstra’s Revenge: Algorithmic challenges in bus selection.
- Voltage Unbalance & Inverter Tripping: The uniquely Australian problem of high-penetration rooftop solar.
- Graph Modeling the Low-Voltage Grid: Network mapping with limited sensor data.
- The Memory Wall: Optimizing the caching layer for streaming calculations.
- The Final Architecture: Tying the engine together for production.
I'm incredibly excited to share this work with you. Whether you're a power systems engineer, a software architect, or just someone fascinated by the intersection of the two, I hope you'll find something valuable in these upcoming posts.
Stay tuned. The first post drops in September.
