Real-Time (RT) Grid Streaming Engine
Abstract
This paper presents the foundational architecture for the Real-Time (RT) Grid Streaming Engine, designed to handle massive telemetry ingestion and run near-instantaneous topology calculations.
Research Context: This is the first of four forthcoming publications analyzing cross-hierarchy network behavior (spanning German CIGRE MV to Australian Wholesale Transmission). These four studies cumulatively establish the 5-Factor Decision Framework for power systems engineers. Specifically, this initial ETECOM 2026 paper proves that in German Medium-Voltage networks, operating state dictates voltage margins far more than local event magnitudes ($r = -0.99$).
Methodology
Content currently under peer review.
- Ingestion Layer: A high-throughput Kafka/gRPC pipeline designed to process real-time telemetry while handling misleading threshold boundaries and data quality issues.
- Processing Engine: Migration from Pandapower (Python) to OpenDSS to overcome the performance bottlenecks in time-series forecasting and real-time state estimation.
- Topology & Graph Modeling: Algorithmic bus selection utilizing optimized Dijkstra pathfinding to map Low-Voltage networks with limited sensor data.
- Streaming Infrastructure: Implementation of a specialized caching layer to break through the "Memory Wall," allowing continuous severity sweeps and real-time generation vs. load correlation.
Expected Outcomes
The implementation demonstrates a capability to run continuous time-series forecasting and congestion mapping (The Redispatch Atlas) without the latency inherent to batch-based Python environments.
By proving that operating state dominates event magnitude at the MV tier, this paper establishes Factor 3 (Relative Variability) of the unified 5-Factor Decision Framework, setting the stage for subsequent research on LV rooftop solar and transmission-level hyperscale data centres.