DRL-Based OSPF Convergence Optimization: From Simulation to Network Emulation
DOI:
https://doi.org/10.66525/ijictt142Keywords:
OSPF, routing convergence, deep reinforcement learning, network emulation, FRRouting, sim-to-real transfer, protocol timersAbstract
Deep reinforcement learning (DRL) has been proposed to tune OSPF convergence timers online, but such agents are trained and evaluated in simulators whose fidelity to real routing stacks is rarely checked. We take DRL-based OSPF convergence optimization from simulation to network emulation. An FRRouting testbed running production ospfd/zebra daemons in Linux namespaces is paired with a protocol-level simulator at three fidelity levels (idealized, FRR-faithful, trace-calibrated) that share scenarios, observations, actions and metrics. Calibration identifies 4.4 ms carrier-loss detection, 2.2 ms per-hop flooding and about 1.0 s of configuration-push skew. On real ISP topologies and traffic matrices, the idealized simulator misestimates the per-episode cost by 62.7% on average and the restoration time of hard failures by a factor of 66, whereas the calibrated simulator stays within 12.6%. We re-implement the DRL-Adapt agent, use its Markov decision process as the common control formulation, and extend it with a feasibility-aware action-branching design. Deployed on the testbed, the re-implemented DRL-Adapt agent lowers the cost by 31.6% (trained in the idealized simulator) and 32.9% (trained in the calibrated simulator) relative to RFC default timers, a gain comparable in magnitude to the convergence-time reduction originally reported for DRL-Adapt. The agent nevertheless converges to a near-static operating point, and a static configuration found by searching the calibrated simulator reduces the cost by 32.7%, within the scenario-to-scenario variability of both DRL-Adapt variants. Our results indicate that DRL-Adapt’s improvement over default timers transfers from simulation to a production routing stack, and that, under realistic protocol constraints, calibrated configuration search captures most of the benefit of adaptive timer control on the evaluated topologies.
Downloads
References
J. Moy, "OSPF version 2," IETF, RFC 2328, Apr. 1998, doi: 10.17487/RFC2328.
M. Goyal, M. Soperi, E. Baccelli, G. Choudhury, A. Shaikh, H. Hosseini, and K. S. Trivedi, "Improving convergence speed and scalability in OSPF: A survey," IEEE Commun. Surveys Tuts., vol. 14, no. 2, pp. 443–463, 2012, doi: 10.1109/SURV.2011.011411.00065.
B. Decraene, S. Litkowski, H. Gredler, A. Lindem, P. Francois, and C. Bowers, "Shortest path first (SPF) back-off delay algorithm for link-state IGPs," IETF, RFC 8405, Jun. 2018, doi: 10.17487/RFC8405.
P. Francois, C. Filsfils, J. Evans, and O. Bonaventure, "Achieving sub-second IGP convergence in large IP networks," ACM SIGCOMM Comput. Commun. Rev., vol. 35, no. 3, pp. 35–44, Jul. 2005.
A. Shaikh, A. Varma, L. Kalampoukas, and R. Dube, "Routing stability in congested networks: Experimentation and analysis," in Proc. ACM SIGCOMM, Stockholm, Sweden, 2000, pp. 163–174.
A. Basu and J. Riecke, "Stability issues in OSPF routing," in Proc. ACM SIGCOMM, San Diego, CA, USA, 2001, pp. 225–236.
F. Zhou, “Moment-constrained side-force allocation for wind-assisted merchant ships: Hydrodynamic performance of retractable anti-leeway fins and flap rudders,” Innovation and Technology Studies, vol. 3, no. 1, pp. 27–41, 2026.
A. Markopoulou, G. Iannaccone, S. Bhattacharyya, C.-N. Chuah, Y. Ganjali, and C. Diot, "Characterization of failures in an operational IP backbone network," IEEE/ACM Trans. Netw., vol. 16, no. 4, pp. 749–762, Aug. 2008.
J. A. Boyan and M. L. Littman, "Packet routing in dynamically changing networks: A reinforcement learning approach," in Adv. Neural Inf. Process. Syst. 6 (NIPS), 1993, pp. 671–678.
A. Valadarsky, M. Schapira, D. Shahaf, and A. Tamar, "Learning to route," in Proc. 16th ACM Workshop Hot Topics Netw. (HotNets), Palo Alto, CA, USA, 2017, pp. 185–191, doi: 10.1145/3152434.3152441.
Z. Xu, J. Tang, J. Meng, W. Zhang, Y. Wang, C. H. Liu, and D. Yang, "Experience-driven networking: A deep reinforcement learning based approach," in Proc. IEEE INFOCOM, Honolulu, HI, USA, 2018, pp. 1871–1879, doi: 10.1109/INFOCOM.2018.8485853.
J. Chen, X. Han, X. Zhang, and Y. Yang, “Exact concept-level explanations for early detection of financial misstatements,” Eurasia Journal of Science and Technology, vol. 5, no. 2, pp. 34–48, 2023.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, "Domain randomization for transferring deep neural networks from simulation to the real world," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), Vancouver, BC, Canada, 2017, doi: 10.1109/IROS.2017.8202133.
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, "Sim-to-real transfer of robotic control with dynamics randomization," in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), Brisbane, QLD, Australia, 2018, doi: 10.1109/ICRA.2018.8460528.
FRRouting Project, "FRRouting (FRR) protocol suite," version 8.4.4. [Online]. Available: https://frrouting.org
S. Knight, H. X. Nguyen, N. Falkner, R. Bowden, and M. Roughan, "The Internet Topology Zoo," IEEE J. Sel. Areas Commun., vol. 29, no. 9, pp. 1765–1775, Oct. 2011, doi: 10.1109/JSAC.2011.111002.
N. Spring, R. Mahajan, and D. Wetherall, "Measuring ISP topologies with Rocketfuel," in Proc. ACM SIGCOMM, Pittsburgh, PA, USA, 2002, pp. 133–145.
S. Gay, P. Schaus, and S. Vissicchio, "REPETITA: Repeatable experiments for performance evaluation of traffic-engineering algorithms," arXiv:1710.08665, 2017.
D. Katz and D. Ward, "Bidirectional forwarding detection (BFD)," IETF, RFC 5880, Jun. 2010, doi: 10.17487/RFC5880.
P. Huang, C. Guo, L. Zhou, J. R. Lorch, Y. Dang, M. Chintalapati, and R. Yao, "Gray failure: The Achilles’ heel of cloud-scale systems," in Proc. 16th Workshop Hot Topics Oper. Syst. (HotOS), Whistler, BC, Canada, 2017, pp. 150–155, doi: 10.1145/3102980.3103005.
D. Teng, “TRIAD: Deadline- and reliability-aware scheduling for post-earthquake structural assessment workflows,” World Journal of Engineering Research, vol. 3, no. 6, pp. 55–66, 2025.
V. Mnih et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529–533, 2015, doi: 10.1038/nature14236.
H. van Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double Q-learning," in Proc. AAAI Conf. Artif. Intell., Phoenix, AZ, USA, 2016, pp. 2094–2100, doi: 10.1609/aaai.v30i1.10295.
Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, "Dueling network architectures for deep reinforcement learning," in Proc. 33rd Int. Conf. Mach. Learn. (ICML), New York, NY, USA, 2016, pp. 1995–2003.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver, "Prioritized experience replay," in Proc. Int. Conf. Learn. Represent. (ICLR), San Juan, Puerto Rico, 2016.
A. Tavakoli, F. Pardo, and P. Kormushev, "Action branching architectures for deep reinforcement learning," in Proc. AAAI Conf. Artif. Intell., New Orleans, LA, USA, 2018, doi: 10.1609/aaai.v32i1.11798.
B. Lantz, B. Heller, and N. McKeown, "A network in a laptop: Rapid prototyping for software-defined networks," in Proc. 9th ACM Workshop Hot Topics Netw. (HotNets), Monterey, CA, USA, 2010.
N. Handigol, B. Heller, V. Jeyakumar, B. Lantz, and N. McKeown, "Reproducible network experiments using container-based emulation," in Proc. 8th Int. Conf. Emerg. Netw. Exp. Technol. (CoNEXT), Nice, France, 2012, pp. 253–264, doi: 10.1145/2413176.2413206.
P. Almasan et al., "Network digital twin: Context, enabling technologies, and opportunities," IEEE Commun. Mag., vol. 60, no. 11, pp. 22–27, Nov. 2022, doi: 10.1109/MCOM.001.2200012.
X. Yue and R. Zhu, “An explainable hybrid machine learning framework for cash-flow-conditioned multi-horizon liquidity risk early warning: An SME-oriented benchmark study,” World Journal of Management Science, vol. 2, no. 1, pp. 63–72, 2024.
J. Ding and D. Teng, “A Risk-Aware Digital Twin Framework for Release-Readiness Validation of Multi-Tenant Zero Trust SASE Systems,” WEP Transactions on Computer Science and Intelligent Systems Research, vol. 13, pp. 247–259, 2026.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Jonah Desai, Elijah Phillips, Seojun Miller (Author)

This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the International Journal of Information and Communication Technology Trends (IJICTT) are distributed under a Creative Commons license. This allows readers to freely access, share, and adapt the content for non-commercial purposes, provided proper attribution is given to the original authors. Authors retain copyright of their work.
