MARL-Converge: Decentralized Multi-Agent Reinforcement Learning for Failure-Aware Routing Convergence
DOI:
https://doi.org/10.66525/ijictt106Keywords:
decentralized control, failure-aware routing, multi-agent reinforcement learning, routing convergence, safe action maskingAbstract
Routing failures demand rapid control-plane convergence, yet uniformly aggressive detection and flooding increases signaling load and can induce unstable parameter oscillations. This paper presents MARL-Converge, a decentralized multi-agent reinforcement learning framework in which every router selects a local conservative, balanced, or aggressive control profile from router-local and one-hop telemetry. A parameter-shared dueling Double Deep Q-Network is trained with shared experience and a global-dominant team reward, while execution remains decentralized. A failure-aware action mask prevents routers without local evidence from blindly escalating to the aggressive profile. We implement a reproducible discrete-time link-state convergence proxy with independent topology views, stochastic failure detection and link-state advertisement diffusion, shortest-path recomputation, and forwarding-loop/drop checks. Evaluation comprises 600 matched episodes on the real Abilene and held-out AARNet topologies from the Internet Topology Zoo. Relative to a static balanced profile, MARL-Converge reduces convergence steps by 19.0% and modeled packet loss by 83.6%; the 6.1% overhead reduction is directionally favorable but not significant at the 0.05 level. Relative to a local heuristic, it is 9.3% faster, uses 14.0% fewer control units, and makes 56.4% fewer profile switches. It is 7.6% slower than always-aggressive control but cuts overhead by 35.4%. These results establish a measured speed-stability-efficiency trade-off without claiming physical-router timing equivalence.
Downloads
References
C. Labovitz, A. Ahuja, A. Bose, and F. Jahanian, “Delayed Internet routing convergence,” in Proc. ACM SIGCOMM, 2000, pp. 175–187, doi: 10.1145/347059.347428.
[2] J. Moy, “OSPF Version 2,” RFC 2328, Apr. 1998, doi: 10.17487/RFC2328.
[3] Y. Rekhter, T. Li, and S. Hares, “A Border Gateway Protocol 4 (BGP-4),” RFC 4271, Jan. 2006, doi: 10.17487/RFC4271.
[4] D. Katz and D. Ward, “Bidirectional Forwarding Detection (BFD),” RFC 5880, Jun. 2010, doi: 10.17487/RFC5880.
[5] A. Atlas and A. Zinin, “Basic specification for IP Fast Reroute: Loop-Free Alternates,” RFC 5286, Sep. 2008, doi: 10.17487/RFC5286.
[6] M. Shand and S. Bryant, “A framework for loop-free convergence,” RFC 5715, Jan. 2010, doi: 10.17487/RFC5715.
[7] B. Wang, Z. Wang, W. Zhao, F. Zhang, and W. Shang, “DRL-Adapt: Deep reinforcement learning for adaptive routing convergence optimization in large-scale networks,” IEEE Open Journal of the Computer Society, vol. 7, pp. 849–863, 2026, doi: 10.1109/OJCS.2026.3687441.
[8] P. François, C. Filsfils, J. Evans, and O. Bonaventure, “Achieving sub-second IGP convergence in large IP networks,” ACM SIGCOMM Computer Communication Review, vol. 35, no. 3, pp. 35–44, 2005, doi: 10.1145/1070873.1070877.
[9] A. Basu and J. G. Riecke, “Stability issues in OSPF routing,” in Proc. ACM SIGCOMM, 2001, pp. 225–236, doi: 10.1145/964723.383077.
[10] Z. M. Mao, R. Govindan, G. Varghese, and R. H. Katz, “Route flap damping exacerbates Internet routing convergence,” in Proc. ACM SIGCOMM, 2002, pp. 221–233, doi: 10.1145/633025.633047.
[11] S. Knight et al., “The Internet Topology Zoo,” IEEE Journal on Selected Areas in Communications, vol. 29, no. 9, pp. 1765–1775, Oct. 2011, doi: 10.1109/JSAC.2011.111002.
[12] M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning,” in Machine Learning Proceedings 1994, 1994, pp. 157–163, doi: 10.1016/B978-1-55860-335-6.50027-1.
[13] F. A. Oliehoek and C. Amato, A Concise Introduction to Decentralized POMDPs. Cham, Switzerland: Springer, 2016, doi: 10.1007/978-3-319-28929-8.
[14] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015, doi: 10.1038/nature14236.
[15] H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with Double Q-learning,” in Proc. AAAI Conf. Artificial Intelligence, vol. 30, no. 1, 2016, pp. 2094–2100, doi: 10.1609/aaai.v30i1.10295.
[16] Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, “Dueling network architectures for deep reinforcement learning,” in Proc. 33rd Int. Conf. Machine Learning, PMLR, vol. 48, 2016, pp. 1995–2003.
[17] T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and S. Whiteson, “QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in Proc. 35th Int. Conf. Machine Learning, PMLR, vol. 80, 2018, pp. 4295–4304.
[18] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 6379–6390.
[19] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in Proc. AAAI Conf. Artificial Intelligence, vol. 32, no. 1, 2018, pp. 2974–2982, doi: 10.1609/aaai.v32i1.11794.
[20] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of PPO in cooperative multi-agent games,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 24611–24624.
[21] K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Başar, “Fully decentralized multi-agent reinforcement learning with networked agents,” in Proc. 35th Int. Conf. Machine Learning, PMLR, vol. 80, 2018, pp. 5872–5881.
[22] J. García and F. Fernández, “A comprehensive survey on safe reinforcement learning,” Journal of Machine Learning Research, vol. 16, no. 42, pp. 1437–1480, 2015.
[23] J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing networks: A reinforcement learning approach,” in Advances in Neural Information Processing Systems, vol. 6, 1994, pp. 671–678.
[24] D. Teng, M. Rhee, Y. Qin, B. Zi, and W. Liu, "SW-SpeedDLM: Sliding window speculative decoding for diffusion language models under long context constraints," Mathematics, 2026.
[25] Z. Wang, J. S. Yang, W. Shang, and J. Ding, "FairPromote: Explainable and fairness-aware talent promotion prediction via adversarial debiasing and SHAP-based interpretation," IEEE Access, 2026.
[26] D. Teng, "TEAS: Token- and Energy-Aware Autoscaling for Cost-Efficient LLM Serving," AI and Data Science Journal, 2025
[27] F. Zhang, Z. Guo, J. Ding, J. Yang, and W. Liu, "Adaptive sensor fusion for robust perception in dense fog: A gated vision and LiDAR integration framework," Sensors, 2026.
[28] B. Zi, "Cloud-native distributed systems for real-time payment intelligence," AI and Data Science Journal, vol. 1, no. 1, pp. 51-56, 2024.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 David Clarke , Mei Huang , Jonas Eriksen (Author)

This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the International Journal of Information and Communication Technology Trends (IJICTT) are distributed under a Creative Commons license. This allows readers to freely access, share, and adapt the content for non-commercial purposes, provided proper attribution is given to the original authors. Authors retain copyright of their work.
