MARL-Converge: Decentralized Multi-Agent Reinforcement Learning for Failure-Aware Routing Convergence

Authors

  • David Clarke Department of Computer Science, Rutgers University, New Brunswick, NJ, USA. Author
  • Mei Huang Department of Electrical and Computer Engineering, Rutgers University, New Brunswick, NJ, USA Author
  • Jonas Eriksen School of Communication and Information, Rutgers University, New Brunswick, NJ, USA. Author

DOI:

https://doi.org/10.66525/ijictt106

Keywords:

decentralized control, failure-aware routing, multi-agent reinforcement learning, routing convergence, safe action masking

Abstract

Routing failures demand rapid control-plane convergence, yet uniformly aggressive detection and flooding increases signaling load and can induce unstable parameter oscillations. This paper presents MARL-Converge, a decentralized multi-agent reinforcement learning framework in which every router selects a local conservative, balanced, or aggressive control profile from router-local and one-hop telemetry. A parameter-shared dueling Double Deep Q-Network is trained with shared experience and a global-dominant team reward, while execution remains decentralized. A failure-aware action mask prevents routers without local evidence from blindly escalating to the aggressive profile. We implement a reproducible discrete-time link-state convergence proxy with independent topology views, stochastic failure detection and link-state advertisement diffusion, shortest-path recomputation, and forwarding-loop/drop checks. Evaluation comprises 600 matched episodes on the real Abilene and held-out AARNet topologies from the Internet Topology Zoo. Relative to a static balanced profile, MARL-Converge reduces convergence steps by 19.0% and modeled packet loss by 83.6%; the 6.1% overhead reduction is directionally favorable but not significant at the 0.05 level. Relative to a local heuristic, it is 9.3% faster, uses 14.0% fewer control units, and makes 56.4% fewer profile switches. It is 7.6% slower than always-aggressive control but cuts overhead by 35.4%. These results establish a measured speed-stability-efficiency trade-off without claiming physical-router timing equivalence.

Downloads

Download data is not yet available.

References

C. Labovitz, A. Ahuja, A. Bose, and F. Jahanian, “Delayed Internet routing convergence,” in Proc. ACM SIGCOMM, 2000, pp. 175–187, doi: 10.1145/347059.347428.

[2] J. Moy, “OSPF Version 2,” RFC 2328, Apr. 1998, doi: 10.17487/RFC2328.

[3] Y. Rekhter, T. Li, and S. Hares, “A Border Gateway Protocol 4 (BGP-4),” RFC 4271, Jan. 2006, doi: 10.17487/RFC4271.

[4] D. Katz and D. Ward, “Bidirectional Forwarding Detection (BFD),” RFC 5880, Jun. 2010, doi: 10.17487/RFC5880.

[5] A. Atlas and A. Zinin, “Basic specification for IP Fast Reroute: Loop-Free Alternates,” RFC 5286, Sep. 2008, doi: 10.17487/RFC5286.

[6] M. Shand and S. Bryant, “A framework for loop-free convergence,” RFC 5715, Jan. 2010, doi: 10.17487/RFC5715.

[7] B. Wang, Z. Wang, W. Zhao, F. Zhang, and W. Shang, “DRL-Adapt: Deep reinforcement learning for adaptive routing convergence optimization in large-scale networks,” IEEE Open Journal of the Computer Society, vol. 7, pp. 849–863, 2026, doi: 10.1109/OJCS.2026.3687441.

[8] P. François, C. Filsfils, J. Evans, and O. Bonaventure, “Achieving sub-second IGP convergence in large IP networks,” ACM SIGCOMM Computer Communication Review, vol. 35, no. 3, pp. 35–44, 2005, doi: 10.1145/1070873.1070877.

[9] A. Basu and J. G. Riecke, “Stability issues in OSPF routing,” in Proc. ACM SIGCOMM, 2001, pp. 225–236, doi: 10.1145/964723.383077.

[10] Z. M. Mao, R. Govindan, G. Varghese, and R. H. Katz, “Route flap damping exacerbates Internet routing convergence,” in Proc. ACM SIGCOMM, 2002, pp. 221–233, doi: 10.1145/633025.633047.

[11] S. Knight et al., “The Internet Topology Zoo,” IEEE Journal on Selected Areas in Communications, vol. 29, no. 9, pp. 1765–1775, Oct. 2011, doi: 10.1109/JSAC.2011.111002.

[12] M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning,” in Machine Learning Proceedings 1994, 1994, pp. 157–163, doi: 10.1016/B978-1-55860-335-6.50027-1.

[13] F. A. Oliehoek and C. Amato, A Concise Introduction to Decentralized POMDPs. Cham, Switzerland: Springer, 2016, doi: 10.1007/978-3-319-28929-8.

[14] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015, doi: 10.1038/nature14236.

[15] H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with Double Q-learning,” in Proc. AAAI Conf. Artificial Intelligence, vol. 30, no. 1, 2016, pp. 2094–2100, doi: 10.1609/aaai.v30i1.10295.

[16] Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, “Dueling network architectures for deep reinforcement learning,” in Proc. 33rd Int. Conf. Machine Learning, PMLR, vol. 48, 2016, pp. 1995–2003.

[17] T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and S. Whiteson, “QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in Proc. 35th Int. Conf. Machine Learning, PMLR, vol. 80, 2018, pp. 4295–4304.

[18] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 6379–6390.

[19] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in Proc. AAAI Conf. Artificial Intelligence, vol. 32, no. 1, 2018, pp. 2974–2982, doi: 10.1609/aaai.v32i1.11794.

[20] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of PPO in cooperative multi-agent games,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 24611–24624.

[21] K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Başar, “Fully decentralized multi-agent reinforcement learning with networked agents,” in Proc. 35th Int. Conf. Machine Learning, PMLR, vol. 80, 2018, pp. 5872–5881.

[22] J. García and F. Fernández, “A comprehensive survey on safe reinforcement learning,” Journal of Machine Learning Research, vol. 16, no. 42, pp. 1437–1480, 2015.

[23] J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing networks: A reinforcement learning approach,” in Advances in Neural Information Processing Systems, vol. 6, 1994, pp. 671–678.

[24] D. Teng, M. Rhee, Y. Qin, B. Zi, and W. Liu, "SW-SpeedDLM: Sliding window speculative decoding for diffusion language models under long context constraints," Mathematics, 2026.

[25] Z. Wang, J. S. Yang, W. Shang, and J. Ding, "FairPromote: Explainable and fairness-aware talent promotion prediction via adversarial debiasing and SHAP-based interpretation," IEEE Access, 2026.

[26] D. Teng, "TEAS: Token- and Energy-Aware Autoscaling for Cost-Efficient LLM Serving," AI and Data Science Journal, 2025

[27] F. Zhang, Z. Guo, J. Ding, J. Yang, and W. Liu, "Adaptive sensor fusion for robust perception in dense fog: A gated vision and LiDAR integration framework," Sensors, 2026.

[28] B. Zi, "Cloud-native distributed systems for real-time payment intelligence," AI and Data Science Journal, vol. 1, no. 1, pp. 51-56, 2024.

Downloads

Published

2026-07-29