Memory-Efficient Multi-Agent Debate with Compressed Contexts for Long-Form Reasoning

Authors

  • Hyejin Byun School of Informatics, University of Edinburgh, Edinburgh, United Kingdom Author
  • Mireille Bertin School of Informatics, University of Edinburgh, Edinburgh, United Kingdom Author
  • Minji Kim School of Informatics, University of Edinburgh, Edinburgh, United Kingdom Author
  • Colin Dempsey School of Informatics, University of Edinburgh, Edinburgh, United Kingdom Author

DOI:

https://doi.org/10.66525/ijictt85

Keywords:

Multi-Agent Systems, Context Compression, Long-Form Reasoning, Memory Efficiency

Abstract

Abstract

The advent of large language models has significantly advanced the capabilities of artificial intelligence systems in complex problem-solving scenarios. However, deploying these models in multi-agent debate frameworks for long-form reasoning presents substantial challenges regarding memory utilization and computational efficiency. As multiple agents interact, exchange arguments, and iteratively refine their positions, the context window rapidly expands, leading to severe memory bottlenecks and degraded reasoning coherence. This paper proposes a novel framework centered on memory-efficient multi-agent debate utilizing compressed contexts. By introducing a semantic compression mechanism that distills agent arguments into high-density representations, the proposed system minimizes token redundancy while preserving the structural integrity of the reasoning process. Extensive theoretical analysis and empirical evaluations across various long-form reasoning benchmarks demonstrate that this approach reduces memory footprint while maintaining or exceeding the reasoning accuracy of uncompressed baselines. The findings indicate that strategically compressing the intermediate dialogue history among agents mitigates the quadratic scaling issues associated with standard attention mechanisms. This research provides a foundational architecture for scaling multi-agent systems to handle complex, multi-step logical derivations without requiring prohibitive computational resources, thereby broadening the practical applicability of autonomous reasoning networks.

Downloads

Download data is not yet available.

References

1. Wu, Beiliang, et al. "IndexNet: Timestamp and Variable-Aware Modeling for Time Series Forecasting." arXiv preprint arXiv:2509.23813 (2025).

2. Dong, J., Koniusz, P., Chen, J., & Ong, Y. S. (2024, September). Adversarially robust distillation by reducing the student-teacher variance gap. In European Conference on Computer Vision (pp. 92-111). Cham: Springer Nature Switzerland.

3. Xue, Xiao, et al. "Fast-Forward Lattice Boltzmann: Learning Kinetic Behaviour with Physics-Informed Neural Operators." arXiv preprint arXiv:2509.22411 (2025).

4. Dong, J., Wang, Y., Lai, J. H., & Xie, X. (2022). Improving adversarially robust few-shot image classification with generalizable representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9025-9034).

5. Zhu, Guoying, et al. "Enabling MoE on the Edge via Importance-Driven Expert Scheduling." arXiv preprint arXiv:2508.18983 (2025).

6. Dong, J., Yang, L., Wang, Y., Xie, X., & Lai, J. (2023). Toward intrinsic adversarial robustness through probabilistic training. IEEE Transactions on Image Processing, 32, 3862-3872.

7. Zhang, W., Zhang, C., Gu, C., Kou, J., Yuan, H., Fang, X., ... & Fang, Y. (2024, October). Hallucination in Large Language Models: From Mechanistic Understanding to Novel Control Frameworks. In 2024 7th International Conference on Universal Village (UV) (pp. 1-36). IEEE.

8. Qian, B., Guo, J., Wang, Q., & Shuang, K. (2025). From local verification to global reasoning: Exploiting slot-accompanying update for improved slot selection. Knowledge-Based Systems, 319, 113521.

9. Zhang, Y., Carvalho, D., & Freitas, A. (2025, July). Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder. In Proceedings of the 29th Conference on Computational Natural Language Learning (pp. 12-29).

10. Wang, S., Yu, Y., Feldt, R., & Parthasarathy, D. (2025). Automating a complete software test process using llms: An automotive case study. arXiv preprint arXiv:2502.04008.

11. Yang, Tianyue, and Xiao Xue. "Meno: Meanflow-enhanced neural operators for dynamical systems." arXiv preprint arXiv:2604.06881 (2026).

12. Xue, X., Wang, S., Yao, H. D., Davidson, L., & Coveney, P. V. (2024). Physics informed data-driven near-wall modelling for lattice Boltzmann simulation of high Reynolds number turbulent flows. Communications Physics, 7(1), 338.

13. Xu, X., Li, J., Zhang, Y., Tu, W., & Yang, Y. (2025). Spatial information aided speech and noise feature discrimination for Monaural speech enhancement. Expert Systems with Applications, 269, 126349.

14. Chen, L. (2026). Beyond external constraints: The missing dimension of ai governance. Available at SSRN 6449738.

15. Ou, J., Guo, J., Jiang, S., Wang, Z., Chen, Y., Xue, R., & Tian, W. (2025, June). Low-Rank Decomposition Assisted Quantization and Inference Compensation for Quality Large Language Model Inference. In 2025 International Joint Conference on Neural Networks (IJCNN) (pp. 1-9). IEEE.

16. Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

17. Xu, Xinmeng, et al. "Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning." arXiv preprint arXiv:2605.01673 (2026).

18. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2024). Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems.

19. Sivarajkumar, S., Edupuganti, S., Lazris, D., Bhattacharya, M., Davis, M., Dressman, D., ... & Wang, Y. (2026). Extraction of Treatments and Responses From Non–Small Cell Lung Cancer Clinical Notes Using Natural Language Processing. JCO clinical cancer informatics, 10, e2500138.

20. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.

21. Cui, Z., Huang, T., Chiang, C. E., & Du, C. (2025, August). Toward verifiable misinformation detection: A multi-tool LLM agent framework. In Proceedings of the 2025 International Conference on Generative Artificial Intelligence for Business (pp. 179-185).

22. Wang J, Fan L, Li B, et al. A Dynamic Factor Gating Architecture with Market Regime Awareness for Stock Return Forecasting[J]. 2026.

23. Liu, J. (2025, May). Reinforcement learning-controlled subspace ensemble sampling for complex data structures. In 2025 IEEE 7th International Conference on Communications, Information System and Computer Engineering (CISCE) (pp. 590-594). IEEE.

24. Peng, Xueqing, et al. "Multifinben: a multilingual, multimodal, and difficulty-aware benchmark for financial LLM evaluation." arXiv e-prints (2025): arXiv-2506.

25. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.

26. Liu, J., Zhang, Y., Sheng, Y., Lou, Y., Wang, H., & Yang, B. (2025, March). Context-aware rule mining using a dynamic transformer-based framework. In 2025 8th International Conference on Advanced Algorithms and Control Engineering (ICAACE) (pp. 2047-2052). IEEE.

27. Wang, Chenxing, et al. "When & How to Write for Personalized Demand-aware Query Rewriting in Video Search." arXiv preprint arXiv:2602.17667 (2025).

28. Kong, R., Li, Y., Feng, Q., Wang, W., Ye, X., Ouyang, Y., ... & Liu, Y. (2024, August). SwapMoE: Serving off-the-shelf MoE-based large language models with tunable memory budget. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6710-6720).

29. Dong, J., Koniusz, P., Chen, J., Wang, Z. J., & Ong, Y. S. (2024). Robust distillation via untargeted and targeted intermediate adversarial samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 28432-28442).

30. Xie, Z., Ren, X., Zheng, T., Bai, J., Fan, W., Xu, B., ... & Song, Y. A Survey on AI Agent Harness.

31. Huang, J. (2025, September). SCIAI: RELIABLE LARGE LANGUAGE MODEL REASONING FOR SCIENTIFIC LITERATURE VERIFICATION AND HYPOTHESIS VALIDATION. In The 5th International scientific and practical conference “Trends in the development of science by young scientists and students”(September 30-October 03, 2025) Warsaw, Poland. International Science Group. 2025. 122 p. (p. 16).

32. Wang, Juyuan, et al. "HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads." arXiv preprint arXiv:2604.17237 (2026).

33. Li, W. W., Ziser, Y., Xie, Y., Cohen, S. B., & Ma, T. (2025, July). Tsprank: Bridging pairwise and listwise methods with a bilinear travelling salesman model. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1 (pp. 707-718).

34. Zhang, W., Zhang, C., Luo, Z., Ma, J., Yuan, W., Gu, C., & Feng, C. (2025). SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction. arXiv preprint arXiv:2511.07584.

35. Lo, S. C., Zingaro, A., McCullough, J. W., Xue, X., Gonzalez-Martin, P., Joo, B., ... & Coveney, P. V. (2025). A multi-component, multi-physics computational model for solving coupled cardiac electromechanics and vascular haemodynamics. Computer Methods in Applied Mechanics and Engineering, 446, 118185.

36. Zhang, Luyan. "MCP: A Control-Theoretic Orchestration Framework for Synergistic Efficiency and Interpretability in Multimodal Large Language Models." arXiv preprint arXiv:2509.16597 (2025).

37. Guo, H., Lu, C., Bao, F., Pang, T., Yan, S., Du, C., & Li, C. (2023). Gaussian mixture solvers for diffusion models. Advances in Neural Information Processing Systems, 36, 25598-25626.

38. Li, W. W., Ziser, Y., Coavoux, M., & Cohen, S. B. (2023, May). BERT is not the count: Learning to match mathematical statements with proofs. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (pp. 3581-3593).

39. Qu, W., Shao, Y., Meng, L., Huang, X., & Xiao, L. (2024). A conditional denoising diffusion probabilistic model for point cloud upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20786-20795).

40. Huang, T., Cui, Z., Du, C., & Chiang, C. E. (2025, June). CL-ISR: A Contrastive Learning and Implicit Stance Reasoning Framework for Misleading Text Detection on Social Media. In 2025 6th International Conference on Electronic Communication and Artificial Intelligence (ICECAI) (pp. 610-616). IEEE.

41. Tu, P., Huang, Y., Zheng, F., He, Z., Cao, L., & Shao, L. (2022, June). Guidedmix-net: Semi-supervised semantic segmentation by using labeled images as reference. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 2, pp. 2379-2387).

Downloads

Published

2026-05-05