SysMoE: Serving Mixture-of-Experts Large Models under Tunable Memory Budgets

Authors

  • Amelia Holmgren Department of Computing, Hong Kong Polytechnic University, Hong Kong, China Author
  • Clara Von Neumann Department of Computing, Hong Kong Polytechnic University, Hong Kong, China Author

DOI:

https://doi.org/10.66525/ijictt82

Keywords:

Large Language Models, Mixture-of-Experts, Model Serving, Memory Management, Inference Optimization

Abstract

The rapid scaling of Large Language Models has introduced unprecedented capabilities in natural language understanding, reasoning, and generation. However, the deployment of these massive models is severely constrained by the memory capacity of modern accelerators. The Mixture-of-Experts architecture has emerged as a promising solution to scale model capacity without a proportional increase in computational cost, achieved by sparsely activating only a subset of network parameters for each input token. Despite this computational efficiency, the aggregate memory footprint of all experts remains a critical bottleneck during inference, as traditional serving systems require the entire model to reside in high-bandwidth memory. To address this challenge, we present SysMoE, a novel serving framework designed to execute Mixture-of-Experts models under strictly tunable memory budgets. SysMoE decouples the computational dependency of experts from their memory residence requirements by introducing a predictive routing mechanism combined with an asynchronous, hardware-aware prefetching engine. By forecasting expert assignments in early layers, SysMoE dynamically orchestrates the transfer of required parameters from host memory to accelerator memory, effectively hiding communication latency. We conduct extensive evaluations across various memory constraints, demonstrating that SysMoE can maintain near-optimal throughput even when the available memory is significantly smaller than the total model size. This framework provides a flexible, cost-effective pathway for deploying massive sparse models on resource-constrained hardware environments.

Downloads

Download data is not yet available.

References

Yi, X. (2025, October). Compliance-by-Design Micro-Licensing for AI-Generated Content in Social Commerce Using C2PA Content Credentials and W3C ODRL Policies. In 2025 7th International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI) (pp. 204-208). IEEE.

Norris, J., He, Z., Qu, Y., Chen, G., Hertzog, C., & Jin, D. (2025, September). An in-network approach for pmu missing data recovery with data plane programmability. In 2025 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) (pp. 1-7). IEEE.

CHEN, R., GUb, B., & YEc, Z. (2025). Design and implementation of big data-driven business intelligence analytics system.

Zhang, Y., Liu, J., Wang, J., Dai, L., Guo, F., & Cai, G. (2025, February). Federated learning for cross-domain data privacy: A distributed approach to secure collaboration. In 2025 8th International Symposium on Big Data and Applied Statistics (ISBDAS) (pp. 824-828). IEEE.

Xu, H., Lei, Y., Chen, Z., Zhang, X., Zhao, Y., Wang, Y., & Tu, Z. (2024). Bayesian diffusion models for 3d shape reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10628-10638).

Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Communications of the ACM, 59(5), 50–57.

Tang, Y., Kojima, K., Gotoda, M., Nishikawa, S., Hayashi, S., Koike-Akino, T., ... & Klamkin, J. (2020, February). InP grating coupler design for vertical coupling of InP and silicon chips. In Integrated Optics: Devices, Materials, and Technologies XXIV (Vol. 11283, pp. 33-38). SPIE.

Yi, X. (2025). Federated Incentive Learning: A Privacy-Preserving Framework for Ad Monetization and Creator Rewards in High-Concurrency Environments. American Journal Of Big Data, 6(3), 60-86.

He, Z., Qu, Y., & Jin, D. (2025, June). Real-Time Power System Event Detection on Programmable Network Switches with Synchrophasor Data. In ICC 2025-IEEE International Conference on Communications (pp. 3918-3923). IEEE.

Huang, Y., Xu, J., Badiu, M. A., Chen, G., Coon, J., & Alouini, M. S. (2026). ISAC-Enabled Low-Overhead Beam Management: Performance Analysis and Pilot Optimization. IEEE Transactions on Wireless Communications.

Xu, Y., Sakata, I., & Ishikawa, I. (2025). Data-Driven Spectral Analysis Through Pseudo-Resolvent Koopman Operator in Dynamical Systems. arXiv preprint arXiv:2512.24953.

Fan, D., Ren, Y., Liu, L., & Wang, Z. (2016, January). An approach based on Bayesian networks for GO-FLOW methodology. In 2016 Annual Reliability and Maintainability Symposium (RAMS) (pp. 1-7). IEEE.

Huang, Y., Chen, G., Badiu, M. A., & Coon, J. (2025, June). ISAC-Enabled Channel-Aware Beam Management in THz Networks with Sensing Errors. In 2025 IEEE International Conference on Communications Workshops (ICC Workshops) (pp. 1389-1394). IEEE.

Dai, Y., Guo, Q., & Wang, X. (2026, January). eHashPipe: Lightweight Top-K and Per-PID Resource Monitoring with eBPF. In 2026 IEEE 23rd Consumer Communications & Networking Conference (CCNC) (pp. 1-6). IEEE.

Ren, Y., Gay, G., Kästner, C., & Jamshidi, P. (2020). Understanding the nature of system-related issues in machine learning frameworks: An exploratory study. arXiv preprint arXiv:2005.06091.

Chen, M., Chen, R., Zhao, Y., Yang, Z., & Guan, Y. L. (2023). Index-modulation OAM detectors resistant to beam misalignment. IEEE Transactions on Vehicular Technology, 73(2), 2836-2841.

Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., et al. (2013). Spanner: Google’s globally-distributed database. ACM Transactions on Computer Systems, 31(3), Article 8.

Yang, Y., Lin, Z., & Wei, L. (2025). ACE-Sync: An Adaptive Cloud-Edge Synchronization Framework for Communication-Efficient Large-Scale Distributed Model Training. arXiv preprint arXiv:2512.18127.

Zhou Z, Leng N, Ma H, et al. Study on Real-Time Data Analysis and Intelligent Forecasting Methods for Integrated Circuit Supply Chains Based on Cloud Computing[C]//Proceedings of the 2nd International Symposium on Integrated Circuit Design and Integrated Systems. 2025: 245-250.

DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., et al. (2007). Dynamo: Amazon’s highly available key-value store. In Proceedings of SOSP.

Ma, F., Liu, L., & Cheng, H. V. (2024). TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability. arXiv preprint arXiv:2405.17678.

Yang, C., Wang, Z., Oh, S. K., Pedrycz, W., & Yang, B. (2022). Ensemble fuzzy radial basis function neural networks architecture driven with the aid of multi-optimization through clustering techniques and polynomial-based learning. Fuzzy sets and systems, 438, 62-83.

He, Z., Qu, Y., Chen, G., Raj, R. S., Lin, H., & Jin, D. (2024, April). Towards secure and resilient synchrophasor networks using p4 programmable switches. In 2024 IEEE Green Technologies Conference (GreenTech) (pp. 17-21). IEEE.

Liu, Z.; Huang, J.; Wang, X; Wu,Y.; Gorbatchev, N.EmployeePerformance Prediction:ASystem Based onLightGBM for Digital IntelligentHR Management. InProceedings of the 2025International Conference onIntelligent Computing and NextGeneration Networks (ICNGN)IEEE,2025,pp.1-5

Chen, Z., Jiao, J., De, X., & Fan, D. (2022). Tradeoff Optimization Technology of Effectiveness‐Cost for Satellite‐Based on CAIV Method. Journal of Sensors, 2022(1), 2888846.

Bao, Z., Zhu, D., Jiang, L., Sheng, S., Wang, Z., & Zhang, H. (2025). SLOFetch: Compressed-Hierarchical Instruction Prefetching for Cloud Microservices. arXiv preprint arXiv:2511.04774.

Huang, Y., Zhang, C., & Pan, C. (2022). Spectrum Sensing for DTMB-A System Using Accumulated Autocorrelation. IEEE Access, 10, 113610-113618.

Fan, D., Feng, Q., Zhang, A., Liu, M., Ren, Y., & Wang, Y. (2023). Optimization of scheduling and timetabling for multiple electric bus lines considering nonlinear energy consumption model. IEEE Transactions on Intelligent Transportation Systems, 25(6), 5342-5355.

Ahmad, N. R. (2025). Exploring the role of digital technologies in enhancing supply chain efficiency: A case study of e-commerce companies. Indus Journal of Social Sciences, 3(1), 226–237. https://doi.org/10.59075/ijss.v3i1.618

Downloads

Published

2026-04-15