CRAI: Calibrated Regret-Aware LLM Invocation for Efficient Cloud Resource Scheduling

Authors

  • Aiman Rahman Faculty of Computer Science and Information Technology, Universiti Malaya, Kuala Lumpur, Malaysia Author
  • Nur Aisyah Karim Faculty of Computer Science and Information Technology, Universiti Malaya, Kuala Lumpur, Malaysia Author

DOI:

https://doi.org/10.66525/ijictt135

Keywords:

cloud resource scheduling, large language models, adaptive invocation, online learning, budgeted control, service-level agreements

Abstract

Large language models can contribute semantic and strategic reasoning to cloud schedulers, but invoking a large model at every scheduling epoch converts inference latency and monetary cost into a control-plane bottleneck. This paper studies the invocation decision itself. We present CRAI, a calibrated regret-aware invocation controller that treats an LLM as an expensive strategic meta-scheduler and calls it only when the estimated loss of reusing the cached policy is likely to exceed a budget-dependent dual price. CRAI combines a compact cluster-state representation, an online ridge predictor with an uncertainty bonus, and dual ascent that tracks a target invocation budget. The LLM-selected policy is never executed directly: a deterministic scheduler enforces CPU and memory feasibility while realizing one of four policy profiles—BALANCE, PACK, EDF, or LOCALITY. We evaluate the controller in a 48-node discrete-event simulator under normal, high-load, and bursty workloads using five independent seeds. Planner outputs are frozen across all runs to isolate invocation timing from provider-side nondeterminism. CRAI invokes the strategic planner for 9.54% of active epochs, a 90.46% reduction relative to always invoking, while average job completion time remains within 1.35% of the always-invoke reference. Against a static policy, CRAI reduces average JCT by 1.46% and SLA misses by 1.08 percentage points; against a state-change trigger, it reduces calls by 79.8% and JCT by 0.30 s. These results show that cloud schedulers can capture most of the value of strategic reasoning without paying its inference cost on routine epochs.

 

Downloads

Download data is not yet available.

References

[1] Yue, X., & Zhu, R. (2024). An explainable hybrid machine learning framework for cash-flow-conditioned multi-horizon liquidity risk early warning: An SME-oriented benchmark study. World Journal of Management Science, 2(1), 63-72.

[2] Jiao, Y., & Liang, Y. (2024). An integrated process mining and explainable machine learning framework for bottleneck and internal control risk detection in the multi entity financial close. Social Science and Management, 1(3), 96-107.

[3] Bian, M., Wang, Y., & Lin, Y. (2024). Explainable supplier risk prediction for critical telecommunications infrastructure procurement: An empirical XGBoost–SHAP framework. Social Science and Management, 1(1), 64-77.

[4] Liang, Y., & Jiao, Y. (2024). Real-time financial fraud detection using a feature-light temporal dynamic graph neural network with contrastive learning. Journal of Trends in Finance and Economics, 1(1), 76–84.

[5] Shang, W., & Chen, T. (2024). PrefixKD-Edge: Real-time locomotion-intent prediction with prefix self-distillation on lightweight temporal networks. Innovation and Technology Studies, 1(1), 30–38.

[6] Rhee, M., Zou, J., & Qin, Y. (2024). CLARITY: A closed loop framework for duplicate aware failure prediction and budgeted automated remediation in cloud infrastructure. AI and Data Science Journal, 1(1), 76-85.

[7] M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes, “Omega: Flexible, Scalable Schedulers for Large Compute Clusters,” in Proc. EuroSys, 2013, pp. 351–364, doi: 10.1145/2465351.2465386.

[8] B. Hindman et al., “Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center,” in Proc. NSDI, 2011.

[9] V. K. Vavilapalli et al., “Apache Hadoop YARN: Yet Another Resource Negotiator,” in Proc. ACM SoCC, 2013, Art. 5, doi: 10.1145/2523616.2523633.

[10] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, Omega, and Kubernetes,” Commun. ACM, vol. 59, no. 5, pp. 50–57, 2016, doi: 10.1145/2890784.

[11] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica, “Dominant Resource Fairness: Fair Allocation of Multiple Resource Types,” in Proc. NSDI, 2011.

[12] R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella, “Multi-Resource Packing for Cluster Schedulers,” in Proc. ACM SIGCOMM, 2014, pp. 455–466, doi: 10.1145/2619239.2626334.

[13] I. Gog, M. Schwarzkopf, A. Gleave, R. N. M. Watson, and S. Hand, “Firmament: Fast, Centralized Cluster Scheduling at Scale,” in Proc. OSDI, 2016, pp. 99–115.

[14] K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica, “Sparrow: Distributed, Low Latency Scheduling,” in Proc. SOSP, 2013, pp. 69–84, doi: 10.1145/2517349.2522716.

[15] C. Delimitrou and C. Kozyrakis, “Quasar: Resource-Efficient and QoS-Aware Cluster Management,” in Proc. ASPLOS, 2014, pp. 127–144, doi: 10.1145/2541940.2541941.

[16] G. Ding, S. Yang, H. Lin, Z. Chen, and J. S. Yang, “LLM-Driven Adaptive Cloud Resource Scheduling: Bridging Reasoning Intelligence With Optimization Guarantees,” IEEE Open Journal of the Computer Society, vol. 7, pp. 560–573, 2026, doi: 10.1109/OJCS.2026.3667549.

[17] E. Cortez, A. Bonde, A. Muzio, M. Russinovich, M. Fontoura, and R. Bianchini, “Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms,” in Proc. SOSP, 2017, pp. 153–167, doi: 10.1145/3132747.3132772.

[18] R. Grandl, M. Chowdhury, A. Akella, and G. Ananthanarayanan, “Altruistic Scheduling in Multi-Resource Clusters,” in Proc. OSDI, 2016, pp. 65–80.

[19] C. Reiss, A. Tumanov, G. R. Ganger, R. H. Katz, and M. A. Kozuch, “Heterogeneity and Dynamicity of Clouds at Scale: Google Trace Analysis,” in Proc. ACM SoCC, 2012, Art. 7, pp. 1–13, doi: 10.1145/2391229.2391236.

[20] M. Tirmazi et al., “Borg: The Next Generation,” in Proc. EuroSys, 2020, Art. 30, pp. 1–14, doi: 10.1145/3342195.3387517.

[21] R. Bhardwaj et al., “Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback,” in Proc. OSDI, 2023, pp. 623–643.

[22] J. Gu et al., “Tiresias: A GPU Cluster Manager for Distributed Deep Learning,” in Proc. NSDI, 2019, pp. 485–500.

[23] Q. Weng et al., “MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters,” in Proc. NSDI, 2022, pp. 945–960.

[24] T. Brown et al., “Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020.

[25] Fang, Q., & Liu, W. (2025). HARLA-ED: Resolving Information Asymmetry and Enhancing Algorithmic Symmetry in Intelligent Educational Assessment via Hybrid Reinforcement Learning. Symmetry, 18(1), 58.

[26] Mai, N. T., Cao, W., & Fang, Q. (2025). A study on how LLMs (eg GPT-4, chatbots) are being integrated to support tutoring, essay feedback and content generation. Journal of Computing and Electronic Information Management, 18(3), 43-52.

[27] Wang, Y., & Xing, S. (2025). AI-driven CPU resource management in cloud operating systems. Journal of Computer and Communications, 13(06), 135-149.

[28] Xing, S., & Wang, Y. (2025). Cross-modal attention networks for multi-modal anomaly detection in system software. IEEE Open Journal of the Computer Society.

[29] Chen, T., Shang, W., & Guo, Z. (2024). Xsla: An SLA-aware heterogeneous XPU scheduling framework for edge AI inference. World Journal of Information Technology, 2(2), 48–59.

[30] Qin, Y., Zou, J., & Rhee, M. (2022). An AI-driven intelligent transaction execution framework for internet finance platforms: Cost-sensitive real-time integrity interdiction. Eurasia Journal of Science and Technology, 4(2), 36–44.

[31] Wang, Y., & Zhang, C. (2023). AERIS: Adaptive executor scoped resource management for privacy preserving data processing in Apache Spark. Eurasia Journal of Science and Technology, 5(2), 25-33.

[32] Fang, Q. (2024). True-EWS: Trustworthy early warning with equalized conformal guarantees for equitable student support in U.S. education. Educational Research and Human Development, 1(2), 44–54.

Downloads

Published

2026-09-24