Alias-Controlled Spectral Sparsification for Million-Token Sequence Models

Authors

  • Ji-Ho Koo Department of Electrical Engineering, School of Electronic and Electrical Engineering, Kyungpook National University, Daegu, Republic of Korea Author
  • Ye-Jun Hwang Department of Electrical Engineering, School of Electronic and Electrical Engineering, Kyungpook National University, Daegu, Republic of Korea Author

DOI:

https://doi.org/10.66525/ijictt128

Keywords:

Sequence Modeling, Spectral Sparsification, Self-Attention, Graph Signal Processing, Anti-Aliasing

Abstract

The rapid evolution of large language models has driven the demand for massively extended context windows, culminating in the pursuit of million-token sequence modeling capabilities. However, scaling the self-attention mechanism to such unprecedented lengths introduces prohibitive computational and memory bottlenecks due to its inherent quadratic complexity. While various sparsification and low-rank approximation techniques have been proposed, they frequently suffer from severe information loss and structural degradation when applied to extremely long sequences. This paper introduces a novel framework termed Alias-Controlled Spectral Sparsification, which leverages principles from graph signal processing to selectively prune attention graphs in the spectral domain. By rigorously analyzing the aliasing artifacts that emerge during aggressive token downsampling, we develop a specialized anti-aliasing filter that preserves the fundamental low-frequency semantic structures while systematically attenuating high-frequency noise. Our methodology enables the compression of million-token sequences into highly tractable spectral representations without sacrificing critical contextual dependencies. Extensive theoretical analysis and empirical evaluations on long-document benchmarks demonstrate that our approach significantly outperforms existing sparse attention mechanisms in both perplexity retention and hardware efficiency. The proposed architecture achieves near-linear time complexity, offering a scalable and robust solution for next-generation sequence models processing ultra-long contexts.

Downloads

Download data is not yet available.

References

1. Zhao, R., Tang, J., Zeng, W., Guo, Y., & Zhao, X. (2025). Towards human-like questioning: Knowledge base question generation with bias-corrected reinforcement learning from human feedback. Information Processing & Management, 62(3), 104044.

2. Zhao, R., Xu, D., Jian, S., Tan, T., Sun, X., & Zhang, W. (2023, July). Quadratic Exponential Decrease Roll-Back: An Efficient Gradient Update Mechanism in Proximal Policy Optimization. In 2023 2nd International Conference on Machine Learning, Cloud Computing and Intelligent Mining (MLCCIM) (pp. 65-70). IEEE.

3. Zhang, J., Shi, Y., Ma, Y., Xu, L., Yu, J., & Wang, J. (2023, June). Ikol: Inverse kinematics optimization layer for 3d human pose and shape estimation via gauss-newton differentiation. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 3, pp. 3454-3462).

4. Liu, Y., Cai, Q., Wang, C., Yang, J., Fan, H., Dong, J., & Chen, S. (2024). Geometry-enhanced attentive multi-view stereo for challenging matching scenarios. IEEE Transactions on Circuits and Systems for Video Technology, 34(8), 7401-7416.

5. Sun, L., Hu, J., Li, M., & Peng, H. (2024, July). R-ode: Ricci curvature tells when you will be informed. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 2594-2598).

6. Wang, Y., Zhang, S., Ye, J., Peng, H., & Sun, L. (2024, October). A mixed-curvature graph diffusion model. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (pp. 2482-2492).

7. Wang, Y., Wang, L., Li, V., & Tu, Z. (2020, November). On the sparsity of neural machine translation models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1060-1066).

8. Gu, C., Zhang, W., Huang, Z., Kou, J., Liu, Z., Zhao, C., Liu, C., Zhang, L., Lin, W., Wang, Z., Deng, J., Xie, Y., Huang, G., Zhang, C., Lu, X., Wang, C., Zhang, Z., Yuan, H., Duan, X., & Fang, Y. (2024). LENS: Layers of evaluation of hallucination in GenAI systems. In 2024 7th International Conference on Universal Village (UV) (pp. 1–85). IEEE. https://doi.org/10.1109/UV63228.2024.11189150

9. Chiu, P. W., Yip, H. C., Xia, X. F., Chan, S. M., Ng, E. K., & Lau, J. Y. (2019). How I do it: Flexible 3‐D endoscope for endoscopic submucosal dissection. Digestive Endoscopy, 31(3), 323-328.

10. Zhang, Z., Yin, Z., Wen, J., Sun, L., Su, S., & Yu, P. S. (2021). DeepBlue: Bi-layered LSTM for tweet popularity estimation. IEEE Transactions on Knowledge and Data Engineering, 34(10), 4737-4752.

11. Huang, Y., Roth, K., Bouniot, Q., Xu, W., & Akata, Z. (2026). Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers. arXiv preprint arXiv:2601.20796.

12. Lu, Y., Cao, C., Zhang, K., Zhang, L., Xue, X., & Tan, K. C. (2026, March). Exploring the Viability of Multi-Paradigm Transfer Optimization. In2026 6th International Symposium on Intelligent Robotics and Systems (ISoIRS)(pp. 1-5). IEEE.

13. Du, X., Chan, K. F., Xia, X., Chiu, P. W. Y., & Zhang, L. (2018, October). A magnetically-triggered soft capsule for on-demand mucus collection. In 2018 IEEE International Conference on Cyborg and Bionic Systems (CBS) (pp. 250-254). IEEE.

14. Wang, X., Suzuki, Y., Li, C. M., & Tanaka, S. (2024). Mechanical quality factor evaluation of damping film materials for polymer/PZT composite MEMS actuator. Journal of Microelectromechanical Systems, 33(5), 631-639.

15. Xue, X., Yang, C., Feng, L., Zhang, K., Song, L., & Tan, K. C. (2025). A scalable test problem generator for sequential transfer optimization.IEEE Transactions on Cybernetics,55(5), 2110-2123.

16. Xue, X., Hu, Y., Feng, L., Zhang, K., Song, L., & Tan, K. C. (2024). Surrogate-assisted search with competitive knowledge transfer for expensive optimization. IEEE Transactions on Evolutionary Computation.

17. Liang, R., Xu, S., Xie, C., Chen, J., Ren, F., Yang, S., & Yabe, T. (2026). Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency. Advances in Neural Information Processing Systems, 38, 99445-99471.

18. Hu, H., Yang, Y., Zhao, Q., & Zhang, C. (2022, June). On the role of discount factor in offline reinforcement learning. In International conference on machine learning (pp. 9072-9098). PMLR.

19. Lin, C. C., Li, G., Hung, C. C., Kuo, Y. C., & Peng, W. C. (2026). Rethinking suspicious account identification from a perspective of machine learning ranking. International Journal of Machine Learning and Cybernetics, 17(3), 84.

20. Xiaoteng Ma, Yiqin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and

Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint

arXiv:2110.09796, 2021b.

21. Zhang, T., Xu, J., Chen, J., Li, J., Xiao, Y., & Wang, Q. (2026). NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning. arXiv preprint arXiv:2608.04381.

22. Lin, Y. (2024). Design of urban road fault detection system based on artificial neural network and deep learning. Frontiers in neuroscience, 18, 1369832.

23. Chai, L., Zhang, J., Li, H., Cui, C., Jiang, J., Zheng, B., ... & Jiang, L. (2021). Investigation of thermomorphogenesis-related genes for a multi-silique trait in Brassica napus by comparative transcriptome analysis. Frontiers in Genetics, 12, 678804.

24. Yu, A., Huang, Y., & Xia, L. (2023). Efficient characterization of phase modulator based on Lyot-Sagnac interferometer. Measurement, 210, 112538.

25. Xia, Y., & Ma, J. (2022). Locality-guided global-preserving optimization for robust feature matching. IEEE Transactions on Image Processing, 31, 5093-5108.

26. Xia, Y., & Ma, J. (2025). Grid-guided sparse Laplacian consensus for robust feature matching. IEEE Transactions on Image Processing, 34, 1367-1381.

27. Zhang, Y., Qiu, C., Liao, X., & Zhang, Y. (2025). Federated hierarchical clustering with automatic selection of optimal cluster numbers. Information Sciences, 122957.

28. Lyu, M., Torii, R., Liang, C., Peach, T. W., Bhogal, P., Makalanda, L., ... & Chen, D. (2024). Treatment for middle cerebral artery bifurcation aneurysms: in silico comparison of the novel Contour device and conventional flow-diverters. Biomechanics and Modeling in Mechanobiology, 23(4), 1149-1160.

29. Xing, Z. (2026, May). Design of High-Bandwidth DDR5 MRDIMM Memory Modules for AI Servers. In 2026 IEEE 6th International Conference on Electronic Technology, Communication and Information (ICETCI) (pp. 307-311). IEEE.

30. Zou, R., Zhang, Y., Zhao, M., Tan, Z., Zhang, Y., & Cheung, Y. M. (2025). SDENK: Unbiased Subspace Density--Clustering. Neurocomputing, 131225.

31. Wang, T., Deng, X., Sheng, S., & Jia, Y. (2024). Mitigating the damage of a global pandemic on the international buyer-supplier relationship: evidence from Chinese suppliers. Journal of Business Research, 172, 114446.

32. Cao, Y., Shi, D., Fu, X., Zou, X., Peng, H., Li, X., ... & Xing, J. (2026, March). Multivariate diffusion transformer with decoupled attention for high-fidelity mask-text collaborative facial generation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 4, pp. 2670-2679).

33. Meng, L., Du, Z., Jia, Z., & Cao, L. (2025). Spatial mechanisms and coupling coordination of cultural heritage and tourism along the Jinzhong segment of the Great Tea Road. Heritage, 9(1), 7.

34. Tian, Y. (2025). Exploration of power market dynamic scheduling algorithm based on reinforcement learning. In Image Processing, Electronics and Computers (pp. 836-845). IOS Press.

35. Li, C., Yu, A., Tian, J., Xiang, P., Zhao, Y., Zuo, Y., ... & Xia, L. (2024). Re-entry fiber optic current sensor based on continuous light source. Journal of Lightwave Technology, 43(1), 390-396.

36. Shi, J., Lin, Y., Hua, Y., Wang, Z., Zhang, Z., Zheng, W., ... & Lu, S. (2026, March). Multiscenario highway lane-change intention prediction: a physics-informed AI framework for three-class classification. In International Conference on Smart Transportation and City Engineering (STCE 2025) (Vol. 14120, pp. 129-145). SPIE.

37. Chen, Y., Zhong, Y., Wang, Z., Shi, J., & Zheng, W. (2026). Applying the Deep Learning–Sector–Governance (DLSG) Framework to the US Healthcare System: Opportunities, Deployment Pathways, and Policy‑Aligned Evaluation. Journal of Technology Innovation and Society, 4(1), 42-59.

38. Shi, J., Xie, Q., Wang, Z., Lin, Y., Zhu, D., Xie, C., ... & Guan, Z. (2026, June). Evolutionary Physics-Informed Temporal Fusion for Lane-Change Intention Prediction. In 2026 6th International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) (pp. 750-764). IEEE.

39. Wang, Y. (2025, June). RAGNet: Transformer–GNN–Enhanced Cox–Logistic Hybrid Model for Rheumatoid Arthritis Risk Prediction. In Proceedings of the 2025 International Conference on Health Informatization and Data Analytics (pp. 90-94).

40. Ling, C., & Wang, Y. (2025). TLFQC: A High-compatible R Shiny based Platform for Automated and Codeless TLFs Generation and Validation. In PharmaSUG 2025 conference proceedings.

Downloads

Published

2026-08-15