Hierarchical Context Compression for Efficient Long-Sequence Diffusion Language Modeling

Authors

  • Maya Srinivasan University of Michigan, Computer Science and Engineering, USA Author
  • Lucas Bernal University of Michigan, School of Information, USA Author

DOI:

https://doi.org/10.66525/ijictt105

Keywords:

diffusion language models, non-autoregressive generation, long-context modeling, key–value cache compression, hierarchical memory, efficient inference

Abstract

Diffusion language models (DLMs) generate text by iteratively denoising an entire sequence in parallel, an appealing alternative to token-by-token autoregressive decoding. On long inputs, however, conditioning every denoising step on the full history costs time and memory that grow linearly with sequence length, and recent segmented decoders that compress each completed window into a small, fixed set of summary tokens still incur a total injected context that scales linearly with the number of windows. We propose Hierarchical Context Compression (HCC), which organizes per-window summaries into a suffix-dyadic tree of “summaries of summaries”: recent windows are kept at fine resolution while progressively older windows are covered by recursively merged, coarser summaries. HCC uses only two learned operators—a leaf compressor and a shared binary MERGE—applied through a segment-tree decomposition of the window history. We prove that HCC reduces the active context injected at each denoising step from O(n) to O(log n) summary vectors while still covering the entire history (no window is dropped), and we give a rate–distortion bound predicting graceful rather than abrupt degradation of far-context fidelity. As full-scale DLM training exceeds our single-CPU budget, we validate the mechanism with controlled and real-text information-retention studies and with direct latency and memory measurements. On histories of 64 windows, HCC retains 5–6× more recoverable information about past windows than a fixed recent-window budget of equal asymptotic footprint, and its measured per-step latency is up to 240× lower than uncompressed full-history attention at 4096 windows while remaining essentially flat as the history grows. HCC directly realizes the hierarchical-compression direction identified as future work by recent segmented-decoding methods.

Downloads

Download data is not yet available.

References

Z. Wang, J. S. Yang, W. Shang, and J. Ding, "FairPromote: Explainable and fairness-aware talent promotion prediction via adversarial debiasing and SHAP-based interpretation," IEEE Access, 2026.

[2] F. Zhang, Z. Guo, J. Ding, J. Yang, and W. Liu, "Adaptive sensor fusion for robust perception in dense fog: A gated vision and LiDAR integration framework," Sensors, 2026.

[3] B. Zi, "Large language models for enterprise workflow automation in financial operations," Innovation and Technology Studies, vol. 1, no. 1, pp. 24-29, 2024.

[4] J. Ding, Z. Shen, and W. Liu, "Game-theoretic cost-sensitive adversarial training for robust cloud intrusion detection against GAN-based evasion attacks," Applied Sciences, vol. 16, no. 8, Art. no. 3944, 2026.

[5] B. Zi, "Cloud-native distributed systems for real-time payment intelligence," AI and Data Science Journal, vol. 1, no. 1, pp. 51-56, 2024.

[6] D. Teng, "TEAS: Token- and Energy-Aware Autoscaling for Cost-Efficient LLM Serving," AI and Data Science Journal, 2025.

[7] H. Zhang, "Physics-informed neural networks for high-fidelity electromagnetic field approximation in VLSI and RF EDA applications," Journal of Computing and Electronic Information Management, vol. 18, no. 2, pp. 38-46, 2025.

[8] Y. Jiao, H. Fan, X. Yue, W. Ping, T. Sun, and J. Wang, "Dynamic heterogeneous graph contrastive learning for uncovering collusive financial fraud," Scientific Reports, 2026.

[9] W. Ping, Y. Jiao, H. Fan, and X. Zhang, "Multimodal fraud detection in financial statements: A trimodal attention network with contrastive evidence chain construction," IEEE Access, 2026.

[10] D. Teng, "PACO: Predictive Auto-Configuration for SLO-Constrained Large Language Model Inference Serving," Innovation and Technology Studies, 2025.

[11] J. Gu, J. Bradbury, C. Xiong, V. O. K. Li, and R. Socher, “Non-autoregressive neural machine translation,” in Proc. Int. Conf. Learning Representations (ICLR), 2018.

[12] M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer, “Mask-Predict: Parallel decoding of conditional masked language models,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2019.

[13] J. Gu, C. Wang, and J. Zhao, “Levenshtein transformer,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2019.

[14] R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv:1904.10509, 2019.

[15] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv:2004.05150, 2020.

[16] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, “Big Bird: Transformers for longer sequences,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2020.

[17] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2022.

[18] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, “RoFormer: Enhanced transformer with rotary position embedding,” Neurocomputing, 2024, arXiv:2104.09864.

[19] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” in Proc. Int. Conf. Learning Representations (ICLR), 2024, arXiv:2309.17453.

[20] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen, “H2O: Heavy-hitter oracle for efficient generative inference of large language models,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2023, arXiv:2306.14048.

[21] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen, “SnapKV: LLM knows what you are looking for before generation,” arXiv:2404.14469, 2024.

[22] Z. Cai et al., “PyramidKV: Dynamic KV cache compression based on pyramidal information funneling,” arXiv:2406.02069, 2024.

[23] X. Ma, R. Yu, G. Fang, and X. Wang, “dKV-Cache: The cache for diffusion language models,” arXiv:2505.15781, 2025.

[24] C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie, “Fast-dLLM: Training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding,” arXiv:2505.22618, 2025.

[25] D. Teng, M. Rhee, Y. Qin, B. Zi, and W. Liu, “SW-SpeedDLM: Sliding window speculative decoding for diffusion language models under long context constraints,” Mathematics, vol. 14, no. 12, art. 2137, 2026, doi: 10.3390/math14122137.

[26] J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap, “Compressive transformers for long-range sequence modelling,” in Proc. Int. Conf. Learning Representations (ICLR), 2020.

[27] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira, “Perceiver IO: A general architecture for structured inputs & outputs,” in Proc. Int. Conf. Learning Representations (ICLR), 2022.

[28] J. Mu, X. L. Li, and N. Goodman, “Learning to compress prompts with gist tokens,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2023.

[29] A. Chevalier, A. Wettig, A. Ajith, and D. Chen, “Adapting language models to compress contexts,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2023.

[30] Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in Proc. Int. Conf. Machine Learning (ICML), 2023.

[31] C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” arXiv:2302.01318, 2023.

[32] T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple LLM inference acceleration framework with multiple decoding heads,” arXiv:2401.10774, 2024.

[33] Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE: Speculative sampling requires rethinking feature uncertainty,” in Proc. Int. Conf. Machine Learning (ICML), 2024.

Downloads

Published

2026-07-29