Visual Speech Enhancement GANs Improve Noisy Telemedicine Consultations with Multimodal Lip Cues and Noisy Audio Streams

Authors

  • Adrian Hamilton TUM School of Engineering and Design, Technical University of Munich, Munich, Germany Author
  • Gavin Morgan TUM School of Engineering and Design, Technical University of Munich, Munich, Germany Author
  • Graham Brooks TUM School of Engineering and Design, Technical University of Munich, Munich, Germany Author

DOI:

https://doi.org/10.66525/ijictt103

Keywords:

Telemedicine, Speech Enhancement, Generative Adversarial Networks, Multimodal Processing

Abstract

The rapid expansion of telemedicine has fundamentally altered the landscape of modern healthcare delivery, enabling remote consultations and expanding access to medical professionals. However, the efficacy of virtual medical consultations is frequently compromised by degraded audio quality stemming from unpredictable background noise in patient environments, low-bandwidth network transmission, and substandard consumer hardware. Traditional speech enhancement algorithms rely exclusively on acoustic features, which often fail in low signal-to-noise ratio conditions typical of telemedicine scenarios. To address this critical gap, this paper introduces a novel Visual Speech Enhancement Generative Adversarial Network designed specifically for noisy telemedicine consultation videos. By leveraging the multimodal nature of video consultations, the proposed architecture incorporates visual cues, specifically lip and facial movements, to guide the acoustic enhancement process. The system employs a dual-stream encoder to extract synchronized audio-visual features, followed by a generator network that reconstructs high-fidelity speech spectrograms, and a discriminator network that enforces realistic speech distributions. Comprehensive evaluations using a newly curated dataset of simulated telemedicine interactions demonstrate that the proposed multimodal approach significantly outperforms unimodal audio enhancement techniques across both objective acoustic metrics and subjective clinical intelligibility scores. The integration of visual data acts as a robust anchor during periods of extreme acoustic masking, preserving vital phonetic details required for accurate medical diagnosis. These findings highlight the transformative potential of audio-visual generative models in ensuring reliable, high-quality communication in remote healthcare settings.

Downloads

Download data is not yet available.

References

1. Li, A., Liu, W., Luo, X., Yu, G., Zheng, C., & Li, X. (2021). A simultaneous denoising and dereverberation framework with target decoupling. arXiv preprint arXiv:2106.12743.

2. Zhang, H., Chen, J., Wu, D., Li, Y., Zhang, Y., Zhang, X. T., ... & Tian, F. (2026). Duplexsla: A full-duplex spoken language model with synchronized speech, language, and action. arXiv preprint arXiv:2605.20755.

3. Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., et al. (2016). Deep Speech 2: End-to-end speech recognition in English and Mandarin. In Proceedings of ICML.

4. Xu, X., Li, J., Zhang, Y., Tu, W., & Yang, Y. (2025). Spatial information aided speech and noise feature discrimination for Monaural speech enhancement. Expert Systems with Applications, 269, 126349.

5. Zhang, J., Chen, C., Chen, X., Yu, H., Xiang, T., Khan, A. S., ... & Adeli, E. (2026). Vibes: A conversational agent with behaviorally-intelligent 3d virtual body. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 39994-40008).

6. Xu, X., Tu, W., & Yang, Y. (2023). CASE-Net: Integrating local and non-local attention operations for speech enhancement. Speech Communication, 148, 31-39.

7. Chen, C., Zhang, J., Lakshmikanth, S. K., Fang, Y., Shao, R., Wetzstein, G., ... & Adeli, E. (2025). The language of motion: Unifying verbal and non-verbal language of 3d human motion. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 6200-6211).

8. Chen, L. (2026). Beyond external constraints: The missing dimension of ai governance. Available at SSRN 6449738.

9. Xu, X., Tu, W., Yang, Y., Li, J., & Zhang, Y. (2025). Interactive Target Positive and Negative Features Modeling for Monaural Speech Enhancement. IEEE Transactions on Audio, Speech and Language Processing, 33, 4856-4869.

10. Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). LibriSpeech: An ASR corpus based on public domain audio books. In Proceedings of ICASSP.

11. Xu, X. (2024). Improving monaural speech enhancement by mapping to fixed simulation space with knowledge distillation. IEEE Signal Processing Letters, 31, 386-390.

12. Zhang, H., Liu, H., Zhang, X., Zhang, Q., Hu, Y., Zhao, J., ... & Chng, E. S. (2025). Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English. arXiv preprint arXiv:2505.17076.

13. Xu, Xinmeng, Weiping Tu, and Yuhong Yang. "Pcnn: A lightweight parallel conformer neural network for efficient monaural speech enhancement." arXiv preprint arXiv:2307.15251 (2023).

14. Xu, X., Zhang, Y., Li, J., Yang, Y., Luo, Y., & Tu, W. (2025). FIRING-Net: A filtered feature recycling network for speech enhancement. In The Thirteenth International Conference on Learning Representations.

15. Xu, X., Tu, W., & Yang, Y. (2023, June). Selector-enhancer: learning dynamic selection of local and non-local attention operation for speech enhancement. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 11, pp. 13853-13860).

16. Li, A., Zheng, C., Zhang, L., & Li, X. (2022). Glance and gaze: A collaborative learning framework for single-channel speech enhancement. Applied Acoustics, 187, 108499.

17. Li, A., Lei, T., Sun, Z., Chen, R., Yin, E., Li, X., & Zheng, C. (2025). Learning neural vocoder from range-null space decomposition. arXiv preprint arXiv:2507.20731.

18. Dai, S., Wu, Y., Chen, S., Huang, R., & Dannenberg, R. B. (2023, November). SingStyle111: A Multilingual Singing Dataset With Style Transfer. In ISMIR (pp. 765-773).

19. Xu, X., Wang, Y., Xu, D., Peng, Y., Zhang, C., Jia, J., & Chen, B. (2022, May). Vsegan: Visual speech enhancement generative adversarial network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7308-7311). IEEE.

20. Li, A., Yu, G., Zheng, C., Liu, W., & Li, X. (2023). A general unfolding speech enhancement method motivated by Taylor's theorem. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 3629-3646.

21. Xu, X., Tu, W., & Yang, Y. (2024). Adaptive selection of local and non-local attention mechanisms for speech enhancement. Neural Networks, 174, 106236.

22. Yu, H., Zhang, J., Chen, C., Xiang, T., Fang, Y., Niebles, J. C., & Adeli, E. (2026, March). Socialgen: Modeling multi-human social interaction with language models. In 2026 International Conference on 3D Vision (3DV) (pp. 1-17). IEEE.

23. Xu, X., Tu, W., Yang, Y., Li, J., Zhang, Y., & Chen, H. (2026). Contribution-aware Dynamic Multi-modal Balance for Audio-Visual Speech Separation. IEEE Transactions on Multimedia.

24. Li, A., Liu, W., Zheng, C., & Li, X. (2022, May). Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6487-6491). IEEE.

25. Lv, Z., Wang, Q., Lyu, T., & Dong, J. (2026). A Deep Learning-Based Model for User Emotion Recognition and Behavior Guidance Design: EMoGuideNet-An Emotion-Aware Multimodal Framework for Enhancing Intelligent User Interaction. Journal of Organizational and End User Computing (JOEUC), 38(1), 1-30.

26. Dai, L., Li, A., Chi, C., Liang, Y., Li, X., & Zheng, C. (2026). GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks. arXiv preprint arXiv:2601.13758.

Downloads

Published

2026-06-25