Performance Evaluation and Error Analysis of Retrieval Augmented Generation for Specialized Question Answering

Authors

  • Robert Harrison Department of Computer Science, ETH Zurich, Zurich, Switzerland Author
  • Amelia Rossi Department of Computer Science, ETH Zurich, Zurich, Switzerland Author
  • Thomas Mitchell Department of Computer Science, ETH Zurich, Zurich, Switzerland Author
  • Samantha Wright Department of Computer Science, ETH Zurich, Zurich, Switzerland Author

DOI:

https://doi.org/10.66525/ijictt78

Keywords:

retrieval augmented generation, question answering, error analysis, performance evaluation

Abstract

The rapid advancement of large language models has fundamentally transformed the landscape of natural language processing, particularly in the domain of question answering. Despite their impressive generative capabilities, these models frequently suffer from factual hallucinations and struggle to incorporate domain specific or temporally updated information. Retrieval augmented generation has emerged as a predominant architectural paradigm to mitigate these deficiencies by grounding the generative process in external knowledge bases. However, evaluating the performance and understanding the specific failure modes of retrieval augmented generation systems in specialized, high stakes domains remains a complex challenge. This paper presents a comprehensive performance evaluation and error analysis of retrieval augmented generation architectures tailored for specialized question answering. By systematically dissecting the system pipeline into retrieval, synthesis, and generation components, we establish a robust analytical framework for identifying and categorizing errors. We investigate the impact of various indexing strategies, retrieval algorithms, and prompt engineering techniques on overall system fidelity. Furthermore, we introduce a detailed error taxonomy that distinguishes between retrieval failures, context misalignment, and generation hallucinations. Our empirical findings demonstrate that while retrieval augmentation significantly enhances factual grounding, the interplay between retrieved noise and language model priors often introduces novel error modalities. This research provides critical insights into the optimization of retrieval augmented systems, offering actionable guidelines for deploying reliable question answering frameworks in specialized academic and professional contexts.

Downloads

Download data is not yet available.

References

Zhang, Y., Carvalho, D., & Freitas, A. (2025, July). Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder. In Proceedings of the 29th Conference on Computational Natural Language Learning (pp. 12-29).

Lou, Y., Liu, J., Sheng, Y., Wang, J., Zhang, Y., & Ren, Y. (2025, March). Addressing class imbalance with probabilistic graphical models and variational inference. In 2025 5th International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA) (pp. 1238-1242). IEEE.

Zeng, D., Yang, Y., Tang, Y., Zhao, L., Wang, X., Yun, D., ... & Lin, H. (2025). Shaping school for childhood myopia: the association between floor area ratio of school environment and myopia in China. British Journal of Ophthalmology, 109(1), 146-151.

Xie, C. (2026). Quantifying the Interplay Between Panic Propagation and Misinformation on Social Media Using Large Language Models. Frontiers in Artificial Intelligence Research, 3(1), 1-8.

Zhang, W., Zhang, C., Gu, C., Kou, J., Yuan, H., Fang, X., ... & Fang, Y. (2024, October). Hallucination in Large Language Models: From Mechanistic Understanding to Novel Control Frameworks. In 2024 7th International Conference on Universal Village (UV) (pp. 1-36). IEEE.

Zhang, K., Shuang, K., Yang, X., Yao, X., & Guo, J. (2023, July). What is overlap knowledge in event argument extraction? APE: A cross-datasets transfer learning model for EAE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 393-409).

Zhang, C., Jiang, X., & Zhao, Y. (2021). Efficient instantaneous channel propagation modeling for aeronautical communications systems with compressed sensing. IEEE Transactions on Antennas and Propagation, 70(2), 1211-1220.

Du, C., Chiang, C. E., Huang, T., & Cui, Z. (2025, September). Adaptive Graph Convolution and Semantic-Guided Attention for Multimodal Risk Detection in Social Networks. In 2025 5th International Conference on Artificial Intelligence, Automation and High Performance Computing (AIAHPC) (pp. 507-512). IEEE.

Li, B., Gu, B., & Ding, Z. (2025). LLM-based Personalized Portfolio Recommender: Integrating Large Language Models and Reinforcement Learning for Intelligent Investment Strategy Optimization. arXiv preprint arXiv:2512.12922.

Liu, F., Geng, K., & Chen, F. (2025). Gone with the wind? Impacts of hurricanes on college enrollment and completion. Journal of Environmental Economics and Management, 133, 103203.

Wang, S., Yu, Y., Feldt, R., & Parthasarathy, D. (2025). Automating a complete software test process using llms: An automotive case study. arXiv preprint arXiv:2502.04008.

Tang, Y., Wang, Y., Zhang, R., & Liu, J. (2024). Linguistic steganalysis via llms: Two modes for efficient detection of strongly concealed stego. IEEE Signal Processing Letters, 32, 541-545.

Zhang, Y., Liu, J., Wang, J., Dai, L., Guo, F., & Cai, G. (2025, February). Federated learning for cross-domain data privacy: A distributed approach to secure collaboration. In 2025 8th International Symposium on Big Data and Applied Statistics (ISBDAS) (pp. 824-828). IEEE.

Vinyals, O., Fortunato, M., & Jaitly, N. (2015). Pointer networks. In Advances in Neural Information Processing Systems.

Yifan, O. U. (2018). Participating in Chinese Social Question and Answer Communities: A Case Study of Zhihu. com.

Ou, Y., de Bruijn, G. J., & Schulz, P. J. (2025). Social media as an emotional barometer: Bidirectional encoder representations from transformers–long short-term memory sentiment analysis on the evolution of public sentiments during Influenza A on Sina Weibo. Journal of Medical Internet Research, 27, e68205.

Zhang, G., Zeng, H., & Jiang, L. (2026). Uni-FinLLM: A Unified Multimodal Large Language Model with Modular Task Heads for Micro-Level Stock Prediction and Macro-Level Systemic Risk Assessment. arXiv preprint arXiv:2601.02677.

Ling, Y., & Liu, W. (2026). SR-DTMA: A Digital Twin-Driven LLM Multi-Agent Framework for Systemic Risk Simulation and Coordinated Decision-Making in Supply Chains. Journal of Computer, Signal, and System Research, 3(1), 123-132.

Tu, P., Huang, Y., Zheng, F., He, Z., Cao, L., & Shao, L. (2022, June). Guidedmix-net: Semi-supervised semantic segmentation by using labeled images as reference. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 2, pp. 2379-2387).

Huang, T., Cui, Z., Du, C., & Chiang, C. E. (2025, June). CL-ISR: A Contrastive Learning and Implicit Stance Reasoning Framework for Misleading Text Detection on Social Media. In 2025 6th International Conference on Electronic Communication and Artificial Intelligence (ICECAI) (pp. 610-616). IEEE.

Shan, B., Fei, X., Shi, W., Wang, A. L., Tang, G., Liao, L., ... & Huang, C. (2024). Mctbench: Multimodal cognition towards text-rich visual scenes benchmark. arXiv preprint arXiv:2410.11538.

Cui, Z., Huang, T., Chiang, C. E., & Du, C. (2025, August). Toward verifiable misinformation detection: A multi-tool LLM agent framework. In Proceedings of the 2025 International Conference on Generative Artificial Intelligence for Business (pp. 179-185).

Fan, D., Sun, B., Dui, H., Zhong, J., Wang, Z., Ren, Y., & Wang, Z. (2022). A modified connectivity link addition strategy to improve the resilience of multiplex networks against attacks. Reliability Engineering & System Safety, 221, 108294.

Hong, J., & Ma, H. (2025). Research on an Automated Data Insight Generation Method Based on Large Language Models. Journal of Industrial Engineering and Applied Science, 3(6), 6-12.

Guo, Y., Sekiguchi, Y., Zeng, W., Ebihara, S., Owaki, D., & Hayashibe, M. (2025). Physics-informed learning framework for lower limb kinematic prediction with sparse sensors and its application in chronic stroke. IEEE Transactions on Neural Systems and Rehabilitation Engineering.

Guo, J., Shuang, K., Li, J., Wang, Z., & Liu, Y. (2022, May). Beyond the granularity: Multi-perspective dialogue collaborative selection for dialogue state tracking. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2320-2332).

Vuruma, S. K. R., Wu, D., Gupta, S. S., Aust, L., Lookingbill, V., Henry, C., ... & Huang, M. (2024). Utilizing large language models to identify reddit users considering vaping cessation for digital interventions. arXiv preprint arXiv:2404.17607.

Li, L., Wang, Y., Fan, J., Li, J., Qin, S., Wen, Q., & Gao, F. (2025). Quantum knowledge distillation for large language models. arXiv preprint arXiv:2505.13205.

Li, K., Shuang, K., & Guo, J. (2026). Bring order to the jumbled input: Layered attention for Large Language Model based Event Argument Extraction. Engineering Applications of Artificial Intelligence, 165, 113399.

Gao, Z., Qu, Y., & Han, Y. (2025). Cross-Lingual Sponsored Search via Dual-Encoder and Graph Neural Networks for Context-Aware Query Translation in Advertising Platforms. arXiv preprint arXiv:2510.22957.

Tang, Y., Kojima, K., Gotoda, M., Nishikawa, S., Hayashi, S., Koike-Akino, T., ... & Klamkin, J. (2020). Design and Optimization of Shallow-Angle Grating Coupler for Vertical Emission from Indium Phosphide Devices.

Ahmad, N. R. (2025). Exploring the role of digital technologies in enhancing supply chain efficiency: A case study of e-commerce companies. Indus Journal of Social Sciences, 3(1), 226–237. https://doi.org/10.59075/ijss.v3i1.618

Downloads

Published

2026-04-05