Comparative Evaluation of Local Llama and Mistral Models in RAG for Helpdesk Documentation

Abstract

The increasing volume of technical documentation and repetitive support requests at PT XYZ has made it difficult for helpdesk personnel to retrieve accurate information efficiently and consistently. The 114 internal applications that PT XYZ oversees have different documentation, troubleshooting protocols, and frequently asked questions, which leads to dispersed knowledge sources and drawn-out problem solving. This study uses locally deployed Large Language Models (LLMs), specifically Llama and Mistral, to design and assess a Retrieval-Augmented Generation (RAG) based chatbot for internal helpdesk knowledge retrieval in order to solve this issue. The suggested solution combines generative language models with semantic document retrieval in PostgreSQL using PGVector. The study compares the effectiveness of local LLMs in RAG and non-RAG configurations, employing 1,068 internal question-answer pairs for knowledge retrieval and 214 evaluation questions for performance assessment. ROUGE, BLEU, and Cosine Similarity measures are used for evaluation. According to experimental findings, RAG greatly enhances both models' performance. While Mistral improved from 0.0984 to 0.8983, Llama ROUGE-1 score rose from 0.1710 to 0.7345. With a ROUGE-1 score of 0.8983, a BLEU-1 score of 0.8161, and a Cosine Similarity score of 0.8916, Mistral with RAG performed the best of all setups. These results show that integrating RAG with locally installed LLMs greatly improves contextual correctness and response relevance, enabling safe and scalable implementation for enterprise helpdesk knowledge management.