Chatbot Admisi Berbasis LLM dengan Retrieval-Augmented Generation, Ekstraksi Data Prospek, dan Dashboard Analitik
-
Kamal Abrar Ramadhan
*
Universitas Pembangunan Jaya, Banten,
Indonesia
- Augury El Rayeb Universitas Pembangunan Jaya, Banten, Indonesia (*) Corresponding Author
Abstract
New-student admission services face high inquiry volumes during enrollment periods and the limitations of rule-based chatbots in handling natural-language variation. This study develops and audits an admission chatbot based on a large language model that augments each request with the entire Firestore FAQ collection (non-selective full-context augmentation rather than ranked retrieval), supported by prospect-data capture, user feedback, an analytics dashboard, and Gemini-assisted knowledge-base scraping. The audit covers the Flask backend, test files, the production Next.js frontend bundle, and read-only Firestore aggregations that do not expose personal data. A 28 September 2026 snapshot recorded 238 FAQs, 191 chat logs, 9 leads, and 14 feedback records; 9 were helpful and 5 unhelpful, yielding a 64.3% helpfulness rate. The earlier 8/8 result is retained only as a test-dataset result rather than the production knowledge-base size. The average Heuristic Lexical Relevance Score of 0.54 and the Lexical Grounding Indicator range of 0.27–0.80 were derived only from illustrative text and are not treated as validated semantic or hallucination-detection measures. An internal Cache Microbenchmark recorded a 99.2% reduction in serialization overhead rather than end-to-end latency, while the previously reported 90% cache hit rate could not be reproduced. The initial audit reported a suspected static administrative token in the frontend bundle; the developer stated that dashboard access uses Firebase Authentication for registered accounts. Token classification and backend authorization still require code verification. Additional public-interface testing produced relevant responses in four UPJ-related cases and refusals in two out-of-domain cases, limited to six purposively selected cases. Prospect conversion and CSAT cannot yet be calculated because the funnel denominator and a CSAT instrument are unavailable. The main contribution is an implementation-grounded framework that separates production observations, test-dataset results, heuristic proxies, and outcomes still requiring experiments.
Downloads
References
Anwar, R., Pratiwi, H., & Wahyuni, W. (2025). Application of large language model for new student admission chatbot. IJISTECH (International Journal of Information System and Technology), 8(6), 319–325. https://doi.org/10.30645/ijistech.v8i6.379
Chen, J., Lin, H., Han, X., & Sun, L. (2024). Benchmarking large language models in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 17754–17762. https://doi.org/10.1609/aaai.v38i16.29728
Chen, Z., Zou, D., Xie, H., Lou, H., & Pang, Z. (2024). Facilitating university admission using a chatbot based on large language models with retrieval-augmented generation. Educational Technology & Society, 27(4), 454–470. https://doi.org/10.30191/ETS.202410_27(4).TP02
Darmawan, F., Purnama, W. G., & Nurcahyo, A. A. (2025). Prototipe sistem chatbot panduan akademik Fakultas Teknik Unpas menggunakan large language model. Jurnal Sistem dan Informatika (JSI), 19(2), 72–82. https://doi.org/10.30864/jsi.v19i2.733
Efendi, G., Mutawalli, L., & Zaen, M. T. A. (2025). Pengembangan chatbot untuk meningkatkan layanan informasi akademik STMIK Lombok menggunakan large language model. ZONAsi: Jurnal Sistem Informasi, 7(3), 956–969. https://doi.org/10.31849/dtsd0p50
Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150–158). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-demo.16
Hoiriyah, H., Rahman, M. S., Ekawati, F., & Wijaya, Y. I. (2025). Chatbot AI sebagai media pencarian informasi dengan menggunakan metode large language models (LLM). Jurnal Sains Komputer dan Teknologi Informasi, 7(2), 70–76. https://doi.org/10.33084/jsakti.v7i2.9952
Itai, K., Hasegawa, S., Yamamoto, Y., Minegishi, G., & Otsuki, M. (2026). LIT-RAGBench: Benchmarking generator capabilities of large language models in retrieval-augmented generation. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (pp. 5482–5493). European Language Resources Association. https://doi.org/10.63317/3ep7jffajytv
Lee, E., Yulianto, A., & Christian, Y. (2026). Penerapan model Transformer dalam pengembangan chatbot layanan informasi kampus. Jurnal Tekno Kompak, 20(1), 335–347. https://doi.org/10.33365/jtk.v20i1.1144
Li, Z., Li, C., Zhang, M., Mei, Q., & Bendersky, M. (2024). Retrieval augmented generation or long-context LLMs? A comprehensive study and hybrid approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track (pp. 881–893). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-industry.66
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638
Maekawa, S., Iso, H., Gurajada, S., & Bhutani, N. (2024). Retrieval helps or hurts? A deeper dive into the efficacy of retrieval augmentation to language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 5506–5521). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.308
Maryamah, M., Irfani, M. M., Tri Raharjo, E. B., Rahmi, N. A., Ghani, M., & Raharjana, I. K. (2024). Chatbots in academia: A retrieval-augmented generation approach for improved efficient information access. In 2024 16th International Conference on Knowledge and Smart Technology (KST) (pp. 259–264). IEEE. https://doi.org/10.1109/KST61284.2024.10499652
Neumann, A. T., Yin, Y., Sowe, S., Decker, S., & Jarke, M. (2025). An LLM-driven chatbot in higher education for databases and information systems. IEEE Transactions on Education, 68(1), 103–116. https://doi.org/10.1109/TE.2024.3467912
Niu, C., Wu, Y., Zhu, J., Xu, S., Shum, K., Zhong, R., Song, J., & Zhang, T. (2024). RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 10862–10878). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.585
Odede, J., & Frommholz, I. (2024). JayBot – Aiding university students and admission with an LLM-based chatbot. In Proceedings of the 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval (pp. 391–395). Association for Computing Machinery. https://doi.org/10.1145/3627508.3638293
Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45–77. https://doi.org/10.2753/MIS0742-1222240302
Qi, Z., Xu, R., Guo, Z., Wang, C., Zhang, H., & Xu, W. (2024). LONG²RAG: Evaluating long-context & long-form retrieval-augmented generation with key point recall. In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 4852–4872). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-emnlp.279
Rau, D., Déjean, H., Chirkova, N., Formal, T., Wang, S., Clinchant, S., & Nikoulina, V. (2024). BERGEN: A benchmarking library for retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 7640–7663). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-emnlp.449
Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 338–354). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.20
Strich, J., Isgorur, E. K., Trescher, M., Biemann, C., & Semmann, M. (2026). T²-RAGBench: Text-and-table benchmark for evaluating retrieval-augmented generation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 165–191). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.eacl-long.8
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Chatbot Admisi Berbasis LLM dengan Retrieval-Augmented Generation, Ekstraksi Data Prospek, dan Dashboard Analitik
Copyright (c) 2026 Kamal Abrar Ramadhan, Augury El Rayeb
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
