Analisis Perbandingan Klasifikasi Intent Chatbot Menggunakan Deep Learning BERT, RoBERTa, dan IndoBERT
-
Aswin Dwiyono
*
Universitas Sriwijaya, Palembang,
Indonesia
- Abdiansah Abdiansah Universitas Sriwijaya, Palembang, Indonesia
- Muhammad Fachrurrozi Universitas Sriwijaya, Palembang, Indonesia (*) Corresponding Author
Abstract
A chatbot is a software application to designed handle user inputs and generate appropriate replies based on those inputs, which are then communicated back to the user. In able to provide accurate responses, the chatbot must be able to understand the intent of the user accurately. An issue in the development of chatbots is how to accurate classify user intent. Incorrectly understanding user intent can result in irrelevant responses. In order to have a conversation with the user, the intent of the user needs to be classified correctly. This paper compares three state-of-the-art transformer-based models BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pretraining Approach), and IndoBERT (Indonesia Bidirectional Encoder Representations from Transformer) for the task of intent classification in chatbot systems. Various performance metrics, including accuracy, F1-score, precision, and recall, were analyzed to determine which model performs more effectively in the same parameter conditions. Performance metrics like accuracy and F1-score were compared to assess model BERT, RoBERTa and IndoBERT performs better in a University Chatbot Dataset in Indonesian language. The BERT model achieved an accuracy of 0.89, RoBERTa model achieved 0.84 and IndoBERT model achieved an accuracy of 0.94. The better performance of IndoBERT compared to BERT and RoBERTa is caused by more language-specific training, more relevant pretraining, and more effective adaptation to Indonesian context and structure.
Downloads
References
Rohim and Zuliarso, “Penerapan Algoritma Deep Learning Untuk Pengembangan Chatbot Yang Digunakan Untuk Konsultasi Dan Pengenalan Tentang Virus Covid-19,” PIXEL, vol. 15, no. 2, pp. 267–278, Dec. 2022, doi: 10.51903/pixel.v15i2.777.
R. C. Hutama, F. Fauziah, and R. T. Komalasari, “Aplikasi Chatbot Berbasis Teks Menggunakan Algoritma Naive Bayes Classifier FAQ GrabAds,” STRING, vol. 6, no. 1, p. 90, Aug. 2021, doi: 10.30998/string.v6i1.9919.
N. Shahin and L. Ismail, “From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation,” 2024, arXiv. doi: 10.48550/ARXIV.2408.14825.
L. Villa, D. Carneros-Prado, A. Sánchez-Miguel, C. C. Dobrescu, and R. Hervás, “Conversational Agent Development Through Large Language Models: Approach with GPT,” in Proceedings of the 15th International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2023), vol. 835, J. Bravo and G. Urzáiz, Eds., in Lecture Notes in Networks and Systems, vol. 835. , Cham: Springer Nature Switzerland, 2023, pp. 286–297. doi: 10.1007/978-3-031-48306-6_29.
D. Griol, Z. Callejas, J. M. Molina, and A. Sanchis, “Adaptive dialogue management using intent clustering and fuzzy rules,” Expert Systems, vol. 38, no. 1, p. e12630, Jan. 2021, doi: 10.1111/exsy.12630.
W. Maeng and J. Lee, “Designing a Chatbot for Survivors of Sexual Violence: Exploratory Study for Hybrid Approach Combining Rule-based Chatbot and ML-based Chatbot,” in Asian CHI Symposium 2021, Yokohama Japan: ACM, May 2021, pp. 160–166. doi: 10.1145/3429360.3468203.
A. Birim and M. Erden, “Robustness to Spelling Errors for Intent Detection,” in 2022 30th Signal Processing and Communications Applications Conference (SIU), Safranbolu, Turkey: IEEE, May 2022, pp. 1–4. doi: 10.1109/SIU55565.2022.9864722.
J. Liu, Y. Li, and M. Lin, “Review of Intent Detection Methods in the Human-Machine Dialogue System,” J. Phys.: Conf. Ser., vol. 1267, no. 1, p. 012059, Jul. 2019, doi: 10.1088/1742-6596/1267/1/012059.
R. A. Sanjaya and E. Winarno, “Pengembangan Chatbot Informasi Pariwisata di Kabupaten Pati Menggunakan Metode Natural Language Processing Berbasis Dialogflow,” Jutisi J. Tek. Sis. Info, vol. 13, no. 1, p. 368, Apr. 2024, doi: 10.35889/jutisi.v13i1.1828.
F. Fatharani, K. P. Kania, J. Hutahaean, and S. R. Wulan, “Deteksi Intensi Chatbot Berbahasa Indonesia dengan Menggunakan Metode Capsule Network,” josh, vol. 3, no. 4, pp. 590–596, Jul. 2022, doi: 10.47065/josh.v3i4.1821.
N. Boudjani, V. Colas, and A. Fotouhi, “Intent Classification: French Recruitment Chatbot Use Case,” in 2023 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA: IEEE, Dec. 2023, pp. 681–685. doi: 10.1109/CSCI62032.2023.00117.
J.-H. Lee, E. H.-K. Wu, Y.-Y. Ou, Y.-C. Lee, C.-H. Lee, and C.-R. Chung, “Anti-Drugs Chatbot: Chinese BERT-Based Cognitive Intent Analysis,” IEEE Trans. Comput. Soc. Syst., vol. 11, no. 1, pp. 514–521, Feb. 2023, doi: 10.1109/TCSS.2023.3238477.
F. Roma, G. Sansonetti, G. D’Aniello, and A. Micarelli, “A BERT-Based Approach to Intent Recognition,” in IEEE EUROCON 2023 - 20th International Conference on Smart Technologies, Torino, Italy: IEEE, Jul. 2023, pp. 568–572. doi: 10.1109/EUROCON56442.2023.10198959.
S. Sayenju et al., “Quantification and Mitigation of Directional Pairwise Class Confusion Bias in a Chatbot Intent Classification Model,” Int. J. Semantic Computing, vol. 16, no. 04, pp. 497–520, Dec. 2022, doi: 10.1142/S1793351X22500040.
Y. Guo et al., “ESIE-BERT: Enriching Sub-words Information Explicitly with BERT for Joint Intent Classification and SlotFilling,” Feb. 02, 2023, arXiv: arXiv:2211.14829. Accessed: Sep. 02, 2024. [Online]. Available: http://arxiv.org/abs/2211.14829
Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” Jul. 26, 2019, arXiv: arXiv:1907.11692. Accessed: Mar. 13, 2024. [Online]. Available: http://arxiv.org/abs/1907.11692
A. Souha, C. Ouaddi, L. Benaddi, and A. Jakimi, “Pre-Trained Models for Intent Classification in Chatbot: Comparative Study and Critical Analysis,” in 2023 6th International Conference on Advanced Communication Technologies and Networking (CommNet), Rabat, Morocco: IEEE, Dec. 2023, pp. 1–6. doi: 10.1109/CommNet60167.2023.10365312.
K. K. Jayanth, G. Bharathi Mohan, R. P. Kumar, and M. Rithani, “Intent Recognition Leveraging XLM-RoBERTa for Effective NLU,” in 2024 3rd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), Salem, India: IEEE, Jun. 2024, pp. 877–882. doi: 10.1109/ICAAIC60222.2024.10575275.
B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” Oct. 08, 2020, arXiv: arXiv:2009.05387. Accessed: Oct. 07, 2024. [Online]. Available: http://arxiv.org/abs/2009.05387
Nirali Vaghani, “Chatbot dataset.” Kaggle, 2024. doi: 10.34740/KAGGLE/DSV/5024271.
DeepL, “DeepL Translator.” [Online]. Available: https://www.deepl.com/
J. H. Tandijaya and I. Sugiarto, “Klasifikasi dalam Pembuatan Portal Berita Online dengan Menggunakan Metode BERT,” vol. Vol 9, No 2 (2021), 2021.
A. Aljabar, “Mengungkap Opini Publik: Pendekatan BERT-based- caused untuk Analisis Sentimen pada Komentar Film,” vol. 5, no. 1, 2024.
R. Khusuma, W. Maharani, and P. H. Gani, “Personality Detection On Twitter User With RoBERTa,” Jurnal Media Informatika Budidarma, vol. 7, 2023.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” Nov. 01, 2020, arXiv: arXiv:2011.00677. Accessed: Oct. 09, 2024. [Online]. Available: http://arxiv.org/abs/2011.00677
L. Geni, E. Yulianti, and D. I. Sensuse, “Sentiment Analysis of Tweets Before the 2024 Elections in Indonesia Using IndoBERT Language Models,” vol. 9, no. 3, 2023.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Analisis Perbandingan Klasifikasi Intent Chatbot Menggunakan Deep Learning BERT, RoBERTa, dan IndoBERT
Pages: 595-606
Copyright (c) 2024 Aswin Dwiyono, Abdiansah Abdiansah, Muhammad Fachrurrozi
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
