Analisis Sentimen Komentar TikTok pada Konten Sound Horeg Menggunakan XLM-RoBERTa, Kamus Bahasa Jawa dan CatBoost
-
Fiona Evelyne Luther
*
Universitas Kristen Satya Wacana, Salatiga,
Indonesia
- Radius Tanone Universitas Kristen Satya Wacana, Salatiga, Indonesia (*) Corresponding Author
Abstract
Comments on TikTok can provide valuable data for analyzing public sentiment toward social and cultural phenomena. However, informal language, multilingual expressions, code-mixing, and limited linguistic resources for regional languages present challenges for sentiment classification. Sound Horeg was selected as the research subject because this social and cultural phenomenon generates diverse public responses on social media, including support and opposition, making it a relevant context for examining public perceptions through sentiment analysis. This study aims to develop a hybrid sentiment analysis pipeline for classifying TikTok comments related to Sound Horeg by integrating XLM-RoBERTa, a Javanese sentiment lexicon, and CatBoost. The methodological novelty of this study lies in integrating the multilingual semantic representations of XLM-RoBERTa with a Javanese lexical resource in the context of Sound Horeg sentiment analysis, while evaluating the contribution of each component through an ablation study. The dataset was collected from five Sound Horeg videos and consisted of 7,330 comments, which were processed through data cleaning, preprocessing, Lexicon-Based sentiment labeling, XLM-RoBERTa embedding extraction, and Random Undersampling. Lexicon-based labeling produced 960 labeled comments, consisting of 559 positive and 401 negative comments, while Random Undersampling produced 802 balanced samples for the main model. The 768-dimensional XLM-RoBERTa embeddings were used as input features for CatBoost. The main model achieved an accuracy of 89.44%, an F1-Macro score of 89.44%, and an ROC-AUC of 90.25%. The ablation study showed that the Lexicon-only configuration achieved an accuracy of 94.79%, followed by XLM-R only at 93.23% and XLM-R + Lexicon at 91.15%. These results indicate that the simpler configurations, particularly Lexicon-only and XLM-R only, achieved higher performance than the main pipeline, suggesting that the hybrid pipeline cannot yet be considered the most optimal configuration for operational implementation. These findings indicate that XLM-RoBERTa can provide effective semantic representations without using skor leksikals as the main features, while the contribution of Javanese vocabulary remains limited because it was detected in only 1.6% of Group A comments and 0.8% of Group B comments. This finding implies that the selection of components in a sentiment analysis system should consider the balance between predictive performance, processing complexity, and the linguistic characteristics of the analyzed data.
Downloads
References
N. Apriliani, N. Suarna, and W. Prihartono, “Analisis sentimen review penggunaan TikTok melalui pendekatan algoritma Naïve Bayes,” JATI (Jurnal Mahasiswa Teknik Informatika), vol. 7, no. 6, 2023, doi: https://doi.org/10.36040/jati.v7i6.8299.
A. M. Saprizal and N. Anisa, “Analisis sentimen TikTok: Wajib militer dengan metode Lexicon Based dan Naive Bayes Classifier,” TAMIKA: Jurnal Tugas Akhir Manajemen Informatika & Komputerisasi Akuntansi, vol. 4, no. 2, pp. 242–246, 2024, doi: https://doi.org/10.46880/tamika.Vol4No2.pp242-246.
N. Apriliani, N. Suarna, and W. Prihartono, “Analisis sentimen review penggunaan TikTok melalui pendekatan algoritma Naïve Bayes,” JATI (Jurnal Mahasiswa Teknik Informatika), vol. 7, no. 6, 2023, doi: https://doi.org/10.36040/jati.v7i6.8299.
M. Isnan, G. N. Elwirehardja, and B. Pardamean, “Sentiment analysis for TikTok review using VADER sentiment and SVM model,” Procedia Computer Science, vol. 227, pp. 168–175, 2023, doi: https://doi.org/10.1016/j.procs.2023.10.514.
Y. Aliyu, A. Sarlan, K. Usman Danyaro, A. S. B. Abd Rahman, and M. Abdullahi, “Sentiment analysis in low-resource settings: A comprehensive review of approaches, languages, and data sources,” IEEE Access, vol. 12, pp. 66883–66909, 2024, doi: https://doi.org/10.1109/ACCESS.2024.3398635.
N. Ahmed, R. Amin, H. Aldabbas, M. Saeed, M. Bilal, and H. Song, “A novel approach for sentiment analysis of a low resource language using deep learning models,” ACM Transactions on Asian and Low-Resource Language Information Processing, 2024, doi: https://doi.org/10.1145/3696789.
L. A. Pha, L. T. Thuy, V. N. Hong, T. T. Phan, and T. K. Tran, “A systematic survey of sentiment analysis in low-resource Southeast Asian languages: Resources, methods, and evaluation challenges,” ICT Express, 2026, doi: https://doi.org/10.1016/j.icte.2026.06.013.
A. Ali, M. Khan, K. Khan, R. U. Khan, and A. Aloraini, “Sentiment analysis of low-resource language literature using data processing and deep learning,” Computers, Materials & Continua, vol. 79, no. 1, pp. 713–733, 2024, doi: https://doi.org/10.32604/cmc.2024.048712.
B. R. P. Darnoto and D. B. Firmawan, “Language-similarity-guided transfer fine-tuning of pre-trained transformer models for sentiment analysis across 12 Indonesian regional languages,” Journal of Computing Theories and Applications, vol. 3, no. 4, pp. 547–563, 2026, doi: https://doi.org/10.62411/jcta.15975.
G. I. Winata et al., “NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 815–834, doi: https://doi.org/10.18653/v1/2023.eacl-main.57.
Z. Fatah and R. A. Ningsih, “Analisis sentimen komentar YouTube terhadap Tragedi Demo 25 Agustus menggunakan pendekatan Lexicon-Based,” JAMASTIKA, vol. 4, no. 2, 2025, doi: https://doi.org/10.35473/jamastika.v4i2.4525.
A. D. Pramesti, K. Umam, and M. R. Handayani, “Identification of buzzers in skincare reviews using a Lexicon-Based sentiment analysis method,” Journal of Applied Informatics and Computing, vol. 9, no. 5, pp. 2598–2606, 2025, doi: https://doi.org/10.30871/jaic.v9i5.11005.
F. Koto, T. Beck, Z. Talat, I. Gurevych, and T. Baldwin, “Zero-shot sentiment analysis in low-resource languages using a multilingual sentiment lexicon,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, Volume 1: Long Papers, 2024, pp. 298–320, doi: https://doi.org/10.18653/v1/2024.eacl-long.18.
A. Conneau et al., “Unsupervised cross-lingual representation learning at scale,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 8440–8451, doi: https://doi.org/10.18653/v1/2020.acl-main.747.
A. Gaurav, B. B. Gupta, S. Sharma, R. Bansal, and K. T. Chui, “XLM-RoBERTa based sentiment analysis of tweets on Metaverse and 6G,” Procedia Computer Science, vol. 238, pp. 902–907, 2024. https://doi.org/10.1016/j.procs.2024.06.110
I. D. G. M. Parawangsa, G. S. Mahendra, and R. N. P. Pinatih, “Implementasi metode XLM-RoBERTa untuk analisis sentimen ulasan pengunjung objek wisata Kerta Gosa Klungkung,” RIGGS: Journal of Artificial Intelligence and Digital Business, vol. 4, no. 4, pp. 9893–9900, 2026, doi: https://doi.org/10.31004/riggs.v4i4.5368.
E. Yulianti and N. K. Nissa, “ABSA of Indonesian customer reviews using IndoBERT: Single-sentence and sentence-pair classification approaches,” Bulletin of Electrical Engineering and Informatics, vol. 13, no. 5, pp. 3579–3589, 2024, doi: https://doi.org/10.11591/eei.v13i5.8032.
A. F. Hidayatullah, “Code-mixed sentiment analysis on Indonesian-Javanese-English text using transformer models,” in 2024 8th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), 2024, pp. 340–345, doi: https://doi.org/10.1109/ICITISEE63424.2024.10730138.
T. D. Purnomo and J. Sutopo, “Comparison of pre-trained BERT-based transformer models for regional language text sentiment analysis in Indonesia,” International Journal Science and Technology, vol. 3, no. 3, pp. 11–21, 2024, doi: https://doi.org/10.56127/ijst.v3i3.1739.
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, “CatBoost: Unbiased boosting with categorical features,” in Advances in Neural Information Processing Systems, vol. 31, pp. 6638–6648, 2018. Available: https://proceedings.neurips.cc/paper_files/paper/2018/hash/14491b756b3a51daac41c24863285549-Abstract.html.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Analisis Sentimen Komentar TikTok pada Konten Sound Horeg Menggunakan XLM-RoBERTa, Kamus Bahasa Jawa dan CatBoost
Pages: 1035-1049
Copyright (c) 2026 Fiona Evelyne Luther, Radius Tanone
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
