Klasifikasi Ujaran Kebencian Menggunakan 5-Fold Ensemble Weighted Probability Averaging berbasis Arsitektur Twitter-RoBERTa
-
Ibra Sahrian Alsa
*
Universitas Islam Negeri Sultan Syarif Kasim Riau, Pekanbaru,
Indonesia
- Surya Agustian Universitas Islam Negeri Sultan Syarif Kasim Riau, Pekanbaru, Indonesia
- Fitra Kurnia Universitas Islam Negeri Sultan Syarif Kasim Riau, Pekanbaru, Indonesia
- Pizaini Pizaini Universitas Islam Negeri Sultan Syarif Kasim Riau, Pekanbaru, Indonesia
- Siska Kurnia Gusti Universitas Islam Negeri Sultan Syarif Kasim Riau, Pekanbaru, Indonesia (*) Corresponding Author
Abstract
Social media platforms have become a critical medium for hate speech propagation at unprecedented scale, with over 66.8 million user reports regarding hateful conduct recorded on platform X during the first half of 2024 alone. This study proposes an end-to-end NLP pipeline for automated hate speech classification using the domain-adapted Twitter-RoBERTa architecture, evaluated on the HASOC (Hate Speech and Offensive Content Identification) English datasets from 2020 and 2021. The core challenge addressed is Transformer fine-tuning instability on relatively small annotated corpora caused by extreme sensitivity to random seed initialization and suboptimal hyperparameter configurations. Three methodological innovations are synergistically integrated: (1) Bayesian Optimization via the Optuna framework for automated adaptive hyperparameter search with 15 trials; (2) Stratified 5-Fold Cross-Validation for robust, reproducible data partitioning; and (3) Weighted Probability Averaging (WPA) as the ensemble aggregation strategy. Results demonstrate that the proposed architecture achieves a Macro F1-Score of 80.99% on Subtask 1A and 64.70% on Subtask 1B, positioning it competitively against 65 international research teams on the official HASOC 2021 leaderboard.
Downloads
References
S. Yoo, E. Jeon, J. Hyeon, and J. Cho, “Adaptive ensemble techniques leveraging BERT based models for multilingual hate speech detection in Korean and english,” Sci. Rep., vol. 15, no. 1, p. 19844, Jun. 2025, doi: 10.1038/s41598-025-88960-y.
D. Hartmann, A. Oueslati, D. Staufer, L. Pohlmann, S. Munzert, and H. Heuer, “Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, New York, NY, USA: ACM, Apr. 2025, pp. 1–26. doi: 10.1145/3706598.3713998.
A. Kumar, P. K. Roy, and S. Saumya, “An Ensemble Approach for Hate and Offensive Language Identification in English and Indo-Aryan Languages,” Sci. Rep 2021. Accessed: Dec. 07, 2025. [Online]. Available: https://ceur-ws.org/Vol-3159/T1-43.pdf
D. Q. Nguyen, T. Vu, and A. T. Nguyen, “BERTweet: A pre-trained language model for English Tweets,” Sci. Rep pp. 9–14, Oct. 2020, Accessed: Dec. 07, 2025. [Online]. Available: http://arxiv.org/abs/2005.10200
D. Antypas and J. Camacho-Collados, “Robust Hate Speech Detection in Social Media: A Cross-Dataset Empirical Evaluation,” Sci. Rep Jul. 2023, [Online]. Available: http://arxiv.org/abs/2307.01680
X. Liu and C. Wang, “An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 2286–2300. doi: 10.18653/v1/2021.acl-long.178.
M. Mosbach, M. Andriushchenko, and D. Klakow, “On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines,” Mar. 2021, [Online]. Available: http://arxiv.org/abs/2006.04884
J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith, “Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping,” Feb. 2020, [Online]. Available: http://arxiv.org/abs/2002.06305
I. E. Kucukkaya and C. Toraman, “Constructing ensembles for hate speech detection,” Natural Language Processing, vol. 31, no. 3, pp. 745–770, May 2025, doi: 10.1017/nlp.2024.44.
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework,” Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.10902
T. Mandla et al., “Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages,” Aug. 2021, [Online]. Available: http://arxiv.org/abs/2108.05927
T. Mandl et al., “Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages,” Dec. 2021, [Online]. Available: http://arxiv.org/abs/2112.09301
F. Barbieri, J. Camacho-Collados, L. Espinosa Anke, and L. Neves, “TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification,” in Findings of the Association for Computational Linguistics: EMNLP 2020, Stroudsburg, PA, USA: Association for Computational Linguistics, Aug. 2020, pp. 1644–1650. doi: 10.18653/v1/2020.findings-emnlp.148.
M. S. Jinan, M. R. Handayani, M. A. Ulinuha, and K. Umam, “DETEKSI CYBERBULLYING MULTIKELAS BERKINERJA TINGGI: ENSEMBLE ROBERTA-LARGE DENGAN PRESISI CAMPURAN,” JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika), vol. 10, no. 3, pp. 2666–2678, Sep. 2025, doi: 10.29100/jipi.v10i3.8056.
Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.11692
M. A. Ganaie, M. Hu, A. K. Malik, M. Tanveer, and P. N. Suganthan, “Ensemble deep learning: A review,” Eng. Appl. Artif. Intell., vol. 115, p. 105151, Oct. 2022, doi: 10.1016/j.engappai.2022.105151.
A. C. Mazari, N. Boudoukhani, and A. Djeffal, “BERT-based ensemble learning for multi-aspect hate speech detection,” Cluster Comput., vol. 27, no. 1, pp. 325–339, Feb. 2024, doi: 10.1007/s10586-022-03956-x.
N. P. Shetty, Y. Singh, V. Hegde, D. Cenitta, and D. K, “Exploring emotional patterns in social media through NLP models to unravel mental health insights,” Healthc. Technol. Lett., vol. 12, no. 1, Jan. 2025, doi: 10.1049/htl2.12096.
U. Iftikhar, S. F. Ali, G. Mustafa, N. Bahar, and K. Ishaq, “Beyond words: a hybrid transformer-ensemble approach for detecting hate speech and offensive language on social media,” PeerJ Comput. Sci., vol. 11, p. e3214, Oct. 2025, doi: 10.7717/peerj-cs.3214.
Z. Zhang, J. Chen, and D. Yang, “Mitigating Biases in Hate Speech Detection from A Causal Perspective,” in Findings of the Association for Computational Linguistics: EMNLP 2023, Stroudsburg, PA, USA: Association for Computational Linguistics, 2023, pp. 6610–6625. doi: 10.18653/v1/2023.findings-emnlp.440.
E. M. Pusung and I. N. Dewi, “Optimasi RoBERTa dengan Hyperparameter Tuning untuk Deteksi Emosi berbasis Teks,” Jurnal Nasional Teknologi dan Sistem Informasi, vol. 10, no. 3, pp. 240–248, Feb. 2025, doi: 10.25077/TEKNOSI.v10i3.2024.240-248.
J. S. Malik, H. Qiao, G. Pang, and A. van den Hengel, “Deep Learning for Hate Speech Detection: A Comparative Study,” Dec. 2023, [Online]. Available: http://arxiv.org/abs/2202.09517
C. Lee, K. Cho, and W. Kang, “Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models,” Jan. 2020, [Online]. Available: http://arxiv.org/abs/1909.11299
S. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” Nov. 2017, Accessed: Mar. 11, 2026. [Online]. Available: http://arxiv.org/abs/1705.07874
B. Aklouche, Y. Bazine, and Z. Ghalia-Bououchma, “Offensive Language and Hate Speech Detection Using Transformers and Ensemble Learning Approaches,” Computación y Sistemas, vol. 28, no. 3, pp. 1031–1039, Sep. 2024, doi: 10.13053/cys-28-3-4724.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Klasifikasi Ujaran Kebencian Menggunakan 5-Fold Ensemble Weighted Probability Averaging berbasis Arsitektur Twitter-RoBERTa
Pages: 199-208
Copyright (c) 2026 Ibra Sahrian Alsa, Surya Agustian, Fitra Kurnia, Pizaini Pizaini, Siska Kurnia Gusti
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
