Evaluasi Kritis Generalisasi Multidataset pada Deteksi Hoax Berbahasa Indonesia Menggunakan BiLSTM dan IndoBERT

Abstract

Machine learning-based Indonesian hoax detection often reports high scores on test data sharing the training data’s distribution, although such scores may misrepresent actual model capability. This critical evaluation, not a new method, analyzes how multidataset composition affects BiLSTM and IndoBERT performance and generalization, and identifies indications of shortcut learning arising from spurious correlations between labels and dataset origin. Three datasets were used: Mendeley (426 samples), Komdigi (3,951 samples), and sh1zuka (8,000 sampled), combined into 12,377 samples (8,256 Valid, 4,121 Hoax). Evaluation employed three schemes: Scheme 1 (stratified 70:15:15 split of the combined data), Scheme 2 (forward cross dataset), and Scheme 3 (reverse cross dataset). In Scheme 1, BiLSTM and IndoBERT achieved Macro F1 scores of  0.9803 and 0.9814, but these fugures cannot be read as successful hoax detection, since under cross dataset schemes Macro F1 dropped to 0.3642 and 0.5624 (Scheme 2) and 0.4231 and 0.4109 (Scheme 3), with accuracy below the majority class baseline. Dataset 1, comprising only 3.4% of the Scheme 1 test data, accounted for 87.9% of BiLSTM errors and 100% of IndoBERT errors, while the accuracy of the word “beredar” as a single label predictor fell from 96.30% on combined data to 42.96% on Dataset 1. These findings indicate that high scores under conventional evaluation reflect recognition of dataset origin more than comprehension of news content; therefore, cross dataset evaluation and source control tests should accompany random splitting on multisource data.