Comparing Machine Learning and BERT for Classifying Grammatical Error Types in Indonesian EFL Essays
- Meily Ivane Esther Neman Universitas Klabat, Minahasa Utara, Indonesia
-
George Morris William Tangka
*
Universitas Klabat, Minahasa Utara,
Indonesia
- Regi Fernando Najoan Universitas Klabat, Minahasa Utara, Indonesia
- Enrico Samuel Djimesha Universitas Klabat, Minahasa Utara, Indonesia
- Daniel Semuel Warouw Universitas Klabat, Minahasa Utara, Indonesia (*) Corresponding Author
Abstract
Automated writing feedback tools are usually trained on learner corpora from outside Indonesia, so their accuracy on Indonesian students' grammatical errors is unknown. This study compares Multinomial Naive Bayes, a linear Support Vector Machine (SVM), Random Forest, and a fine-tuned BERT model for classifying nine grammatical error types. The models were trained on 48,011 error edits from the BEA-2019 W&I+LOCNESS and FCE corpora, re-annotated with ERRANT 3, and tested in-domain on 5,129 edits and cross-corpus on 1,900 edits from 66 essays by 33 Indonesian university students in the ICNALE Edited Essays. BERT achieved the highest macro-F1 on both test sets, 0.917 in-domain and 0.849 on the Indonesian essays, and McNemar tests confirmed its advantage over each classical model. All models lost 0.035 to 0.068 macro-F1 under the corpus shift. Noun number errors formed 24% of the Indonesian errors, three times their training share, while word form and pronoun were the hardest classes. This study contributes a cross-corpus benchmark of classical and Transformer-based error-type classifiers on Indonesian students' essays, a reproducible pipeline that labels both corpora with the same ERRANT version, and a class-level reliability profile for classroom feedback. For English teachers in Indonesia, the results support explicit, contrastive practice on articles and plural marking and the use of BERT for immediate feedback on reliably recognised classes, while its word form and pronoun suggestions are confirmed by the teacher.
Downloads
References
M. H. Hamdani and N. Abid, “Indonesian EFL students' writing challenges: A qualitative meta-analysis”, Getsempena English Education Journal, vol. 12, no. 1, pp. 14–36, 2025. https://doi.org/10.46244/geej.v12i1.3091
Fitrawati and D. Safitri, “Students' grammatical errors in essay writing: A pedagogical grammar reflection”, International Journal of Language Education, vol. 5, no. 2, pp. 74–88, 2021. https://doi.org/10.26858/ijole.v5i2.16128
M. I. E. Neman and V. S. Kamuh, “Errors in English descriptive text produced by high school students in Manado”, ELS Journal on Interdisciplinary Studies in Humanities, vol. 7, no. 1, pp. 17–21, 2024. https://doi.org/10.34050/elsjish.v7i1.32782
Y. Mubarok and T. Budiono, “An error analysis on EFL students' writing”, Englisia: Journal of Language, Education, and Humanities, vol. 9, no. 2, pp. 187–203, 2022. https://doi.org/10.22373/ej.v9i2.11386
Puspita and Suswanto, “An analysis of English article errors and pedagogical implications in students' writing skill”, Buletin Poltanesa, vol. 24, no. 1, pp. 60–67, 2023. https://doi.org/10.51967/tanesa.v24i1.1972
G. Dizon and J. M. Gayed, “A systematic review of Grammarly in L2 English writing contexts”, Cogent Education, vol. 11, no. 1, Art. no. 2397882, 2024. https://doi.org/10.1080/2331186X.2024.2397882
A. R. S. Tambunan, W. Andayani, W. S. Sari, and F. K. Lubis, “Investigating EFL students' linguistic problems using Grammarly as automated writing evaluation feedback”, Indonesian Journal of Applied Linguistics, vol. 12, no. 1, pp. 16–27, 2022. https://doi.org/10.17509/ijal.v12i1.46428
E. Kasneci et al., “ChatGPT for good? On opportunities and challenges of large language models for education”, Learning and Individual Differences, vol. 103, Art. no. 102274, 2023. https://doi.org/10.1016/j.lindif.2023.102274
S. Lin, “Evaluating LLMs' grammatical error correction performance in learner Chinese”, PLoS ONE, vol. 19, no. 10, Art. no. e0312881, 2024. https://doi.org/10.1371/journal.pone.0312881
A. Pack, A. Barrett, and J. Escalante, “Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability”, Computers and Education: Artificial Intelligence, vol. 6, Art. no. 100234, 2024. https://doi.org/10.1016/j.caeai.2024.100234
D. F. E. Meluwu and M. I. E. Neman, “Students' perception towards the use of ChatGPT as a tool for learning English”, Global Education Journal, vol. 3, no. 2, pp. 206–216, 2025. https://doi.org/10.59525/gej.v3i2.712
D. S. Warouw and M. I. E. Neman, “The use of English language learning strategies in learning as foreign language”, ELS Journal on Interdisciplinary Studies in Humanities, vol. 7, no. 1, pp. 32–42, 2024. https://doi.org/10.34050/elsjish.v7i1.32847
C. Bryant, Z. Yuan, M. R. Qorib, H. Cao, H. T. Ng, and T. Briscoe, “Grammatical error correction: A survey of the state of the art”, Computational Linguistics, vol. 49, no. 3, pp. 643–701, 2023. https://doi.org/10.1162/coli_a_00478
Y. Wang, Y. Wang, K. Dang, J. Liu, and Z. Liu, “A comprehensive survey of grammatical error correction”, ACM Transactions on Intelligent Systems and Technology, vol. 12, no. 5, Art. no. 65, 2021. https://doi.org/10.1145/3474840
C. Bryant, M. Felice, and T. Briscoe, “Automatic annotation and evaluation of error types for grammatical error correction”, in Proc. 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, 2017, pp. 793–805. https://doi.org/10.18653/v1/P17-1074
C. Bryant, M. Felice, Ø. E. Andersen, and T. Briscoe, “The BEA-2019 shared task on grammatical error correction”, in Proc. Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy, 2019, pp. 52–75. https://doi.org/10.18653/v1/W19-4406
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding”, in Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, 2019, pp. 4171–4186. https://doi.org/10.18653/v1/N19-1423
Q. Li et al., “A survey on text classification: From traditional to deep learning”, ACM Transactions on Intelligent Systems and Technology, vol. 13, no. 2, Art. no. 31, 2022. https://doi.org/10.1145/3495162
B. Min et al., “Recent advances in natural language processing via large pre-trained language models: A survey”, ACM Computing Surveys, vol. 56, no. 2, Art. no. 30, 2023. https://doi.org/10.1145/3605943
I. M. A. R. Antara and N. P. Y. Anggreni, “Analyzing grammatical errors in EFL students' opinion essays using DeepSeek AI: Patterns and pedagogical implications”, Stilistika: Jurnal Pendidikan Bahasa dan Seni, vol. 13, no. 2, pp. 210–218, 2025. https://doi.org/10.59672/stilistika.v13i2.4560
M. Qin, “A study on automatic correction of English grammar errors based on deep learning”, Journal of Intelligent Systems, vol. 31, no. 1, pp. 672–680, 2022. https://doi.org/10.1515/jisys-2022-0052
A. Musyafa, Y. Gao, A. Solyman, C. Wu, and S. Khan, “Automatic correction of Indonesian grammatical errors based on transformer”, Applied Sciences, vol. 12, no. 20, Art. no. 10380, 2022. https://doi.org/10.3390/app122010380
P. Sabilla and R. Dhenabayu, “Comparative complaint text classification using SVM, CNN, and BERT with priority scoring for NYC 311 service requests”, Jurnal Riset Informatika, vol. 8, no. 4, pp. 576–590, 2026. https://doi.org/10.34288/jri.v8i4.589
P. Elisa and A. R. Isnain, “Comparison of random forest, support vector machine and naive bayes algorithms to analyze sentiment towards mental health stigma”, Jurnal Teknik Informatika (JUTIF), vol. 5, no. 1, pp. 321–329, 2024. https://doi.org/10.52436/1.jutif.2024.5.1.1817
W. G. Mokodaser, T. I. Soewignyo, G. M. W. Tangka, and F. Soewignyo, “Forecasting health insurance payer income: A comparative analysis of decision tree and SVR algorithms”, Jurnal Riset Informatika, vol. 7, no. 3, pp. 210–218, 2025. https://doi.org/10.34288/jri.v7i3.369
K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4396–4415, 2023. https://doi.org/10.1109/TPAMI.2022.3195549
V. Starchenko et al., “Fighting evaluation inflation: Concentrated datasets for grammatical error correction”, Journal of Language and Education, vol. 10, no. 4, pp. 112–129, 2024. https://doi.org/10.17323/jle.2024.22272
G. M. W. Tangka, I. R. Chrisanti, J. Waworundeng, R. C. Maringka, and G. A. Sandag, “Sentiment analysis and topic detection on post-pandemic healthcare challenges: A comparative study of Twitter data in the US and Indonesia”, CogITo Smart Journal, vol. 10, no. 2, pp. 561–579, 2024. https://doi.org/10.31154/cogito.v10i2.819.561-579
I. N. Oktavianti and I. Prayogi, “Discourse functions of lexical bundles in Indonesian EFL learners' argumentative essays: A corpus study”, Studies in English Language and Education, vol. 9, no. 2, pp. 761–783, 2022. https://doi.org/10.24815/siele.v9i2.23995
S. Ishikawa, The ICNALE Guide: An Introduction to a Learner Corpus Study on Asian Learners' L2 English. London: Routledge, 2023.
H. Yannakoudakis, T. Briscoe, and B. Medlock, “A new dataset and method for automatically grading ESOL texts”, in Proc. 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, 2011, pp. 180–189.
L. Tunstall, L. von Werra, and T. Wolf, Natural Language Processing with Transformers: Building Language Applications with Hugging Face, Revised ed. Sebastopol, CA: O'Reilly Media, 2022.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Comparing Machine Learning and BERT for Classifying Grammatical Error Types in Indonesian EFL Essays
Pages: 167-178
Copyright (c) 2026 Meily Ivane Esther Neman, George Morris William Tangka, Regi Fernando Najoan, Enrico Samuel Djimesha, Daniel Semuel Warouw
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
