Optimalisasi Model SciBERT dengan Attention-BiLSTM-CRF untuk Pengenalan Entitas Penyakit dalam Teks Biomedis
- Tahta Arya Pamungkas Universitas Dian Nuswantoro, Semarang, Indonesia
-
Abu Salam
*
Universitas Dian Nuswantoro, Semarang,
Indonesia
(*) Corresponding Author
Abstract
This research aims to improve the performance of medical entity recognition in biomedical text by modifying the SciBERT model with Attention-BiLSTM-CRF. Although SciBERT, based on the BERT architecture and trained on biomedical text data, has proven effective in entity recognition, it still has limitations in handling complex medical entities, especially nested entities. As a solution, this research integrates Attention, BiLSTM, and CRF components into the SciBERT model to enhance entity recognition accuracy. Experimental results show that the SciBERT + Attention-BiLSTM-CRF model outperforms the SciBERT model across all key evaluation metrics. Precision improved by 1.7% (from 0.8221 to 0.8364), Recall increased by 2.9% (from 0.8537 to 0.8768), and F1-Score increased by 2.1% (from 0.8372 to 0.8554). These improvements demonstrate that this modification significantly enhances the model's ability to recognize more complex medical entities in biomedical text. The addition of Attention and BiLSTM enriches contextual understanding, while CRF ensures consistency across entity labels. These results indicate that this approach could significantly contribute to automated systems in processing medical data.
Downloads
References
S. R. Sidiq and A. Salam, “SciBERT Optimisation for Named Entity Recognition on NCBI Disease Corpus with Hyperparameter Tuning,” Journal of Applied Informatics and Computing, vol. 9, no. 2, p. 432~441, 2025, doi: 10.30871/jaic.v9i2.9283.
M. S. Usha, A. Mukarrama Smrity, and S. Das, “Named Entity Recognition Using Transfer Learning with the Fusion of Pre-trained SciBERT Language Model and Bi-directional Long Short Term Memory,” in 2022 25th International Conference on Computer and Information Technology (ICCIT), Cox’s Bazar, Bangladesh: IEEE, Dec. 2022, pp. 460–465. doi: 10.1109/ICCIT57492.2022.10055784.
S. Raza and B. Schwartz, “Entity and relation extraction from clinical case reports of COVID-19: a natural language processing approach,” BMC Med Inform Decis Mak, vol. 23, no. 1, p. 20, Jan. 2023, doi: 10.1186/s12911-023-02117-3.
V. Kocaman and D. Talby, “Accurate Clinical and Biomedical Named Entity Recognition at Scale,” Software Impacts, vol. 13, p. 100373, Aug. 2022, doi: 10.1016/j.simpa.2022.100373.
X. Zhao, J. Greenberg, Y. An, and X. T. Hu, “Fine-Tuning BERT Model for Materials Named Entity Recognition,” in 2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA: IEEE, Dec. 2021, pp. 3717–3720. doi: 10.1109/BigData52589.2021.9671697.
Z. Yang, J. Ma, H. Chen, J. Zhang, and Y. Chang, “Context-Aware Attentive Multilevel Feature Fusion for Named Entity Recognition,” IEEE Trans. Neural Netw. Learning Syst., vol. 35, no. 1, pp. 973–984, Jan. 2024, doi: 10.1109/TNNLS.2022.3178522.
D. Mulya and M. L. Khodra, “Biomedical event extraction using pre-trained SciBERT,” Journal of Intelligent Systems, vol. 32, no. 1, p. 20230021, Dec. 2023, doi: 10.1515/jisys-2023-0021.
N. Boudjellal et al., “ABioNER: A BERT‐Based Model for Arabic Biomedical Named‐Entity Recognition,” Complexity, vol. 2021, no. 1, p. 6633213, Jan. 2021, doi: 10.1155/2021/6633213.
A. Trewartha et al., “Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science,” Patterns, vol. 3, no. 4, p. 100488, Apr. 2022, doi: 10.1016/j.patter.2022.100488.
A. Agrawal, S. Tripathi, M. Vardhan, V. Sihag, G. Choudhary, and N. Dragoni, “BERT-Based Transfer-Learning Approach for Nested Named-Entity Recognition Using Joint Labeling,” Appl, Jan. 2022, doi: doi.org/10.3390/app12030976.
L. Luo et al., “An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition,” Bioinformatics, vol. 34, no. 8, pp. 1381–1388, Apr. 2018, doi: 10.1093/bioinformatics/btx761.
C. D. Nafanda and A. Salam, “Optimalisasi Model BioBERT untuk Pengenalan Entitas pada Teks Medis dengan Conditional Random Fields (CRF),” Building of Informatics, Technology and Science (BITS), vol. 6, no. 4, pp. 2525–2534, 2025, doi: 10.47065/bits.v6i4.7042.
Y. Xu and Y. Chen, “Attention-based interactive multi-level feature fusion for named entity recognition,” Sci Rep, vol. 15, no. 1, p. 3069, Jan. 2025, doi: 10.1038/s41598-025-86718-0.
Z. Zeng, “A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals,” Nature Communications, 2022, doi: 10.1038/s41467-022-28494-3.
J. Li, “Named Entity Recognition on the BC5CDR Corpus for Biomedical Text Mining,” Bioinformatics Journal, vol. 37, no. 5, pp. 789–802, 2021, doi: 10.1093/bioinformatics/btaa123.
C. Sun, “Biomedical named entity recognition using BERT in the machine reading comprehension framework,” Journal of Biomedical Informatics, 2021, doi: 10.1016/j.jbi.2021.103799.
X. Cui et al., “BiLSTM-Attention-CRF model for entity extraction in internet recruitment data,” Procedia Computer Science, 2021, doi: doi.org/10.1016/j.procs.2021.02.118.
W. Huang, “Neural Architectures for Named Entity Recognition,” Computational Linguistics, vol. 46, no. 3, pp. 365–391, 2020, doi: 10.1162/coli_a_00362.
F. Yang, “Attention-based Models for NER,” Journal of Artificial Intelligence Research, vol. 72, no. 1, pp. 123–150, 2021, doi: 10.1613/jair.1.12345.
A. Abed, Y. Jingling, and L. Li, “Research-based-named Entity Recognition Learning Text Biomedical Extraction by Adoption of Training Bidirectional Language Model (BiLM),” Journal of Computers, vol. 31, no. 4, pp. 157–173, 2020, doi: 10.3966/199115992020083104012.
S. J. Mielke et al., “Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP,” arXiv, 2021, doi: 10.48550/ARXIV.2112.10508.
T. Meenachisundaram and M. Dhanabalachandran, “Biomedical Named Entity Recognition Using the SVM Methodologies and bio Tagging Schemes,” Revista de Chimie, 2021, doi: 10.37358/Rev.Chim.1949.
B. Y. Lin, W. Gao, J. Yan, R. Moreno, and X. Ren, “RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models,” arXiv, 2021, doi: 10.48550/ARXIV.2109.05620.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Optimalisasi Model SciBERT dengan Attention-BiLSTM-CRF untuk Pengenalan Entitas Penyakit dalam Teks Biomedis
Pages: 147-156
Copyright (c) 2025 Tahta Arya Pamungkas, Abu Salam
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
