Penerapan CNN dan RNN untuk Pembuatan Deskripsi Konten Visual Menggunakan Deep Learning
-
Aldy Agil Hermanto
*
Universitas Amikom Purwokerto, Purwokerto,
Indonesia
- Giat Karyono Universitas Amikom Purwokerto, Purwokerto, Indonesia
- Imam Tahyudin Universitas Amikom Purwokerto, Purwokerto, Indonesia (*) Corresponding Author
Abstract
The development of technology in the field of image and sound processing has had a significant impact on increasing the accessibility of information for various groups, especially for individuals with visual impairments. One of the innovations that emerged was the image to speech system, which allows the conversion of images into sounds that can be understood by its users. The main problem lies in the low accuracy of object recognition in images with high variability, such as poor lighting or complex backgrounds, as well as the challenge of producing suitable text descriptions to be converted into audio. The method used involves extracting image features using InceptionV3-based CNN and forming a sequence of descriptive texts through RNN with an attention mechanism. The dataset consists of 40,455 captions and 8,091 images, processed using text and image pre-processing techniques before being trained using the teacher forcing technique. The evaluation results show a very low BLEU score (5.154827976372712e-153), indicating the model's inability to replicate the original caption well. However, the audio from the text-to-speech conversion using Google Text-to-Speech is quite clear. Future solutions include increasing the dataset, applying regularization, and adjusting the model architecture to improve the accuracy of caption prediction and audio relevance to the image. With these improvements, it is hoped that the system can provide more inclusive visual information accessibility for individuals with visual impairments.
Downloads
References
A. I. Pradana and W. Wijiyanto, “Identifikasi Jenis Kelamin Otomatis Berdasarkan Mata Manusia Menggunakan Convolutional Neural Network (CNN) dan Haar Cascade Classifier,” G-Tech: Jurnal Teknologi Terapan, vol. 8, no. 1, 2024, doi: 10.33379/gtech.v8i1.3814.
M. F. Prayuda, “Classification of Sad Emotions and Depression Through Images Using Convolutional Neural Network (CNN),” Jurnal Informatika Universitas Pamulang, vol. 6, no. 1, 2021, doi: 10.32493/informatika.v6i1.8433.
D. Iswantoro and D. Handayani UN, “Klasifikasi Penyakit Tanaman Jagung Menggunakan Metode Convolutional Neural Network (CNN),” Jurnal Ilmiah Universitas Batanghari Jambi, vol. 22, no. 2, 2022, doi: 10.33087/jiubj.v22i2.2065.
I. K. Trisiawan and Y. Yuliza, “Penerapan Multi-Label Image Classification Menggunakan Metode Convolutional Neural Network (CNN) Untuk Sortir Botol Minuman,” Jurnal Teknologi Elektro, vol. 13, no. 1, 2022, doi: 10.22441/jte.2022.v13i1.009.
F. A. Febriyanti, “Image Processing Dengan Metode Convolutional Neural Network (CNN) Untuk Deteksi Penyakit Kulit Pada Manusia,” Kohesi: Jurnal Sains dan Teknologi, vol. 3, no. 10, pp. 21–30, 2024, doi: https://doi.org/10.3785/kohesi.v3i10.4088.
E. Oktafanda, “Klasifikasi Citra Kualitas Bibit dalam Meningkatkan Produksi Kelapa Sawit Menggunakan Metode Convolutional Neural Network (CNN),” Jurnal Informatika Ekonomi Bisnis, 2022, doi: 10.37034/infeb.v4i3.143.
Y. A. P. Vandalis, S. Soim, and Lindawati, “Pengembangan Algoritma Convolutional Neural Networks (CNN) untuk Klasifikasi Objek dalam Gambar Sampah,” Building of Informatics, Technology and Science (BITS), vol. 6, no. 2, pp. 797–806, 2023, doi: https://doi.org/10.47065/bits.v6i2.5585.
A. Akram, K. Fayakun, and H. Ramza, “Klasifikasi Hama Serangga pada Pertanian Menggunakan Metode Convolutional Neural Network,” Building of Informatics, Technology and Science (BITS), vol. 5, no. 2, 2023, doi: 10.47065/bits.v5i2.4063.
N. Lubis, Mhd. Z. Siambaton, and R. Aulia, “Implementasi Algoritma Deep Learning pada Aplikasi Speech to Text Online dengan Metode Recurrent Neural Network (RNN),” Sudo Jurnal Teknik Informatika, vol. 3, no. 3, pp. 113–126, 2024, doi: https://doi.org/10.56211/sudo.v3i3.583.
Y. C. Adi, W. Priharti, and I. Hidayat, “Implementasi Pengenal Tulisan Tangan Menggunakan Optical Character Recognition Dengan Metode Cnn Dan Rnn Pada Dokumen Resi Dan Kuitansi,” e-Proceeding of Engineering, vol. 11, no. 1, pp. 32–38, 2024.
D. T. Adherda, M. Hikmatyar, and Ruuhwan, “Gender Classification Based On Voice Using Recurrent Neural Network (Rnn),” Antivirus : Jurnal Ilmiah Teknik Informatika, vol. 17, no. 1, 2023, doi: 10.35457/antivirus.v17i1.3049.
Y. A. Suwitono and F. J. Kaunang, “Implementasi Algoritma Convolutional Neural Network (CNN) Untuk Klasifikasi Daun Dengan Metode Data Mining SEMMA Menggunakan Keras,” Jurnal Komtika (Komputasi dan Informatika), vol. 6, no. 2, 2022, doi: 10.31603/komtika.v6i2.8054.
Herdianto and D. Nasution, “Implementasi Metode Cnn Untuk Klasifikasi Objek,” METHOMIKA: Jurnal Manajemen Informatika & Komputersisasi Akuntansi, vol. 7, no. 1, pp. 54–60, 2023, doi: https://doi.org/10.46880/jmika.Vol7No1.pp54-60.
J. V. P. Putra, F. Ayu, and B. Julianto, “Implementasi Pendeteksi Penyakit pada Daun Alpukat Menggunakan Metode CNN,” Stains (Seminar Nasional Teknologi & Sains), vol. 2, no. 1, 2023, doi: https://doi.org/10.29407/stains.v2i1.2888.
R. H. Alfikri, M. S. Utomo, H. Februariyanti, and E. Nurwahyudi, “Pembangunan Aplikasi Penerjemah Bahasa Isyarat Dengan Metode Cnn Berbasis Android,” Jurnal Teknoinfo, vol. 16, no. 2, pp. 183–197, 2022, doi: https://doi.org/10.33365/jti.v16i2.1752.
F. N. Cahya, N. Hardi, D. Riana, and S. Hadiyanti, “Klasifikasi Penyakit Mata Menggunakan Convolutional Neural Network (CNN),” SISTEMASI, vol. 10, no. 3, 2021, doi: 10.32520/stmsi.v10i3.1248.
Y. B. E. Purba, N. F. Saragih, A. P. Silalahi, S. Sitepu, and A. Gea, “Perancangan Alat Pendeteksi Kematangan Buah Nanas Dengan Menggunakan Mikrokontroler Dengan Metode Convolutional Neural Network (CNN),” Methotika: Jurnal Ilmiah Teknik Informatika, vol. 2, no. 1, 2022.
A. Zalvadila, “Klasifikasi Penyakit Tanaman Bawang Merah Menggunakan Metode SVM dan CNN,” Jurnal Informatika: Jurnal Pengembangan IT, vol. 8, no. 3, 2023, doi: 10.30591/jpit.v8i3.5341.
J. R. Aisya and A. Prasetiadi, “Klasifikasi Penyakit Daun Kentang dengan Metode CNN dan RNN,” Jurnal Tekno Insentif, vol. 17, no. 1, 2023, doi: 10.36787/jti.v17i1.888.
Diki Hananta Firdaus, Bahtiar Imran, Lalu Darmawan Bakti, and Emi Suryadi, “Klasifikasi Penyakit Katarak Berdasarkan Citra Menggunakan Metode Convolutional Neural Network (Cnn) Berbasis Web,” Jurnal Kecerdasan Buatan dan Teknologi Informasi, vol. 1, no. 3, 2022, doi: 10.69916/jkbti.v1i3.6.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Penerapan CNN dan RNN untuk Pembuatan Deskripsi Konten Visual Menggunakan Deep Learning
Pages: 2573-2581
Copyright (c) 2025 Aldy Agil Hermanto, Giat Karyono, Imam Tahyudin
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
