Smart Assistant System for Blind Users Based on Vision-Language Models and Contextual Location Assistance
- Clay Aiken Mangeber Jr Universitas Klabat, Minahasa Utara, Indonesia
- Angeli Marselin Kolmate Universitas Klabat, Minahasa Utara, Indonesia
-
Semmy Wellem Taju
*
Universitas Klabat, Minahasa Utara,
Indonesia
- Regi Fernando Najoan Universitas Klabat, Minahasa Utara, Indonesia (*) Corresponding Author
Abstract
In this study, we designed and developed an AI-based Smart Assistant for visually impaired users at Panti Sosial GMIM Baertemeus. The proposed smart assistant system integrates a Vision-Language Model (VLM) to identify and describe visual objects, and Contextual Location Assistance (CLA) to provide directional and positional information based on geographic data. This research followed the Prototyping method which consists of some user needs that had been identified, such as system design, prototype development and testing with visually impaired users. The implementation results show that the proposed system can recognize objects and their surroundings and deliver contextual voice feedback adapted to the user’s location. The CLA feature also allows administrators to monitor user positions in real time via a web dashboard. This system improves the independence and safety of visually impaired individuals in daily activities and contributes to the development of inclusive AI-based assistive technologies through the integration of VLM and CLA for accessibility enhancement.
Downloads
References
M. A. Shera et al., “Usability Evaluation of Blind and Visually Impaired Interface in Solving the Accessibility Problems,” 4th International Conference on Innovative Computing, ICIC 2021, no. November, 2021, doi: 10.1109/ICIC53490.2021.9693084.
C. Zhu, “Artificial Intelligence Applications in Everyday Life,” in Proceedings of the 5th International Conference on Signal Processing and Machine Learning, 2025, pp. 191–197. doi: 10.54254/2755-2721/121/2025.20074.
K. Chemnad and A. Othman, “Digital accessibility in the era of artificial intelligence—Bibliometric analysis and systematic review,” Frontiers in Artificial Intelligence, vol. 7, p. 1349668, 2024, doi: 10.3389/frai.2024.1349668.
R. R. A. Bourne et al., “Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: The Right to Sight: An analysis for the Global Burden of Disease Study,” Lancet Glob. Health, vol. 9, no. 2, pp. e144–e160, 2021, doi: 10.1016/S2214-109X(20)30489-7.
R. Susanto, W. Atmadja, R. Eko Prabowo, N. Julian, and U. Bina Nusantara, “Asisten Navigasi untuk Orang dengan Gangguan Penglihatan Menggunakan Ultra Sonic (Sonar Vision) Untuk Yayasan Mitra Netra,” SEEIJ (Social Economics and Ecology International Journal), vol. 4, no. 1, pp. 9–13, 2020.
A. Lavric, C. Beguni, E. Zadobrischi, A. M. Căilean, and S. A. Avătămăniței, “A Comprehensive Survey on Emerging Assistive Technologies for Visually Impaired Persons: Lighting the Path with Visible Light Communications and Artificial Intelligence Innovations,” Sensors, vol. 24, no. 15, 2024, doi: 10.3390/s24154834.
L. Triyono, R. Gernowo, and S. R. Cholil, “A Systematic Review of Artificial Intelligence in Assistive Technology for People with Visual Impairment,” Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control, vol. 8, no. 4, 2023, doi: 10.22219/kinetik.v8i4.1772.
J. Chai, H. Zeng, A. Li, and E. W. T. Ngai, “Deep learning in computer vision: A critical review of emerging techniques and application scenarios,” Machine Learning with Applications, vol. 6, no. August, p. 100134, 2021, doi: 10.1016/j.mlwa.2021.100134.
S. Alhazmi, M. Kutbi, S. Alhelaly, U. Dawood, R. Felemban, and E. Alaslani, “Utilizing Artificial Intelligence Techniques for Assisting Visually Impaired People: A Personal AI-based Assistive Application,” International Journal of Advanced Computer Science and Applications, vol. 13, no. 8, pp. 813–820, 2022, doi: 10.14569/IJACSA.2022.0130894.
A. McGibney, R. Pospisil, K. O’Mahony, J. F. Martinez, and S. Rea, “Spatial Mapping for Visually Impaired and Blind Using BLE Beacons,” in Proceedings of the 6th Collaborative European Research Conference (CERC 2020), 2020, pp. 15–29. [Online]. Available: https://www.ariadnegps.eu/
N. Tyagi, D. Sharma, J. Singh, B. Sharma, and S. Narang, “Assistive Navigation System for Visually Impaired and Blind People: A Review,” Proceedings - 2021 1st IEEE International Conference on Artificial Intelligence and Machine Vision, AIMV 2021, 2021, doi: 10.1109/AIMV53313.2021.9670951.
Z. Merchant, A. Anwar, E. Wang, S. Chattopadhyay, and J. Thomason, “Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision People.”, arXiv preprint arXiv:2407.08219, Jul. 202, doi: 10.48550/arXiv.2407.08219.
B. Fu, A. Hadid, and N. Damer, “Generative AI in the context of assistive technologies: Trends, limitations and future directions,” Image Vis. Comput., vol. 154, Feb. 2025, doi: 10.1016/j.imavis.2024.105347.
R. Adnin and M. Das, “‘I look at it as the king of knowledge’: How Blind People Use and Understand Generative AI Tools,” in The 26th International ACM SIGACCESS Conference on Computers and Accessibility, New York, NY, USA: ACM, Oct. 2024, pp. 1–14. doi: 10.1145/3663548.3675631.
Yuan, Zhiqiang, Ting Zhang, Yeshuang Zhu, Jiapei Zhang, Ying Deng, Zexi Jia, Peixiang Luo, Xiaoyue Duan, Jie Zhou, and Jinchao Zhang. "Walkvlm: Aid visually impaired people walking by vision language model." In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9845-9854. IEEE, 2025, doi: 10.1109/ICCV51701.2025.00918.
M. S. A. Baig, S. A. Gillani, S. M. Shah, M. Aljawarneh, A. A. Khan, and M. H. Siddiqui, “AI-based Wearable Vision Assistance System for the Visually Impaired: Integrating Real-Time Object Recognition and Contextual Understanding Using Large Vision-Language Models,” arXiv preprint arXiv:2412.20059, 2024, doi: 10.48550/arXiv.2412.20059.
D. Plikynas, A. Indriulionis, A. Laukaitis, and L. Sakalauskas, “Indoor-Guided Navigation for People Who Are Blind: Crowdsourcing for Route Mapping and Assistance,” Applied Sciences (Switzerland), vol. 12, no. 1, Jan. 2022, doi: 10.3390/app12010523.
Z. Chen, Z. Liu, K. Wang, K. Wang, and S. Lian, “A Large Vision-Language Model based Environment Perception System for Visually Impaired People,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, doi: 10.1109/IROS58592.2024.10801813.
R. Nishant, M. Kennedy, and J. Corbett, “Artificial intelligence for sustainability: Challenges, opportunities, and a research agenda,” Int. J. Inf. Manage., vol. 53, p. 102104, Aug. 2020, doi: 10.1016/J.IJINFOMGT.2020.102104.
D. Su and P. Fung, “Improving Spoken Question Answering Using Contextualized Word Representation,” ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2020-May, pp. 8004–8008, May 2020, doi: 10.1109/ICASSP40776.2020.9053979.
L. Manirajee, S. Q. Hanis Shariff, and S. M. Mohd Rashid, “Assistive Technology for Visually Impaired Individuals: A Systematic Literature Review (SLR),” International Journal of Academic Research in Business and Social Sciences, vol. 14, no. 2, 2024, doi: 10.6007/ijarbss/v14-i2/20827.
F. Atoillah, A. R. Gonzales, and J. Amirulloh, “Analisis Penggunaan Perangkat Android Dengan Perbandingan Parameter Benjamin Sparkman Fakultas Teknologi Informasi Yadika,” Jurnal Sistem Informasi Aplikasi Teknologi Informasi, vol. 1, no. 2, pp. 141–153, 2024.
Z. Huang and M. Benyoucef, “A systematic literature review of mobile application usability: addressing the design perspective,” Univers. Access Inf. Soc., vol. 22, no. 3, pp. 715–735, Aug. 2023, doi: 10.1007/S10209-022-00903-W.
A. Obaid, “Using Prototypes in Agile Software Development,” International Journal of Computers and Informatics, vol. 3, no. 2, pp. 23–38, Feb. 2024, doi: 10.59992/ijci.2024.v3n2p2.
K. Kurniawan and Y. Yulhendri, “Pemanfaatan Framework React Native dalam Perancangan Aplikasi Penjualan Merchandise,” Nucleus, vol. 4, no. 2, pp. 84–97, 2023, doi: 10.37010/nuc.v4i2.1369.
S. Abdelhadi, K. Elbahnasy, and M. Abdelsalam, “A PROPOSED MODEL TO PREDICT AUTO INSURANCE CLAIMS USING MACHINE LEARNING TECHNIQUES,” J. Theor. Appl. Inf. Technol., vol. 98, p. 22, 2020, [Online]. Available: www.jatit.org
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Smart Assistant System for Blind Users Based on Vision-Language Models and Contextual Location Assistance
Pages: 80-91
Copyright (c) 2026 Clay Aiken Mangeber Jr, Angeli Marselin Kolmate, Semmy Wellem Taju, Regi Fernando Najoan
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
