Sistem Lost and Found Berbasis Vision-Language Retrieval dengan Explainable AI untuk Lingkungan Smart Campus
- Enrico Samuel Djimesha Universitas Klabat, Minahasa Utara, Indonesia
-
Regi Fernando Najoan
*
Universitas Klabat, Minahasa Utara,
Indonesia
- Argha Orion Silitonga Universitas Klabat, Minahasa Utara, Indonesia
- Wilsen Grivin Mokodaser Universitas Klabat, Minahasa Utara, Indonesia
- George Morris William Tangka Universitas Klabat, Minahasa Utara, Indonesia (*) Corresponding Author
Abstract
Evidence for combined image-text retrieval in campus lost-and-found systems remains limited, and zero-shot CLIP's ability to distinguish item attributes and identities is uncertain. This study evaluates CLIP ViT-B/32 on an item-image collection covering eight categories. The controlled multimodal evaluation uses 404 images, 85 metadata-defined relevance groups as proxy ground truth, and 237 leave-one-out queries. Separate image-only validation uses 387 images from 79 provenance-based Object IDs after duplicate removal and quality control; each Object ID contains multiple images of the same physical item. In Object-ID validation, CLIP achieved Recall@1 of 0.674, Recall@10 of 0.935, and mAP of 0.419 (95% object-cluster bootstrap CI: 0.369–0.469). In multimodal evaluation, image-only achieved mAP 0.624, text-only 0.409, and image+text reached the best mAP of 0.732 at α=0.5, selected in four of five group cross-validation folds. Ablation showed that the single fused-index outperformed the three-index configuration (mAP 0.715 versus 0.692); therefore, three-index is not claimed as an accuracy improvement. Direct Indonesian input to English CLIP achieved mAP 0.163, compared with 0.363 using multilingual CLIP and 0.409 through translation. Attention Rollout is positioned only as a supporting visualization because its deletion-curve AUC of 0.871 lacks a random-occlusion baseline. These results indicate the benefit of image-text fusion under controlled evaluation but do not establish performance for free-form user descriptions or production readiness.
Downloads
References
S. Y. Tan and C. R. Chong, “An Effective Lost and Found System in University Campus,” Journal of Information System and Technology Management, vol. 8, no. 32, pp. 99–112, 2023, doi: 10.35631/JISTM.832007.
J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 8, pp. 5625–5644, 2024, doi: 10.1109/TPAMI.2024.3369699.
F.-L. Chen et al., “VLP: A Survey on Vision-Language Pre-Training,” Machine Intelligence Research, vol. 20, no. 1, pp. 38–56, 2023, doi: 10.1007/s11633-022-1369-5.
S. R. Dubey, “A Decade Survey of Content-Based Image Retrieval Using Deep Learning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 5, pp. 2687–2704, 2022, doi: 10.1109/TCSVT.2021.3080920.
W. Xiong, H. Liu, S. Mi, and Y. Zhang, “Asymmetric Bi-Encoder for Image-Text Retrieval,” Multimed. Syst., vol. 29, no. 6, pp. 3805–3818, 2023, doi: 10.1007/s00530-023-01162-2.
S. Yang et al., “Semantic Completion and Filtration for Image-Text Retrieval,” ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 19, no. 4, pp. 1–20, 2023, doi: 10.1145/3572844.
Z. Liu, F. Chen, J. Xu, W. Pei, and G. Lu, “Image-Text Retrieval With Cross-Modal Semantic Importance Consistency,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 5, pp. 2465–2476, 2023, doi: 10.1109/TCSVT.2022.3220297.
H. Zhu, C. Zhang, Y. Wei, S. Huang, and Y. Zhao, “ESA: External Space Attention Aggregation for Image-Text Retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 10, pp. 6131–6143, 2023, doi: 10.1109/TCSVT.2023.3253548.
M. Cao, Y. Bai, Z. Cao, L. Nie, and M. Zhang, “Efficient Image-Text Retrieval via Keyword-Guided Pre-Screening,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 6, pp. 5132–5145, 2024, doi: 10.1109/TCSVT.2023.3339489.
L. Meng, F. Zhang, X. Zhang, and C. Xu, “Prototype Local-Global Alignment Network for Image-Text Retrieval,” Int. J. Multimed. Inf. Retr., vol. 11, no. 4, pp. 525–538, 2022, doi: 10.1007/s13735-022-00258-1.
P. Gao et al., “CLIP-Adapter: Better Vision-Language Models With Feature Adapters,” Int. J. Comput. Vis., vol. 132, no. 2, pp. 581–595, 2024, doi: 10.1007/s11263-023-01891-x.
Y. Zhu et al., “Enhancing Few-Shot CLIP With Semantic-Aware Fine-Tuning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 36, no. 7, pp. 12349–12362, 2025, doi: 10.1109/TNNLS.2024.3443394.
X. Wang, J. Jin, C. Li, J. Tang, C. Zhang, and W. Wang, “Pedestrian Attribute Recognition via CLIP-Based Prompt Vision-Language Fusion,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 35, no. 1, pp. 148–161, 2025, doi: 10.1109/TCSVT.2024.3454366.
R. Chen et al., “CLIP-Based Semantic Refinement Method for Image-Text Retrieval,” Comput. J., vol. 68, no. 10, pp. 1377–1385, 2025, doi: 10.1093/comjnl/bxaf044.
J. Wu and G. Feng, “CLIP-UG: CLIP-Driven Vision-Language Model for UAV-View Geo-Localization,” IEEE Transactions on Consumer Electronics, vol. 71, no. 4, pp. 10096–10107, 2025, doi: 10.1109/TCE.2025.3600486.
T. Yao, S. Peng, L. Wang, Y. Li, and Y. Sun, “Cross-Modality Interaction Reasoning for Enhancing Vision-Language Pre-Training in Image-Text Retrieval,” Applied Intelligence, vol. 54, no. 23, pp. 12230–12245, 2024, doi: 10.1007/s10489-024-05823-1.
T. Abhiram, A. K. EB, M. KM, V. AP, and A. A, “CampusTrace: Application for Lost and Found Items on Campus,” Journal of Trends in Computer Science and Smart Technology, vol. 6, no. 2, pp. 168–179, 2024, doi: 10.36548/jtcsst.2024.2.006.
M. Zhou, I. Fung, L. Yang, N. Wan, K. Di, and T. Wang, “LostNet: A Smart Way for Lost and Find,” PLoS One, vol. 19, no. 10, p. e0310998, Oct. 2024, doi: 10.1371/journal.pone.0310998.
T. Alpay, S. Magg, P. Broze, and D. Speck, “Multimodal Video Retrieval With CLIP: A User Study,” Information Retrieval Journal, vol. 26, no. 1–2, p. 6, 2023, doi: 10.1007/s10791-023-09425-2.
Z. Cheng, Y. Wu, Y. Li, L. Cai, and B. Ihnaini, “A Comprehensive Review of Explainable Artificial Intelligence (XAI) in Computer Vision,” Sensors, vol. 25, no. 13, p. 4166, 2025, doi: 10.3390/s25134166.
A. Radford et al., “Learning Transferable Visual Models from Natural Language Supervision,” in Proceedings of the 38th International Conference on Machine Learning (ICML), 2021, pp. 8748–8763. Accessed: Aug. 21, 2026. [Online]. Available: https://proceedings.mlr.press/v139/radford21a
A. Dosovitskiy et al., “An Image Is Worth 16 x 16 Words: Transformers for Image Recognition at Scale,” in Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021. Accessed: Aug. 21, 2026. [Online]. Available: https://arxiv.org/pdf/2010.11929
S. Abnar and W. Zuidema, “Quantifying Attention Flow in Transformers,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 4190–4197. doi: 10.18653/v1/2020.acl-main.385.
V. Petsiuk, A. Das, and K. Saenko, “RISE: Randomized Input Sampling for Explanation of Black-Box Models,” in Proceedings of the British Machine Vision Conference (BMVC), 2018. doi: https://doi.org/10.48550/arXiv.1806.07421.
F. Carlsson, P. Eisen, F. Rekathati, and M. Sahlgren, “Cross-Lingual and Multilingual CLIP,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France: European Language Resources Association, 2022, pp. 6848–6854. [Online]. Available: https://aclanthology.org/2022.lrec-1.739/
J. Johnson, M. Douze, and H. Jégou, “Billion-Scale Similarity Search with GPUs,” IEEE Trans. Big Data, vol. 7, no. 3, pp. 535–547, Jul. 2021, doi: 10.1109/TBDATA.2019.2921572.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Sistem Lost and Found Berbasis Vision-Language Retrieval dengan Explainable AI untuk Lingkungan Smart Campus
Pages: 1-12
Copyright (c) 2026 Enrico Samuel Djimesha, Regi Fernando Najoan, Argha Orion Silitonga, Wilsen Grivin Mokodaser, George Morris William Tangka
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
