Evaluasi Arsitektur Long Short-Term Memory untuk Klasifikasi Gestur Tangan Dinamis Berbasis MediaPipe Sebagai Kendali Presentasi

Abstract

Advances in Human-Computer Interaction (HCI) are driving the use of more natural interaction methods through hand gesture recognition technology. Dynamic gestures have an advantage over static gestures because they contain temporal information such as direction and movement patterns hat is more representative in conveying a command. However, the development of dynamic gesture recognition models still faces challenges in maintaining generalization capabilities for new users, necessitating an evaluation scheme capable of measuring model performance more representatively. This study aims to develop a dynamic hand gesture classification model for controlling PowerPoint presentations based on MediaPipe Hands, MediaPipe Pose, and Long Short-Term Memory (LSTM). The research stages included video data collection, hand and shoulder landmark extraction, data normalization using the midpoint of both shoulders as a reference point, training of eight variations of the LSTM architecture, and evaluation using the Leave-One-Subject-Out Cross-Validation (LOSO-CV) scheme, such that each participant took turns serving as test data to evaluate the model’s generalization ability toward new users. Test results show that the Baseline 1 (B1) architecture delivers the best performance with an average accuracy of 95.77%, precision of 96.03%, recall of 95.90%, and an F1-score of 95.84%, Analysis of the confusion matrix shows that most gestures were correctly recognized, while misclassifications occurred primarily in the “idle” class and for some gestures with similar hand poses. The results of the study indicate that the combination of MediaPipe and LSTM is capable of building a dynamic hand gesture classification model that maintains consistent classification performance in cross-subject testing.