A cross-subject multimodal static gesture dataset for a vision-based gestural calculator: Image–landmark Fusion

This paper proposes a multimodal approach dedicated to static hand gestures, integrated into a gesture recognition system based on computer vision. The system combines normalized RGB images, segmented landmarks, and 3D information, extracted using the MediaPipe library, to simultaneously capture both the visual appearance and the joint geometry of the hand. The dataset comprises 20 subjects, 13 classes of symbolic gestures, and 62,400 paired samples, with each instance consisting of a normalized RGB image of the hand and a vector of 63 normalized coordinates. Two unimodal branches are evaluated in a strict 5-fold cross-validation protocol: EfficientNet-B0 for the image and a lightweight MLP for the landmarks. The results show that the landmark branch achieves 98.99% accuracy, compared to 96.36% for the image branch. Late probability fusion offers the best overall performance, with 99.02% accuracy, 99.02% Macro-F1, and 99.82% Top-2 accuracy. These results validate, on the one hand, the strong discriminative power of hand geometry for static gestures, and establish, on the other hand, that visual information provides a significant complementary contribution, thereby contributing to overall robustness. Beyond the numerical results, the article discusses the role of cross-subject validation and the place of multimodal fusion in real-time natural interfaces in HCI and HRI.