Data collection and annotation framework for vision-based finger intent recognition

Abstract

Abstract. This study presents a functionally grounded framework for finger-intention recognition in robotic prototype, focusing on the rigorous collection, processing and annotation of egocentric visual data. Recordings were captured using smart glasses in realistic home environments, simulating object manipulation scenarios relevant to pediatric assistive applications. To mitigate the subjectivity inherent in user intention, a contact-based annotation logic was developed, ensuring objective class boundaries between transitional hand states. The methodology prioritizes data-centric principles, employing region of interest extraction, class balancing and multi-stage augmentation to enhance model robustness against occlusion and lighting variations. Training of MobileNetV2 and ResNet50V2 architectures on the curated dataset demonstrated stable learning dynamics and accurate detection of pre-shaping movements. The proposed framework closes the gap between raw visual input and real-time control requirements, supporting reliable vision-based actuation grounded in observed hand-object interactions.