Gesture Detection and Capturing Using Brainypi

AI is a branch of computer science that deals with the creation of intelligent agents, which are systems that can reason, learn, and act autonomously. AI is used in a wide range of applications, including healthcare, finance, and transportation.

Computer vision is a field of computer science that deals with the extraction of meaningful information from digital images or videos. Computer vision is used in a wide range of applications, including self-driving cars, facial recognition, and medical imaging.

Gesture detection is a subfield of computer vision that deals with the identification and tracking of human gestures. Gesture detection is used in a wide range of applications, including gaming, virtual reality, and human-computer interaction.

These three technologies are often combined to create systems that can understand and respond to human gestures. For example, a self-driving car might use AI to identify and track other cars, pedestrians, and cyclists. It might also use computer vision to recognize traffic signs and signals. And it might use gesture detection to allow the driver to control the car with their hands or voice.

The combination of AI, computer vision, and gesture detection is opening up a new world of possibilities for human-computer interaction. These technologies are making it possible to create systems that are more natural and intuitive to use. They are also making it possible to create systems that can interact with us in new and innovative ways.

This project specifically involves the use of Brainypi to create a running software of gesture detection of multiple gestures and recognition models for live detection and capturing based on gestures.

• Libraries used : TensorFlow, Keras and OpenCV
• Brainypi
• Inbuilt - camera for live detection

We trained multiple individual models and models using multi-task learning to create a software which can capture and collect images for the gestures given in the models.

The first step was to collect the data, done by multiple data repositories and customized data collection by us. Next, we cleaned the data, segregated, adjusted the data according to the frames and sizes required by us for the model training. The images were defined into a specific size and the models were trained. The model involved deep CNN, pipeline models and detection models such as Faster RCNN. Although the use of HaarCascade xml files helped in detection of individual models, a pipeline and Faster RCNN were used separately for combined/multitasking models.

1. Capture Hand Gesture: The first step in the workflow is to capture an image or video of the user's hand gesture using a camera or sensor. The image or video is then fed into the hand gesture recognition software for analysis.

2. Preprocessing: Before the captured image or video can be analyzed, preprocessing is performed to enhance the image quality and remove any noise or unwanted elements. This may involve tasks such as background subtraction, image filtering, and normalization.

3. Feature Extraction: Once the image or video has been preprocessed, the hand gesture recognition software extracts features from the data that are relevant for gesture recognition. These features may include hand shape, finger position, and motion trajectory.

4. Classification: After the features have been extracted, the hand gesture recognition software uses a machine learning algorithm to classify the gesture based on the extracted features. The machine learning algorithm may be trained on a dataset of hand gestures to recognize common gestures such as a thumbs up, a wave, or a peace sign.

5. Gesture Interpretation: Once the gesture has been classified, the hand gesture recognition software interprets the gesture based on its context and intended meaning. For example, a thumbs up gesture may indicate agreement or approval in some contexts, while a wave gesture may indicate a greeting or goodbye.

6. Action: Finally, based on the interpretation of the gesture, the hand gesture recognition application performs an action or triggers an event. This could be as simple as displaying a message on the screen or as complex as controlling a robot or virtual reality environment. Once the gesture is detected, capturing takes place and the frame with the gesture is saved into the storage drive.

• Gestures : Smile detection, thumbs up, thumbs down, peace symbol and another 8 different types of simple hand gestures.

1. Remote Login to Brainypi : Using Linux or Ubuntu, login into Brainpy after the authentication process.
2. Clone the Repository : Clone the required repository with all the models and required files.
3. Training : Train and test the model on different sets of images to try and optimize any variations in the model.
4. Final testing : Ensure the model runs in an optimized with correct results using Brainypi.

One of the gesture results on Brainypi.

• BrainyPi
• TensorFlow
• Keras