Speech Cum Gesture Control Music System

Whenever we think about any smart and intelligent device or system then JARVIS(iron man) like system comes in our mind.but presently the smart system like SIRI , CORTONA they only used speech as input.i think gesture play an important role to make the device intelligent. when we talk to each other speech and gesture both are as important in conversation.

so i made a small music system which work on unique combination of speech as well as gesture to understand us better.

here a used raspberry pi since it has GPIO pin for understanding different gesture and inbuilt good quality audio processor.

here i divided the whole process in 3 different step

1. gesture

2.speech recognition

3 controlling music system(play pause volume + /- )

1. Raspberry pi

2. 3 pair of IR sensor(gesture)

3. good quality of speaker

4. webcam(as mic for speech recognition)

5. power bank

here i try to use simplest type of gesture.here i try to replicate the system that are present in normal music system with gesture.In normal music system their are three button or switch +, || , - (volume increase ,pause/play, volume decrease).when we press the volume buttom single times to change volume and press long times to change the song.so in total it forward or backward the songs,volume increase or decrease,pause play the song.

therefore i am trying to do the same work by using 3 IR sensors.One sensor is in left side (L),one in the middle(m) and one in the right side(R).

IR sensor gives output HIGH when any object come near to it (IR reflect back from the object ).sensors are connected to the GPIO pin of raspberry pi. Since raspberry pi works 3.3 v logic means takes max 3.3v input so if the sensors are 5v logic then you must change it first into 3.3v logic by using simple voltage voltage divider circuit.In step 4 control unit i am discussed in details

There are many way to do the Speech recognition in raspberry pi. for example

1.Jasper – Voice Recognition Software.

2.Raspberry Pi Voice Recognition by Oscar Liang.

3. Raspberry Pi Voice Control by Steven Hickson.

Out of these three software i used Steven Hiskson voice command v3.0 . it most precise because it used google voice API .it gives JARVIS like interface .you have to change its command according to the command required.during installation you may face some problem so read all its comments there you will get your solution.here webcam is used as mic for user voice input.

After the setup of gesture and speech recognition ,now you need a platform to used both.

i used python IDE in raspberry pi to write code.for controlling music system Omxplayer is best software which is inbuilt .you can get all its command of omxplayer in link. http://elinux.org/Omxplayer .

Now at last in python script you have write a simple code that when system recognized a specific gesture or voice command than called respective command of Omxplayer .so that the song changed or other function happened.

I am doing in my own way its up to you how you set it for example as per the video you can see that

1. moving hand left to right --- L- (high) --(0.4s) --M (high) - then- forward song

2 moving hand right to left --- H-(high)--0.4s-M(high) ---then---previous song.

3 placing hand in the middle --M - (high) -- stop or change the playlist

4 single move M-high or L ---high-- pause or resume

5. placing hand left side more then 0.5s (long press) -- L -high- after 0.5s again L --high-- then volume decrease.

6. placing hand right side more then 0.5s (long press)---H-high after 0.5s again H --high ----then volume increase.

when ever only M is high voicecommand v3.0 start and wait for any voice command from the user.

see the video for better understanding

This system is completely independent here computer is only to show the command window .Through this music system i wants to so that how Gesture and speech recognition together make the things smart and intelligent.you can used this system in different purpose.may this project help you make the things easier.thank you