Measuring Finger Positions on a Violin With ESP32

As a violin player I always wanted an app or tool that could show me the position of my fingers on the violin very precise. With this project I tried to build this. Although this is a prototype and you could still add many features.

I also tried to separate the ESP32 and the rPI and thus I made the ESP32 send data wireless to the rPi. Which is probably the most difficult thing about this project.

It is also very important that at the end of this project nothing is stored on your computer but it is either on the rPI or ESP32.

Before getting into the specifics of building this project we need a few things.

1. 4x Linear Softpot: Linear Potentiometers to measure the position of a finger (a violin has 4 strings)
2. ESP32: An ESP32 module to read the data from the linear softpots.
3. a 4/4 violin: a violin to place the linear softpots on top.
4. a Raspberry Pi with an SD card: a raspberry pi which will store our database and website.
5. 10k potentiometer: a potentiometer for the brightness of the LCD
6. LCD-screen: an LCD screen to show to the ip addres of the rPi
7. Soldering Kit: For soldering all the elements togheter
8. Male-to-male wires and male-to-female wires: Cables for connecting all the elements
9. Micro USB cable: For powering the ESP32

First of all we need to connect our softpots to the esp32. We connect the left and right pins to the 5V and GND respectively. We connect the middle pin to an analog pin on the ESP32. We also need to connect the middle pin with a resistance of 10k ohm and connect this to the GND. This is so that our output of the softpots doesn't return random value.

Then we connect the ESP32 with the micro usb cable to our pc so that we can upload code to it. We will use the Arduino IDE for programming the ESP32. But first we need to install the Arduino core for the ESP32 so that we can upload to it. This can be done here.

Then we can begin to write code.

First we need to assign our pins to which we connected our middle pin of the softpots.

const int SOFT_POT_PIN1 = 34;

const int SOFT_POT_PIN2 = 35;

const int SOFT_POT_PIN3 = 32;

const int SOFT_POT_PIN4 = 33;

unsigned long onTime;

unsigned long softPotTime;

Then we can set up our pins. And we need to start our serial monitor and our time.

void setup() {

onTime = millis();

Serial.begin(115200);

Serial.println("Program start");

pinMode(SOFT_POT_PIN1, INPUT);

pinMode(SOFT_POT_PIN2, INPUT);

pinMode(SOFT_POT_PIN3, INPUT);

pinMode(SOFT_POT_PIN4, INPUT); }

void getdata(byte pdata[]) {

// Read in the soft pot's ADC value

Then we need to read our pins so we can receive our data.

int softPotADC1 = analogRead(SOFT_POT_PIN1);

nt softPotADC2 = analogRead(SOFT_POT_PIN2);

int softPotADC3 = analogRead(SOFT_POT_PIN3);

int softPotADC4 = analogRead(SOFT_POT_PIN4);

Then we put the values in a list so we can easily output it later.

for (int i=0; i < 4; i++){

int Names[] = {softPotADC1,softPotADC2,softPotADC3,softPotADC4};

int softpot = Names[i];

if (softpot > 10){

pdata[0] = i;

pdata[1] = softpot;

pdata[2] = millis();

} } }

}

For connecting the ESP32 and RPI wirelessly, we will use a library called websocket. To install this library, we can get the files here. We will need to change some code in the files itself in order to use this library for the ESP32.

We will need to change the MD5.c and MD5.h.

• MD5Init to MD5InitXXX
• MD5Update to MD5UpdateXXX
• MD5Final to MD5FinalXXX

We will also need to delete the avr/io.h lines in the sha1 files.

Then we can add the library to our Arduino IDE by sketch > include library > add .ZIP library and then we can select your library in a zip file.

After that we can start to write our code.

First for the ESP32:

Including our library

#include
#include

Assigning our pins again.

const int SOFT_POT_PIN1 = 34;

const int SOFT_POT_PIN2 = 35;

const int SOFT_POT_PIN3 = 32;

const int SOFT_POT_PIN4 = 33;

Assigning our wifi server

WiFiServer server(80);

Starting our websocket server

WebSocketServer webSocketServer;

Assigning our SSID and password of your wifi

const char* ssid = "your wifi SSID";

const char* password = "your wifi password";

void setup() {

Setting up your serial monitor

Serial.begin(115200);

Setting up your softpots

pinMode(SOFT_POT_PIN1, INPUT);

pinMode(SOFT_POT_PIN2, INPUT);

pinMode(SOFT_POT_PIN3, INPUT);

pinMode(SOFT_POT_PIN4, INPUT);

Starting our wifi and connecting to it

WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) {

delay(1000);

Serial.println("Connecting to WiFi.."); }

Serial.println("Connected to the WiFi network");

Serial.println(WiFi.localIP());

server.begin(); delay(100); }

void getdata(char *pdata) {

Reading your data

// Read in the soft pot's ADC value

int softPotADC1 = analogRead(SOFT_POT_PIN1);

int softPotADC2 = analogRead(SOFT_POT_PIN2);

int softPotADC3 = analogRead(SOFT_POT_PIN3);

int softPotADC4 = analogRead(SOFT_POT_PIN4);

Placing the data in a list and converting it to hexadecimal.

sprintf(pdata, "%x,%x,%x,%x,%x", softPotADC1, softPotADC2, softPotADC3, softPotADC4, millis());

}

void loop() {

Connecting your client (the rPI)

WiFiClient client = server.available();

if (client.connected()) {

delay(10);

if (webSocketServer.handshake(client)) {

Serial.println("Client connected");

Sending and receiving data.

while (client.connected()) {

char data[30];

getdata(data);

Serial.println(data);

webSocketServer.sendData(data);

delay(10); // Delay needed for receiving the data correctly }

Serial.println("The client disconnected");

delay(100); }

else {

Serial.println("shitsfuckedyo");

} } }

Then for the rPI in python:

Importing our libraries

import websocket
import time

Assasigning a globale variabel i

i = 0

Setting a maximum of 200 messages that we can receive

nrOfMessages = 200

class Websocket():

def __init__(self):

Initialising our websocket and connecting it to our ESP32

self.ws = websocket.WebSocket()

self.ws.connect("ws://172.30.248.48/")

Receiving our data

def work(self):

self.ws.send("message nr: 0" )

result = self.ws.recv() time.sleep(0.5) return result

Closing the websocket after receiving everything

def close(self):

self.ws.close()

As for connecting our Database and website, you will first of all need to create your database on the pi by installing mariadb: sudo apt install mariadb.

Then you can acces it by doing: sudo mariadb.

Then you will also need to create your website. You can do this however you like, but you have to use Flask and you need to have a form in your HTML for stopping and starting your data.

Then you can insert this code to connect your database and your website (both your website and database have to both be on your pi, this can be done by using the deployment tab in settings of pycharm)

from flaskext.mysql import MySQL

app.config["MYSQL_DATABASE_HOST"] = "localhost"

app.config["MYSQL_DATABASE_DB"] = "your database name"

app.config["MYSQL_DATABASE_USER"] = "your database user"

app.config["MYSQL_DATABASE_PASSWORD"] = "your database password"

Function for getting data out of our database.

def get_data(sql, params=None):

conn = mysql.connect()

cursor = conn.cursor()

print("getting data")

try:

print(sql)

cursor.execute(sql, params)

except Exception as e:

print(e)

return False

result = cursor.fetchall()

data = []

for row in result:

data.append(list(row))

cursor.close()

conn.close()

return data

Function for inserting data into our database

def set_data(sql, params=None):

conn = mysql.connect()

cursor = conn.cursor()

try:

log.debug(sql)

cursor.execute(sql, params) conn.commit()

log.debug("SQL uitgevoerd")

except Exception as e:

log.exception("Fout bij uitvoeren van sql: {0})".format(e))

return False

cursor.close()

conn.close()

return True

We will also need to thread our application so that you can do other things while you are recording.

class ThreadedTask(threading.Thread):

def __init__(self, ):

Setting up thread

threading.Thread.__init__(self)

Creating a list for holding all of your received data

self.data_all =[]

def run(self):

time.sleep(5)

Import your own python code where your receive the data

import receive_websocket

Receive your data

w = receive_websocket.Websocket()

Append your data in your list and print it.

for i in range(0, 200):

self.data_all.append(w.work().split(","))

print(self.data_all)

task = ThreadedTask()

Then you can do task.run() to start your Thread and begin to receive data.

For running your website from your Pi you have to use a service:

[Unit]
Description=uWSGI instance to serve project1 web interface

After=network.target

BindsTo=mysqld.service

After=mysqld.service

[Service]

Change to your user

User=pi

Group=www-data

Here you have to enter your directory of your Flask file

WorkingDirectory=/home/pi/project1/web

Directory of your ini file that can be found later.

ExecStart=/usr/bin/uwsgi --ini /home/pi/project1/conf/uwsgi-flask.ini

[Install]

WantedBy=multi-user.target

uwsgi-flask.ini that you need to place in the directory you specified in ExecStart above

[uwsgi]
module = web:app virtualenv = /home/pi/project1/env

master = true processes = 5

plugins = python3

socket = project1.sock chmod-socket = 660 vacuum = true

die-on-term = true

Now you can read your data and display it on your website.

We can connect an LCD screen so that we can show the ip-adress of our Pi for our website.

import RPi.GPIO as GPIO
import time

import commands

GPIO.cleanup()

D0 = 22

D1 = 5

D2 = 6

D3 = 13

D4 = 19

D5 = 26

D6 = 20

D7 = 21

list = [22,5,6,13,19,26,20,21]

E = 24

RS = 23

class Screen:

def __init__(self):

GPIO.setmode(GPIO.BCM)

self.setup()

#Function set self.stuur_instructie(0x3f) #Display self.stuur_instructie(0x0c) #On + cursor self.stuur_instructie(0x01) @staticmethod def setup(): GPIO.setup(list, GPIO.OUT) GPIO.setup([E,RS],GPIO.OUT)

def stuur_instructie(self,byte):

GPIO.output(E,GPIO.HIGH)

GPIO.output(RS,GPIO.LOW)

self.set_GPIO_bits(byte)

time.sleep(0.005)

GPIO.output(E,GPIO.LOW)

def stuur_teken(self,char):

temp = ord(char)

GPIO.output(E,GPIO.HIGH)

GPIO.output(RS,GPIO.HIGH)

self.set_GPIO_bits(temp)

time.sleep(0.005)

GPIO.output(E,GPIO.LOW)

def set_GPIO_bits(self,byte):

for i in range(0,8):

if (byte & (2**i)) == 0:

GPIO.output(list[i],GPIO.LOW)

else:

GPIO.output(list[i], GPIO.HIGH)

def main():

s = Screen()

teken = "Local IP address:"

for letter in teken:

s.stuur_teken(letter)

teken2 = commands.getoutput("ip addr show wlan0 | grep -Po 'inet \K[\d.]+'")

print(teken2)

s.stuur_instructie(0xc0)

for letter2 in teken2:

s.stuur_teken(letter2)

if __name__ == '__main__': #Program starting from here

try:

main()

except KeyboardInterrupt:

pass

Then we can create a service to start the LCD on start-up.