How to Make Smart Glasses! (Revised)
by drjhxie in Circuits > Raspberry Pi
2856 Views, 11 Favorites, 0 Comments
How to Make Smart Glasses! (Revised)
Hello, and today we are going to make a DIY Smart Glass! Some of the code is from this instructable, but some of it is outdated and it just does not work(From my side), so in this instructable, I am going to revise and fix the code and equipment.
Supplies
- Raspberry Pi Zero W (~$10 at Micro Center)
- Bluetooth Earbuds. (Any type will work) (~$5-$15)
- Lipo battery 1100 mAh (This is what I used)
- Adafriut 500c (This is what I used)
- Clear OLED display. (This is what I used)
And of course, a 3d printer like an ender 3, and a soldering iron.
If you don't have a 3d printer, you can email drjhxie@gmail.com and I'll print them for you at a cheap price!
Get the Applications & APIs
Applications you will need:
PuTTY is the software we will use to ssh into the RaspberryPI before accessing the GUI desktop. To download it click the link and hit 'here'. This will take you to a new page where you can install either the 32bit or 64bit version.
We will need BalenaEtcher to flash the Raspbian OS to the SD-card. Click the link to get to their website where you can choose your specific operating system and 32bit or 64bit version.
- Microsoft Remote Desktop Classic
Remote Desktop Classic is what we need to access the RaspberryPI GUI. Remote Desktop Classic is available for download in the Microsoft store. If you are on mac a possible alternative is Chrome Remote Desktop.
After connecting the RaspberryPI 0w to the internet we will need to access it via its IP address. Angry IP Scanner scans all the devices on your wireless network. To install it go to the link and click 'free download' then '32/64-bit Installer'. This installer will auto-detect whether you have a 32bit or 64bit system.
Cura is the slicing software needed to convert the STL files to G-code files which will then be loaded onto your 3D printer. Upon visiting their website you must click download and then select your operating system. Cura is only available on 64bit systems, but there are alternatives like Slic3r. Although I recommend using Cura if possible.
APIs you'll need:
One of the many features the smart glasses include sending SMS or text messages. To accomplish this task we will use Twillio which is a 'free' to use communication API that is capable of sending or receiving SMS or calls. The reason I say 'free' in brackets is that this service is only free until the graciously provided trial balance of $15.00 runs out; although this might not sound like a lot on the surface until you realize that sending SMS only costs a little over one cent per message. That is approximately five-hundred messages.
The set-up process is simple and easy but may take a while at first. Go to the website and sign-up for a free trial account. After verifying your phone number Twilio will ask you some questions. For the first two questions just put other then select 'with code' then 'Python' and finally 'No, I want to use my own hosting service'. Now you are inside your console which will display anything and everything about your account and your apps. Click 'Get a trial number' this is your phone number Twilio will use to send SMS. For the trial account, we have to verify the numbers we would like to send SMS to. In order to do that go to your console and on the right you should see a link that says 'Verified Numbers' your number will already be there, but if you are ever going to want to message someone else you will have to add their phone number here. That should be it, for now, leave the console open we will come back to it later when we build the SMS feature in the code.
Last but not least is Wolfram Alpha which is a computational search engine. Wolfram Alpha will be the brains of the smart glasses. Wolfram Alpha allows us to get answers to any question we may ask or math problem we present it with. Sign-up is very straight forward just head to the link and sign-up for an account then click on your profile in the top right then 'My Apps (API)'. Now click 'Get AppID' where it will ask you for some description about your app then finally it will give you your app-ID. Keep this screen open we will come back to it later in the code.
Setting Up the Raspberry PI
If you are new to installing an Operating system on a Raspberry Pi, follow these steps:
- Go to www.raspberrypi.com/software/
- Download the imager by clicking the download button when you scroll down
- Open the App, click "Operating System," click Raspberry Pi OS (Other)
- Scroll down and click Raspberry Pi OS Lite (32-bit or legacy)
- Insert SD Card
- Click Storage and click on your Micro SD card.
- Click Write
- Insert SD card to the Raspberry Pi
- Connect a good power supply to the Raspberry Pi.
SSH Into the Raspberry Pi
Accessing with SSH
Now that your RaspberryPI is ready to connect to your network power it up by connecting the micro-usb port labeled 'PWR' to your laptop. Please note that the RaspberryPI 0w is a 2.4ghz device meaning that if you only have a 5ghz network you won't be able to connect to the RaspberryPI wirelessly; however, most networks have a 2.4ghz and 5ghz band. If your computer isn't already connected to the 2.4ghz band go ahead and connect it. Next, you are going to have to figure out the IP address of the RaspberryPI by scanning all of the devices on your network. Open Angry IP Scanner and scan your network by clicking the start button in the top right. Your RaspberryPI should have a hostname of 'raspberrypi', so find that device in the list and note the IP address. Now that you have the IP-address open PuTTY. Enter the IP address in the text bar at the top and check the box labeled ssh below it then click open.
Updating the RaspberryPI 0w
You should now be prompted with a window that says log in. The default username is 'pi' and the password is 'raspberry', be careful when typing the password as you won't be able to see it for security reasons.
As with any new RaspberryPI we need to do some updates first. Type these commands into the terminal one by one.
sudo apt-get update sudo apt-get upgrade sudo apt-get autoremove sudo reboot
Note: This could take a while, so make sure you have a stable internet connection before you start.
Enabling the GUI
Before we enable the GUI we need to change the hostname so if you ever get another RaspberryPI 0w you won't have to worry about connecting to the wrong one. Type 'sudo raspi-config' after which you will be prompted with a new window. Using your arrow keys navigate to the second option and hit enter then type in your new hostname. Don't exit out of this window quite yet, we also need to enable the SSH server so go to 'Interfacing options' then scroll down to where it says SSH then hit enter. For now, we are done working inside this window so click finishes and then reboot. Now SSH is enabled and ready to use, but we still have to install another application, xrdp. This essentially communicates to the remote desktop application on your computer. To install it start another Putty session and then type 'sudo apt-get install tightvncserver' and then 'sudo apt-get install xrdp'.
Viewing the GUI
You are now good to go ahead and access the GUI, so close out of that Putty session and head over to the remote desktop. Hit 'add' in the top right corner then 'Desktop' and enter the IP address of the RaspberryPI 0w. As long as your RaspberryPI is powered up click on the tab with the IP address and it will wirelessly access the GUI desktop.
Downloading Libraries & Music
Installing Libraries
Although Python comes pre-installed on the RaspberryPI we still need to install some libraries necessary for the glasses. Below is the command for installing all of the packages not included in the initial Python install.
pip install bs4 urllib3 luma.core luma.oled wolframalpha SpeechRecognition PyAudio wikipedia picamera dropbox twilio
Transfering the Music
Now we have to get some music files onto the RaspberryPI, so we can play some dank tunes on our spectacular glasses. We are going to first download them from our PC and then copy and paste them onto the RaspberryPI. Technically, you could download them straight from the Pi, but that would be very slow. First, download this song and convert it to wav format with an online converter. Now rename it to 'Imagine.mp3' and copy and paste it onto the RaspberryPi desktop. Although you can download any song you like just make sure you save it on the desktop in wav format and you change the name in the code so it matches the file name.
Connecting Wireless Headphones/Earbuds
In order to get user input and provide feedback we will need to connect the wireless headphones to the RaspberryPI 0w. There are multiple ways to do this, I will show you two ways, so if the first method doesn't work you can always try the second option.
First Method
This method is by far the easiest as you can accomplish it with only two clicks. First make sure your bluetooth headphones are in pair mode, if you are using the same ones that I recommended just hold down the power button until the LED starts rapidly flashing. Now that your headphones are in pair mode click on the bluetooth icon in the top right of the screen and select tab labeled GS. If it pairs you should see a green check mark in that menu and the LED should stop flashing. If you were able to get this method to work then you can move on to the next step. For the rest of us who weren't so lucky try the second method.
Second Method
This method is a bit more in-depth, but shouldn't be to difficult.
First we need to initialize the Bluetooth agent. Type these commands into the shell editor one-by-one.
bluetoothctl power on agent on default-agent
Turn the headphones to pair mode by holding down the power button. Now type this command.
scan on
While it's scanning you should see the MAC address of your headset appear. If you don't know what the MAC address of your device is I recommend connecting them to your phone where it should show the address in your wireless device settings. After you have seen your MAC address type this command to stop the scan.
scan off
Finally, all that is left is to pair the device and connect it. Type these commands in the shell editor one-by-one making sure to replace the asterisk with the MAC address for your device.
pair * trust * connect *
The Raspberry Pi Speaks!
Text-To-Speech
The main way the smart glasses will relay the information is through the chat bot. The chat bot will work with what is called a text-to-speech library that just takes a normal input string or text and converts that into an MP3 file. Originally, I wanted to use the pyttsx3 library to drive the chat bot and I have gotten it to work in the past on the RaspberryPi 0w; however, when I tried it more recently it didn't work. I believe the problem arose from updates to the RaspberryPI's operating system and various Python libraries. After some quick research I discovered a different speech engine, Espeak, and while I don't think Espeak is as good as pyttsx3 it will accomplish what we need it to.
Download
To download it open the shell editor and enter the following command.
sudo apt-get install espeak
To test Espeak we will give it some text which it will hopefully dictate back to us! If your headphones aren't connected anymore go ahead and power them on. Type the following line into the shell editor and you should hear a talking Pi!
Note: Wherever you see a 'XX:XX:XX:XX:XX:XX' replace it with the MAC address of your wireless headset.
espeak "Hello World" -ven+f3 -k5 -s125 --stdout | aplay -D bluealsa:DEV=XX:XX:XX:XX:XX:XX,PROFILE=sco<br>
Modify the Voice
On it's own Espeak has a pretty robotic sound to it, but thankfully we can modify the tone, gender, speed, and much more. Below are a couple of modifications you can make, but if you would like a full list visit this link. All it takes a couple of modifications and an imagination, so make it yours!
- -ven ~ Specifies the language. Change the en to the abbreviation of whatever language you should desire.
- -f ~ Speaks a text file
- -a ~ Set amplitude, range 0-200.
- -s ~ Changes the speed. Lower limit 80 and no maximum.
- -g ~ Increases the pause between words in 10ms.
- -f3 ~ Changes to the third female voice. There are ten options, five male and five female. (e. m1, f2...)
Setting Up the Camera
Setup
Another one of the amazing features of the smart glasses includes taking videos and pictures! Taking pictures with the RaspberryPI 0w is very straightforward! All you have to do is plug the camera into the ribbon cable connector by pulling back on the black lever located near the end. To actually take the picture we first have to enable the camera, so click on the RaspberryPI icon in the top left and scroll down to preferences->Raspberry PI Configuration->Interfaces. If you are wondering where you are this is just the GUI version of the raspi-config menu we accessed earlier in the tutorial. Inside the menu, you should see an option to enable the camera, then we need to reboot. Go to the shell editor and type sudo reboot.
Taking Pictures
You are now ready to start taking beautiful pictures. Open up the shell editor and type the following.
raspistill -o Image.jpg
After hitting enter the indicator LED on the camera should turn on for a few seconds then the picture named Image.jpg will get saved in /home/pi directory.
Setting Up the OLED Display
Enabling I2C
Finally, we have made it to one the coolest parts about the smart glasses, the transparent OLED. This is where we are able to display information to the user such as, time, temperature, and just general info. As I said previously, this display will work via I2C and the luma.core python library. By default, I2C isn't enabled, so we must enable it. Go to the same place you enabled the camera module and enable I2C then reboot.
Wire It!
To connect up the wires me must solder them from the driver to the RaspberryPi. I recommend a soldering temperature of 725°F/385°C. Don't forget to power off your RaspberryPi and disconnect any peripherals. Here are the connections.
OLED---RaspberryPI
GND-GND
3v3-3v3
SDA-GPIO_2(SDA)
SCL-GPIO3_(SCL)
The Code!
I edited the code, making it simpler to use, made it up to date, added a few more features, and most of all, made it actually work.
Note: I could not make the dropbox work, so I just made it so the pics and videos save to the desktop.
########Section_1########## import sys import bs4 from bs4 import BeautifulSoup as soup from urllib.request import urlopen import os import sys #from luma.core.interface.serial import i2c #from luma.core.render import canvas #from luma.oled.device import ssd1306, ssd1325, ssd1331, sh1106 import time import datetime import wolframalpha import speech_recognition as sr import pyaudio import wikipedia from picamera import PiCamera import dropbox import picamera from twilio.rest import Client from time import sleep camera = PiCamera() ##########Section_2########### contacts= {'Mom': "111-111-1111", 'mom': "111-111-1111", 'someone': "111-111-1111", 'Someone': "111-111-1111"} i = int(0) x= 0 v= 0 #Wolfram app_id = (' ') #WolfRam Alpha ID #Dropbox dropbox_access_token= " " #Unique Dropbox token #Twilio twilio_account_sid = ' ' #You will find this information in your Twilio account twilio_auth_token = ' ' twilio_client = Client(twilio_account_sid, twilio_auth_token) client = wolframalpha.Client(app_id) res = client.query('what is the temperature in Vienna') #Replace with the name of your city answer = next(res.results).text str1 = answer str2 = str1.split('(', 1)[0] bad_chars = [';', '|', '(', ')', '+', '=', '1', "'", '"'] #We use this to remove all of the characters that Espeak can't say num = ["05", "10", "15", "20", "25", "30", "35", "40", "45", "50", "55", "0"] #At these points in every hour the glasses will refresh the weather data #######Section_3################# def util2(): cdt = datetime.datetime.now() min1 = str(cdt.minute) hour = str(cdt.hour) with canvas(device) as draw: draw.text((0, 0), hour, fill = "blue") draw.text((11, 0),":", fill = "blue") draw.text((15, 0), min1, fill = "blue") draw.text((0, 0), "___________", fill = "yellow") draw.text((0, 9), date, fill = "white") draw.text((33, 0), str2, fill = "white") draw.text((0, 115), "...", fill = "white") def util3(): cdt = datetime.datetime.now() min1 = str(cdt.minute) hour = str(cdt.hour) with canvas(device) as draw: draw.text((0, 0), hour, fill = "blue") draw.text((11, 0),":", fill = "blue") draw.text((15, 0), min1, fill = "blue") draw.text((0, 0), "___________", fill = "yellow") draw.text((0, 9), date, fill = "white") draw.text((33, 0), str2, fill = "white") draw.text((0, 115), "...", fill = "white") def listen(): recognizer = sr.Recognizer() ''' recording the sound ''' with sr.Microphone() as source: print("Adjusting noise ") recognizer.adjust_for_ambient_noise(source, duration=1) print("Recording for 7 seconds") recorded_audio = recognizer.listen(source, timeout=100) print("Done recording") ''' Recorgnizing the Audio ''' try: print("Recognizing the text") val = recognizer.recognize_google(recorded_audio, language="en-US") print(val) except Exception as ex: print(ex) def say(statment): statment1 = statment.replace(" ", "_") os.system('espeak ' + str(statment1) ) #Change the MAC address here as well def query(query): client = wolframalpha.Client(app_id) res = client.query(query) answer = next(res.results).text answer1 = answer.partition('\n')[0] print(answer1) def start_up(): hour1 = int(datetime.datetime.now().hour) if hour1 >= 0 and hour1 < 12: #print('computer: good morning') say('good_morning') elif hour1 >= 12 and hour1 < 20: #print('computer: good afternoon') say('good_afternoon') else: #print('computer: good evening') say('good_evening') say('how may i help you') def send_sms(person, msg_to_text): if person in contacts: #say('What would you like to say') #listen() message = twilio_client.messages \ .create( body= msg_to_text, from_='+19383003788', #PUT YOUR TWILIO PHONE NUMBER HERE, sry for caps status_callback='http://postb.in/1234abcd', to= contacts[person] ) serial = i2c(port=1, address=0x3C)#Put in the address of your display device = ssd1306(serial, rotate=1) text = ("What would you like to say? ") text1 = ('\n'.join([text[i:i+11] for i in range(0, len(text), 11)])) cdt = datetime.datetime.now() date = str(cdt.day) + "/" + str(cdt.month) + "/" + str(cdt.year) start_up() with canvas(device) as draw: draw.text((0, 0), hour, fill = "blue") draw.text((11, 0),":", fill = "blue") draw.text((15, 0), min1, fill = "blue") #draw.text((0, 0), "___________", fill = "yellow") #draw.text((0, 9), date, fill = "white") draw.text((33, 0), str2, fill = "white") draw.text((0, 115), "...", fill = "white") if min1 in num: #client = wolframalpha.Client(app_id) res = client.query('what is the temperature in Vienna') answer = next(res.results).text str1 = answer str3 = str1.split('(', 1)[0] draw.text((33, 0), str3, fill = "white") def util(func1): cdt = datetime.datetime.now() min1 = str(cdt.minute) hour = str(cdt.hour) with canvas(device) as draw: draw.text((0, 0), hour, fill = "blue") draw.text((11, 0),":", fill = "blue") draw.text((15, 0), min1, fill = "blue") draw.text((0, 0), "___________", fill = "yellow") draw.text((0, 9), date, fill = "white") draw.text((33, 0), str2, fill = "white") draw.text((0, 64), func1, fill = "white") ########Section_4############# while i < 3: recognizer = sr.Recognizer() ''' recording the sound ''' with sr.Microphone() as source: print("Adjusting noise ") recognizer.adjust_for_ambient_noise(source, duration=1) say("Recording") print("Recording for 7 seconds") recorded_audio = recognizer.listen(source, timeout=100) print("Done recording") say("Done Recording") ''' Recorgnizing the Audio ''' try: print("Recognizing the text") val = ("Sorry I could not pick up what you said") val = recognizer.recognize_google(recorded_audio, language="en-US") print(val) except Exception as ex: print(ex) if val=="Sorry I could not pick up what you said": print("Sorry I could not pick up what you said") say('Sorry I could not pick up what you said') i = int(0) if val=="news": news_url="https://news.google.com/news/rss" Client=urlopen(news_url) xml_page=Client.read() Client.close() soup_page=soup(xml_page,"xml") news_list=soup_page.findAll("item") # Print news title, url and publish date r = 0 for news in news_list: print(news.title.text + "\n") say(news.title.text) #print(news.link.text) #print(news.pubDate.text) #print("-"*60) r += 1 time.sleep(1) i = int(0) if r == 3: i = int(0) break time.sleep(3) i = int(0) if val == "exit program": os.system('pkill -f main.py') del val if val == "SMS": del val say('who would you like to send a text to?') recognizer = sr.Recognizer() with sr.Microphone() as source: print("Adjusting noise ") recognizer.adjust_for_ambient_noise(source, duration=1) print("Recording for 7 seconds") recorded_audio = recognizer.listen(source, timeout=100) print("Done recording") ''' Recorgnizing the Audio ''' try: print("Recognizing the text") val = recognizer.recognize_google(recorded_audio, language="en-US") print(val) except Exception as ex: print(ex) dude = str(val) del val say('what would you like to say') recognizer = sr.Recognizer() with sr.Microphone() as source: print("Adjusting noise ") recognizer.adjust_for_ambient_noise(source, duration=1) print("Recording for 7 seconds") recorded_audio = recognizer.listen(source, timeout=100) print("Done recording") ''' Recorgnizing the Audio ''' try: print("Recognizing the text") val = recognizer.recognize_google(recorded_audio, language="en-US") print(val) except Exception as ex: print(ex) msg_to_send = str(val) del val send_sms(dude, msg_to_send) time.sleep(3) i = int(0) if val == "pause for one minute": say("pauseing for one minute") time.sleep(61) if val == "pause for 2 minutes": say("pauseing for two minutes") time.sleep(122) if val == "shutdown": say("shutdown") os.system("sudo shutdown now") if val == "time": cdt1 = datetime.datetime.now() h = cdt1.hour m = cdt1.minute print(str(h) + ':' + str(m)) say(str(h)) say(str(m)) del val time.sleep(3) i = int(0) if val == "picture": #camera = PiCamera() x = x+1 globals()['image%s' % x] = 'image%s' % x print('image%s' % x) camera.capture('/home/pi/Desktop/' + 'image%s' % x + '.jpg') print('image saved to desktop') say('image saved to desktop') time.sleep(3) i = int(0) if val == "play belever": os.system('aplay /home/pi/Desktop/Belever.wav') if val == "play happy": os.system('aplay /home/pi/Desktop/Happy.wav') if val == "play happier": os.system('aplay /home/pi/Desktop/Happier.wav') if val == "play pop music": os.system('aplay /home/pi/Desktop/Pop_music.wav') if val == "video": # v = v+1 globals()['video%s' % v] = 'video%s' % v print('video%s' % v) # say('recording for 15 seconds') camera.start_recording('/home/pi/Desktop/' + 'video%s' % v + '.h264') sleep(16) camera.stop_recording() print('video saved to desktop') say('video save to desktop') time.sleep(1) i = int(0) #############section_5###################### try: client1 = wolframalpha.Client(app_id) res1 = client1.query(val) answer1 = next(res1.results).text answer2 = answer1.partition('\n')[0] for i in bad_chars: answer2 = answer2.replace(i, '') print(answer2) say(answer2) i = int(0) except: try: wiki = wikipedia.summary(val, sentences = 1) #print(wikipedia.summary(val, sentences = 1)) print(wiki) #say(wikipedia.summary(val, sentences = 1)) say(wiki) i = int(0) except: print('') del val #del val ################THE_END##########################
What the Code Does
This part is what the code does so if you want to change anything, you'll know where to find it:
#1 Importing the Libraries
These are the libraries that we installed earlier in the tutorial, so all you have to do is copy pasta.
#2 AppIDs and Authorization Tokens
Here are all of our App_IDs and account tokens as well as some other variables. Replace the contact numbers and names with people you would like to send SMS to. Put all of your specific appIDs and auth tokens that we got from step three into their indicated places.
#3 Functions
In this section, we define a lot of tasks for the display that would be rather tedious to type out each time. This is also where the start-up sequence is defined and initiated.
#4 While Loop and If Statements
This is the part of the code that infinitely loops until you end it with the keyword, 'exit program'. Taking a deeper look into this section we see the glasses can tell you the news, send SMS, tell time, take pictures, take videos, and play music based on the trigger words inside of the if-statement condition.
#5 Catch-All
If you didn't say any of the trigger words then the program will search whatever you said on Wolfram, but if that fails it will search Wikipedia and return a short summary.
How to Use the Code
Before trying to execute the code make sure your wireless headphones are connected. If all goes well the display should turn on and the chatbot will ask what you would like for it to do. It will listen for your response whenever the three dots at the bottom of the screen are present. Below is a full list of the commands and what they will do. When you are done just say, 'exit program' then flip the power switch. It's that simple!
The default commands are:
news - The chatbot will tell you the top three headlines from Google News.
exit program - This will stop the script and then all you have to do is flip the switch.
SMS - This will send any message to anyone you tell it to.
Pause for one minute - it will pause for one minute
Pause for 2 minutes - it will pause for 2 minutes
Time - This one is pretty self-explanatory and is a little more descriptive than the clock on the screen.
Picture - This will take a picture and save it to the raspberry pi desktop.
Play music_name_here- This will play a song on the desktop of the raspberry pi named music_name_here
Video - This will take a 15-sec video then save it to the desktop
Making the Glasses
As noted in the intro, this instructable has been a base for my project, and steps 14-16 work well, so you can use steps 14-16 on that instructable.
You may want to make the glasses frame a bit bigger, about 5%-15% bigger depending on your head size: On Cura, select all the objects, and on the sidebar, click the scale button. (Screenshot above)
Also, after the glasses are created, try them on. If it falls off, tie a rubber band at the end (or hot glue them on), and they should stay on now.
You're Done!
Congratulations! You have successfully built your very own smart glasses with the Raspberry Pi 0w. Thank you for reading my Instructable. I hope it has helped you make your own smart glasses!
If you have any issues with the code or anything, email me at drjhxie@gmail.com or post a comment down below.
Above is my improved version of the smart glasses!