IoT: Control the HoloLens Using Your Eyebrows (EMG)

This project was a part of the University of Colorado Boulder's NASA SUITS project that was presented and tested at NASA JSC in April 2019. For that year's project, I was the project lead of hardware development and this was one of my contributions. Read more about the NASA SUITS challenge here.

With this project, I wanted to allow anyone (in this case EVA Astronauts) to interface with a Heads-Up-Display (HUD) deployed in the Microsoft HoloLens without using the built-in hand gestures or voice inputs. I tackled this goal from an accessibility standpoint, I wanted to develop a peripheral/wearable that would allow for EVA astronauts to interface with their display without occupying communications with ground control, and because of their highly pressurized gloves gesture interfacing is unreasonable. While this is a domain I'm somewhat unfamiliar with, I can also see this peripheral being useful for those with disabilities, allowing them to use the HoloLens or other AR/VR devices and forgo the interfacing inputs that are exclusionary or daunting.

While the item list for this project is exclusionary itself (the HoloLens is very expensive!), the peripheral can be used with other AR/VR Devices.

If you have any questions, want to keep up with my work, or just toss around ideas, please do so on my Twitter: @4Eyes6Senses.

Microsoft HoloLens (or other AR/VR device)

Particle Photon

MyoWare Muscle Sensor - The MyoWare sensor can be substituted with other EMG breakout boards. If you're not sure whar EMG is, I suggest you read more about it here

Biomedical Sensor Pads

Unity (Free)

Some Experience with Creating Projects in Unity

Wire

Connecting the Particle Photon to the MyoWare board is fairly straightforward. For this step, you will need to solder the connections between the Particle Photon and the MyoWare board. Make sure you have strong connections such that the cables don't break while wearing the device. To avoid breaks, I recommend braiding the cables before soldering them into the Photon.

- Solder a wire into the "+" port of the MyoWare board, then solder the other end of the cable into the Photon's "3v3" pin.

- Solder a wire into the "-" port of the MyoWare board, then solder the other end of the cable into the Photon's "GND" pin.

- Solder a wire into the "SIG" port of the MyoWare board, then solder the other end of the cable into the Photon's "A0" pin.

Using the Particle Photon's IDE, Upload the .ino file. Place the MyoWare board on a muscle group and check the values to make sure your device is working. When looking at the code you'll notice that there's a "threshold" variable already set, this variable is the minimum value the MyoWare reads from my muscle when I raise my eyebrow fully. I also set the variable to "600" because it allowed me to have normal conversations while wearing the MyoWare without creating any false positives (accidental triggering), you may want to play around with the values until you find the desired threshold for your uses.

For our NASA SUITS project, I selected to place the MyoWare Board above the eyebrow. I was inspired to place the MyoWare there after looking at the "'Sup Brows" project done by Adafruit. After some testing, it became clear that the eyebrows are a great location for this specific project. It's a great location because the sensors wouldn't be affected by the constant muscle movements through the arms, legs, and torso that EVA astronauts do during spacewalks.

Now it's time for the final step, let's connect the MyoWare with Unity! Included in this Instructable is the script that you'll need to include into your Unity project. But first, you will need to change a few things in the code. First, you'll need to add the JSON Object Unity plugin into your project. Next, you'll need to add your own device id and access token into line 19: particleURI = "https://api.particle.io/v1/devices/[Insert your device id]/Next?access_token=[Insert your access token]". You can get your device id from the Particle IDE, following figure 2, click on your devices tab (red box) and look under your device name to find your id (blue box). Next, to find your access token click on the settings tab in the IDE.

After finishing your Unity project, you will have an EMG controlled HoloLens!

If you have any in-depth questions, want to learn about human augmentation, want to keep up with my work, or just toss around ideas, please do so on my Twitter:

@4Eyes6Senses

If you are interested in working with our team for the 2019 - 2020 NASA SUITS challenge, please email me at:

chhi5098@colorado.edu

Special thanks to the model in the first picture, Darren, who also tested the design at NASA. Another thanks to my programming partner for the project AJ, who made a lot of this possible.