Turn Your Raspberry Pi Into a Fully Customizable Smart Home!

Why buy an Alexa smart home device when you can build one yourself? Raspberry Pi Smart Home is just that.

At its heart is a virtual assistant. built with one of the world's largest AI Models. This model not only has the ability to understand the user's intent to a level that wasn’t possible before but is also capable of having an intelligent conversation with the user!

This system has a local network address, accessible not only thru your computer, but also thru your phone. It gives you the full ability to read, write and rename your components, not to mention lets you to access the virtual assistant as well as easily train it. All within an extremely easy to use and simple user interface.

A raspberry pi (Any kind, but must have WIFI connection) (raspberrypi.org)

• A pack of LEDs (sparkfun)
• A 7" RPI LCD Touchscreen (Optional) (sparkfun)

Items that are optional, but useful for prototyping:

• A pack of Resistors (adafruit, sparkfun)
• GPIO Breakout board (sparkfun)
• Breadboard (sparkfun)
• Jumper wires (adafruit)

This part is completely optional but having a touchscreen can add a sleek look to your smart home hub.

First, we want to 3D print the touchscreen mount.

The File was designed in 2 parts, using Autodesk Fusion 360. (You can install fusion 360 here: Download Fusion 360 for Free | Free Trial | Autodesk)

Download the f3d file, select each individual component and select "Save as Mesh" then open the files in the slicer of your choice to 3D print them.

• This model was designed to be mounted to a surface. I am using metal shafts to Elevate the screen.

Once the touchscreen has been built, mount the raspberry pi to the touchscreen, add the GND and 5V wires to the pi's GPIO, and attach the ribbon cable to the "DISPLAY" cable insert.

First, we need to Download Apache NetBeans.

Download it here: Welcome to Apache NetBeans

Once it has been installed, we want to create a new Java project with Ant.

Ant is important because it is a more flexible platform, and it supports Remote Platform Projects.

We need to make sure the package list is updated before we get started:

sudo apt update
sudo apt upgrade

To install Java 11 on the raspberry pi:

sudo apt install default-jdk

Now check to see if it has been installed:

java –version

A Remote platform project allows you to execute your project on the Raspberry pi while being able to debug and see the output on your PC. When you start the application, it automatically sends the project to your raspberry pi via SSH and executes it.

In order to start, you need to enable SSH on your Raspberry Pi Configuration.

The following steps to setup a remote platform:

1. Right click on your project in the projects bar and select "Project Properties"
2. Go down into the run tab and select "Manage Platforms"
3. Click "Add Platform" in the new window
4. In the new window, select "Remote Java Standard Edition"
5. A new screen will open up and you will be able to enter your PI's IP address, password and username.

• Your Pi's IP Adress

Enter your Pi's terminal and type:

hostname -I

The output is your RPI local IP Adress. Hold on to this information, because you will need it for later

• Unless you have changed it, your Raspberry Pi's username is probably going to be "pi",

• Remote JRE Path

The Remote JRE path can be found by using the command:

whereis javac.

Enter only the path that is shared by all paths. See attached image.

1. After you are done, test the remote platform to see if the Raspberry Pi will connect.

Download the Smart Home Software source code, and follow along for the parts you will need to configure:

zipCoder933/RpiSmartHomeSystem (github.com)

Download the code, unzip the file and paste it directly into your source packages.

CONFIGURAITION

In your source code go to Main.java.

IP Adress

Set the IP address in your code to your Raspberry Pi's IP Adress.

Using Screen

set the "usingScreen" variable to false, since executing the code from NetBeans is in headless mode, (headless mode means you can't use any UI, or things to that nature).

If you are using a screen, you can only set this variable to "true" once you are ready to execute the code from the raspberry pi itself.

There are 2 required custom libraries needed for this project. One is an API wrapper for the weather API and the other is an API wrapper for the codex API.

You can download them here: zipCoder933/Dependencies-for-RPI-Smart-Home-System: The custom libraries needed for Raspberry Pi Smart Home System (github.com)

To install the libraries to your ANT Project:

1. Right click on the libraries section of your project hierarchy. Select add JAR/Folder and navigate to the Jar folder you want to add.
2. Note: Do not add the entire folder. Import each jar file separately.

We could build A smart home assistant with relatively the amount of intelligence as Alexa or Google, but for this project, I thought it would be a better idea if a much more powerful solution was implemented.

OpenAI Codex is a Natural Language model, trained on huge amounts of code. It can speak English as well as every computer language. In this project, we will be using it to convert text commands into a code that can be applied to the smart home components.

Codex has the amazing ability to understand the user's intent at a level that has never been possible before. Unlike Alexa where you would have to program virtually every request to a list of responses in its database, codex only needs a few examples until it is capable of telling you the time, switching on the lights, or actually having a conversation with you!

In order to get started you need to access the private beta: Form (hsforms.com)

Once you have access, find your API Key here: Account API Keys - OpenAI API

Once you have access, find a file located in the resources directory of your code called "CodexAPIKey.txt" and enter your API key into that file.

If you want to have access to weather data for your smart home system, you will need access to a weather API.

For this project, we will be using Open Weather Map API Сurrent weather and forecast - OpenWeatherMap

It may take up to 2 hours for you to gain access to the API. Once you have, you can get your API Key here: Members (openweathermap.org)

In your resource's directory, find the file called "OpenWeatherAPIKey.txt" and enter your API key there.

The resources directory is where your program will go for any resources, whatsoever.

Because of this, we need to tell the program where to find it and and we also need to make sure the PI as access to it.

The resources directory already comes with the source code you downloaded, but you will have to transfer it to your Raspberry pi and place it in a designated area, before the PI can have access to it.

The resources directory is the only thing that NetBeans will not be able to transfer to the raspberry pi directly, so in order to transfer the resources directory to your raspberry, pi you will need to do it via jump drive. Make sure you have everything you need on it first because it will not be as easy to update the resources directory directly on the PI.

CONFIGURAITION

In the source code you go to Main.java.

Resource Path

Set this to the path of your resource directory on your raspberry pi.

Local Resource Path

Set this to the path of your resource directory on your computer. This should be the path to the resource directory embedded in your project.

Find the resources package in your source code. That is your resources directory. You can find the link by right-clicking on the folder and clicking "Properties" in the sub-menu. There you will find the file-path.

This part is optional, but recommended, as it is a good idea to test the code on your PC before deploying it to the raspberry pi.

We want to be able to speak to our smart home system and have it talk back to us. To do this, we have a few things we need to do:

Speech recognition

With speech recognition, you can command the home assistant without having to type it out on a phone or computer.

From The Ultimate Guide To Speech Recognition With Python – Real Python

The speech recognition code is written in python. The python script (stt.py in the resources directory) uses the below libraries. Enter the commands below to install them:

pip install SpeechRecognition
pip install sounddevice
pip install numpy

The speech recognition python code listens for a certain threshold of noise and then listens for voice input. The command is then sent to the Codex Assistant and carried out in the same way, as if you were to type in on the user interface.

In order to properly enable speech recognition functionality:

• You need to create a new "USB Microphone" component in the interface,
• A Microphone must be plugged in via USB, (obviously).
• The software will auto detect what port the microphone is plugged into, so you dont have to configure that.
• The changes will only be applied when you restart the software.

To install TTS

If we want to talk to your raspberry pi, we also need it to be able to talk back to us. The audio is outputted thru the Raspberry pi AUX cable. So just plug in the speaker of your choice.

REQUIRED PACKAGES

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install alsa-utils
sudo apt-get install mplayer

Now is the point where you can decide what voice you want. The 2 options, I have programmed are espeak and google TTS. Espeak isn't as good voice quality as google, but I have noticed some errors when trying to speak with google, and hence, I recommend espeak more.

Installing Espeak

sudo apt-get install espeak

If you want to change to google TTS

in the terminal, cd into the resource's directory on your raspberry pi and enter "nano speech.sh"

Uncomment the google text to speech code and comment the espeak code:

#!/bin/bash


#FOR GOOGLE TEXT TO SPEECH
#THIS ONE IS BETTER, BUT SOMETIMES FAILS.
say() { local IFS=+;/usr/bin/mplayer -ao alsa -really-quiet -noconsolecontrols>
say $*


#ESPEAK (LESS QUALITY, BUT OFFLINE AND IS BUG FREE)
#espeak $*

Now it is time to deploy your application to the raspberry pi. This version will not need to be executed on NetBeans, but we will execute on NetBeans anyway just to get the JAR file onto the PI.

1. Set the "usingScreen" variable we discussed earlier in Main.java to "true" and run the application in Remote
2. wait until the application has started and stop it. Unless you have changed it,
3. .JAR is the file format our program is in, and it is the file we will need to find, once it is on the raspberry pi.
4. The Jar file is probably going to be somewhere in /home/pi/NetBeansProjects.
5. Go there and search for a jar file under the name of your project. Once you find it, copy the link to the jar file.

Running the Application at startup.

One of the ways to run code on startup, on a raspberry pi is to add to the "bashrc" file.

Do it with this command:

sudo nano /home/pi/.bashrc

Add the following text to the end of the file:

sudo java -jar PATH_TO_YOUR_NETBEANS_JAR_FILE.jar

Press Ctrl+O to save the file and Ctrl+X to exit.

Enter:

sudo reboot

To make sure it was done successfully.

Now it's finally time to get started!

I have connected the Raspberry pi GPIO expansion board and a breadboard to the raspberry pi for easier access to the pins. And have attached 2 LEDs. One red LED and one RGB LED connected to 220-ohm resistors.

GPIO

The GPIO Interface used by the software is called pigpio pigpio library (abyz.me.uk). It should come pre-installed with recent Raspbian images. If it's not, you can install it with the below command:

wget https://github.com/joan2937/pigpio/archive/master.zip
unzip master.zip
cd pigpio-master
make
sudo make install

This part is totally optional:

unless you already know what each of the pins represent, it might be a good idea to test the LEDs before we punch them into our user interface. Specifically, the RGB led.

pigpio uses a command line interface called pigs. To start pigs, we need to start the pigpio daemon:

sudo pigpiod

We won't have to start the daemon again, unless we restart the PI.

In order to test a pin, we first need to set it to writable mode. We do that using the below command:

pigs m YOUR_PIN w

Be careful, as the pigs library does not stop you from reading/writing to unsafe pins. Messing with unsafe pins can result in anything from nothing happening, to having to re-flash your Raspbian image due to file corruption. (See pigpio library (abyz.me.uk) to learn more)

Activate your pin with:

pigs p YOUR_PIN 255

The command uses PWM, so you can set it to anything in-between 0 and 255.

Now you can test each pin on the RGB LED to see what colors correspond to what pins and write them down.

First experience with the User Interface

The user interface allows you to change your component configuration, specifically, add (configure), remove, and rename components as well as allowing you to read all component data and write to your components.

In your computer, or on your phone, enter your PI's IP address in the browser of your choice, along with the port: 8500. Your URL should look something like this:

192.168.0.12:8500

• Hit enter and wait for the page to load.
• Add a new component,
• And test the new component by setting the PWM values up and down.

This is the fun part!

To train your smart home assistant, you need to teach it how to do its job. Luckily, because of Codex. The training process is extremely simple, the AI model only needs a few examples to understand what to do. You should only have to give it 4-6 examples for it to work properly. If it performs improperly, you can add more examples until it understands. The video will step you through how to train your AI Assistant.

Steps to train

1. Repeat 4-6 the following steps times...
2. Go into the settings and clear all training data. It is recommended to retrain your model every time you change your component configuration, so that it doesn't get confused by its past memory.
3. Set the command mode to "train" and close the window.
4. Enter your prompt into the command bar, manually set the relevant component properties, and submit the command by pressing enter in the command bar. Remember, you need to set the desired properties before you submit a training example.
5. In the settings, set the command mode back to command or reload the page.
6. Enter some commands and see what it can do!

This is not recommended. But if you are feeling adventurous, and want to create your own custom components, you not only need to have a good understanding of Java but will have a lot of work to do programming wise. I will try to cover everything, but be wary, that you could potentially run into bugs and errors that prevent the smart home system from being able to operate.

You will need to get comfortable with the code. Take some time to get familiar with how the software works before proceeding.

1. Create a new class that extends the Component class.
2. Add your configuration properties (add screen properties) as shown in the image
3. Add your properties as shown in the image.
4. Set the title of your component via the super() method. The component label is auto generated and is the same thing as the title, except all lowercased, and spaces are replaced with dashes.
5. append your custom component in the newComponent.html file in the resources package and add your custom component's title and label at the end of the html tag.
6. append your custom component in the Configuration.java class in the "getComponentByClassLabel" method. This method helps other parts of software find the component class by its label.
7. Append an Enum named after your component to the end of the CopmonentType Enum in the Configuration package.
8. in your component class, set the setMethod() value to the new Enum value you created. This lets the ComponentMethodHandler class to know what component it is.
9. in the ComponentMethodHandler class in the Configuration package:
10. append the gatherData method with your new component.
11. The gatherData method is called for your component to gather any sensor data.
12. append the initializeComponent method with your component.
13. The initialization gets it ready right after it was created.
14. append the setValue method with your component.
15. the setValue method is called when a new component value is set by the user.
16. Take a look around at the existing utilities for GPIO faculties and other resources in the utilities package. Create new utilities if you need one.
17. Copy the resources from your PC to the Raspberry pi, like we discussed before.
18. Run your code locally and on the PI to make sure it works. Stress test the software to make sure its foolproof.

This is just the beginning.

So far, you have built a system, with an incredibly simple and easy-to-use user interface, accessible on your phone and available from anywhere within your house. From giving you complete control of your home components, to letting you easily change your component configurations with as little fuss as possible, to the instant access of a smart home assistant, with its intelligence and its understanding of user intention that far surpasses Google, Alexa or Siri.

The potential of this system stretches even farther! From smart wall plugs to automatic door locks, you could truly innovate your home!

All powered by just a single Raspberry Pi.