Virtual Assistant / Chatbot (AI) With PictoBlox

PictoBlox is a Scratch-Based graphical programming software where young makers can program robots, make interactive games, and animations, and program AI/ML projects. This beginner-friendly instructable will guide you on making your own Virtual Assistant / Chatbot using the Artificial Intelligence extension on PictoBlox.

The Virtual Assistant / Chatbot we will be building today will be capable of:

• Facial Recognition
• Speech Recognition
• Text-To-Speech
• Machine Learning

Before we begin, let's get a few things out of the way. We must acknowledge what we will be making. Unless you have some experience with coding, you will not be creating something as advanced as J.A.R.V.I.S. or ChatGPT, not even close. We're going to be building the bare minimum of AI, with speech and facial recognition, and machine learning. However, with this basic AI framework, it's possible to expand your knowledge of computer science and add more to this base. Consider this an Artificial Intelligence guide for Dummies. Let's begin.

• PictoBlox (Download Link: https://thestempedia.com/product/pictoblox/download-pictoblox/)
• Headset (with Microphone)
• Webcam

First, you must install PictoBlox using the link above. Select your Operating System (OS), and then download the PictoBlox installer. Run the .exe file, select the options appropriate for your needs, and install the software.

After installing the software, open PictoBlox.

The extension button is a purple block with 2 blocks and a plus symbol in the bottom-left corner of the block palette. Click on it to open the extensions catalog. Install the following extensions:

• Artificial Intelligence
• Face Detection
• Text to Speech

You may then click the "Back" button and return to the main project interface.

We're almost ready to begin programming our virtual assistant/chatbot. Let's start with designing the interface/dashboard of our AI. At this stage in development, you don't need to design very futuristic/detailed visuals

Once we finish writing the basic code of our AI, that's when you can get creative on how you want your AI's visuals and interface to look.

For now, make some basic visuals and buttons. Here's a list of buttons/visuals you'd want to make.

• Face_Recog (Initiates Facial Recognition, Needs ON and OFF costumes)
• Speech_Prompt (Initiates Speech Recognition)
• Train (Adds the face on screen as a "class")
• Bounding Box (for Facial Recognition)

Personally, when I make bases/prototypes for later projects, I like to design my interfaces and visuals to look simplistic, and basic. But please, don't let me stop you from designing your AI's interface the way you wish to.

Let's make some variables now, these are going to be really important later on in development.

Here are some variables you're gonna need to make

• Face_X (For the Bounding Box sprite only)
• Face_Y (For the Bounding Box sprite only)
• Face_# (Number of Faces)
• Face_ID (Face Identified)
• recog_durat (The duration of the speech recording)
• Size (For the Bounding Box sprite only, Slider variable)
• toggle (For the Face_Recog button only)
• Voice (TTS Voice)

Before we can program the AI's facial recognition abilities, we need to set up the button that will initiate the process.

The Face_Recog button should be toggleable, so it's easy to use.

Program the Face_Recog button using the code above.

This code will:

• Allow the Face_Recog to be toggleable
• Switch the button's costumes from "ON" to "OFF" depending on the state the button's in.
• Set the default state of the Face_Recog button to "OFF"

If the button is now able to toggle between states, switch between being ON and OFF, and turn your webcam on and off without any errors or bugs, proceed to the next step.

If not, double-check the code to make sure there are no errors or mistakes in it, then try testing it again.

Now, this is where those variables we made earlier are going to come in handy.

You're gonna need these variables to make the readouts for your facial recognition.

• Face_X
• Face_Y
• Face_ID
• Face_#

Since both Face_X and Face_Y can only be accessed with the Bounding Box sprite, we will be programming the readouts in that sprite.

Use the code above to code in the readouts.

This code will:

• Show and hide the readouts and bounding box depending on the Face_Recog button's state.
• Allow the user to optimize the size of the bounding box.

Make sure that when the readouts are visible, they are positioned in the following order:

• Face_X
• Face_Y
• Face_ID
• Face_#

If the readouts are now able to switch between being visible and invisible based on the state of the Face_Recog button, and the bounding box size can be easily optimized with the slider, proceed to the next step.

We're finally ready to code our facial recognition capabilities into our AI!

Let's start with the basics first

It's called a "bounding box" for a reason.

The bounding box should track and follow your face.

Use the code above to code the bounding box.

This code will:

• Allow the Bounding Box to track and follow your face.

Now test the code, if the bounding box is now following your face wherever it goes, proceed to the next step.

Now, let's set up the basic training algorithm for our AI.

The goal here is to program a process for the AI, so it can recognize a face on an image or in real-time, and save the face as a "class" with a name based on user input.

This process can only be initiated when the train button we made earlier has been clicked, so this is where that's gonna come in handy.

First, make a list named "face_recog database"

Now, use the code above to code the train button.

This code will:

• Allow the AI to analyze and detect a face on an image or in real-time.
• Save the face detected as a "class" with a name based on user input.

Now that we have the training algorithm set up, we can now set up its face-matching system.

This is by far one of the most important parts of our AI's facial recognition ability.

Our goal is to utilize the training algorithm we've set up and allow the AI to "recognize" faces.

Use the code above to program the face-matching system.

This code will:

• Analyze the face on screen.
• Initiate a face-matching process.
• Display the face/class name on a readout.

If you're able to save faces/classes on a database in your AI,

and if your AI is now able to recognize faces, proceed to the next step.

Great work! We're almost done programming our AI's Facial Recognition features.

All we have to do now is program the readouts we made earlier.

The goal of this step is to display information gathered by our AI's analysis on screen for the user to see.

Use the code above to program the readouts.

This code will be programmed in the Bounding Box sprite.

This code will:

• Display the face's X position on the Face_X readout.
• Display the face's Y position on the Face_Y readout.
• Display the number of faces on the screen on the Face_# readout.

If your AI can now display information gathered from its analysis on your face, proceed to the next step.

Alright! We're now ready to get started on our AI's speech recognition features.

Let's start by setting up the button that will initiate the process.

This is where the "recog_durat" variable we made earlier comes in.

Make sure it's a slider variable before we get started on the coding.

Use the code above to program the Speech_Prompt button.

This code will:

• Show the "recog_durat" slider variable.
• Record speech for "recog_durat" seconds.

If your AI is now able to record your speech based on the number of seconds you inputted, proceed to the next step.

Alright! Believe it or not, this is the last step of programming our AI's speech recognition features.

Instead of using preprogrammed voice commands, we're going to be setting up an algorithm where if it doesn't know how to respond to the user's command, the user can "train" it so it knows what to say the next time the user says the same command.

This model will be programmed in the Speech_Prompt button.

First, make two lists named "input" and "output".

Then, use the code above to program the speech recognition model.

If your AI is now able to "learn" how to respond to your commands, and responds correctly, proceed to the next step.

Well, it appears we've reached the end of this tutorial.

By now, you should have a fully-functional AI system.

Now is your time to name your AI and get creative with it!

Customize the visuals, and change up the code a bit, it's yours to tinker with now!

If you're still experiencing some bugs, go ahead and comment and I'll try to get back to you ASAP.

Enjoy your AI!

- TheTerminator