What can Mark Zuckerberg’s Jarvis do as home assistant?

Photo credit: Kibeom Kim.

Many of us love Jarvis from Iron Man and wish it to be part of our lives. Mark Zuckerberg made it happen (to an extent) by building an AI assistant for his home.

It controls home systems—adjusts temperature, plays music, operates lights and appliances, and has a door camera with face recognition, iOS voice app, Messenger bot, speech recognition, and language processing tools. While Mark claims to have spent about 100 hours in building this system, he also admits that there are many aspects of the AI that are unexplored and will take repeated experiments and breakthroughs to unveil.

For now, we’ll take a better look at what Jarvis is capable of performing:

Natural language

According to Mark, Jarvis learned natural language through two steps: first is by text, then a speech feature was added to it. But how could it possibly differentiate natural language from codes? Here came the need to add relevant words like “bedroom” or “living room” in the program.

These words were then given context because the AI needed to be acquainted with the context to function effectively. For example, a context understands who is requesting for what (differences in word connotations), obeying a request in a specific room by a specific person, and performing open-ended requests through trial and error.

Visual and face recognition

It was difficult for the AI to figure out the difference between a vision by itself and what it meant, so the next problem was face recognition. But Facebook has succeeded in getting that theory right (Facebook can identify your friends in a photo and give you friend suggestions at the same time), so it was just a repeat implementation for Jarvis.

Face recognition is easier when faces are clearly focused in images. So, Mark installed cameras in his home at different angles to capture the faces of the people arriving at his door. Jarvis now can identify visitors by checking their names on the guest list, and then, if the answer is positive, inform Mark about their arrival.

Messenger bots

Messenger bots, as opposed to an app, are convenient in many ways as they work across various platforms like iOS and Android. They instantly push and receive messages, handle control or permissions for activities to be performed, and more. Also, as compared to speech instructions, bots for Messenger seem a great way for personal communication as they are less disruptive and maintain a quiet ambiance.

Given the convenience they provide, both speech and text controls are now imperative for future AI projects.

Voice and speech recognition

While it is agreeable that texting is a more subtle and quieter way of communication, speech has advantages in being fast and hands-free. In the same way, the Jarvis app comes in handy and can be installed in many phones at home. Although its major limitations were figuring out languages and structural sentences spoken by a person, and understanding the meaning of these, repeated processing brought in enhanced results.

Providing context

The clearer the context, the smarter an AI will be. Of course, there will be a lot of trial and error apart from the repeated conversational training involved. But building an excellent context will result in a breakthrough, making an AI learn new skill sets without following the mainstream training module or programmed training.

Jarvis has enough of it to run Mark’s home system, but the Facebook founder undoubtedly wishes to explore this more. Built entirely in Php and Facebook-like codes, it is still the most innovative personal project to date.

While Instagram competes with Snapchat and Facebook’s growth slows down due to advancements that can trace fake Facebook accounts, I guess Mark is exploring other avenues to expand his company.