The truth behind Facebook AI inventing its own language

Photo credit: Downloadsource.fr.

There have been so many articles about Facebook shutting down its robots after they developed their own language. The media is just loving these clickbait titles. Some of these articles would let you believe that this was a very close call—that scientists at Facebook barely shut down the AI before it could take over the world.

Fortunately, there are still sane people out there, so there have been quite a few articles explaining why all of the doomsday talk is complete nonsense (like the ones published by Snopes or CNBC). Even some of the media that originally offered a very scandalous version of this event eventually edited the content to be less dramatic (like The Independent).

The problem is that false but catchy news is much easier to spread than anything else. I also find it curious that none of the articles actually explained what happened in terms that people would understand. I tried to explain the situation to some of my friends and eventually decided that it was worth writing down. Maybe, it will help people sleep without thinking of Skynet from the movie The Terminator. However, my goal is to educate—to show how so-called AI works—and not to take sides.

Facebook AI Research (FAIR)

So, what was Facebook actually doing and how did the robots “almost become sentient?” The whole project is well-documented and available to the general public. Anyone can actually download and run this AI as well as observe the new language on their own. (Just please be careful and shut it down in time like the Facebook engineers did.)

The system tries to simulate dialog and negotiation. The so-called robot is given a set of items (consisting of books, hats, and balls) and some preferences for which items it wants more than others. Then, it is supposed to negotiate with its counterparty, be it a human or another robot, about how to split the treasure among themselves.

The research was published in June, including all code and training data used for the experiment. If you are interested in more details, read the official article or just get the code from Github.

So how does it work in simple terms?

Machine learning

I will not bother you with all the technical details, but it is important to understand some basic principles about how this technology works.

When developing a robot like this, you start with something called a “training data set.” This consists of well-described examples of the behavior that the robot is trying to simulate.

In the particular case of the Facebook negotiation chatbot, you give it examples of negotiation dialogs with the whole situation properly annotated—what the initial state was, the preferences of the negotiator, what was said, what the result was, etc. The program analyzes all these examples, extracts some features of each dialog, and assigns a number to these features, representing how often dialogs with that feature ended in positive results for the negotiator. To better imagine what a feature is, think words, phrases, and sentences. It is more complicated than that in reality but it is good enough to get the principle.

To be more specific, if the robot wants hats, the phrase “You can have all the hats” will have a really low score because this sentence ended with a bad result in every scenario from the training data—the negotiator did not get what he wanted.

A diagram representing a sample dialog evaluation. From the study “Deal or No Deal? End-to-End Learning for Negotiation Dialogues” by Mike Lewis, et al., 2017.

In the end, it’s nothing surprising