Black boxes, human factors, and the unintended consequences of today’s AI

Photo credit: Pixabay.

Aftermath is a 2017 movie based on the Uberlingen air disaster of 2002. It tells the revenge story of one of the victims’ grieving parents who went on to kill an air controller blamed for the crash. The movie had reviewers rolling their eyes at Arnold Schwarzenegger trying to do an emotive role. But the true story behind it has an important takeaway on managing new risks as AI becomes a part of everyday life.

Uber’s product lead from San Francisco Alexis Zheng brought up the Uberlingen story at a panel discussion I moderated earlier this month at Seoul’s Startup Festival. Uberlingen has little to do with the ride-hailing company apart from the name. It’s the German city over which two planes collided – a Russian Tupolev passenger jet and a DHL Boeing 757 cargo jet.

For humans to work with humans is hard enough; for humans to work with machines is going to be even harder.

Both planes were equipped with traffic collision avoidance systems (TCAS), which automatically directed one plane to fly 1,000 feet higher and the other one to go 1,000 feet lower. But in this case, a Swiss air controller also spotted the impending disaster and gave the pilots the opposite command. The Russian pilot listened to the human command, whereas the DHL pilot from Western Europe was trained to listen only to the TCAS and disregard the contrary human instruction. The result was that the the two opposite commands neutralized one another and the planes came back on a collision course. All aboard the Tupolev and the cargo jet died in the crash.

Today this is used to illustrate challenges in machines and humans working together, says Zheng, who leads a team building machine learning and AI products at Uber. “The biggest challenge in my job is when a machine is making a decision and a human is making a decision at the same time; that’s an area less talked about today, but it’s going to be very important going forward,” she says. “For humans to work with humans is hard enough; for humans to work with machines is going to be even harder.”

Fear of an AI being running amok has been a popular theme in science fiction and movies. But rather than a Frankenstein or Matrix, what we’re starting to see are myriads of AI applications that become part of our everyday lives in ways that we may not even be aware of: from our Uber rides to news feeds and Netflix recommendations. Add the internet of things to the mix, and we have a scenario where almost every asset in our lives is intelligent, knows about us, and is thinking of ways to be more efficient and personalized. “The term AI will disappear because it’s everywhere,” says Zheng.

The challenge for her and others developing and deploying AI applications then is to anticipate unintended consequences and prevent their pernicious effects. Startups and larger tech companies jumping in to use AI in their products and services have a lot to gain from real-time automation. But how much attention are they paying to how AI applications will behave in various real-life situations?

See: The AI haves are pulling ahead of the have-nots, McKinsey study finds

AI panel discussion at Startup Festival 2017 in Seoul. Photo credit: Jeffrey Broer.

Design problem

It’s a concern that is cropping up in different forums.

Because every decision is expressed at a mathematical level, it forces us as humans to anticipate these problems when we design intelligent machines.

A debate has been raging on the extent to which fake news manufactured in Russia and posted on social media might have influenced the outcome of the 2016 US election. Facebook boss Mark Zuckerberg dismissed it as a “crazy idea” when it first surfaced last year but has since then agreed to share with US congressional investigators thousands of political ads linked to fake accounts in Russia. He has promised to double down on a team “working on election integrity.”

Common sense