Researchers from MIT have said that moral responses to unavoidable damage vary greatly around the world in a way that poses a big challenge for companies planning to build driverless cars.
They presented variations of the classic “trolley problem” thought experiment almost 40m times to millions of volunteers from all around the world.
In the traditional thought experiment, participants are asked to consider whether they would reroute a runaway trolley car which is about to hit and kill five people, directing it on to a siding where it would kill only one person. In the new quiz, dubbed “Moral Machine”, the researchers instead asked volunteers to consider what a self-driving car should do in examples from more than 26 million variations of the same question.
Should a car with three occupants, an adult man and woman and a child, swerve into a wall, killing them all, in order to avoid hitting three elderly people, two men and a woman? Should an unoccupied car swerve and kill an unemployed adult man, a child and a cat in order to save an adult man and woman and a child? Does the answer change if the pedestrian light is red? What if one of the people is unfit, or pregnant?
Responses to those questions varied greatly around the world. In the global south, for instance, there was a strong preference to spare young people at the expense of old – a preference that was much weaker in the far east and the Islamic world. The same was true for the preference for sparing higher-status victims – those with jobs over those who are unemployed.
When compared with an adult man or woman, the life of a criminal was especially poorly valued: respondents were more likely to spare the life of a dog (but not a cat).
The researchers, whose work is published in the journal Nature, also note “some striking peculiarities, such as the strong preference among those in the global south for sparing women and fit characters.
“Only the (weak) preference for sparing pedestrians over passengers and the (moderate) preference for sparing the lawful over the unlawful appear to be shared to the same extent in all clusters.”
The data comes with caveats, of course. Unlike traditional polling, the volunteers were entirely self-selected, reached in large numbers thanks to the viral nature of the “Moral Machine” quiz, which was covered by technology news sites like The Next Web and Business Insider. That means that, for instance, the data is likely to skew towards the wealthy in nations with weak internet penetration. More generally, they write, “most users on Moral Machine are male, went through college, and are in their 20s or 30s”.
Nonetheless, they argue that the findings should prompt policymakers and auto engineers to consider embedding some moral intuitions into self-driving cars. “Before we allow our cars to make ethical decisions, we need to have a global conversation to express our preferences to the companies that will design moral algorithms, and to the policymakers that will regulate them,” they write.
Among some autonomous vehicle engineers, however, that view is disputed. Speaking to the Guardian just after the Moral Machine quiz was first released, Andrew Chatham, a principal engineer on Google’s self-driving car project, said the problem has little bearing on actual design.
“It takes some of the intellectual intrigue out of the problem, but the answer is almost always ‘slam on the brakes’,” he said. “You’re much more confident about things directly in front of you, just because of how the system works, but also your control is much more precise by slamming on the brakes than trying to swerve into anything. So it would need to be a pretty extreme situation before that becomes anything other than the correct answer.”
