This is an edited repost from Human Factors Blog from 2015
Recently, some very prominent scientists and other figures have warned of the consequences of autonomous weapons, or more generally artificial intelligence run amok.
The field of artificial intelligence is obviously a computational and engineering problem: designing a machine (i.e., robot) or software that can emulate thinking to a high degree. But eventually, any AI must interact with a human either by taking control of a situation from a human (e.g., flying a plane) or suggesting courses of action to a human.
I thought this recent news item about potentially dangerous AI might be a great segue to another discussion of human-automation interaction. Specifically, to a detail that does not frequently get discussed in splashy news articles or by non-human-factors people: degree of automation. This blog post is heavily informed by a proceedings paper by Wickens, Li, Santamaria, Sebok, and Sarter (2010).
First, to HF researchers, automation is a generic term that encompasses anything that carries out a task that was once done by a human. Such as robotic assembly, medical diagnostic aids, digital camera scene modes, and even hypothetical autonomous weapons with AI. These disparate examples simply differ in degree of automation.
Let's back up for a bit: Automation can be characterized by two independent dimensions:
- STAGE or TYPE: What is it doing and how is it doing it?
- LEVEL: How much it is doing?
Stage/Type of automation describes the WHAT tasks are being automated and sometimes how. Is the task perceptual, like enhancing vision at night or amplifying certain sounds? Or is the automation carrying out a task that is more cognitive, like generating the three best ways to get to your destination in the least amount of time?
The second dimension, Level, refers to the balance of tasks shared between the automation and the human; is the automation doing a tiny bit of the task and then leaving the rest to the user? Or is the automation acting completely on its own with no input from the operator (or ability to override)?
See Figure 1. If you imagine STAGE/TYPE (BLUE/GREEN) and LEVEL (RED) as the X and Y of a chart (below), it becomes clearer how various everyday examples of automation fit into the scheme. As LEVEL and/or TYPE increase, we get a higher degree of automation (dotted line).
Mainstream discussions of AI and its potential dangers seem to be focusing on a hypothetical ultra-high degree of automation. A hypothetical weapon that will, on its own, determine threats and act. There are actually very few examples of such a high level of automation in everyday life because cutting the human completely "out of the loop" can have severely negative human performance consequences.
Figure 2 shows some examples of automation and where they fit into the scheme:
Wickens et al., (2010) use the phrase, "the higher they are, the farther they fall." This means that when humans interact with greater degrees of automation, they do fine if it works correctly, but will encounter catastrophic consequences when automation fails (and it always will at some point). Why? Users get complacent with high DOA automation, they forget how to do the task themselves, or they loose track of what was going on before the automation failed and thus cannot recover from the failure so easily.
You may have experienced a mild form of this if your car has a rear-backup camera. Have you ever rented a car without one? How do you feel? That feeling of being "out of the loop" tends to get magnified with higher degrees of automation. More on this in an upcoming throwback post.
So, highly autonomous weapons (or any high degree of automation) is not only a philosophically bad/evil idea, it is bad for human performance!
For more discussion on the degree and level of automation, see Arathi's recent Throwback post.