Robotic weapon systems are dangerous in several ways. The obvious way is as a threat to life and limb. A recent major "accident" with a robotic weapon (described below) is the occasion and inspiration for this blog entry. There are two more significant threats, neither of which is as easy to avoid as a buggy system (and fixing a buggy system is not easy).
The threats I'm concerned with are:
1. The rapidly advancing technology of "autonomous systems" and the readiness of the military to employ autonomy in weapons systems.
2. The "ethical distance" that is introduced when autonomous systems are used; a human in many cases is no long required to make a decision to kill
First, more about the recent accident:
Robot Cannon Kills 9, Wounds 14
We're not used to thinking of them this way. But many advanced military weapons are essentially robotic -- picking targets out automatically, slewing into position, and waiting only for a human to pull the trigger. Most of the time. Once in a while, though, these machines start firing mysteriously on their own. The South African National Defence Force "is probing whether a software glitch led to an antiaircraft cannon malfunction that killed nine soldiers and seriously injured 14 others during a shooting exercise on Friday."
Autonomous Robotic Weapons
Are we only a few steps from "Skynet"? A "character" (liberally speaking) in the Terminator series of movies, Skynet was the product of a tightly integrated autonomous defense system which became "sentient" (I won't define that for you) and then (reasonably) determined that humans were a threat to its own existence.
No, we are more than a few steps. Nevertheless, weapon systems are becoming not only more automated (the capability to execute complex instructions) but more autonomous (the ability to independently determine when to perform a course of action).
As an expert in autonomous systems, I am very concerned about the accelerating trend to arm robotic systems. A good example is the unmanned combat autonomous vehicle, or UCAV, an emerging aircraft system beyond the high capability UAVs such as GlobalHawk.
UCAVs are the product of three converging technologies:
1. Aircraft are becoming more autonomous and their computers are taking over more of the functions of targeting and weapons delivery;
2. Weapons such as the Tomahawk cruise missile are becoming more sophisticated and capable of functions, such as target and aim point selection, that before now belong to the pilot or what is called "the delivery platform";
3. Rapidly increasing performance levels are incorporated into UCAV design as new technology becomes available.
Normally I would applaud #3, but the increasing pace of technology development leaves us little time to think of the consequences of our actions. Fortunately with respect to UCAVs, this discussion is underway within and outside the military. Important issues include the "level of autonomy" (degree to which human intervention is required) for capability, weapon release authority, and de-confliction with other platforms.
Consider the question of "how much" autonomy to give to a robotic system for weapon release in a given type of war zone (e.g, rear of the front line vs. active areas of combat with the enemy). This is more or less equivalent to giving human soldiers "rules of engagement", or simply, when to shoot and when not to shoot. The maturity of deployable autonomous systems is at a level now where there is
no technical reason why such systems could not choose their own target and destroy it without human intervention. For example, see Lockheed's "
Low Cost Autonomous Attack System". Classified as a "smart munition", Lockheed says of it, "Target aimpoint and warhead mode [based on type of target] are automatically determined." This weapon system autonomously finds and attacks targets once release by a weapons system such as an aircraft, rocket or missile. No human tells the LOCAAS what to hit. Humans just release the weapon in the locality of the enemy.
The greatest danger is of course deadly friendly fire. (A good, but somewhat dated military overview of the issues surrounding autonomous weapons can be found
here.)
The Ethical Distance From Killing
Consider the case of a modern cruise missile. A person in the chain of command makes a decision to launch the weapon at a primary target. The ethical responsibility lies with that person. Such weapons have the capability to be "reprogrammed" in flight, that is, told to halt the attack on the primary target and to proceed to attack a secondary target at certain coordinates. This could be done by someone else in a different chain-of-command. Now the original person who launched the missile is no longer responsible; the person who made the re-targeting decision is responsible.
Now let us suppose that the re-targeting is done automatically by systems on-board the missile. For example, on arrival in the target area the missile does not detect the artillery that was its primary target. The missile is pre-programmed to "loiter" in the area and look for a valid target. A tank is detected by the missile which then proceeds to destroy it. (This technology exists and may already be deployed.) Who now has the ethical responsibility for making the decision to kill the people in the tank? The person who originally launched the missile, but has no idea of what it actually attacked? The programmers of the "search and destroy" automation on-board the missile? The military program manager who decided to develop and deploy such systems? It is very easy to see how the responsibility for the decision to kill, in particular, has been blurred by the use of an autonomous weapons system. By taking away that clear responsibility, are we making it easier to kill?
I think the answer is yes, because no decision actually needs to be made. Consider the case of a robotic "sentry" which is deployed to guard a particular area, i.e., deny use of the area to the enemy. The DMZ in Korea provides a real-world example of such an area, where "enemy troops" (e.g., from North Korea) entering the DMZ could signal an invasion. A robot sentry recently developed by Samsung (Ref 1, Ref 2, Ref 3 - translated to English with video) has the ability to automatically detect a person at a distance of 4 kilometers in the daytime, 2 kilometers at night, in a 180 degree radius. A person in camo or only partially visible (hiding) can be detected, even in inclimate weather. Once detected, an audible warning is sounded, although just how audible at 4 kilometers is open to question. Reportedly, the robot has the ability to use voice recognition to detect "surrender" (again, one wonders). The robot then makes a decision to use its 5.5mm machine gun to take down the target. The visual recognition system of this robot reportedly has the ability to detect the "shedding of blood" to determine whether additional shots are required for a kill. No humans, other than the enemy combatant, are involved.
South Korea is beginning the deployment of this system, and other countries are also interested.
The "ethical distance" in the case of the robot sentry is enormous because nobody actually makes a decision to kill. After all, there is no enemy in the "exclusion zone", only the possibility of an enemy, who "should" be killed, at some time in the future.
On reflection, I suppose that this type of weapon is not all that different from land mines with respect to the ethical issues (and long-lasting land mines give rise to even more ethical issues). However, the methodical combination of technologies in an autonomous robotic system – the decision to kill and individual – especially worries me.
Labels: artificial intelligence, autonomous systems, ethics, robotics, war, weapon
