The Interlocks They Took Out

The first safety analysis carried out on the Therac-25 did not include the software. That is in the report, in one sentence, with the relevant fact in brackets: the analysis did not include software, although nearly full responsibility for safety rested on the software.

That sentence is the origin of everything that followed, and it is missing from almost every account of the case.

The machine

The Therac-25 was a medical linear accelerator built by Atomic Energy of Canada Limited, a dual mode machine that could fire either a low current electron beam directly at a tumour or drive a high current beam into a metal target to generate X-rays. The two modes are not similar. The X-ray mode uses a beam roughly a hundred times more powerful, because most of it is meant to be absorbed by the target and the flattener sitting between the beam and the patient.

If the machine is set for X-ray power and the target is not in place, what reaches the patient is an unattenuated electron beam.

Eleven Therac-25s were installed, five in the United States and six in Canada. The machine was controlled by a DEC PDP-11. The software was written in PDP-11 assembly language, by a single person.

What was left out

The earlier machines in the family, the Therac-6 and the Therac-20, had independent hardware interlocks: physical circuits whose job was to notice that the turntable was in the wrong position and refuse to fire, regardless of what the computer thought.

The Therac-25 did not duplicate them. The manufacturer took the view, then becoming common, that if the computer was watching, the hardware did not need to. As the investigation put it, some companies decide that hardware interlocks and backups are not worth the cost.

Some of the Therac-20's software routines were carried over into the Therac-25. A defect later found in that shared code was present in the Therac-20 as well. It had never hurt anybody there, because the Therac-20 had the hardware interlocks, and the interlocks caught what the software missed. Nobody noticed the bug existed until people had been injured by the machine that had nothing to catch it.

What happened

Between June 1985 and January 1987 there were six known accidents in which patients received massive overdoses. The investigators described it as the worst series of radiation accidents in the thirty five year history of medical accelerators. Some of the patients died.

The doses are the part that is hard to hold in the head. A therapeutic treatment is around 200 rads. Whole body exposure of about 500 rads kills half the people who receive it. Patients on the Therac-25 were estimated to have received something between 13,000 and 17,000 rads in one case and between 16,500 and 25,000 in another, delivered to a small area in a fraction of a second.

The mechanism most often described is a timing fault in the data entry screen. An operator who finished entering the prescription, then went back up the screen to change the mode from X-ray to electron, and returned to the command line within about eight seconds, could leave the machine in a state where the magnets had been set for one configuration and the beam for the other. Eight seconds is roughly how long the magnets take to move. An experienced operator typing quickly is exactly the person who triggers it.

That is a real fault and it was really there. It is also, according to the people who spent years investigating it, close to useless as an explanation.

What the investigation actually said

Nancy Leveson and Clark Turner published their account in IEEE Computer in July 1993, and a good deal of it is an argument against the way the story was already being told.

"It is nearly as useless to ascribe the cause of an accident to a computer error or a software error," they wrote. "Certainly software was involved in the Therac-25 accidents, but it was only one contributing factor. If we assign software error as the cause of the Therac-25 accidents, we are forced to conclude that the only way to prevent such accidents in the future is to build perfect software that will never behave unexpectedly."

They were equally hard on the fix. "Another mistake was the assumption that fixing a particular error, eliminating the current software bug, would prevent future accidents. There is always another software bug."

And on where the failure actually sat: "A lesson to be learned from the Therac-25 accidents is not to remove standard hardware interlocks when adding computer control." Software, they argued, should be treated as a single component. Designing a dangerous system so that one failure can lead to an accident violates basic system engineering, whether the failing component is a relay or a routine.

The report is also sharp about software reuse, which was then being offered as a safety argument in itself. Reusing modules that have been exercised for years does not make the new system they are dropped into safe, and can produce designs that are both awkward and dangerous. The Therac-20 code is the demonstration: the same routines, in a machine without interlocks, killed people.

There is a further observation in the paper that has aged unusually well. The Therac-25 accidents were unusual, they wrote, in actually involving coding errors. Most computer related accidents do not. They involve errors in the requirements: omissions, and mishandled environmental conditions and system states. The code does what it was asked to do, and what it was asked to do was wrong.

The part about the company

The engineering failure was matched by an organisational one, and the report is precise about it. When the accidents began, investigators assumed the cause must be hardware and looked only at hardware, because software was not the sort of thing that failed. There was no hazard logging, no incident tracking, no procedure that turned a phone call from a worried physicist into an investigation. The first such call should have started one. So should the first lawsuit.

The machine was recalled in 1987, about five years after the first commercial version became available and a year and a half or more after the first patient was hurt by it.

The sentence the report ends on is the one worth keeping. Preventing accidents, Leveson and Turner wrote, requires paying attention to all the root causes, not just the precipitating event in a particular circumstance. Accidents are unlikely to happen twice in exactly the same way. If we patch only the symptoms, or fix only the specific cause of one accident, we have not done the work.

Thirty years on, the Therac-25 is taught almost everywhere as a story about a race condition. It is filed under bugs. The people who investigated it wrote, at length and in the same journal issue, that this is the wrong drawer.

Sources

• Nancy G. Leveson and Clark S. Turner, "An Investigation of the Therac-25 Accidents", IEEE Computer volume 26 number 7, July 1993 (the six accidents between June 1985 and January 1987 and their description as the worst series of radiation accidents in the thirty five year history of medical accelerators; the eleven machines installed, five in the United States and six in Canada, and the 1987 recall; control by a DEC PDP-11 with software written in PDP-11 assembly language by a single person; the decision not to duplicate the hardware safety mechanisms and interlocks present on the Therac-6 and Therac-20; the reused Therac-20 routines containing a defect that caused no harm there because of those interlocks; the estimated doses of 13,000 to 17,000 and 16,500 to 25,000 rads against therapeutic doses of around 200 and a fatal whole body dose of about 500; the editing sequence and the roughly eight seconds the magnets take to set; the statement that the first safety analysis did not include software although nearly full responsibility for safety rested on it; the passages on ascribing accidents to software error, on there always being another software bug, and on not removing standard hardware interlocks when adding computer control; the observation that most computer related accidents involve requirements errors rather than coding errors; the absence of hazard logging and incident tracking; and the closing argument that preventing accidents requires attention to all the root causes rather than the precipitating event).
• Massachusetts Institute of Technology, course 6.033, "An Investigation of the Therac-25 Accidents, parts I to V" (the same investigation set as HTML and used as a teaching text, including the accident narratives and the software analysis).
• University of California eScholarship, "An investigation of the Therac-25 accidents" (the institutional copy of the paper deposited by the authors' university).
• Massachusetts Institute of Technology, course 6.033, "An Investigation of the Therac-25 Accidents, part V" (the lessons section, including the arguments on system engineering, software reuse and the treatment of software as a single component).