Here is my proposition. If an engineering project is very large and/or complex, was costly to produce and deploy, is expensive to maintain, and has been operating successfully without failure, something bad will happen. In due course, its owners and operators will discover failure modes and mechanisms no one was aware of at the design phase. But with continued operating success (and often with concomitant financial success), attention to the known failure mechanisms will wane. Then the system will fail, and “everyone” will be surprised.

The syndrome was well described by author Diane Vaughn in her analysis of the 1986 Challenger explosion. The deteriorating propulsion-rocket joint seal that caused the tragedy was well known to NASA engineers, who examined the degree of degradation after each successful flight, but did little to correct the fault. Vaughn described this acceptance of poor design in which NASA continued to “get by” as “normalizing deviation.”

Failure, Then What?

In the post-accident phase of a major technical system failure, most of the key actors will emphasize the anticipated low probability that the system would have failed. Because of this projected remote risk of failure, an important factor will have been de-emphasized — namely, the consequences of a failure should it occur (or, more realistically, when it occurs). And what to do after it occurs.

Let’s test this proposition in the case of the April 2010 BP/Transocean Deepwater Horizon oil rig disaster. The blind shear ram (part of a 54-foot-high blowout prevention system resting on the ocean floor) was acknowledged to be the rig’s last line of defense. During the accident investigation it was often referred to as a “fail-safe” element — a misnomer since by definition the failure of a fail-safe device would result in the benign shutdown of a system. The shear ram was designed as a “pincher” to cut the drill pipe and stop the flow of oil and gas beyond the preventer. Its failure would permit the oil and gas to rise uncontrollably to the rig itself.

“Inconceivable”

Spokesmen for BP called the April event unprecedented and one that no one foresaw. One said it “seemed inconceivable” that the blowout preventer would fail. Yet the federal agency responsible for regulating offshore drilling, the Minerals Management Service (MMS), in 1999 reported 319 failures of blowout preventers in offshore U.S. drilling between 1992 and 1998, and 19 blowouts in the Gulf of Mexico from 2007 to 2009. The consequences of most of these failures were insufficient to make the front pages. On the other hand, it took two months to plug a leaky spill in Australia in 2009. Prior to the current accident, the largest spill in the Gulf of Mexico occurred on 4 June 1979. It continued at the rate of 10,000 to 30,000 barrels daily until 23 March 1980, when it was successfully capped. An attempt by divers to activate the blowout preventer had been unsuccessful.

Ram Failures 

With the passage of time, the shortcomings of the ram blowout preventer had become well known to the oil drilling industry. Det Noske Veritus (a Norwegian firm) found that in 11 cases of potential deepwater blowouts between 1980 and 2006, only six were prevented when the blowout preventers were activated. Expert consulting firms, many of them hired by the industry itself, had made recommendations for changes in the configuration of blowout preventers and in the methods and frequencies of their testing and maintenance.

West Engineering Services, in studies made in 2002 and 2004, concluded that the shear ram could fail to cut pipe even when properly activated because modern pipe is twice the strength of older pipe and faces additive pressures in deep, frigid water. Also, if the ram happened to close on the coupling between sections of the pipe, it would be virtually impossible to cut through it. Because of this, plus the well-known failure mechanisms of the shear ram, offshore drillers by 2001 had begun equipping their blowout preventers with a second, redundant shear ram. Transocean, the contractor in the April blowout, reported that 11 of its 14 rigs in the gulf now have redundant shear rams. But the Deepwater Horizon did not. In 2001 a Scandinavian research group, in a study commissioned by MMS, concluded that all subsea blowout preventers used for deepwater drilling should be equipped with two blind shear rams. MMS did not follow up on this requirement.

Among the vulnerabilities of the shear ram is the hydraulic shuttle valve that activates the ram blades. If it leaks or jams, the ram will malfunction. A leak was suspected in the hydraulic system of the Deepwater Horizon. The engineer who activated the blowout preventer reported that he immediately checked the hydraulic flowmeters; they indicated no flow. He thought it time to abandon ship.

Lax Testing

The New York Times reported that one industry-financed study described a mentality among rig operators that “I don’t want to find problems; I want to do the minimum necessary to obtain a good test” in a draft that contended that companies cut corners on federally mandated tests of blowout preventers.

 In 2003, the MMS mandated that companies submit test data that confirmed the shear rams could work on the specific drill pipe and at a particular site at the pressures they would encounter. But in 2009, an MMS engineer with decades of experience approved a BP permit without requiring such test data, saying he was never told to do so, and adding that he had approved hundreds of other permits in the gulf without such proof.

What may be learned from system failure case histories? It appears that not only managers but engineers, too, are lulled by a series of low-profile failures, especially if they don’t impose serious financial penalties. They may then give insufficient attention to known failure mechanisms and, with time, discount the possibility of a major failure and so be unprepared when it happens.

Sources

For more on risk analysis:

Flyvbjerg, B., N. Bruzelius, and W. Rothengatter, Megaprojects and Risk, Cambridge University Press, 2003.

Wilson, R., and E.A.C. Crouch, Risk Benefit Analysis, Harvard University Press.

Christiansen, D., "A NASA Design Defect," IEEE Spectrum, April 1986.

For more on blowout preventers:

Marine Riser Systems and Subsea Blowout Preventers, Petroleum Extension Service, The University of Texas, Austin.

The Cameron U Blowout Preventer http://www.c-a-m.com/content/products/product_detail.cfm?pid=2797

“Oil spill investigators find critical problems in blowout preventer,” The Washington Post, 12 May 2010.

Fountain, H., “Focus Turns to Well-Blocking System,” The New York Times, 11 May 2010.

“Gulf oil spill: Drilling technology explained,” Los Angeles Times, 29 April 2010.

Blowout Preventers   http://www.blowout-preventers.com/

Barstow, D., et al., “Between Blast and Spill, One Last Hope,” The New York Times, 21 June 2010.