Drift Into Failure Read online

  DRIFT INTO FAILURE

  Drift Into Failure

  From Hunting Broken Components to Understanding Complex Systems

  Sidney Dekker

  CRC Press

  Taylor & Francis Group

  6000 Broken Sound Parkway NW, Suite 300

  Boca Raton, FL 33487–2742

  © 2011 by Sidney Dekker

  CRC Press is an imprint of Taylor & Francis Group, an Informa business

  No claim to original U.S. Government works

  Printed on acid-free paper

  Version Date: 20160226

  International Standard Book Number-13:978-1-4094-2222-8 (Hardback) 978-1-4094-2221-1 (Paperback)

  This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

  Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

  For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978–750–8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

  Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

  Visit the Taylor & Francis Web site at

  http://www.taylorandfrancis.com

  and the CRC Press Web site at

  http://www.crcpress.com

  Contents

  Cover

  Half Title

  Title

  Copyright

  Contents

  List of Figures

  Acknowledgments

  Reviews for Drift into Failure

  Preface

  1 Failure is Always an Option Who messed up here?

  Technology has developed more quickly than theory

  Complexity, locality and rationality

  Complexity and drift into failure

  A great title, a lousy metaphor

  References

  2 Features of Drift The broken part

  The outlines of drift

  A story of drift

  References

  3 The Legacy of Newton and Descartes Why did Newton and Descartes have such an impact?

  So why should we care?

  We have Newton on a retainer

  References

  4 The Search for the Broken Component Broken components after a hailstorm

  Broken components to explain a broken system

  Newton and the simplicity of failure

  References

  5 Theorizing Drift Man-made disasters

  High reliability organizations

  Goal interactions and production pressure

  Normalizing deviance, structural secrecy and practical drift

  Control theory and drift

  Resilience engineering

  References

  6 What is Complexity and Systems Thinking? More redundancy and barriers, more complexity

  Up and out, not down and in

  Systems thinking

  Complex systems theory

  Complexity and drift

  References

  7 Managing the Complexity of Drift Complexity, control and influence

  Diversity as a safety value

  Turning the five features of drift into levers of influence

  Drifting into success

  Complexity, drift, and accountability

  A post-Newtonian ethic for failure in complex systems

  References

  Bibliography

  Index

  List of Figures

  Figure 5.1 Control structure as originally envisioned to guarantee water quality in Walkerton

  Figure 5.2 Safety control structure at the time of the water contamination incident at Walkerton

  Figure 7.1 The Herfindahl Index

  Acknowledgments

  I want to thank Paul Cilliers and Jannie Hofmeyr at the Centre for Studies in Complexity at the University of Stellenbosch, South Africa, for our fascinating discussions on complexity, ethics and system failure. I also want to thank Eric Wahren and Darrell Horn for their studious reading of earlier drafts and their helpful comments for improvement.

  Reviews for Drift into Failure

  "'Accidents come from relationships, not broken parts." Sidney Dekker's meticulously researched and engagingly written Drift into Failure: From Hunting Broken Parts to Understanding Complex Systems explains complex system failures and offers practical recommendations for their investigation and prevention from the combined perspectives of unruly technology, complexity theory, and post-Newtonian analysis. A valuable source book for anyone responsible for, or interested in, Organizational safety.'

  Steven P. Bezman, Aviation Safety Researcher

  'Dekker's book challenges the current prevalent notions about accident causation and system safety. He argues that even now, what profess to be systemic approaches to explaining accidents are still caught within a limited framework of 'cause and effect' thinking, with its origins in the work of Descartes and Newton. Instead, Dekker draws his inspiration from the science of complexity and theorises how seemingly reasonable actions at a local lavel may promulgate and proliferate in unseen (and unknowable) ways until finally some apparent system "failure " occurs. The book is liberally illustrated with detailed ease studies to articulate these ideas. As with all Dekker's books, the text walks a fine line between making a persuasive argument and provoking an argument. Love it or hate it, you can't ignore it.'

  Don Harris, HFI Solutions Ltd

  'Dekker's book contributes to the growing debate around the nature of retrospective investigations of safety-critical situations in complex systems. Both provocative and insightful, the author shines a powerful light on the severe limits of traditional linear approaches. His call for a diversity of voices and narratives, to deepen our understanding of accidents, will be welcomed in healthcare. Dekker's proposal that we shift from going "down and in " to "up and out" suggests a paradigm shift in accident investigation.'

  Rob Robson, Healthcare System Safety and Accountability, Canada

  'Professor Dekker explodes the myth that complex economic, technological and environmental failures can be investigated by approaches fossilized in linear, Newtonian-Cartesian logic. Today nearly 7 billion people unconsciously reshape themselves, their organizations, and societies through the use of rapidly-evolving, proliferating and miniaturizing technologies powered by programs that supersede the intellectual grasp of their developers. Serious proponents of the next high reliability organizations would do well to absorb Drift into Failure.'

  Jerry Poje, Founding Board Member of the U.S. Chemical Safety and Hazard Investigation Board

  'Today, catastrophic accidents resulting from failure of simple componen
ts confound industry. In Drift into Failure, Dekker shows how reductionist analysis – breaking the system down until we find the "broken part" – does not explain why accidents in complex systems occur. Dekker introduces the systems approach. Reductionism delivers an inventory of broken parts; Dekker's book offers a genuine possibility of future prevention. The systems approach may allow us to Drift into Success.'

  John O'Meara, HAZOZ

  Preface

  When I was in graduate school for my doctorate, we always talked about the systems we studied as complex and dynamic. Aviation, nuclear power, medicine, process control – these were the industries that we were interested in, and that seemed to defy simple, linear modeling – industries that demand of us, researchers, safety analysts, a commitment to penetrate the elaborate, intricate and live ways in which their work ebbs and flows, in which human expertise is applied, how organizational, economic and political forces suffuse and Constrain their functioning over time.

  Back then, and during most of my work in the years since, I have not encountered many models that are complex or dynamic. Instead, they are mostly simple and static. Granted, models are models for a reason: they are abstractions, simplifications, or perhaps no more than hopes, projections. Were a perfect model possible, one that completely and accurately represented the dynamics and complexity of its object, then its very specificity would defeat the purpose of modeling.

  So models always make sacrifices of some kind. The question, though, is whether our models sacrifice inconsequential aspects of the worlds we wish to understand and control, or vital aspects.

  During my first quarter in graduate school I took five classes, thinking this would be no problem. Well, actually, I didn't really think about it much at all. What concerned me was that I wanted as much value for money as I could get. I paid for my first quarter in grad school myself, which, for an international student, was a significant outlay (from there on I became a Graduate Research Assistant and the tuition was waived, otherwise I would not be writing this or much of anything else). For that first quarter, the earnings from a summer consulting job were burnt in one invoice. Then something interesting happened. Somehow I found out that with the four classes I had to take, I had reached a kind of maximum level beyond which apparently not even Ohio State could morally muster to extort more money from its international students. I could, in other words, throw in a class for the fun of it.

  I did.

  It became a class in non-linear dynamic systems. The choice was whimsical, really, a hint from a fellow student, and a fascinating title that seemed to echo some of the central labels of the field I was about to pursue a PhD in. The class was taught at the Department of Psychology, mind you. The room was small and dark and dingy and five or six students sat huddled around the professor.

  The first class hit me like the blast of a jet engine.

  The differences between static and dynamic stability were the easy stuff. You know, like done and over with in the first three minutes. From there, the professor galloped through an increasingly abstruse, dizzying computational landscape of the measurement of unpredictability, rotating cylinders and turning points, turbulence and dripping faucets, strange attractors, loops in phase space, transitions, jagged shores and fractals. And the snowflake puzzle.

  I didn't get an A.

  It was not long after James Gleick had published Chaos, and a popular fascination with the new science of complexity was brewing. The same year that I took this class, 1992, Roger Lewin published the first edition of Complexity, a first-person account of the adventures of people at the Santa Fe Institute and other exciting places of research.

  Taking this class was in a sense a fractal, a feature of a complex system that can be reproduced at any scale, any resolution. The class talked about the butterfly effect, but it also set in motion a butterfly effect. In one sense, the class was a marginal, serendipitous footnote to the subsequent years in grad school. But it represented the slightest shift in starting conditions. A shift that I wouldn't have experienced if it hadn't been for the tuition rules (or if I hadn't been reminded of that particularly arcane corner of the tuition rules or met that particular student who suggested the class, or if the psychology department hadn't had a professor infatuated with computation and complexity), an infinitesimal change in starting conditions that might have enormous consequences later on.

  Well, if you consider the publication of yet another book "enormous." Hardly, I agree. But still, I was forced to try to wrap my arms around the idea that complex, dynamic systems reveal adaptive behavior more akin to living organisms than the machines to which most safety models seem wedded. By doing this, the seeds of complexity and systems thinking were planted in me some 20 years ago.

  Drifting into failure is a gradual, incremental decline into disaster driven by environmental pressure, unruly technology and social processes that normalize growing risk. No organization is exempt from drifting into failure. The reason is that routes to failure trace through the structures, processes and tasks that are necessary to make an organization successful. Failure does not come from the occasional, abnormal dysfunction or breakdown of these structures, processes and tasks, but is an inevitable by-product of their normal functioning. The same characteristics that guarantee the fulfillment of the organization's mandate will turn out to be responsible for undermining that mandate.

  Drifting into failure is a slow, incremental process. An organization, using all its resources in pursuit of its mandate (providing safe air-travel, delivering electricity reliably, taking care of your savings), gradually borrows more and more from die margins that once buffered it from assumed boundaries of failure. The very pursuit of the mandate, over time, and under the pressure of various environmental factors (competition and scarcity most prominently), dictates that it does this borrowing – does things more efficiently, does more with less, perhaps takes greater risks. Thus, it is the very pursuit of the mandate that creates the conditions for its eventual collapse. The bright side inexorably brews the dark side – given enough time, enough uncertainty, enough pressure. The empirical base is not very forgiving: Even well-run organizations exhibit this pattern.

  This reading of how organizations fail contradicts traditional, and some would say simplistic, ideas about how component failures are necessary to explain accidents. The traditional model would claim that for accidents to happen, something must break, something must give, something must malfunction. This may be a component part, or a person. But in stories of drift into failure, organizations fail precisely because they are doing well – on a narrow range of performance criteria, that is – the ones that they get rewarded on in their current political or economic or commercial configuration. In the drift into failure, accidents can happen without anything breaking, without anybody erring, without anybody violating the rules they consider relevant.

  I believe that our conceptual apparatus for understanding drift into failure is not yet well-developed. In fact, most of our understanding is held hostage by a Newtonian–Cartesian vision of how the world works. This makes particular (and often entirely taken-for-granted) assumptions about decomposability and the relationship between cause and effect. These assumptions may be appropriate for understanding simpler systems, but are becoming increasingly inadequate for examining how formal-bureaucratically organized risk management, in a tightly interconnected complex world, contributes to the incubation of failure.

  The growth of complexity in society has outpaced our understanding of how complex systems work and fail. Our technologies have got ahead of our theories, We are able to build things whose properties we understand in isolation. But in competitive, regulated societies, their connections proliferate, their interactions and interdependencies multiply, their complexities mushroom.

  In this book, I explore complexity theory and systems thinking to better understand how complex systems drift into failure. I take some of the ideas from that early class in complexity theory, like sensitive dep
endence on initial conditions, unruly technology, tipping points, diversity – to find that failure emerges opportunistically, non-randomly, from the very webs of relationships that breed success and that are supposed to protect organizations from disaster. I hope this book will help us develop a vocabulary that allows us to harness complexity and find new ways of managing drift.

  1

  Failure is Always an Option

  Accidents are the effect of a systematic migration of organizational behavior under the influence of pressure toward cost-effectiveness in an aggressive, competitive environment.1

  Rasmussen and Svedung

  Who Messed up Here?

  If only there was an easy, unequivocal answer to that question. In June 2010, the U.S. Geological Survey calculated that as much as 50,000 barrels, or 2.1 million gallons of oil a day, were flowing into the Gulf of Mexico out of the well left over from a sunken oil platform. The Deepwater Horizon oil rig exploded in April 2010, then sank to the bottom of the sea while killing 11 people. It triggered a spill that lasted for months as its severed riser pipe kept spewing oil deep into the sea.

  Anger over the deaths and unprecedented ecological destruction turned to a hunt for culprits – Tony Hayward, the British CEO of BP, which used the rig (the rig was run by Transocean, a smaller exploration company) or Carl-Henric Svanberg, its Swedish chairman, or people at the federal Minerals Management Service. As we wade deeper into the mess of accidents like these, the story quickly grows murkier, branching out into multiple possible versions. The "accidental" seems to become less obvious, and the roles of human agency, decision-making and organizational trade-offs appear to grow in importance. But the possible interpretations of why these decisions and trade-offs caused an oil rig to blow up are book-ended by two dramatically different families of versions of the story.