Safeware: system safety and computers
Leveson
Notes on:
Safeware: system safety and computersLeveson |
|
|
Safeware: system safety and computers, Nancy Leveson, Addison-Wesley, Reading MA, 1995. 680 pages+, ISBN 0-201-11972-2.
This book puts software safety into the context of traditional industrial safety engineering. Or, put another way, it is designed as a way for software practitioners to learn to think about things that safety engineers think about. This book is not a definitive practitioner's guide to software safety (as many people had hoped it would be). But, it is extremely useful for both managers and engineers who are new to computer-based safety-critical systems, and serves as a way to refresh the "big picture" in those who are already practicing in the field.
The book begins with a discussion of describing computer-based systems as the fit into the "safety culture" (traditionally consisting of civilian aerospace, chemical refinery, nuclear power, and medical radiology applications). It then discusses system safey fundamentals, terminology, and models. It then prescribes the elements of a safety-critical software program including analyses, design, and verification. An appendix describes accidents from the various safety-critical system domains.
The strong points of this book are that it pays particular attention to the human element as it relates to largely automated systems. Another strong point is that it clearly spells out the differences between safety and other aspects such as dependability and security. The major weak point is that the book strongly reflects the opinions of the author, who is considered by some to be controversial (I myself tend to agree with most of what she says, but it is important to realize that in the software safety field there are as of yet few definitive answers). That being said, it is an essential part of a balanced reading diet for the serious student of computer-based safety critical systems. (Another part of this balanced diet should be Storey's Safety-Critical Computer Systems, which complements Leveson's work)
In the author's words (from a comp.risks posting): "Virtually all system safety engineers agree that software-related accidents are not any different than those in which computers are not used---software engineers are causing the same accidents that other engineers learned how to avoid years ago. They (and I) feel that unless the software engineers learn those basic safety concepts that have been accumulated over decades by engineers, we are going to repeat the accidents of the past and kill thousands of people unnecessarily."
Topic coverage: (*** = emphasized; ** = discussed with some detail; * = mentioned)
| *** | Dependability | Electronic Hardware | ** | Requirements | |||||
| *** | Safety | *** | Software | ** | Design | ||||
| * | Security | Electro-Mechanical Hardware | Manufacturing | ||||||
| Scalability | * | Control Algorithms | * | Deployment | |||||
| Latency | *** | Humans | Logistics | ||||||
| * | Affordability | Society/Institutions | Retirement |
Publisher Comments:
We are building systems today - and using computers to control them that have the potential for large-scale destruction of life and environment. More than ever, software engineers and system developers, as well as their managers, must understand the issues and develop the skills needed to anticipate and prevent accidents before they occur. Professionals should not require a catastrophe to happen before taking action.
Addressing this need in her long-awaited book, Nancy Leveson examines what is currently known about building safe electromechanical systems and looks at past accidents to see what practical lessons can be applied to new computer-controlled systems.
Safeware:
- Demonstrates the importance of integrating software safety efforts with system safety engineering
- Describes models of accidents and human error that underlie particular approaches to safety problems
- Presents the elements of a safeware program, including management, hazard analysis, requirements analysis, design for safety, design of the human-machine interface, and verification.
Reviews:
Related links:
PART ONE: THE NATURE OF RISK 1
1. RISK IN MODERN SOCIETY 3
1.1 Changing Attitudes toward Risk 5
1.2 Is Increased Concern Justified? 6
1.3 Unique Risk Factors in Industrialized Society 7
1.4 How Safe Is Safe Enough? 13
2. COMPUTERS AND RISK 21
2.1 The Role of Computers in Accidents 22
2.2 Software Myths 26
2.3 Why Software Engineering is Difficult 33
2.4 The Reality We Face 38
3. A HIERARCHICAL VIEW OF ACCIDENTS 39
3.1 The Concept of Causality 39
3.2 Subjectivity in Ascribing Causality 43
3.3 Oversimplification in Determining Causality 44
3.4 A Hierarchical Approach to Causality 48
4. ROOT CAUSES OF ACCIDENTS 53
4.1 Flaws in the Safety Culture 53
4.2 Ineffective Organizational Structure 74
4.3 Ineffective Technical Activities 77
4.4 Summary 88
5. HUMAN ERROR AND RISK 91
5.1 Do Humans Cause Most Accidents? 92
5.2 The Need for Humans in Automated Systems 100
5.3 Human Error as Human-Task Mismatch 102
5.4 Conclusions 107
6. THE ROLE OF HUMANS IN AUTOMATED SYSTEMS 109
6.1 Mental Models ill
6.2 The Human as Monitor 113
6.3 The Human as Backup 120
6.4 The Human as Partner 123
6.5 Conclusions 126
PART TWO: INTRODUCTION TO SYSTEM SAFETY 127
7. FOUNDATIONS OF SYSTEM SAFETY 129
7.1 Safety Engineering Before World War 11 129
7.2 Systems Theory 135
7.3 Systems Engineering 139
7.4 Systems Analysis 143
8. FUNDAMENTALS OF SYSTEM SAFETY 145
8.1 Historical Development 145
8.2 Basic Concepts 150
8.3 Software System Safety 156
8.4 Cost and Effectiveness of System Safety 159
8.5 Other Approaches to Safety 161
PART THREE: DEFINITIONS AND MODELS 169
9. TERMINOLOGY 171
9.1 Failure and Error 172
9.2 Accident and Incident 175
9.3 Hazard 176
9.4 Risk 179
9.5 Safety 181
9.6 Safety and Security 182
9.7 Summary 184
10. ACCIDENT AND HUMAN ERROR MODELS 185
10.1 Accident Models 186
10.2 Human Task and Error Models 204
10.3 Summary 224
PART FOUR: ELEMENTS OF A SAFEWARE PROGRAM 225
11. MANAGING SAFETY 227
11.1 The Role of General Management 228
11.2 Place in the Organizational Structure 235
11.3 Documentation 239
12. THE SYSTEM AND SOFTWARE SAFETY PROCESS 249
12.1 The General Tasks 250
12.2 Examples 261
13. HAZARD ANALYSIS 287
13.1 The Hazard Analysis Process 289
13.2 Types of System Models 305
13.3 General Types of Analysis 306
13.4 Limitations and Criticisms of Hazard
Analysis 309
14. HAZARD ANALYSIS MODELS AND TECHNIQUES 313
14.1 Checklists 314
14.2 Hazard Indices 315
14.3 Fault Tree Analysis 317
14.4 Management Oversight and Risk Tree Analysis 326
14.5 Event Tree Analysis 327
14.6 Cause-Consequence Analysis 332
14.7 Hazards and Operability Analysis 335
14.8 Interface Analyses 340
14.9 Failure Modes and Effects Analysis 341
14.10 Failure Modes, Effects, and Criticality
Analysis 344
14.11 Fault Hazard Analysis 344
14.12 State Machine Hazard Analysis 346
14.13 Task and Human Error Analysis Techniques 350
14.14 Evaluations of Hazard Analysis Techniques 357
14.15 Conclusions 358
15. SOFTWARE HAZARD AND REQUIREMENTS ANALYSIS 359
15.1 Process Considerations 360
15.2 Requirements Specification Components 362
15.3 Completeness in Requirements Specifications 363
15.4 Completeness Criteria for Requirements
Analysis 364
15.5 Constraint Analysis 391
15.6 Checking the Specification Against the
Criteria 393
16. DESIGNING FOR SAFETY 395
16.1 The Design Process 397
16.2 Types of Design Techniques and Precedence 400
16.3 Hazard Elimination 403
16.4 Hazard Reduction 414
16.5 Hazard Control 439
16.6 Damage Reduction 445
16.7 Design Modification and Maintenance 446
17. DESIGN OF THE HUMAN-MACHINE INTERFACE 447
17.1 General Process Considerations 449
17.2 Matching Tasks to Human Characteristics 452
17.3 Reducing Safety-Critical Human Errors 462
17.4 Providing Appropriate Information and
Feedback 464
17.5 Training and Maintaining Skills 481
17.6 Guidelines for Safe HMI Design 485
18. VERIFICATION OF SAFETY 489
18.1 Dynamic Analysis 492
18.2 Static Analysis 495
18.3 Independent Verification and Validation 507
18.4 Conclusions 508
EPILOGUE: THE WAY FORWARD 509
Appendix A. MEDICAL DEVICES: THE THERAC-25 STORY 515
A.1 Introduction 515
A.2 Background 515
A.3 Events 521
A.4 Causal Factors 548
Appendix B. AEROSPACE: APOLLO 13, THE DC-10,
AND CHALLENGER 555
B.1 The Approach to Safety in Civil Aviation 555
B.2 Apollo 13 558
B.3 The DC-1 0 Cargo Door Saga 563
B.4 The Space Shuttle Challenger Accident 569
Appendix C. THE CHEMICAL INDUSTRY: SEVESO,
FLIXBOROUGH, BHOPAL 581
C.1 Safety in the Chemical Process Industry 581
C.2 Seveso 584
C.3 Flixborough 591
C.4 Bhopal 598
Appendix D. NUCLEAR POWER: WINDSCALE, THREE
MILE ISLAND, AND CHERNOBYL 609
D.1 Background 609
D.2 Windscale 616
D.3 Three Mile Island 619
D.4 Chernobyl 640
REFERENCES 649
CREDITS 669
INDEX 671
Go to: other books | resource page
Philip Koopman: koopman@cmu.edu