Handbook of Software Reliability Engineering
Michael Lyu (Editor)
Notes on:
Handbook of Software Reliability EngineeringMichael Lyu (Editor) |
|
|
Handbook of Software Reliability Engineering, Michael Lyu (Editor), IEEE Computer Society Press, 1996. (850+ pages).
This is a comprehensive collection of chapters on Software Reliability, and is an excellent place to review the current state of knowledge in the area. Topics covered include a survey or reliability modeling techniques, discussion of operational profiles (which are central to many modeling approaches), current practice & experience, data analysis, and several emerging techniques.
Topic coverage: (*** = emphasized; ** = discussed with some detail; * = mentioned)
| *** | Dependability | Electronic Hardware | * | Requirements | |||||
| Safety | *** | Software | ** | Design | |||||
| Security | Electro-Mechanical Hardware | *** | Manufacturing | ||||||
| Scalability | Control Algorithms | * | Deployment | ||||||
| Latency | Humans | * | Logistics | ||||||
| * | Affordability | Society/Institutions | Retirement |
"Features contributions from the world's leading reliability experts, this book/CD-ROM package offers you the most comprehensive and up-to-date resource on software reliability engineering available today.
"The Handbook takes you step by step through software reliability measurement and prediction ... the attributes and metrics of product design, development process, software operational environment, and their effects on reliability ... and the application of this information in software development, acquisition, use, and maintenance. You'll find detailed coverage of:
- "Software reliability engineering techniques involving modeling, measurement, process framework, systems operations, data collection, and more
- "Benchmark case studies from AT&T, JPL, Bellcore, IBM, Nortel, NASA, Hitachi, ALCATEL, and other European organizations
- "Emerging research methods including software metrics, testing schemes, fault-tolerant software, fault-tree analysis, process simulation, and neural networks
"This indispensable reference comes packaged with an MS/DOS CD-ROM loaded with more than $250 worth of popular software reliability tools: SMERFS, CASRE, and SoftRel-plus a valuable repository of software project failure data sets."
Related material:
Foreword by Alfred V. Aho xix
Foreword by Richard A. DeMillo xxi
Preface xxiii
Chapter 1. Introduction
Michael R. Lyu (AT&T Bell Labs.)
1.1 The Need for Reliable Software 3
1.2 Software Reliability Engineering Concepts 5
1.3 Book Overview 8
1.4 Basic Definitions 12
1.5 Technical Areas Related to the Book 19
1.5.1 Fault Prevention 19
1.5.2 Fault Removal 20
1.5.3 Fault Tolerance 20
1.5.4 Fault/Failure Forecasting 21
1.5.5 Scope of this Handbook 21
1.6 Summary 22
Problems 22
Chapter 2. Software Reliability and System Reliability
Jean-Claude Laprie and Karama Kanoun (LAAS-CNRS, France)
2.1 Introduction 27
2.2 The Dependability Concept 28
2.2.1 Basic Definitions 28
2.2.2 On the Impairments to Dependability 28
2.2.3 On the Attributes of Dependability 32
2.2.4 On the Means for Dependability 33
2.3 Failure Behavior of an X-Ware System 35
2.3.1 Atomic Systems 35
2.3.2 Systems Made up of Components 41
2.4 Failure Behavior of an X-Ware System with Service
Restoration 49
2.4.1 Characterization of System Behavior 50
2.4.2 Maintenance Policies 51
2.4.3 Reliability Modeling 53
2.4.4 Availability Modeling 60
2.5 Situation with Respect to the State-of-the-Art in
Reliability Evaluation 64
2.7 Summary 68
Problems 68
Chapter 3. Software Reliability Modeling Survey
William Farr (Naval Surface Warfare Center)
3.1 Introduction 71
3.2 Historical Perspective and Implementation 72
3.2.1 Historical Background 72
3.2.2 Model Classification Scheme 73
3.2.3 Model Limitations and Implementation Issues 76
3.3 Exponential Failure Time Class of Models 77
3.3.1 Jelinski-Moranda "De-Eutrophication" Model 77
3.3.2 Nonhomogeneous Poisson Process Model 80
3.3.3 Schneidewind's Model 82
3.3.4 Musa's Basic Execution Time Model 87
3.3.5 Hyperexponential Model 90
3.3.6 Others 92
3.4 Weibull and Gamma Failure Time Class of Models 93
3.4.1 Weibull Model 93
3.4.2 S-Shaped Reliability Growth Model 95
3.5 Infinite Failure Category Models 98
3.5.1 Duane's model 98
3.5.2 Geometric Model 99
3.5.3 Musa-Okumoto Logarithmic Poisson 102
3.6 Bayesian Models 104
3.6.1 Littlewood-Verrall Reliability Growth Model 105
3.5.3 Other Bayesian Models 109
3.7 Model Relationships 109
3.7.1 Generalized Exponential Model Class 109
3.7.2 Exponential Order Statistic Model Class 111
3.8 Software Reliability Prediction in Early Phases
of the Life Cycle 111
3.8.1 Phase-Based Model 111
3.8.2 Predicting Software Defects from Ada Design 112
3.8.3 Rome Laboratory Work 113
3.9 Summary 114
Problems 115
Chapter 4. Techniques for Prediction Analysis and Recalibration
Sarah Brocklehurst, Bev Littlewood (City University of London)
4.1 Introduction 119
4.2 Examples of Model Disagreement and Inaccuracy 120
4.2.1 Simple Short Term Predictions 120
4.2.2 Longer Term Predictions 123
4.2.3 Model Accuracy Varies from Data Source to
Data Source 126
4.2.4 Why We Cannot Select the Best Model a 126
Priori
4.2.5 Discussion - a Possible Way Forward 127
4.3 Methods of Analyzing Predictive Accuracy 128
4.3.1 Basic Ideas - Recursive Comparison of
Predictions with Eventual Outcomes 128
4.3.2 The Prequential Likelihood Ratio (PLR) 131
4.3.3 The U-Plot 135
4.3.4 The Y-Plot 140
4.3.5 Discussion: the Likely Nature of Prediction
Errors, and How We can Detect Inaccuracy 141
4.4 Recalibration 145
4.4.1 The U-Plot as a Means of Detecting 'Bias' 145
4.4.2 The Recalibration Technique 146
4.4.3 Examples of the Power of Recalibration 147
4.5 A Worked Example 150
4.6 Discussion 156
4.6.1 Summary of the Good News: Where We Are Now 156
4.6.2 Limitations of Present Techniques 159
4.6.3 Possible Avenues for Improvement of Methods 160
4.6.4 Best Advice to Potential Users 162
4.7 Summary 163
Problems 164
Chapter 5. The Operational Profile
John Musa, Bruce Juhlin, Gene Fuoco, Diane Kropfl, and Nancy
Irving
(AT&T Bell Labs.)
5.1 Introduction 167
5.2 Concepts 168
5.3 Development Procedure 170
5.3.1 Customer Type List 173
5.3.2 User Type List 173
5.3.3 System Mode List 174
5.3.4 Functional Profile 176
5.3.5 Operational Profile 183
5.4 Test Selection 194
5.4.1 Selecting Operations 195
5.4.2 Regression Test 196
5.5 Special Issues 197
5.5.1 Indirect Input Variables 197
5.5.2 Updating the Operational Profile 197
5.5.3 Distributed Systems 198
5.6 Other Uses 199
5.7 Application to DEFINITY 200
5.7.1 Project Description 200
5.7.2 Development Process Description 200
5.7.3 Describing Operational Profiles 201
5.7.4 Implementing Operational Profiles 203
5.7.5 Conclusion 204
5.8 Application to FASTAR (Fast Automated Restoration) 204
5.8.1 System Description 204
5.8.2 FASTAR: SRE Implementation 206
5.8.3 FASTAR: SRE Benefits 210
5.9 Application to the Power Quality Resource System 210
5.9.1 Project Description 210
5.9.2 Developing the Operational Profile 211
5.9.3 Testing 213
5.9.4 Conclusion 214
5.10 Summary 215
Problems 215
Chapter 6. Best Current Practice of SRE
Mary Donnelly, Bill Everett, John Musa, and Geoff Wilson
(AT&T
Bell Labs.)
6.1 Introduction 219
6.2 Benefits and Approaches to SRE 220
6.2.1 Importance and Benefits 221
6.2.2 An SRE Success Story 221
6.2.3 SRE Costs 222
6.2.4 SRE Activities 223
6.2.5 Implementing SRE Incrementally 223
6.2.6 Implementing SRE on Existing Projects 224
6.2.7 Implementing SRE on Short-Cycle Projects 226
6.3 SRE During Feasibility and Requirements Phase 226
6.3.1 Feasibility Stage 226
6.3.2 Requirements Stage 228
6.4 SRE during Design and Implementation Phase 232
6.4.1 Design Stage 232
6.4.2 Implementation Stage 233
6.5 SRE during the System Test and Field Trial Phase 235
6.5.1 Determine Operational Profile 236
6.5.2 System Test Stage 237
6.5.3 Field Trial Stage 241
6.6 SRE during Post-Delivery and Maintenance Phase 242
6.6.1 Project Post-Release Staff Needs 242
6.6.2 Monitor Field Reliability vs. Objectives 243
6.6.3 Track Customer Satisfaction 245
6.6.4 Time New Feature Introduction by Monitoring
Reliability 245
6.6.5 Guide Produce and Process Improvement with
Reliability Measures 246
6.7 Getting Started with SRE 246
6.7.1 Prepare Your Organization for SRE 247
6.7.2 Find More Information or Support 250
6.7.3 Do an SRE Self-Assessment 250
6.8 Summary 252
Problems 253
Chapter 7. Software Reliability Measurement Experience
Allen Nikora (Jet Propulsion Laboratory) and Michael R. Lyu
(AT&T Bell Labs.)
7.1 Introduction 255
7.2 Measurement Framework 256
7.2.1 Establishing Software Reliability
Requirements 259
7.2.2 Setting up a Data Collection Process 266
7.2.3 Defining Data to be Collected 267
7.2.4 Choosing a Preliminary Set of Software
Reliability Models 272
7.2.5 Choosing Reliability Modeling Tools 273
7.2.6 Model Application and Application Issues 273
7.2.7 Dealing with Evolving Software 276
7.2.8 Practical Limits in Modeling
Ultrareliability 277
7.3 Investigation at JPL 278
7.3.1 Project Selection and Characterization 278
7.3.2 Characterization of Available Data 280
7.3.3 Experimental Results 280
7.4 Investigation at Bellcore 281
7.4.1 Project Characteristics 281
7.4.2 Data Collection 284
7.4.3 Application Results 285
7.5 Linear Combination of Model Results 289
7.5.1 Statically-Weighted Linear Combinations 290
7.5.2 Weight Determination Based on Ranking Model
Results 290
7.5.3 Weight Determination Based on Changes in
Prequential Likelihood 291
7.5.4 Modeling Results 291
7.5.5 Overall Project Results 292
7.5.6 Extensions and Alternatives 295
7.5.7 Long-Term Prediction Capability 298
7.6 Summary 299
Problems 300
Chapter 8. Measurement Based Analysis of Software Reliability
Ravi K. Iyer (University of Illinois) and Inhwan Lee (Tandem,
Inc.)
8.1 Introduction 303
8.2 Framework 304
8.2.1 Overview 304
8.2.2 Operational vs. Development Phase
Evaluation 306
8.2.3 Past Work 306
8.3 Measurement Techniques 307
8.3.1 On-Line Machine Logging 308
8.3.2 Manual Reporting 310
8.4 Preliminary Analysis of Data 312
8.4.1 Data Processing 312
8.4.2 Fault and Error Classification 314
8.4.3 Error Propagation 317
8.4.4 Error and Recovery Distributions 320
8.5 Detailed Analysis of Data 323
8.5.1 Dependency Analysis 324
8.5.2 Hardware-Related Software Errors 327
8.5.3 Evaluation of Software Fault Tolerance 328
8.5.4 Recurrences 329
8.6 Model Identification and Analysis of Models 333
8.6.1 Impact of Failures on Performance 333
8.6.2 Reliability Modeling in the Operational
Phase 335
8.6.3 Failure/Error/Recovery Model 339
8.6.4 Multiple Error Model 344
8.7 Impact of System Activity 345
8.7.1 Statistical Models from Measurements 345
8.7.2 Overall System Behavior Model 348
8.8 Summary 352
Problems 353
Chapter 9. Orthogonal Defect Classification
Ram Chillarege (IBM Research)
9.1 Introduction 359
9.2 Measurement and Software 360
9.2.1 Software Defects 361
9.2.2 The Spectrum of Defect Analysis 364
9.3 Principles of ODC 367
9.3.1 The Intuition 367
9.3.2 The Design of Orthogonal Defect
Classification 370
9.3.3 Necessary Condition 371
9.3.4 Sufficient Conditions 373
9.4 The Defect-Type Attribute 374
9.5 Relative Risk Assessment Using Defect Types 376
9.5.1 Subjective Aspects of Growth Curves 377
9.5.2 Combining ODC and Growth Modeling 379
9.6 The Defect Trigger Attribute 384
9.6.1 The Trigger Concept 384
9.6.2 System Test Triggers 387
9.6.3 Review and Inspection Triggers 387
9.6.4 Function Test Triggers 388
9.6.5 The Use of Triggers 389
9.7 Multidimensional Analysis 393
9.8 Deploying ODC 396
9.9 Summary 398
Problems 399
Chapter 10. Trend Analysis
Karama Kanoun and Jean-Claude Laprie (LAAS-CNRS, France)
10.1 Introduction 401
10.2 Reliability Growth Characterization 402
10.2.1 Definitions of Reliability Growth 403
10.2.2 Graphical Interpretation of the
Subadditive Property 404
10.2.3 Subadditive Property Analysis 406
10.2.4 Subadditive Property and Trend Change 407
10.2.5 Some Particular Situations 408
10.2.6 Summary 409
10.3 Trend Analysis 410
10.3.1 Trend Tests 410
10.3.2 Example 419
10.3.3 Typical Results That Can Be Drawn from
Trend Analyses 422
10.3.4 Summary 424
10.4 Application to Real Systems 424
10.4.1 Software of System SS4 425
10.4.2 Software of System S27 427
10.4.3 Software of System SS1 427
10.4.4 Software of System SS2 429
10.4.5 SAV 429
10.5 Extension to Static Analysis 431
10.5.1 Static Analysis Conduct 431
10.5.2 Application 433
10.6 Summary 433
Problems 435
Chapter 11. Field Data Analysis
Wendell Jones (BNR, Inc.) and Mladen Vouk (NCSU)
11.1 Introduction 439
11.2 Data Collection Principles 441
11.2.1 Introduction 441
11.2.2 Failures, Faults, and Related Data 442
11.2.3 Time 444
11.2.4 Usage 445
11.2.5 Data Granularity 446
11.2.6 Data Maintenance and Validation 447
11.2.7 Analysis Environment 448
11.3 Data Analysis Principles 449
11.3.1 Plots and Graphs 450
11.3.2 Data Modeling and Diagnostics 454
11.3.3 Diagnostics for Model Determination 455
11.3.4 Data Transformations 458
11.4 Important Topics in Analysis of Field Data 459
11.4.1 Calendar Time 461
11.4.2 Usage Time 461
11.4.3 An Example 462
11.5 Calendar-Time Reliability Analysis 463
11.5.1 Case Study (IBM Corp.) 464
11.5.2 Case Study (Hitachi) 466
11.5.3 Further Examples 468
11.6 Usage-Based Reliability Analysis 469
11.6.1 Case Study (Northern Telecom
Telecommunication Systems) 469
11.6.2 Further Examples 470
11.7 Special Events 472
11.7.1 Rare Event Models 473
11.7.2 Case Study (Space Shuttle Flight Software)476
11.8 Availability 479
11.8.1 Introduction 479
11.8.2 Measuring Availability 480
11.8.3 Empirical Unavailability 481
11.8.4 Models 483
11.9 Summary 486
Problems 487
Chapter 12. Software Metrics for Reliability Assessment
John Munson (University of Idaho) and Taghi Khoshgoftaar (Florida
Atlantic University)
12.1 Introduction 493
12.2 Static Program Complexity 495
12.2.1 Software Metrics 495
12.2.2 A Domain Model of Software Attributes 496
12.2.3 Principal Components Analysis 497
12.2.4 The Usage of Metrics 499
12.2.5 Relative Program Complexity 500
12.2.6 Software Evolution 502
12.3 Dynamic Program Complexity 504
12.3.1 Execution Profile 505
12.3.2 Functional Complexity 505
12.3.3 Dynamic Aspects of Functional Complexity 507
12.3.4 Operational Complexity 509
12.4 Software Complexity and Software Quality 510
12.4.1 An Overview 510
12.4.2 An Application and Its Metrics 512
12.4.3 Multivariate Analysis in Software Quality
Control 514
12.4.4 Fault Prediction Models 518
12.4.5 Enhancing Predictive Models with
Increased Domain Coverage 520
12.5 Software Reliability Modeling 523
12.5.1 Reliability Modeling with Software
Complexity Metrics 524
12.5.2 The Incremental Build Problem 526
12.6 Summary 527
Problems 527
Chapter 13. Software Testing and Reliability
Joseph R. Horgan (Bellcore) and Aditya P. Mathur (Purdue
University)
13.1 Introduction 531
13.2 Overview of Software Testing 532
13.2.1 Kinds of Software Testing 532
13.2.2 Concepts from White-Box and Black-Box
Testing 532
13.3 Operational Profiles 534
13.3.1 Difficulties in Estimating the
Operational Profile 535
13.3.2 Estimating Reliability 537
13.4 Time/Structure Based Software Reliability
Estimation 539
13.4.1 Definitions and Terminology 539
13.4.2 Basic Assumptions 540
13.4.3 Testing Methods and Saturation Effect 541
13.4.4 Testing Effort 541
13.4.5 Limits of Testing Methods 542
13.4.6 Empirical Basis of the Saturation Effect 543
13.4.7 Reliability Overestimation due to
Saturation 545
13.4.8 Incorporating Coverage in Reliability
Estimation 546
13.4.9 Filtering Failure Data Using Coverage
Information 547
13.4.10 Selecting the Compression Ratio 551
13.4.11 Handling Rare Events 553
13.5 A Microscopic Model of Software Risk 554
13.5.1 A Testing-Based Model of Risk Decay 554
13.5.2 Risk Assessment: An Example 555
13.5.3 A Simple Risk Computation 558
13.5.4 A Risk Browser 560
13.5.5 The Risk Model and Software Reliability 561
13.6 Summary 563
Problems 563
Chapter 14. Fault-Tolerant Software Reliability Engineering
David McAllister and Mladen Vouk (NCSU)
14.1 Introduction 567
14.2 Present Status 568
14.3 Principles and Terminology 569
14.3.1 Result Verification 570
14.3.2 Redundancy 574
14.3.3 Failures and Faults 575
14.3.4 Adjudication by Voting 577
14.3.5 Tolerance 578
14.4 Basic Techniques 581
14.4.1 Recovery Blocks 581
14.4.2 N-Version Programming 582
14.5 Advanced Techniques 583
14.5.1 Consensus Recovery Block 583
14.5.2 Acceptance Voting 584
14.5.3 N Self-Checking Programming 584
14.6 Reliability Modeling 585
14.6.1 Diversity and Dependence of Failures 586
14.6.2 Data-Domain Modeling 589
14.6.3 Time-Domain Modeling 594
14.7 Reliability in the Presence of Inter-Version
Failure Correlation 596
14.7.1 An Experiment 596
14.7.2 Failure Correlation 598
14.7.3 Consensus Voting 599
14.7.4 Consensus Recovery Block 601
14.7.5 Acceptance Voting 603
14.8 Development and Testing of Multi-Version
Fault-Tolerant Software 604
14.8.1 Requirements and Design 605
14.8.2 Verification, Validation and Testing 606
14.8.3 Cost of Fault-Tolerant Software 607
14.9 Summary 609
Problems 609
Chapter 15. Software Reliability Analysis using Fault Trees
Joanne Bechta Dugan (University of Virginia)
15.1 Introduction 615
15.2 Fault Tree Modeling 615
15.2.1 Cutset Generation 617
15.2.2 Fault Tree Analysis 619
15.3 Fault Trees as a Design Aid for Software Systems 622
15.4 Safety Validation Using Fault Trees 623
15.5 Analysis of Fault Tolerant Software Systems 627
15.5.1 Fault Tree Model for Recovery Block
System 629
15.5.2 Fault Tree Model for N-Version Programming
System 630
15.5.3 Fault Tree Model for N Self-Checking
Programming System 632
15.6 Qualitative Analysis of Fault Tolerant Software 635
15.6.1 Methodology for Parameter Estimation from
Experimental Data 635
15.6.2 A Case Study in Parameter Estimation 639
15.6.3 Comparative Analysis of Three Software
Fault Tolerant Systems 642
15.7 System-Level Analysis of Hardware and Software
System 645
15.7.1 System Reliability/Safety Model for DRB 647
15.7.2 System Reliability/Safety Model for NVP 648
15.7.3 System Reliability/Safety Model for NSCP 650
15.7.4 A Case Study in System-Level Analysis 651
15.8 Summary 657
Problems 657
Chapter 16. Software Reliability Simulation
Robert Tausworthe (Jet Propulsion Laboratory) and Michael R. Lyu
(AT&T Bell Labs.)
16.1 Introduction 661
16.2 Reliability Simulation 662
16.2.1 The Need for Dynamic Simulation 663
16.2.2 Dynamic Simulation Approaches 664
16.3 The Reliability Process 665
16.3.1 The Nature of the Process 666
16.3.2 Structures and Flows 667
16.3.3 Interdependencies among Elements 668
16.3.4 Software Environment Characteristics 669
16.4 Artifact-Based Simulation 669
16.4.1 Simulator Architecture 670
16.4.2 Results 675
16.5 Rate-Based Simulation 676
16.5.1 Event Process Statistics 677
16.5.2 Single-Event Process Simulation 678
16.5.3 Recurrent Event Statistics 679
16.5.4 Recurrent Event Simulation 681
16.5.5 Secondary Event Simulation 682
16.5.6 Limited Growth Simulation 683
16.5.7 The General Simulation Algorithm 684
16.6 Rate-Based Reliability 686
16.6.1 Rate Functions of Conventional Models 686
16.6.2 Simulator Architecture 687
16.6.3 Display of Results 689
16.7 The Galileo Project Application 690
16.7.1 Simulation Experiments and Results 691
16.7.2 Comparisons with Other Software
Reliability Models 694
16.8 Summary 696
Problems 697
Chapter 17. Neural Networks for SRE
Nachimu Karunanithi (Bellcore) and Yashwant Malaiya (Colorado
State
University)
17.1 Introduction 699
17.2 Neural Networks 700
17.2.1 Processing Unit 700
17.2.2 Architecture 702
17.2.3 Learning Algorithms 705
17.2.4 Backpropagation Learning 705
17.2.5 Cascade-correlation Learning Architecture 707
17.3 Application of Neural Networks for Software
Reliability 709
17.3.1 Dynamic Reliability Growth Modeling 709
17.3.2 Identifying Fault-Prone Modules 710
17.4 Software Reliability Growth Modeling 710
17.4.1 Training Regimes 712
17.4.2 Data Representation Issue 712
17.4.3 A Prediction Experiment 713
17.4.4 Analysis of Neural Network Models 718
17.5 Identification of Fault-Prone Software Modules 718
17.5.1 Identification of Fault-Prone Modules
Using Software Metrics 719
17.5.2 Data Set Used 719
17.5.3 Classifiers Compared 720
17.5.4 Data Representation 722
17.5.5 Training Data Selection 723
17.5.6 Experimental Approach 723
17.5.7 Results 723
17.6 Summary 726
Problems 726
Appendix A. Software Reliability Tools 729
Appendix B. Review of Reliability Theory, Analytical
Techniques, and Basic Statistics 747
References 781
Go to: other books | resource page
Philip Koopman: koopman@cmu.edu