Measuring Robustness
Software testing heritage:
- “Dirty” test cases -- see if correct error response is generated
- Can significantly out-number “clean” test cases (4:1 or 5:1) Ž expensive!
Fault tolerance heritage: fault injection
- Insert an intentional defect and observe how gracefully the system responds
- Potentially automated (potentially cheap)
- But, there are challenges
- Creating a non-intrusive injection mechanism
- Combinational explosion of potential interactions
- Repeatability / determinism
- Portability to compare systems / requirement for special hardware