Comparing Operating Systems using Robustness Benchmarks

Philip Koopman
John Sung
Christopher Dingman
Daniel Siewiorek
Ted Marz

Carnegie Mellon University
Pittsburgh, Pennsylvania

Paper presented at: 16th IEEE Symposium on Reliable Distributed Systems, Durham, NC, October 22-24, 1997, pp. 72-79.


When creating mission-critical distributed systems using off-the-shelf components, it is important to assess the dependability of not only the hardware, but the software as well. This paper proposes a way to test operating system dependability. The concept of response regions is presented as a way to visualize erroneous system behavior and gain insight into failure mechanisms. A 5-point CRASH scale is defined for grading the severity of robustness vulnerabilities encountered. Test results from five operating systems are analyzed for robustness vulnerabilities, and exhibit a range of dependability. Robustness benchmarking comparisons of this type may provide important information to both users and designers of off-the-shelf software for dependable systems.

KEYWORDS: software reliability, software testing

A preprint of the paper is available: 432500 bytes: pdf file

BALLIST HOME PAGE Ballista Home Page