pdf file (114 KB)Nathan P. Kropp, Philip J. Koopman, Daniel P. Siewiorek
Institute for Complex Engineered Systems
Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
Published in the Proceedings of FTCS’98, June 23-25, 1998 in Munich, Germany.
Mission-critical system designers may have to use a Commercial Off-The-Shelf (COTS) approach to reduce costs and shorten development time, even though COTS software components may not specifically be designed for robust operation. Automated testing can assess component robustness without sacrificing the advantages of a COTS approach. This paper describes the Ballista methodology for scalable, portable, automated robustness testing of component interfaces. An object-oriented approach based on parameter data t ypes rather than component functi onality essentially eliminates the need f or function- specific test scaffolding. A f ull- scal e implementation that automatically tests the robustness of 233 operating system software components has been ported to ten POSIX systems. Between 42% and 63% of components tested had robustness problems, with a normalized failure rate ranging from 10% to 23% of tests conducted. Robustness testing could be used by developers to measure and improve robustness, or by consumers to compare the robustness of competing COTS component libraries.
Paper:
pdf file (114 KB)Slides from conference presentation:
On-line slide show
pdf file (221 KB)![[summary results picture]](results.gif)
