Automated Robustness Testing of Off-the-Shelf Software Components

Nathan P. Kropp, Philip J. Koopman, Daniel P. Siewiorek

Institute for Complex Engineered Systems
Carnegie Mellon University, Pittsburgh, Pennsylvania, USA

Published in the Proceedings of FTCS’98, June 23-25, 1998 in Munich, Germany.


Mission-critical system designers may have to use a Commercial Off-The-Shelf (COTS) approach to reduce costs and shorten development time, even though COTS software components may not specifically be designed for robust operation. Automated testing can assess component robustness without sacrificing the advantages of a COTS approach. This paper describes the Ballista methodology for scalable, portable, automated robustness testing of component interfaces. An object-oriented approach based on parameter data t ypes rather than component functi onality essentially eliminates the need f or function- specific test scaffolding. A f ull- scal e implementation that automatically tests the robustness of 233 operating system software components has been ported to ten POSIX systems. Between 42% and 63% of components tested had robustness problems, with a normalized failure rate ranging from 10% to 23% of tests conducted. Robustness testing could be used by developers to measure and improve robustness, or by consumers to compare the robustness of competing COTS component libraries.


Slides from conference presentation:

[summary results picture]

BALLIST HOME PAGE Ballista Home Page