The Ballista Software Robustness Testing Service

John DeVale, Philip Koopman & David Guttendorf

ECE Department & Institute for Complex Engineered Systems
Carnegie Mellon University, Pittsburgh, Pennsylvania, USA

Published in the Proceedings of TCS’99, Washington DC.


Abstract

Computers are becoming essential to everyday life in modern society, but are not necessarily as dependable as one would like, especially with respect to software robustness. Cost and time constraints often limit testing to the important area of functional correctness, but leave few resources to determine a software system’s robustness in the face of exceptional conditions. This paper describes the Ballista Project – a methodology and Web server that remotely test software modules in linkable object code form. By focusing on module interfaces rather than functional specifications, we have been able to build a scalable robustness evaluation framework having a test development cost sub-linear with respect to the number of modules to be tested. A client-server framework exercises the module under test on the developer’s computer. The Ballista server remotely directs testing, eliminating the need to port server or application code, and maintaining confidentiality of the source code under test. Stand-alone tests to replicate failures are generated and viewed on the World Wide Web. The Ballista server can be used to create or augment test suites, and to collect data to better understand the exceptional conditions that induce software robustness failures.


Paper:

Slides from conference presentation:

For results of testing Microsoft Windows, see:
Shelton, C. & Koopman, P., "Robustness Testing of the Microsoft Win32 API, Dependable Systems and Networks Conference 2000/FTCS-30, New York City, June 26-28 2000.


BALLIST HOME PAGE Ballista Home Page

koopman@cmu.edu