The Ballista® Project:
COTS Software Robustness Testing
® Carnegie Mellon University
Prof. Phil Koopman
The Ballista® Project:
|
|
® |
| Carnegie Mellon University |
The Ballista® automated robustness testing approach probes software to see how effective it is at exception handling. For example, Ballista testing can find ways to make operating systems crash, and can make other software packages suffer abnormal termination instead of gracefully returning error indications. Ballista testing enables users to test the robustness of home-grown and third party software on their own computers. It is primarily intended as a "black box" software testing tool, and is especially well suited to use on the APIs of Commercial Off-The-Shelf (COTS) software. Ballista started as a DARPA-funded research project, and is now largely completed. This web site is an archive of Ballista-related papers, publicly available software tools, and other information.
The Ballista® automated robustness testing methodology characterizes the exception handling effectiveness of software modules. For example, Ballista testing can find ways to make operating systems crash in response to exceptional parameters used for system calls, and can find ways to make other software packages suffer abnormal termination instead of gracefully returning error indications. The Ballista testing server enables users to test the robustness of their own and third party software via the Internet. Ballista is a "black box" software testing tool, and is works well on testing the APIs of Commercial Off-The-Shelf (COTS) software.
The success of many products depends on the robustness of not only the product software, but also operating systems and third party component libraries. But, until now, there has been no way to quantitatively measure robustness. Ballista changes this by providing a simple, repeatable way to directly measure software robustness without requiring source code or behavioral specifications. As a result, product developers can use robustness metrics to compare off-the-shelf software components, and component developers can measure their effectiveness at exception handling.
The Ballista testing approach is both scalable and portable across a wide variety of application domains. No behavior specification is required for testing - the implicit specification of “doesn’t crash; doesn’t hang” suffices. Additionally, tests are created based on the data types of the parameter list rather than based on module functionality, exploiting the fact that in most APIs there are fewer data types than functions/calls.
Originally developed as an internet-based service, a stand-alone version of the Ballista testing tool can be downloaded under GPL. The Ballista testing tool tests for exceptional condition handling effectiveness. The testing tool comes with the source code for not only tests, but the tools we used to create those tests. So, a knowledgeable user can modify the tests or extend them to test other functions/APIs. To use the service, the parameter signature information for the software component to be tested is compiled to form a test harness module, and links this harness to the software component to be tested. The tools use C++ for the test harness, but can be used to test any linkable component.
After the test tools exercise some combinations of input values for the software component under test, they analyze the robustness results and prepares a list of specific, reproducible test cases that cause the component to exhibit robustness failures (crashes and hangs).
The publicly distributed test tool includes predefined data types and other definition files to test common operating system calls and C library functions. The tests are extensible, although doing so requires substantial knowledge and is not intended for computing novices. (Please note, if you choose to extend or modify the tests we are unable to support you; however others have done this successfully and we provide enough information to get you pointed in the right direction to do this.)
As with any code downloaded from a university web site, the user should realize that this is code created as part of a project where students are still learning how to create software systems. While we understand that Ballista has been adopted by a number of companies, their first step is usually to re-write the code to conform with their internal quality standards. Under no circumstances should this student project code be mistaken for an actual product, and this code should not be used as-is for high integrity, mission critical, or safety-critical systems.
The initial funding for the Ballista project came from
DARPA ITO, Embeddable Systems program,
under contract DABT63-96-C-0064. The contents of this web site are the product
of Ballista researchers, and are not necessarily endorsed by DARPA.
The Ballista team thanks the following additional organizations for providing support:
Additional research partners:
Web server equipment subsidized by an equipment grant from
Compaq (formerly Digital)
; some testing performed on equipment donated by Intel
with software donated by Microsoft.
![[ECE Logo]](pics/ecelogo.gif)
The Ballista project is conducted by the Institute for Complex Engineered Systems (ICES; formerly EDRC); the Institute for Software Research, International; and the Electrical and Computer Engineering Department at Carnegie Mellon University.
"Ballista®" is a registered trademark of Carnegie Mellon University.
Web site by Phil Koopman.
Comments to: koopman@cmu.edu