Ballista OS Robustness Test Suite

Ballista OS Robustness Test Suite - Download & Execute

If you have not yet obtained the os test suite, you need to download the os test suite. Please understand that
" By downloading this software the user agrees to accept this software entirely as-is, with absolutely no warranty whatsoever, expressed or implied. The responsibility for ensuring fitness and correctness for any purpose lies entirely with the user."
and click on I Agree/Download to continue.
Please untar and unzip ostest.tar.gz after downloading it. It will create its own directory. For those of you who are unfamiliar with a the unix archiving utilities tar and gzip, you may use the following command to do this:
```
gunzip -c ostest.tar.gz |tar -xvf - 
```
You will notice that a directory ballista was created as part of the untar. Change to the Ballista directory.
```
cd ballista 
```
Execute the tests:
```
perl ostest.pl 
```

Do not run as root. It is possible that running the test suite as root may corrupt your system.
Answer the questions: When the script first starts it will ask you a couple questions or tell you difficulties that it encountered. If you want robustness failure rates to compare with other systems then hit ENTER when given the choice between ENTER and a percentage.
The package will automatically test every supported POSIX function in your operating system and return the results. NOTE: if a system crash occurs, you must reboot your machine, comment-out the line in the file callTable.all containing the function that caused the crash, and restart the test. (Additional details available)
The testing will take several hours. (We have seen 18-24 hour run times.) The test results are stored in the ../ballista/outfiles directory. In this directory you will notice a file corresponding with each POSIX function that was tested. The format for the function test results is outfile.function_name.param1.param2.... In addition to the function files are the os_results.txt, and os_results.html files, which contain formatted summaries of the os tests.
Important platform specific notes.

Generate stand alone code: Once the tests have been run there will be the option to create stand alone C++ code to reproduce a detected robustness failure. The details for generating code are outlined in the previously mentioned file ../ballista/outfiles/os_results.html. This step is optional but can be useful for bug reports.

Verify Testing: After the Ballista OS Robustness Test Suite completes then you should investigate the associated problems (if any exist). If the Test Suite encounters problems testing a function, it will log the function in .../ballista/outfiles/faillog, display an appropriate message, and skip testing the associated function. Generally these functions will be skipped because:
- The function is not defined on your operating system.
- The files are not in the expected location
- The required programs/utilities are not available
- The default compile options do not work properly for your system
- The datatype definition files use datatypes or constants which are inappropriate or unavailable for your system
- The OS function definition differs from the expected signature.
- other
If after investigating the reason the test suite skipped a function, you make OS specific changes, then the next step is reevaluating the function. The simplest way to perform this evaluation is to rerun the entire test suite. You may however, desire a less time consuming method for checking your changes. One shortcut is temporarily shortening the list of functions listed in the ../ballista/callTable.all file to the functions impacted by your changes. Remember to move ../ballista/outfiles to a temporary name if you want to keep your previous results before rerunning ostest.pl.
Testing is complete: At this point you have:
- The statistics necessary to compare the tested POSIX Operating System with other Ballista tested POSIX Operating Systems. (Including previous versions of the same OS)
- The individual test results and generated code that detail and reproduce the robustness failures detected by the test suite. This information should facillitate generating bug reports and eliminating the robustness failures.
Compare your results: to reported results on other operating systems. If your results differ from those listed or you are testing an unrepresented OS please consider submiting your results.

The Ballista OS Robustness Test Suite is distributed under the terms of the GNU General Public License

Update: We have updated the released version to be one that compiled and executed properly under Fedora Core 2 (although that isn't all that new now either). Unfortunately, this project has long since run out of funding. You are welcome to send email ballista@ece.cmu.edu if you encounter difficulties, but in most cases we won't have resources to address significant issues that might arise. It is expected that over time various parts of the tool chain we rely upon will evolve to the point that the code doesn't compiler or execute properly any more. If you want to use this code, you should expect to spend some time porting it to your current tool chain. Thanks for your understanding.