Ballista OS Robustness Test Suite - Download & Execute

  1. If you have not yet obtained the os test suite, you need to download the os test suite. Please understand that
    " By downloading this software the user agrees to accept this software entirely as-is, with absolutely no warranty whatsoever, expressed or implied. The responsibility for ensuring fitness and correctness for any purpose lies entirely with the user."
    and click on I Agree/Download to continue.
    Download

  2. Please untar and unzip ostest.tar.gz after downloading it. It will create its own directory. For those of you who are unfamiliar with a the unix archiving utilities tar and gzip, you may use the following command to do this:

    gunzip -c ostest.tar.gz |tar -xvf - 
  3. You will notice that a directory ballista was created as part of the untar. Change to the Ballista directory.

    cd ballista 
  4. Execute the tests:

    perl ostest.pl 
  5. Generate stand alone code: Once the tests have been run there will be the option to create stand alone C++ code to reproduce a detected robustness failure. The details for generating code are outlined in the previously mentioned file ../ballista/outfiles/os_results.html. This step is optional but can be useful for bug reports.

  6. Verify Testing: After the Ballista OS Robustness Test Suite completes then you should investigate the associated problems (if any exist). If the Test Suite encounters problems testing a function, it will log the function in .../ballista/outfiles/faillog, display an appropriate message, and skip testing the associated function. Generally these functions will be skipped because:

    If after investigating the reason the test suite skipped a function, you make OS specific changes, then the next step is reevaluating the function. The simplest way to perform this evaluation is to rerun the entire test suite. You may however, desire a less time consuming method for checking your changes. One shortcut is temporarily shortening the list of functions listed in the ../ballista/callTable.all file to the functions impacted by your changes. Remember to move ../ballista/outfiles to a temporary name if you want to keep your previous results before rerunning ostest.pl.

  7. Testing is complete: At this point you have:

    • The statistics necessary to compare the tested POSIX Operating System with other Ballista tested POSIX Operating Systems. (Including previous versions of the same OS)
    • The individual test results and generated code that detail and reproduce the robustness failures detected by the test suite. This information should facillitate generating bug reports and eliminating the robustness failures.

  8. Compare your results: to reported results on other operating systems. If your results differ from those listed or you are testing an unrepresented OS please consider submiting your results.


The Ballista OS Robustness Test Suite is distributed under the terms of the GNU General Public License


Update: We have updated the released version to be one that compiled and executed properly under Fedora Core 2 (although that isn't all that new now either). Unfortunately, this project has long since run out of funding. You are welcome to send email ballista@ece.cmu.edu if you encounter difficulties, but in most cases we won't have resources to address significant issues that might arise. It is expected that over time various parts of the tool chain we rely upon will evolve to the point that the code doesn't compiler or execute properly any more. If you want to use this code, you should expect to spend some time porting it to your current tool chain. Thanks for your understanding.