Ballista^® OS Robustness Test Suite:
Expanding to Non-POSIX Functions

The Ballista test suite can be used to test Non-POSIX functions after some modifications. The nature and degree of the modifications vary but are normally relatively manageable. The following steps are guidelines which outline the possible changes you will need to make.

• Ensure that the Ballista test suite runs correctly under your system's configuration for the POSIX functions.
  (If you run into a snag we promise this step will save you lots of effort.)




• Add your function(s) to the list of functions to be tested. The test suite uses a single master file called callTable.all or callTable<system>.all to register each file to be tested. (where system corresponds with your operating system i.e. DUNIX)

  The callTable.all or callTable<system>.all files have a strict format. Each line in the file corresponds with a single function entry. The function entry consists of fields:

  include_file	Exactly one file name of a system header file (.h) to include in the compilation. Do not include the angle brackets as they are implied. If no system .h file is required, use "stdlib.h" as a placeholder. User-supplied .h files are specified in the userIncludes.h file. Occasionally, system files are in sub-directories instead of main directory for system header files. To accomodate these alternately located files prefix the header file name with the sub-directory. (for example sys/stat.h)
  call_type	There are only two options for this field. Either the keyword "function" for calling a c function or the keyword "constructor" for calling a C++ constructor. The only difference in processing the different call_types is that "constructor" inserts the keyword "new" before the function name in the generated code.
  return_type	C/C++ return type of the function. This field cannot contain any whitespace. For example please use "void*" instead of "void *".
  function_name	The name of the function to be tested.
  [+]b_arg	A space-separated list of one or more Ballista datatypes corresponding, in order, to the arguments of the function. Any datatype preceded with a "+" is a "phantom" parameter. Phantom parameters are used for variable construction/destruction purposes and do not correspond with a parameter from the function's parameter list. For more on phantom parameters. Note: In addition to the functional entry lines, the callTable.all/callTable<system>.all file can include comments, which are lines that begin with the "#" character. Note: The current implementation of Ballista does not support embedded phantom parameters. Currently all parameters that follow the leftmost "+" are processed as phantom parameters. Note: Ballista is not tolerant of spelling mistakes in the callTable.all/callTable<system>.all file. Spelling mistakes do not necessarily produce error codes. Please double-check the spelling of functional entry lines if you have problems.

• Adding a new datatype to the test suite. This procedure can be a bit tricky. Ballista datatypes are defined in files called template files. Template files are stored in /templates subdirectory and have the .tpl extension. The format for a template file is strict and the template compiler is unforgiving. First define a new datatype and then modify the callTable.all entry for the functions you wish to reference the new datatype. When run, the test suite will automatically process the new template. Creating a simplified callTable.all with just a single entry referencing the new template can be helpful if you run into difficulties with compilation of the template.




• Adding pre or post processing to the test suite. Occasionally there are situations where a state must be reached before any realistic testing can occur or a state should be broken down after testing occurs.
  There are two ways to do this:

  userSetup and userShutdown

  is the simplist method. This method creates static scaffolding code that is run before and after each test case. The same state is reached with each test case using this method.

  phantom parameters

  is a bit more complex but is also more versatile. This method treats the setup/tear down code as variable that can be randomly varied as a parameter to the test case. Phantom parameters are defined in the same manner as ballista datatypes. The definition of the Ballista datatypes in this case should also set/tear down the state. For example, when testing the POSIX function rand() then it might make sense to use a phantom parameter that sets the seed. In order to do this you would need to define a Ballista datatype that varied the seed values, and reference this new datatype in the rand entry in callTable.all. The new Ballista datatype should define different dial settings, and call srand with the setting.




• Adding additonal includes to the Ballista test suite to support your function. Addition of include files requires modification of a file in the /ballista directory called userIncludes.h.

  The userIncludes.h file contains a list of header files (“.h” files) and potentially other information needed to compile the function under test. The userIncludes.h file is treated as being entirely C++ code, so comments can be inserted with “/* … */” and “//” as in any C++ program. The userIncludes.h file applies globally to all functions in the callTable.

  The Ballista test suite automatically processes anything placed in the file userIncludes.h in the /ballista directory.




• Adding additional libraries to the Ballista test suite to support compilation of your function. Addition of libraries requires modification of the file userLibs located in the /ballista directory.

  The userLibs file contains a listing of object files to be included when compiling, with one filename per line in the the file. The userLibs file can contain comments, indicated by a line starting with ‘#’.

  The Ballista test suite automatically processes anything placed in the file userLibs in the /ballista directory.