Ballista^® OS Robustness Test Suite - Templates

The effectiveness of robustness testing depends on having appropriate testing objects for each argument to a MuT. While testing can almost always be accomplished with basic data types such as pointer and integer, thorough testing requires some design effort and the definition of testing objects tailored to a particular system. This webpage discusses how to create customized testing objects using template files and how to use these custom testing objects when performing Ballista^® testing.

Template File Overview

A template file (a ".tpl" file) defines the test values for a testing object. It uses a special syntax described in following sections of this webpage to organize the snippets of C++ code that constitute various parts of a test value (the constructor, the commitor, and the destructor). Each template file corresponds to a single testing object. Testing objects are also referred to as Ballista datatypes.

Note:

The format of template files is strict and the compiler can be unforgiving. Often folks refer to existing templates files when creating a new Ballista datatype.

The Ballista testing client software includes .tpl files for commonly used data types in the templates/ subdirectory. This chapter describes how to create new template files that, in most cases, inherit test values from the pre-supplied template files. Users are encouraged to study the predefined template files for examples to supplement the information in this user manual.

Note:

The name of a Ballista testing object must be exactly the same as the filename of the template file (without the .tpl extension). Both the testing object name and file name are case sensitive.

Note:

The standard location for template files is in the /templates subdirectory. This location is automatically supported by the Ballista Test Suite and is the recommended directory for new template files.

name C_type B_type;
parent t_object;
includes [ {  includes_list } ]
global_defines [ {  defines_list  } ]
dials[  dial_defn; … ]
access[ dial_posn { dial_code; }  … ]
commit[ dial_posn { dial_code; }  … ]
cleanup[ dial_posn { dial_code; }  … ]

All sections are required to be in a template file. However, some sections may have null contents. The various sections of the template will be described in turn.

1. Testing object name

   The first non-commented entry in a template file must be the testing object name.

   Syntax:
   

   name C_type B_type;

   C_type	is the C++ data type of the testing object, corresponding to the type the testing object must be cast to for C++ compilation purposes.
   B_type	is the name of the testing object. This must exactly correspond to the filename of the testing object (but without the .tpl extension in the file name).

   Note:

   Neither C_type nor B_type may have any embedded whitespace. For example, "void*" should be used instead of "void *". Occasionally, there are cases where white space must be embedded, such as "unsigned int". (see Template Type with White Space for details on handling this unusual case.)

   For example:

   name int bInt;

   This example creates a testing object named bInt (which must be in file bInt.tpl). It is permissible to use bInt for any argument of C++ type int. However, there may be other template files that actually represent the argument more completely.

   Note:

   The template filename must have this same name information for proper compilation.




2. Parent name

   The second entry in a template file must be the name of the parent testing object, or "paramAccess" (the root of the inheritance tree) if there is no parent object.

   Syntax:
   

   parent t_object;

   t_object

   is the name of the testing object. This must exactly correspond to the filename of some other testing object (but without the .tpl extension in the file name).

   For example:

   Parent paramAccess;

   Note:

   All the test values defined for the parent and the parent's ancestry will be inherited for this template. Therefore, if a test value such as NULL is defined in the inheritance tree then it should not be duplicated in the this template.




3. Includes list

   The includes section contains C++ code that #includes any header files necessary to compile all code in the testing object. This section is completely independent of the include section of callTable entries. This eliminates the need for users of a testing object to have to know what include files are necessary for any particular data type.

   Syntax:
   

   includes [ {
   includes_list
   } ]

   includes_list

   is C++ code pasted into the testing object. Normally this is a list of #include lines to be pasted in to the testing object, with one file name per line in the file.

   For example:

   Includes[ {
   #include “values.h” // comment goes here
   #include “bTypes.h”
   }]

   Note:

   The includes section must contain the line #include "bTypes.h" if the parent listed is paramAccess otherwise include the header file associated with the parent (for example #include "b_float.h") along with any other necessary headers.




4. Global defines

   The global section contains C++ code which defines any global variables, typedefs, compilation variables, supporting subroutines, and other similar code. When a single test case program is generated by the web server, this section is pasted into the resulting code regardless of which test case is used. A particularly important use for this section is to retain information about resources allocated by a constructor (for example, a memory pointer) so that the corresponding test case destructor can release those resources cleanly.

   Syntax:
   

   global_defines [ {
   defines_list
   } ]

   defines_list

   is C++ code that is pasted into the testing object before the routine having test case code.

   For example:

   global_defines [ {
   #define filename “btestfile”
   int* allocated_pointer;
   } ]

   Note:

   Remember to include this section even if nothing is defined within the [ ].




5. Dial definitions

   Fine-grain testing is supported by the “dials” concept. Each testing object can be thought of as a composite of different, orthogonal “dial” settings, analogous to a row of dials on a control panel. While the high level description of Ballista states that combinations of test values are tested, in fact there is another level of combinations below that level. What really happens is that combinations of possible dial values are tested (with each testing object having one or more dials associated with it).

   As an example of using dials, consider a file descriptor testing object. In the absence of dials, each test value would have to make several different decisions when setting up a file for test (e.g., file name, file open/closed status, file permissions). In the absence of further decomposition, this leads to a potentially huge number of possible test values that must be written to achieve thorough testing coverage. Instead, in this case dials could be used to decompose the different aspects of a file descriptor to let the Ballista testing system select combinations of dial settings automatically (e.g., one dial for different legitimate or illegal file names, one dial for file open/closed status, one dial for file permissions).

   Every Ballista testing object must have one or more dials declared. Dials are categorical.Care must be taken in selecting the names and position-names of dials to avoid conflicts with C++ keywords and other compilation names. Keep in mind that template files are preprocessed, but any names selected are eventually fed through a C compiler.

   Syntax:
   

   dials[ dial_type dial_name : position_name, … ; … ]

   dial_type	Specifies what type of dial is desired. Permissible values are: enum_dial (categorial) int_dial (obsolete)
   dial_name	The name of the dial, which gets compiled into a C++ variable name. All dial names must be distinct. There must be at least one dial_name in the dials clause.
   position_name	A comma-separated list of one or more dial positions. Each dial-position is compiled into a C++ procedure name. Exactly one dial position from every dial_name is selected to form a single test case. A semicolon terminates the list of position_names and permits the start of a new dial definition beginning with a dial_type.

   Note:

   Although there is no limit to the number of dials you can declare or to the number of settings included in each dial, you must declare at least one dial and at least one setting per dial.

   Note:

   There must be a space on both sides of the colon or a syntax error will result. So, "HVAL : ONE" is valid, but "HVAL: ONE" would be invalid.

   Note:

   All dial_name and position_name symbols must be unique within a testing object to avoid compilation problems. Furthermore, they should not conflict with other symbols defined in C++. Therefore, for example, avoid the use of NULL as a position_name and int as a dial_name.

   For example:

   dials
   [
   enum_dial MODE : READ, WRITE, APPEND, READ_PLUS, WRITE_PLUS, APPEND_PLUS;
   enum_dial PERMISSIONS : R_USER, W_USER, X_USER, RWX_USER, R_GRP, NONE;
   enum_dial EXISTANCE : EXIST, CLOSED, DELETED;
   enum_dial STATE : EMPTY, BEGINNING, EOF, PAST_EOF;
   ]

   This example is from the file template, b_ptr_file.tpl. It creates 4 dials for the 4 orthogonal properties being tested for file.



6. Access/Constructor definitions

   The access clause defines the constructor associated with each position for each dial. The special variable “_theVariable” is used to hold results of the construction operation, and is the value fed to the function under test during testing. All code between the "{" and the "}" is fed to the C++ compiler unaltered.

   Syntax:
   

   access[
   { init_code }
   dial_posn
   { posn_code }
   …
   dial_posn
   { posn_code }
   ]

   init_code	This is code executed /compiled before every posn_code. It is commonly used to declare variables used by various constructors, but can be any C++ code.
   posn_code	This is C++ code that accomplishes the constructor phase of data value creation. Code for positions of all dials is mingled in a single clause. Code segments are executed in the order that dials were defined, regardless of the order in which they appear in the access clause.

   For example:

   access
   [
   {
      //this section will be associated with all values 
      _theVariable = 0;  
   }
      F_OK, FRWX_OK  
      {
         _theVariable |= F_OK;
      }
      R_OK
      {
         _theVariable |= R_OK;
      }
      RWX_OK
      {
         _theVariable |= R_OK | W_OK | X_OK;
      }
   ]
   
   

   This example is from b_int_amode.tpl and shows the use of common section where _theVariable is initialized to 0. Of additional notice is that the dial settings F_OK and FRWX_OK share the same posn_code block.

   It is the user’s responsibility to ensure that the special variable "_theVariable" is set appropriately by the time that code for all dial positions has been executed. Note that only a single value of _theVariable is provided for each Ballista datatype to supply exactly one value for calling the function under test per Ballista datatype. The type of _theVariable is the C++ type of the testing object name in the "name" clause at the beginning of the template file.



7. Commit definitions

   The commit clause is identical in format and operation to the access clause previously described (except it starts with "commit" instead of "access"). The only difference is that all access clauses are executed before any commit clause executes. This provides a two-phase creation cycle that is useful for preventing unintentional recycling of allocated and de-allocated resources. The commit clause must be included, but may be empty.

   As an example of using the commit phase, consider a situation in which the function under test takes two memory buffers as arguments and a test value of "pointer to freed memory buffer". If the access code for that test value malloc’ed a memory buffer and then free’d it, it is possible that consecutive executions of that test value for two different memory buffers would point to the same memory location, which may not be the desired result. Instead, the access phase should perform the malloc, and the commit phase should perform the free.

   Syntax:
   

   commit[
   { init_code }
   dial_posn
   { posn_code }
   …
   dial_posn
   { posn_code }
   ]

   Note:

   This is the same format specification as ACCESS except for the replacement of the keyword "access" with "commit".

   For example:

   commit
   [
    FREED
       {
          free(ptrRef);
       }
   ]
   
   

   This example is from b_ptr_void.tpl.

   Commit clauses may query the state of _theVariable and global variables defined in the access clause.

   Note:

   Reserve using the commit section for special cases in which the second cycle of processing is beneficial.




8. Cleanup/destructor definitions

   The cleanup clause is identical in syntax to the access and commit clauses (except that it starts with "cleanup" instead of "access" or "commit"). Cleanup clauses are executed after the function under test is executed (except in the case of a robustness failure, in which case the testing task is terminated before the cleanup constructor can be executed).

   Cleanup clauses may use the value of _theVariable set by the access or commit phases, and additionally may use any global variables declared and set by the access or commit phases.

   In general, the cleanup clause should attempt to release resources acquired by the access or commit phases, checking to be sure that the resources were not already released as a side-effect of testing the function.

   Syntax:
   

   cleanup[
   { init_code }
   dial_posn
   { posn_code }
   …
   dial_posn
   { posn_code }
   ]

   Note:

   This is the same format specification as ACCESS except for the replacement of the keyword "access" with "cleanup".

   For example:

   cleanup
   [
    DYNAMIC
       {
          if (cp_ptrRef)
          {
             free(cp_ptrRef);
             ptrRef = NULL;
          }
       }
   ]
   

   This example is from b_ptr_void.tpl.

• simple example
• complex example

• b_ptr_file.tpl
• b_ptr_dir.tpl