18649 - Testing Requirements

*Please submit all project-related questions to {email}

-- thanks!

This document describes the procedure and requirements for unit, integration, and acceptance testings. Over the course of the project, you will use these testing requirements to test your entire elevator. Be sure to read this document carefully.

Introduction: A Note on the Usefulness of Testing

Testing is an important, but often neglected, part of the design process. It's not as fun as design or writing code, so it is even more important that you start early and make sure you allocate enough time to be thorough. Like the rest of the process, its usefulness is directly related to the amount of effort you put in to writing your tests. If you write bad tests, and the lesson you should take away from that experience is "it is important to write good tests", NOT "testing is a waste of time".

If you patch together a unit or integration test file that will "probably work," and then run the test, see where it fails, and adjust the test accordingly, then you probably won't find any bugs in your tests because you've tuned your test to match your design. In the real world, it's common for testing to be done by a separate team for just this reason. Although you do not have the luxury of having a separate team for testing, you can certainly have someone other than the designer write the tests. Write unit tests while only looking at statecharts, and integration tests while only looking at sequence diagrams. Be as thorough as you can. If you write good tests, you'll catch bugs and be able to refine the design early on. If you just go through the motions of unit and integration tests, and you may still be able to grind out the major bugs by debugging acceptance tests, but it will take a long time, and there is a good chance you'll overlook some subtle bugs.

Simulator Testing Framework

Unit and Integration Testing Framework

The simulator can be configured to take a plain text file describing messages (and physical inputs) to be sent and actually sending them out at specific times. It can also be configured with assertions to check the value of physical state and network messages at specified times. You can use the -cf flag to specify the configuration (which object is instantiated for the unit test), and the -mf flag to specify the message inputs for the unit test.

The message input file format is described in the simulator command line reference (run the simulator with no parameters), but there are five basic line types:

Input network message
Input framework message
Assertion on a network message value
Assertion on a framework value
Assertion on the controller's internal state

Because of jitter in the network and the way the assertions are checked, you must wait longer than two message period to be sure that the assertion has the updated value. That is to say, if the verbose output of your car call controller indicates that it has set the output to "true" at time 3.0s and the message (or framework value) is being updated at 100ms, you are not guaranteed that the assertion will read the update value until after 3.2s (e.g. 3.21 s).

The assertion format only allows you to check the value of one field. If you wish to check multiple values, you can create multiple assertions.

The message injector file has a simple macro syntax: #DEFINE <macro> <value>. There is also an include directive: #INCLUDE <filename>. These are explained in more detail in the command line documentation. You can use the -pd option to print a list of #DEFINE statements for all network and framework messages. We STRONGLY advise you to use this list in an include file, since your CAN ids will change throughout the semester, and it will make it much easier to keep your tests up to date.

You can refer back to the test files in the testlight example to see a working example of message injection and assertion and controller state assertions. The example is a unit test, but this framework can also be used for integration testing by using injections for arcs that come from system objects (these are test inputs) and assertions to check the arcs that come from control objects (these are test outputs).

Acceptance Testing Framework

Acceptance testing consists of instantiating the entire elevator, including the network and framework message architecture, all system objects and smart sensors, and all distributed controllers. This is handled automatically by the simulator. Once the elevator is instantiated, passenger objects interact with the physical interface by pressing buttons, passing through doors (sometimes triggering reversals), and adding weight to the elevator.

Acceptance tests can be started using the -pf flag. Acceptance test files consist of a list of passengers. Each passenger is defined by the landing they enter the system at and their destination. Read the simulator command line reference for more information.

Unit Testing

If you have not already done so, go read the Introduction and Simulator Testing Framework sections at the beginning of this document.

Unit test for this project means testing the behavior of a single system object such as a door controller or the dispatcher. These tests are derived from the system design, and shall completely cover all states and transitions in your state diagrams to ensure that code that is written actually implements the intended design.

You are responsible for implementing and testing the objects you've designed. You are not responsible for testing environmental objects like the Safety object or the smart sensors. You can assume that we will test these objects.. By the end of the project, you will design tests for all seven of your controllers.

Summary of Unit Testing:

Writing Unit Tests
Traceability for Unit Tests
Peer Review
Unit Test Log
Unit Test Summary File

Writing Unit Tests

The idea with unit tests is to generate a set of files that are sufficient to completely test a single controller implementation. We are going to leave the details of how to accomplish things to you, but to be correct a test shall address the following points:

Each set of test files shall
- Be associated with one and only one control object.
- Include in the comments the name of module it is supposed to be testing, your group number and member names
- Include comments that describe the states and arcs within the module that are tested and indicate where the transitions occur in the test (this is an extension of traceability -- now you can trace from requirements through design and into unit test).
- Use assertions to test all the outputs of the control object in each state. For each state, there should be an assertion to check each and every output defined in the output interface of the object being tested.
- Have assertions formulated in a positive sense -- that is, each assertion indicates "PASSED" when the correct behavior is observed.
- Be located in the unit_test subdirectory of the portfolio and may NOT be placed in a subdirectory of that folder or in any other location.

A complete unit test shall
- Include all states and execute all transitions in your state diagrams to ensure that code that is written actually implements the intended design
- - When a guard condition on a transition consists of two or more terms ORed together, you must separately test each of the ORed terms.
- Test the module in such a way that every line of code that is reachable from the timerExpired control loop method is executed at least once.
- Execute correctly when no other control objects are present in the system (e.g., DoorControl unit test works when Dispatcher, DriveControl, LanternControl, HallButtonControl, CarButtonControl, CarPositionControl are not instantiated)
- Contain a comment line that describes the message being sent if that message uses a generic translator (e.g. BooleanCanPayloadTranslator)
Additional Guidelines:
- Your test must completely cover the statechart (all states and states and transitions), but you need not cover every state and transition in every file.
- - The unit tests from the soda machine example contain several cases where the unit test is broken down across several message injector files.
  - For simple controllers (like the buttons), you can probably cover the whole statechart in one test.
  - For more complex controllers (like the DoorControl), one long test file may be too cumbersome, but you probably don't need more than three or four test files to thoroughly test any elevator module.
- Designing these tests may require some assumptions about timing. You should include in comments describing any timing-sensitive conditions that you are testing.

Traceability for Unit Tests

Unit tests must cover all states and transitions in the statechart of the object being tested. To accomplish this, you will add two specific types of traceability comments to your message injector (.mf) files. This is in addition to the comments in the header that describe the test.

Before every set of message injections that exercise a transition, you shall include a comment of the form:
;#transition 'T2.1'
where T2.1 is the number that corresponds to the transition being exercised.
Before every set of assertions that test the outputs of a state, you shall include a comment of the form:
;#state 'S2.1 IDLE'
where S2.1 Idle is the number and name of the state being tested.

Peer Reviews for Unit Tests

Note that it is possible (likely!) that your unit tests will have bugs. For that reason, someone else in the group (other than the author of the unit test) must review the unit test for accuracy. Each unit test must be reviewed. The review shall include the following checks:

Is each statechart arc exercised?
Is each statechart state exercised?
If there are multiple OR conditions on each branch, are they exercised?
Are all the inputs and outputs mentioned in the behavioral requirements exercised?
Do the tests exercise all code reachable from the timerExpired control loop?

For each test, add the checklist, the unit test author's name, and the name of the person who performed the review to the peer review section of the Unit Test Log in your portfolio.
Note: If a peer review results in changes to the unit tests, those tests shall be logged in the issue log.

Unit Test Log

Instead of simply running tests, finding bugs and fixing them, we will ask you to create a test log. This log will allow you and us to quickly see what tests you have run, their outcome and any changes you had to make. A test log serves as a simple tool to measure the coverage of your tests and to highlight areas of your design where there were problems.

A template for the unit test log has been included in the portfolio template. Be sure that you include the following items:

Name of the object being tested
Name of (and link to) control file and message file for the test.
Which states and transitions of your state chart are tested.
Number of assertions passed and failed
If you failed the test, list the source of the failure: whether the test is buggy, the implementation is wrong, or the design is flawed. If the error is in the implementation, give the file name and line number, and then trace the source of the failure. This must also be recorded in your defect tracking.
A link to the simulator output of this test. The output must include at least the PASS/FAIL lines for each assertion and the summary of assertion statistics and it must include the random seed that ran your test to completion. When we grade your tests, we will use other random seeds, but we will verify that the seed you give also runs the test to completion. If your test is failing, including the seed will aid the other engineers of your team in reproducing the bug.

Note: For the unit tests that you have written but are not required to execute, you may put "N/A" in the boxes for test results.

Unit Test Summary File

In order to facilitate automated testing, we require a machine-readable text input format that summarizes your unit tests. The requirements for the test summary file are as follows. Note that this requirement is in addition to the HTML summary of your tests in the Unit Test Log of the portfolio.

For every project submission, the test summary file shall be up-to-date and contain a reference to every unit test listed in the Unit Test Log.
The unit test summary file shall be called unit_tests.txt.
It shall be located in the unit_test folder of your project portfolio.
This file shall contain a line for each unit test you wrote. The line shall contain three text fields with the following format:

<object tested> <configuration file (.cf)> <message injector file (.mf)>

Note that this requirement precludes the use of filenames with spaces.

The portfolio template contains a placeholder file with the correct name and location and an example entry.

You can use the test_verify.sh script to check your summary file. The syntax is
./test_verify.sh <summary file name>
This script must be run from the unit_test/ directory of your portfolio.

Integration Tests / Sequence Diagram Tests

If you have not already done so, go read the Introduction and Simulator Testing Framework sections at the beginning of this document.

Throughout the project writeup, we will use the terms Integration Test and Sequence Diagram Test interchangeably. With respect to the simulator framework, integration tests work much the same way as unit tests. For each sequence diagram, you will generate a .cf file with a list of controllers for that test. You will also generate a .mf file which defines messages to be injected and assertion to be checked. Consult the command line documentation and the discussion in the unit testing section above for more details.

Summary of Integration Testing:

Writing Integration Tests
Traceability
Peer Reviews
Test Log
Test Summary File

Writing Integration Tests

The idea with integration tests is to generate a test that is sufficient to completely test one sequence diagram. Although you may only instantiate one controller (depending on which sequence diagram you choose), this is a form of integration testing because it tests the interaction between various objects in the system. We're going to leave the details of how to accomplish things to you, but to be correct a test must somehow address all the below points:

The test files must include the name of sequence diagram it is supposed to be testing.
The test must instantiate all the controller objects that are included in the sequence diagram. This means any of the seven controllers that you have implemented (e.g. DoorControl, DriveControl, etc.). The test may not instantiate any other objects, including environmental objects and smart sensors.
The test must inject a message for each arc in the sequence diagram that goes to any of the instantiated objects (this corresponds to a message sent to the object).
The test may inject additional messages at the beginning of the test to manipulate the controllers into the state described in the preconditions. These messages should be clearly marked (using comments) as corresponding to the preconditions.
After each message (or set of messages) required to trigger a message output from a controller, include a comment that describes what you expect to observe happening, if anything, that constitutes a "pass" of the test. Or in some cases it will be easier to say what constitutes a "failure". Include one or more assertions to check the outputs you observe. Note that assertions can only check the physical state and network messages, they cannot check the internal state of the controllers.
Message injections and assertions that use generic translators (e.g. BooleanCanPayloadTranslator) must be accompanied by a comment that describes the message that is being sent.
The test files shall be located in the integration_test subdirectory of the portfolio and may NOT be placed in a subdirectory of that folder or in any other location.

Traceability for Integration Tests

Each network or framework message injection corresponds to an arc originating from a system object. For arcs that originate from the controller objects, each arc is tested with an assertion. Each of these injections or assertions shall be preceded by a comment of the form:

;#arc '1A/1b'

Where 1A refers to the sequence diagram being tested and 1b is the number of the arc being tested or injected.

Peer Reviews for Integration Tests

For each sequence diagram test, have someone other than the test author perform the following peer review on each test:

Does each sequence diagram arc that originates from an environmental sensor (such as a DoorClosed or AtFloor sensor) correspond to one or more injected messages in the message file (if more than one, they must all be of the same type, e.g. several DoorOpened messages)?
Does each injected message correspond to at least one sequence diagram arc that originates from an environmental sensor or to the preconditions?
Does each sequence diagram arc that originates from one of the seven control objects (such as DoorControl) correspond to one or more assertions in the message file?
Does each assertion correspond to at least one sequence diagram arc that originates from a control object OR a pre/post condition on a controller object?

For each test, add the checklist, the integration test author's name, and the name of the person who performed the review to the peer review section of the Integration Test Log in your portfolio.

Integration Test Log

Instead of simply running tests, finding bugs and fixing them, we will ask you to create a test log. This log will allow you and us to quickly see what tests you have run, their outcome and any changes you had to make. A test log serves as a simple tool to measure the coverage of your tests and to highlight areas of your design where there were problems.

A template for the integration test log has been included in the portfolio template. Be sure that you include the following items:

Sequence diagram being tested.
Name of (and link to) control file and message file for the test.
Number of passed and failed assertions.
If your test failed, list the source of the failure (the test file itself, the code, or the design): give the file name, line number, and trace the source of the failure. This must also be recorded in your defect tracking.
A link to the simulator output of this test. The output must include at least the PASS/FAIL lines for each assertion and the summary of assertion statistics and it must include the random seed that ran your test to completion. When we grade your tests, we will use different random seeds, but we will verify that the seed you give also runs the test to completion. If your test is failing, including the seed will aid the other engineers of your team in reproducing the bug.

Note: For the sequence diagram tests that you have written but are not required to execute, you may put "N/A" in the boxes for test results.

Integration Test Summary File

For every project submission, the test summary file shall be up-to-date and contain a reference to every unit test listed in the Integration Test Log.
The unit test summary file shall be called integration_tests.txt
It shall be located in the integration_test folder of your project portfolio.
This file shall contain a line for each integration test you wrote. The line shall contain three text fields with the following format:

<sequence diagram name> <configuration file (.cf)> <message injector file (.mf)>

Note that this requirement precludes the use of filenames with spaces.

Acceptance Tests

If you have not already done so, go read the Introduction and Simulator Testing Framework sections at the beginning of this document.

Acceptance testing involves instantiating the entire elevator control system and testing its operation with simulated passenger objects. Throughout the project, you will be provided with various acceptance tests. Occasionally, you will also be required to write some of your own acceptance tests.

In addition to the required tests, you are encouraged to make up your own tests to exercise your elevator system. Very simple (e.g. 1 passenger) acceptance tests that exercise a certain problem may be simpler to analyze and debug. Additional complex acceptance tests can help you identify additional problems with your implementation.

The acceptance test file (.pass file) defines the basic parameters for one or more passengers. The system acceptance test file format is:

<entry time> <start_floor> <start_hallway> <destination_floor> <destination_hallway> ; <comment_field>

The command line reference contains a more complete description of these fields in the description of the -pf flag. Tests with this input file are run with all modules in the system loaded. The entire system is initialized before the testing begins. An example input for the system integration test is:

15s 1 BACK 3 FRONT ; person A going up
15s 7 BACK 1 FRONT ; person B going down

Which means that at the same time, t = 15, two people show up, 1 person on the 1st floor back hallway wanting to go to the 3rd floor front hallway, and 1 person on the 7th floor back hallway wanting to go down to the 1st floor back hallway. Keep in mind that certain floors do not have certain hallways, and if you mistakenly assign people to hallways that do not exist, you will receive an error! Here is the hoistway setup. Keep this handy when performing your acceptance tests.

Floor #
8 FH
7 FH and BH
6 FH
5 FH
4 FH
3 FH
2 BH
1 FH and BH

There are no peer reviews for acceptance tests.

Because of the pseudorandom behavior of the simulator, you may find that some bugs occur in a test when run with one random seed, but not when run with another. For this reason, you should consider running tests multiple times with different random seeds.

Acceptance Test Log

Instead of simply running tests, finding bugs and fixing them, you will create an Acceptance Test Log. This log will allow you, your teammates, and the course staff to quickly see what tests you have run, their outcome and any changes you had to make as a result of testing. A test log serves as a simple tool to measure the coverage of your tests and to highlight areas of your design where there were problems.

All test files and output files must be included in the acceptance_test/ directory, and may NOT be placed in a subdirectory or any other location.

You shall save the results of the acceptance tests and include them in your project portfolio. A template for the acceptance test log has been included in the project portfolio. Be sure that you include the following items:

Name of (and link to) passenger file for the test. You should include a local copy of the test file with your portfolio (in the acceptance test folder).
Brief description of passenger load (distilled from .pass file), such as, "heavy up-peak traffic, more than 20 passengers in 2 minutes." This will allow you to see how your elevator performs under many situations.
Random seed used -- ultimately, you will want to run your acceptance tests with several different random seeds. Put a separate row in the table for each random seed/acceptance test combination you test.
Test outcome ("all passengers delivered," "some passenger not delivered," or "passengers delivered, unsatisfactory performance"). At this stage you can begin to tune your elevator for passenger workloads. If you are satisfied with your elevator performance now, know that you will have to improve it later!
Minutes and seconds required to deliver all passengers.
Performance score as computed by the simulator.
If you failed to deliver passengers: trace to the source of the problem, including as much detailed information as possible. You should also mention which entries in the issue log were created as a result of running the acceptance testing.
A link to the simulator output of this test. The output must include at least the .stats output file and it must include the random seed that ran your test to completion. When we grade your tests, we may use different random seeds, but we will verify that the seed you give also runs the test to completion. If your test is failing, including the seed will aid the other engineers of your team in reproducing the bug.

Back to Project 5