While the terms verification and validation are used interchangably in papers and texts, there are distinct differnences in their terminology. According to the IEEE Standard Glossary of Software Engineering Terminology, verification is defines as "The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase." Validation, on the other hand, is defined as "The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements." So verification simply demonstrates whether the output of a phase conforms to the input of a phase as opposed to showing that the output is actually correct. Verification will not detect errors resulting from incorrect input specification and these errors may propagate without detection through later stages in the development cycle. It is not enough to only depend on verification, so validation is necessary to check for problems with the specification and to demonstrate that the system is operational. Finally, certification is "A written guarantee that a system or compnent complies with its specified requirements and is acceptable for operational use."
There are many different verification techniques but they all basically fall into 2 major categories - dynamic testing and static testing.
There are also numerous validation techniques, including formal methods, fault injection, and dependability analysis. Validation usually takes place at the end of the development cycle, and looks at the complete system as opposed to verification, which focuses on smaller sub-systems.
Verification and validation can be performed by the same organization performing the design, development, and implementation but sometimes it is performed by an independent testing agency. This is called independent verification and validation (IV & V). These agencies usually need to be accredited by a higher organization, to be sure that their results are dependable. For example, in the United Kingdom, the National Measurement Accreditation Service has begun to accredit companies for testing computer software used in safety-critical systems. The first company was accredited in 1994. The testing methods approved includes a suite of in-house procedures including static and dynamic testing techniques. [Storey96]
Verification and validation is a very time consuming process as it consists of planning from the start, the development of test cases, the actual testing, and the analysis of the testing results. It is important that there are people specifically in charge of V & V that can work with the designers. Since exhaustive testing is not feasible for any complex system, an issue that occurs is how much testing is enough testing. Sure, the more testing the better but when do the cost and time of testing outweigh the advantages gained from testing. The amount of time and money spent on V & V will certainly vary from project to project. In many organizations, testing is done until either or both time and money runs out. Whether this method is effective or not, it is a technique used by many companies.
Verification and validation are part of the long certification process for any embedded system. There are different reasons why a product needs certification. Sometimes certification is required for legal reasons. For example, before an aircraft is allowed to fly, it must obtain a license. Being certified would also be important for commercial reasons like having a sales advantage. One of the main reasons for certification is to show competence in specific areas. Certification are usually carried out by government agencies or other organizations with a national standing.
Certification can be applied to either organizations or individuals, tools or methods, or systems or products. Certification of organizations aims at assuring that the organization achieves a certain level or proficiency and that they agree to certain standards or criterias. This however, is not applicable to all areas because while it is easy to measure the procedures of a company, it is much harder to measure the competence with which they are performed. So certification is usually applied to areas such as quality assurance and testing as opposed to design. Certification may also apply to individuals where workers must be certified in order to be a certain profession. This usually applies to workers such as doctors, lawyers, accountants, and civil engineers. Tools or methods may also be certified. For example, although DO-178B does not specifically define the tools that must be used, it does give certain requirements of tools used to gain certification. Finally, systems or products may also be certified. [Storey96] In certification, there is always the issue of whether artifacts or methodology be certified. This becomes an issue in the certification of products containing software. Because software testing is so difficult, certification must be based on the process of development and on the demonstrated performance. This is a case where the methodology (development process) is certified instead of the artifact (software).
Even though certification does not occur until the end of a system development cycle, the planning starts from the very beginning. Because certification is a complicated process between the developer and the regulatory agency, the certification liason between the parties must be established early on in the process. Next, the developer should submit a verification plan for approval by the regulatory agency. After the submission, discussion takes place between the developer and regulatory agency to resolve areas of misunderstanding and disagreement. Changes to the methods used have to be approved by the regulatory body to insure that certification will not be affected. Throughout the entire development life cycle of the product, documentation must be continually submitted to show that the certification plan is satisfied. The regulating authority will also hold a series of reviews to discuss the submitted material. At the end, if the terms of the certification plan have been satisfied, then a certificate or license is issued. [Storey96]
The safety case is an important document used to support certification. It contains a set of arguments supported by analytical and experimental evidence concerning the safety of a design. It is created early in the development cycle and is then expanded as issues important to safety comes up. In the safety case, the regulatory authority will look to see that all potential hazards have been identified, and that appropriate steps have been taken to deal with them. In addition, the safety case must also demonstrate that appropriate development methods have been adopted and that they have been performed correctly. Items that should be included in the safety case includes, but are not limited to the following: specification of safety requirements, results of hazard and risk analysis, verification and validation strategy, and results of all verification and validation activities. The CONTESSE Test Handbook, which is applicable in the United Kingdom, lists a number of items that should be included in a safety case. [Storey96]
A potential problem with certification is that manufacturers use it to avoid its legal and moral obligations. An important aspect of certification is that it does not prove that the system is correct. Certification only proves that a system has met certain standards set by the certifying agency. The standards show that a product has met certain guidelines, but it does not mean that the system is correct. Any problem with the system is ultimately the responsibility of the designer and manufacturer, not the certification agency.
In the United States, different government organizations are responsible for the certification of different products. For example, the FDA is in charge of the certification of medical devices and the FAA is in charge of the certification of aircraft. Specifically, the FAA software certification is based on the standard RTCA/DO-178B. The standard provides information about all aspects of the software certification process including the following sections: software planning process, software development process, software verification process, and the certificate liason process. The software verification process includes more than testing, since testing in general cannot show the absence of errors. Therefore, the software verification process is usually a combination of review, analyses, and testing. Review and analyses are performed on the following different components. [RTCA92]
In addition, there is also a section in the standard about alternative methods. This section includes information that were not in the previous section because of immaturity at time of print. Some alternative verification methods include the use of formal methods and exhaustive input testing. [RTCA92] Research has been performed on formal methods and the certification of critical systems. There are two reasons why formal methods might be used to support certification. The first reason is to use formal methods for reasons other than improved quality control and assurance. This can be achieved by three ways: to supplement traditional processes and documentation, to substitute formal specifications for some traditional documentation, and to substitute formal proofs for some traditional reviews and analyses. The second reason is to use formal methods to improve quality control and assurance. [Rushby93] However, it is not always possible to formally prove all pieces of software. Another alternative method of testing is exhaustive input testing. This method has limitations too, as it is only feasible if the software component is simple and isolated.
The certification process is greatly assisted by and sometimes requires the use of guidelines and standards. Some documents are specific to a particular industry while others are generic. Several standards will be briefly mentioned below. [Storey96]
Table 1: Use of testing methods throughout the development life cycle [Storey96]
Static | Dynamic |
Requirements analysis and functional specification | |
walkthroughs | |
design reviews | |
checklists | |
Top-level design | |
walkthroughs | |
design reviews | |
checklists | |
formal proofs | |
fagan inspection | |
Detailed design | |
walkthroughs | |
design reviews | |
control flow analysis | |
data flow analysis | |
symbolic execution | |
checklists | |
fagan inspection | |
metrics | |
Implementation | |
static analysis | functional testing |
boundary value analysis | |
structured-based testing | |
probabilistic testing | |
error guessing | |
process simulation | |
error seeding | |
Integration testing | |
walkthroughs | functional testing |
design reviews | time and memory tests |
sneak circuit analysis | boundary value analysis |
performance testing | |
stress testing | |
probabilistic testing | |
error guessing | |
Validation | |
functional testing |
There are many organizations and companies that perform independent verification and validation. For example, NASA has a Software Independent Verification & Validation Facility which provides IV & V technical analyses for NASA programs, industries, and other government agencies. [NASA99]
With all the work dealing with the Y2K problem, many Y2K companies have embraced a new role as IV & V companies. One such example is SEEC. SEEC has an IV & V Workbench which is a comprehensive, integrated solution that includes year 2000 remediation. The workbench brings together powerful tool and processes such as the SEEC COBOL Analyst 2000 and SEEC Smart Change 2000 for verification and the SEEC COBOL Slicer and SEEC/TestDirector for validation. [SEEC99]
There are also several issues concerning the certification process. The first issue is should artifacts be certified or the methodology certified? The advantage of certifying the methodology is that it is applicable to different products. So if the same methodology is applied to different products, then each product does not need to be re-certified. The advantage of certifying the artifact is that if the methodology used to develop the artifact changes, the product may not have to be re-certified.
In addition, certification does not prove correctness. If a product receives certification, it simply means that it has met all the requirements needed to be met for certification. It does not mean that the product is error free. Therefore, the manufacturer cannot use certification to avoid assuming it's legal or moral obligations.
While verification, validation, and certification are important in the development of any system, they are even more important in the development of safety-critical embedded systems. Tests such as those for electromagnetic compatibility prevent electronic systems from harmful interference with its surroundings, which may include humans. Certification by the FCC also guarantees that products meet certain safety limits.
Future work in this area includes the standardization of certification methods used in different industries. Currently, these methods vary considerably. Therefore, not only does this situation limit the exchange of information between different industries, but it also limits the full use of the available human resources. The design of IEC 1508 has helped industries to not only maintain a common approach to safety but also the ability to still produce their own standards. The use of formal methods in software certification is also a relatively new area and debate is still occuring as to whether formal methods can accurately verify and validate safety-critical embedded systems.
- Andriole, Stephen J., editor, Software Validation, Verification, Testing, and Documentation, Princeton, NJ: Petrocelli Books, 1986.
This book presents an overview of the software verification and validation process including the planning stage, testing stage, and documentation stage.
- Neumann, B. de, editor, Software Certification, London, New York: Elsevier Applied Science, 1986.
This book contains a collection of papers dealing with the certification of computer software.