Philip Koopman, Carnegie Mellon University
The SAE J3016:2021 standard defines terminology for automated vehicles including the famous SAE Automation Levels. It is widely referenced in discussions, other standards, and even government regulations. Unfortunately, what is said about J3016 is too often inaccurate, misleading, or just plain incorrect.
Misinterpreting the SAE Levels can lead to misunderstandings about what the standard actually says, the technology incorporated into a car, and a driver's expectations. It's important to get statements in standards and regulations right. Moreover, it's important when referring to J3016 to understand that it says what it says, not what some author might want it to say, what might seem optimal for safety, or what other documents state that it says. (While this might seem obvious, perpetuation of misunderstandings is rampant.)
The scope of this article is to help readers understand what is in J3016, and what is not. In particular, it serves to debunk common myths and misconceptions about the contents of J3016. Even with an attempt to keep things brief, this article is complicated, simply because J3016 is a complicated standard.
This article is not actually the standard itself, so it can't be perfect in every respect. However, it is intended to be useful orientation for anyone trying to interpret the standard. Anyone who is going to refer to the SAE Levels in any writing should actually read the standard, using this article as a guide so you know what to look for. You can get J3016:2021 for free(SAE membership may be required):
Beyond the Levels, SAE J3016 defines other related terminology. That terminology should be used when possible to help reduce terminology fragmentation, but is not the primary focus of this article.
It is important to note in this article that numerous volunteers spent very significant amounts of time creating the versions of SAE J3016 and in particular tried to make what that standard says precise. Their efforts are highly appreciated.
Resources:
We've all seen overviews of the SAE Levels. Unfortunately, many of them are incorrect for reasons both subtle and obvious - after you've read the actual standard. Here is a summary that (hopefully) is entirely correct, although necessarily missing some of the nuance of the 41-page full standard. (This table is inspired by J3016 Table 1.)
"Other safety" is not defined by J3016, but refers in general to operational safety considerations outside the scope of J3016 such as ensuring that children are properly secured before the vehicle moves and whether a trip route or destination is expected to be acceptably safe during trip planning. Note that for Level 3 the "driver" is more precisely called the "fallback-ready user" in J3016, who becomes the driver when performing the fallback task.
Key terminology definitions below are paraphrased for brevity. The actual J3016 standard text should be used when complete accuracy rather than intuitional understanding is required.
J3016 Table 1 contains a precise explanation of the levels, although additional background from the rest of the standard is needed to interpret it correctly. If you need to get things right, start with that table.
Restating the dense language in J3016 Table 1 for a broader audience:
A shocking amount of descriptive text written about SAE J3016 in social media, news outlets, and even material that could lead to regulations, is meaningfully incomplete, misleading, or just plain wrong. Here are some of the more common myths debunked. If you're writing something about the SAE Levels please use this as a checklist to avoid being called out on social media for getting it wrong or, even worse, being accused of #autonowashing.
An Operational Design Domain is more than just the geographic area of operation of a vehicle. It includes all the factors that must be true of the environment for the ADS to function properly. That potentially includes lighting, precipitation, road stripe paint condition, whether roads are icy, and so on. While it is likely that early vehicles will have geographic or route restrictions as a matter of practicality (especially if high definition maps are needed), that is only one possible limitation inherent in an ODD. (J3016 3.21, 6., Figure 11, Figure 12)
J3016 is not a technical specification for what should be built. It simply describes something that could be built. Moreover, something that does meet a defined Level might be entirely reasonable to build. And something that might precisely meet J3016 is not guaranteed to be acceptably safe. (J3016 8.1) In particular, Levels 2 and 3 present significant safety issues if implemented as defined with nothing extra, because they put humans in charge of supervising automation without also requiring effective driver monitoring. Additionally, Level 2 does not enforce ODD restrictions, resulting in likely automation misuse or abuse. Legislators and regulators that say they are using an "SAE Safety Standard" to check the safety box for new regulations via incorporating J3016 are making misleading statements.
Higher level numbers have more automation, but they might not be safer or otherwise "better," especially in the near term. (J3016 8.3)
Example: A Level 2-engaged vehicle with a poorly performing driver monitoring system might have a higher crash rate than a Level 1-engaged vehicle due to driver automation complacency resulting in poor performance of the driver supervision task.
Each level is a different type of vehicle. The tradeoffs for Levels 1 & 2 are quite different than for Levels 4 & 5 due to the different role of the human driver. Improved performance within a particular level's features does not equate to increasing the level.
Example: A notionally "perfect" Level 3 feature would not be expected to qualify as a Level 4 feature, because it has no requirement to be able to detect and respond to non-ADS "evident" vehicle failures.
There is no such thing as "Level 2+" or "Level 2.5" etc. Fractional or "+" terminology is specifically prohibited by J3016 8.3.
It is possible that someone using the term "Level 2+" is describing a feature that does not fall into any of the J3016 levels. Or it is possible that the "Level 2+" designation is simply marketing puffery.
Despite an SAE released graphic (not part of the standard) that suggests otherwise, the ADS is not required to issue a Fallback request, and is not required to continue to operate the vehicle during a handoff period if there is an "evident" vehicle equipment failure that is "kinesthetically apparent."
J3016 3.22 Note 2:
J3016 5.4 Notes 2 & 3:
A Level 3 feature is not required to notify the driver explicitly if there is a non-ADS failure, and is similarly not required to provide a transition time.
Note: SAE J3016 uses the term "fallback-ready user" instead of "driver" while a Level 3 feature is engaged. I use "driver" here because I believe that being "receptive" is a task that requires the "fallback-ready" person to be performing an active monitoring rather than passive "receptivity" task, and thus is still a "driver." That having been said, precise standard terminology is "fallback-ready user."
This topic is subtle, but makes a huge difference to human driver responsibilities for Level 3 operation.
To begin with, J3016 does not say that Level 3 means "eyes off road" anywhere. That is a concept that is entirely outside the scope of the standard. (A text search reveals the words "eye" or "eyes" only appears in 3.18.2 NOTE in the context of stating that a driver of a conventional vehicle sometimes takes eyes off road momentarily to tune the radio, etc. Nothing to do with Level 3 at all.)
J3016 considers the driver to be "receptive to vehicle conditions that adversely affect the performance of the DDT." (J3016 3.18.3 Note) and refers to the definition of Receptivity at 3.22. The gist of receptivity is that the driver is supposed to notice that an "evident" or "kinesthetically apparent" vehicle failure happens even if not specifically paying attention, as well as receptive to an annunciated request to intervene by the ADS. Neither "evident" nor "kinesthetically apparent" are concretely defined in the standard.
Relevant concrete examples of an "evident" vehicle system failure failure in J3016 include:
J3016 3.18.1 Notes 1 & 2 taken together indicate the user monitoring (sometimes known as driver monitoring) is useful for Level 3 due to the potential of "misuse or abuse of driving automation technology" which includes over-reliance due to complacency. J3016 8.2 specifically designates the Level 3 driver falling asleep as "improper."
This means that if, for example, there is a broken tie rod, it is entirely on the driver to notice that the failure happened, presumably due to the car vibrating strangely or other "kinesthetic" cue. The driver must then intervene by immediately taking over operation of the vehicle to prevent a crash. Similarly a tire blow-out would require essentially instant take-over. As a practical matter, safety would likely require the driver to maintain situational awareness even when the ADS is driving to be able to react to the failure without hitting other vehicles or driving off the roadway due the vehicle suddenly becoming difficult to control.
Given the driver responsibilities of Level 3 it is difficult to see how safe operation can be achieved if the driver is continuously looking away from the road ("eyes off road") even if J3016 does not specifically require such attention. The potential for total consumption of driver attention by non-driving activities (e.g., engrossed in a mobile phone game or watching a movie) resulting in not noticing a potentially subtle "kinesthetic" cue gives further pause for thought.
Note: SAE J3016 uses the term "fallback-ready user" instead of "driver" while a Level 3 feature is engaged. I use "driver" here because I believe that being "receptive" is a task that requires the "fallback-ready" person to be performing an active monitoring rather than passive "receptivity" task, and thus is still a "driver." That having been said, precise standard terminology is "fallback-ready user."
In Level 3 operation the ADS is completely responsible for not only driving, but also observing the external world, events, and other road users. That means that Level 3 has no stated requirement for the driver to monitor or anticipate the behavior of other road users.
However, the human driver is responsible for noticing "evident" or "kinesthetically apparent" vehicle failures that impair the ability of the ADS to drive the vehicle safely, and for immediately operating the vehicle if that happens, even if the ADS does not issue an explicit request for intervention. This suggests a requirement to maintain situational awareness, but not a requirement to notice odd behavior by other road users.
Note: SAE J3016 uses the term "fallback-ready user" instead of "driver" while a Level 3 feature is engaged. I use "driver" here because I believe that being "receptive" is a task that requires the "fallback-ready" person to be performing an active monitoring rather than passive "receptivity" task, and thus is still a "driver." That having been said, precise standard terminology is "fallback-ready user."
A Level 3 feature can only be engaged when within its ODD. If engagement is possible outside the ODD, it is not Level 3. Same for Level 4. (J3016 Table 2, entries for Levels 3 & 4.)
Any automation capability for which a driver can be blamed for operation outside the ODD is either Level 1 or 2 by definition.
The presence of a safety driver does not change the level. A level 2 production feature and a level 4 test feature both require a driver. (Level 2 is a driver; Level 4 is a safety test driver -- two very different roles.)
Vehicle features with a Level 4 "design intent" that are being tested with a safety driver are still Level 4. (J3016 8.2):
While Level 5 features are said to have "unlimited" ODD, there are still two inherent limitations that designation:
In other words "unlimited ODD" (defined operationally in J3016 5.6 Note 1) at best means the ADS can drive on public roads anywhere a human can, which is not the same as drives anywhere a human can. ("On-road" is defined as publicly accessible roadways in J3016 Section 1 Scope.)
A further limitation is that if a Level 5 feature encounters a road that it is unable to operate on, it can still be called Level 5 (J3016 8.2).
Another limitation to Level 5 is that the ADS is not expected to perform "strategic" aspects of vehicle operation, such as route planning. (J3016 8.10, 8.11)
Another limitation is that some geo-fencing due to legal or business constraints still permits a Level 5 designation (J3016 8.8). It is unclear if this could be stretched to claim Level 5 within only a single city due to "business constraints."
The ANSI/UL 4600 autonomous product safety standard addresses safety beyond the scope of J3016 Level 5. ANSI/UL 4600 is not referenced by SAE J3016. For this reason, it is incorrect to say that Level 5 is sufficient to create a fully autonomous vehicle such as a robo-taxi. Drivers do more than just perform DDT and Fallback to keep a vehicle safe. All the things put out of scope by J3016 Level 5 -- but included in ANSI/UL 4600 -- still need to be dealt with for system-level safety.
While it might be that some dedicated use vehicles such as Level 4-capable taxi services won't need a steering wheel, others will.
Consider a vehicle that can only operate at Level 4 on divided highways. A human driver might be responsible for driving the vehicle onto the highway and pressing the "Level 4 go" dispatching button. The human driver can then do other activities (since at Level 4 the ADS is responsible for both driving and Fallback). When it's time to exit the highway, the ADS would need to arrange a transition back to human driving, with contingencies in place if the human driver does not respond to any request. The concept of a Sub-Trip Feature (J3016 3.7.2, Figure 1) generally discusses this topic. A similar situation exists for Level 5-capable dual-mode vehicles (J3016 3.25 NOTE, 3.31.2 Example 3, 3.32 Note 3).
Additionally, there are likely to be situations in which it is impractical to expect a Level 4-capable or even Level 5-capable vehicle to be capable of managing its own maneuvers such as for example positioning itself onto a maintenance lift or parking in a hay field at a special event. It is likely that there will need to be at least an auxiliary controller for Level 4-capable and Level 5-capable vehicles, even if that is a plug-in controller or remote controller rather than a traditional steering wheel. (See J3016 3.32.3 Note 3)
There is no safety analysis required by J3016. Section 3.16 Note 3 gives considerable flexibility in options, including expressly permitting a "stop within its current travel path." While this might generally be expected to be safer than continuing driving with a faulty vehicle, issues such as risk of being hit by passing high speed traffic or coming to a stop on top of the proverbial train tracks at a railroad crossing are not discussed. (Note: the 2021 edition of J3016 requires that the MRC result in a "stable stopped condition" in 3.16, while Note 3 also admits the possibility that the MRC might include automatically returning the vehicle to the dispatching facility before that stop is accomplished.)
J3016 8.5.iii states that "The minimal risk condition depends on both the vehicle condition and its operating environment at the time that fallback is triggered and could follow a degraded mode strategy that considers the relative risks associated with continuing operation, pulling off the road, or stopping in place." This is not to say that any particular vehicle would be unsafe, but rather simply emphasizes that J3016 is not a safety standard (see J3016 8.1), and that an MRC is more of an aspirational concept than a statement that any particular vehicle in any particular situation will actually achieve acceptable safety.
The length of warning for Level 3 is "at least several seconds" (J3016 3.12 Note 3; 3.17 Example 2). Nowhere in the standard is there a requirement that it be at least 10 seconds (or any other specific number).
Note: The ALKS standard does impose a 10 second minimum (ALKS 5.4.4.1.) However, there is no requirement in SAE J3016 that vehicles conform to that nor other aspects of ALKS that are not identical to what is in J3016.
There is no requirement to ensure that the "at least several seconds" of delay (J3016 3.12 Note 3) is long enough for the human driver in any particular situation to safely regain operation. Rather, the requirement is that the fallback-ready user "is then expected to resume manual vehicle operation, or to achieve a minimal risk condition" In other words, there is no requirement that the "several seconds" be long enough to provide safety in realistic conditions. Nor is there any requirement that the vehicle ensure that the fallback-ready user is in fact fallback ready.
There is no requirement that the human driver actually be ready to intervene. Rather, the human driver "is expected" to intervene if necessary after a delay, if such a delay has been provided.
J3016 describes the possibility of a "failure mitigation strategy designed to bring the vehicle to a controlled stop wherever the vehicle happens to be." in the current lane of travel when exiting an ODD. This is an optional behavior (J3016 8.6) and is not necessarily implemented in any particular Level 3 feature.
The "should" situation is Level 4. This is true even though such vehicles are currently being marketed as being equipped with Level 3 features. (J3016 3.12 Note 4, 3.25 Note, 5.5 Note 2)
ECE/TRANS/WP.29/2020/81 ALKS does not reference J3016, and does not use the word "level" in a way that references any J3016 Level. An ALKS feature might or might not be operating at Level 3 (or even Level 4). While detailed interpretation of that standard might (or might not) necessarily result in a specific J3016 Level designation, the document itself does not do so. Detailed analysis to determine which J3016 Level - if any - is described by ALKS has not been performed.
There is no such thing as a "Level 2 vehicle" or any "Level X vehicle" per J3016 terminology.
Levels are associated with driving automation features, not with vehicles. (J3016 Section 1). Vehicles can have multiple features that operate at different levels. So the vehicle is operating at a particular Level at a particular time based on which feature has been activated, but that can change even during a single drive cycle (see J3016 3.7.3 and Figure 1).
The proper use of terminology is "Level [1 or 2] driving automation system-equipped vehicle" or "Level [3, 4, or 5] ADS-equipped vehicle" (J3016 7.2). (Strictly speaking an "ADS" supports only Levels 3-5, while "driving automation system" refers to technology that supports any driving level (J3016 3.2).) Related approved phrases are "Level [1 or 2] driving automation system-engaged vehicle" or "Level [3, 4, or 5] ADS-operated vehicle" (J3016 7.2).
Related terminology: an ADS-DV vehicle is always dispatched in driverless operation, but a dual-mode vehicle can be operated with or without a human driver. (J3016 3.7). It is acceptable to say "Level X ADS-DV" for levels 3, 4, 5 (e.g., "Level 4 ADS-DV") for vehicles which normally run full trips with Level 3 or higher features engaged. For example, a campus shuttle with no human operator might be called a "Level 4 ADS-DV."
In informal writing if I want to keep things compact and not use ADS-DV, I change "Level 4 vehicle" to "Level 4-capable vehicle" to respect the difference without being too wordy. This is not officially endorsed by J3016 but some might consider it an adequate workaround to avoid longer phrases, especially for vehicles that are equipped with multiple features at different Levels. Other handy informal phrasing approaches are "Level 4 operation" and "Level 4-engaged." Again, these are informal phrases that should not be used in regulations or technical specifications, but might be helpful in making writing more accessible for less rigorously technical purposes and to non-technical audiences.
J3016 is a terminology standard, not a safety standard. For style issues the document does not say this in a straightforward way, but discussions about potential changes proposed to the standards comittee I have made reinforce this position in every discussion.
What the standard does say is that it "does not provide specifications, or otherwise impose requirements on, driving automation systems (for further elaboration, see 8.1)" In turn, section 8.1 says that the standard "By itself, this document imposes no requirements, nor confers or implies any judgment in terms of system performance." In other words, this document does not tell you if a system corresponding to a particular level will be safe.
J3016 section 5.2 note 3 uses the term ADAS to describe "warnings and/or momentary intervention" ... "as well as some convenience features that involve level 1 driver support features..." This is in the section on Level 1, "Driver Assistance," which is the only place the term ADAS is used in the entire standard beyond its appearance in the list of abbreviations.
While J3016 prefers not to use the term ADAS as a description, there is no support whatsoever in the document for calling Level 2 "ADAS."
There are extensive notes and examples in J3016 that need to be understood to fully appreciate the nuances of the standard. The Myths section above covers some of those. A few major potential surprises include:
There are some scope limitations of J3016 that are not always appreciated in interpreting what the standard says and doesn't say.
J3016 Section 7 contains details regarding deprecated terms (i.e., terms that should not be used in the context of J3016 Levels.) Some key terms include:
A specific type of terminology misuse in practice is failing to distinguish between the real world and the ODD.
Thus, when the real world (OD) is outside the intended operational design environment (ODD), the vehicle has exited the ODD and is no longer in an environment it was designed to operate within.
Additionally, there is by definition only one ODD for a driving automation feature. However, there can be multiple driving features. Thus, it is incorrect for example to say that a Level 4 highway feature has "multiple ODDs of day and night with fair weather" when what should be said is that the feature has "an ODD that includes both day and night in fair weather." However, that same ADS might at the same time support a completely different feature of Level 3 highway operation with an all-weather daytime ODD. In other words, each driving feature is associated with its own, exactly one ODD. J3016 Section 6 and Figure 12 delve into this complexity in more detail. See also J3016 8.4. It's complicated.
While this is primarily and explanation of J3016 and not a critique, there are some problematic points that simply can't be ignored.
To be clear, the group that has spent many hours of discussion and debate to create the standard has done its best, especially in terms of reaching precise wordings of definitions. However, some overall approaches of the standard are problematic in terms of achieving safety (which is in fact a subject that is explicitly out of scope for J3016).
These points are the author's opinion, and not simply a purely factual interpretation of the standard:
In short, while J3016 provides definitions that are actionable by design engineers, it is asking a lot for ordinary drivers to understand the true implications of the Levels. Moreover, it seems likely that building Level 2 and 3 vehicles precisely as defined (and no more, for example without any driver monitoring since it is not required) could lead to unreasonably dangerous vehicles.
For these reasons I think it is a mistake to use the SAE Levels (especially Level 2 and Level 3) for regulatory purposes. (More on that here)
(The J3016 drafting committee has kindly put items from this list that they consider to be in scope for J3016 into the "parking lot" list for consideration in the next version of the standard. However the standard as currently issued does not, in my opinion, adequately address these issues.)
Thanks for reviews and suggestions to: Jackie Erickson, Gil Amid, Stefan Benz on content. Sebastian Holmqvist for a usability trick.
The myths listed are not the fault of J3016 per se. Rather, they are the result of people writing and talking about J3016 without being fully aware of the contents of that standard.
SAE J3016 is so complex it is difficult to get everything right, even for me. If you see something that is incorrect please let me know, but please do tell me exactly where in the standard I'm conflicting. Conflicting with some other article would be no surprise, because so many of them buy in to one of the myths (or one I haven't run into yet).
Phil Koopman: koopman@cmu.edu
Updated September 4, 2021.