SAE J3016 User Guide

Philip Koopman, Carnegie Mellon University

The SAE J3016:2021 standard defines terminology for automated vehicles including the famous SAE Automation Levels. It is widely referenced in discussions, other standards, and even government regulations. Unfortunately, what is said about J3016 is too often inaccurate, misleading, or just plain incorrect.

Misinterpreting the SAE Levels can lead to misunderstandings about what the standard actually says, the technology incorporated into a car, and a driver's expectations. It's important to get statements in standards and regulations right. Moreover, it's important when referring to J3016 to understand that it says what it says, not what some author might want it to say, what might seem optimal for safety, or what other documents state that it says. (While this might seem obvious, perpetuation of misunderstandings is rampant.)

The scope of this article is to help readers understand what is in J3016, and what is not. In particular, it serves to debunk common myths and misconceptions about the contents of J3016. Even with an attempt to keep things brief, this article is complicated, simply because J3016 is a complicated standard.

This article is not actually the standard itself, so it can't be perfect in every respect. However, it is intended to be useful orientation for anyone trying to interpret the standard. Anyone who is going to refer to the SAE Levels in any writing should actually read the standard, using this article as a guide so you know what to look for. You can get J3016:2021 for free (SAE membership may be required):

Beyond the Levels, SAE J3016 defines other related terminology. That terminology should be used when possible to help reduce terminology fragmentation, but is not the primary focus of this article.

It is important to note in this article that numerous volunteers spent very significant amounts of time creating the versions of SAE J3016 and in particular tried to make what that standard says precise. Their efforts are highly appreciated.

Resources:

J3016 Overview

We've all seen overviews of the SAE Levels. Unfortunately, many of them are incorrect for reasons both subtle and obvious - after you've read the actual standard. Here is a summary that (hopefully) is entirely correct, although necessarily missing some of the nuance of the 41-page full standard. (This table is inspired by J3016 Table 1.)

{LevelsTable}

"Other safety" is not defined by J3016, but refers in general to operational safety considerations outside the scope of J3016 such as ensuring that children are properly secured before the vehicle moves and whether a trip route or destination is expected to be acceptably safe during trip planning. Note that for Level 3 the "driver" is more precisely called the "fallback-ready user" in J3016, who becomes the driver when performing the fallback task.

Terminology summary

Key terminology definitions below are paraphrased for brevity. The actual J3016 standard text should be used when complete accuracy rather than intuitional understanding is required.

Summary of the J3016 Automation Levels

J3016 Table 1 contains a precise explanation of the levels, although additional background from the rest of the standard is needed to interpret it correctly. If you need to get things right, start with that table.

Restating the dense language in J3016 Table 1 for a broader audience:

Myths

A shocking amount of descriptive text written about SAE J3016 in social media, news outlets, and even material that could lead to regulations, is meaningfully incomplete, misleading, or just plain wrong. Here are some of the more common myths debunked. If you're writing something about the SAE Levels please use this as a checklist to avoid being called out on social media for getting it wrong or, even worse, being accused of #autonowashing.

Myth #1: Limited ODD (Levels 2-4) just means geo-fencing

An Operational Design Domain is more than just the geographic area of operation of a vehicle. It includes all the factors that must be true of the environment for the ADS to function properly. That potentially includes lighting, precipitation, road stripe paint condition, whether roads are icy, and so on. While it is likely that early vehicles will have geographic or route restrictions as a matter of practicality (especially if high definition maps are needed), that is only one possible limitation inherent in an ODD. (J3016 3.21, 6., Figure 11, Figure 12)

Myth #2: SAE J3016 Levels define safe vehicles

J3016 is not a technical specification for what should be built. It simply describes something that could be built. Moreover, something that does meet a defined Level might be entirely reasonable to build. And something that might precisely meet J3016 is not guaranteed to be acceptably safe. (J3016 8.1)

Myth #3: Higher SAE J3016 Levels are safer

Higher level numbers have more automation, but they might not be safer or otherwise "better," especially in the near term. (J3016 8.3)

Example: A Level 2-engaged vehicle with a poorly performing driver monitoring system might have a higher crash rate than a Level 1-engaged vehicle due to driver automation complacency resulting in poor performance of the driver supervision task.

Myth #4: Progress involves moving up the levels from 1 to 5

Each level is a different type of vehicle. The tradeoffs for Levels 1 & 2 are quite different than for Levels 4 & 5 due to the different role of the human driver. Improved performance within a particular level's features does not equate to increasing the level.

Example: A notionally "perfect" Level 3 feature would not be expected to qualify as a Level 4 feature, because it has no requirement to be able to detect and respond to non-ADS "evident" vehicle failures.

Myth #5: SAE J3016 Level 2+ is better than Level 2

There is no such thing as "Level 2+" or "Level 2.5" etc. Fractional or "+" terminology is specifically prohibited by J3016 8.3.

It is possible that someone using the term "Level 2+" is describing a feature that does not fall into any of the J3016 levels. Or it is possible that the "Level 2+" designation is simply marketing puffery.

Myth #6: SAE J3016 Level 3 features alway notify the driver to take over via an ADS request to intervene

Despite an SAE released graphic (not part of the standard) that suggests otherwise, the ADS is not required to issue a Fallback request, and is not required to continue to operate the vehicle during a handoff period if there is an "evident" vehicle equipment failure that is "kinesthetically apparent."

J3016 3.22 Note 2:
{J3016 3.22 Note2

J3016 5.4 Notes 2 & 3:
{J3016 5.4 Notes 2&3}

A Level 3 feature is not required to notify the driver explicitly if there is a non-ADS failure, and is similarly not required to provide a transition time.

Note: SAE J3016 uses the term "fallback-ready user" instead of "driver" while a Level 3 feature is engaged. I use "driver" here because I believe that being "receptive" is a task that requires the "fallback-ready" person to be performing an active monitoring rather than passive "receptivity" task, and thus is still a "driver." That having been said, precise standard terminology is "fallback-ready user."

Myth #7: SAE J3016 Level 3 means the driver can perform non-driving activities ("eyes off the road")

This topic is subtle, but makes a huge difference to human driver responsibilities for Level 3 operation.

To begin with, J3016 does not say that Level 3 means "eyes off road" anywhere. That is a concept that is entirely outside the scope of the standard. (A text search reveals the words "eye" or "eyes" only appears in 3.18.2 NOTE in the context of stating that a driver of a conventional vehicle sometimes takes eyes off road momentarily to tune the radio, etc. Nothing to do with Level 3 at all.)

J3016 considers the driver to be "receptive to vehicle conditions that adversely affect the performance of the DDT." (J3016 3.18.3 Note) and refers to the definition of Receptivity at 3.22. The gist of receptivity is that the driver is supposed to notice that an "evident" or "kinesthetically apparent" vehicle failure happens even if not specifically paying attention, as well as receptive to an annunciated request to intervene by the ADS. Neither "evident" nor "kinesthetically apparent" are concretely defined in the standard.

Relevant concrete examples of an "evident" vehicle system failure failure in J3016 include:

J3016 3.18.1 Notes 1 & 2 taken together indicate the user monitoring (sometimes known as driver monitoring) is useful for Level 3 due to the potential of "misuse or abuse of driving automation technology" which includes over-reliance due to complacency. J3016 8.2 specifically designates the Level 3 driver falling asleep as "improper."

This means that if, for example, there is a broken tie rod, it is entirely on the driver to notice that the failure happened, presumably due to the car vibrating strangely or other "kinesthetic" cue. The driver must then intervene by immediately taking over operation of the vehicle to prevent a crash. Similarly a tire blow-out would require essentially instant take-over. As a practical matter, safety would likely require the driver to maintain situational awareness even when the ADS is driving to be able to react to the failure without hitting other vehicles or driving off the roadway due the vehicle suddenly becoming difficult to control.

Given the driver responsibilities of Level 3 it is difficult to see how safe operation can be achieved if the driver is continuously looking away from the road ("eyes off road") even if J3016 does not specifically require such attention. The potential for total consumption of driver attention by non-driving activities (e.g., engrossed in a mobile phone game or watching a movie) resulting in not noticing a potentially subtle "kinesthetic" cue gives further pause for thought.

Note: SAE J3016 uses the term "fallback-ready user" instead of "driver" while a Level 3 feature is engaged. I use "driver" here because I believe that being "receptive" is a task that requires the "fallback-ready" person to be performing an active monitoring rather than passive "receptivity" task, and thus is still a "driver." That having been said, precise standard terminology is "fallback-ready user."

Myth #8: SAE J3016 Level 3 means the driver monitors other road users and obstacles in case the ADS misses them

In Level 3 operation the ADS is completely responsible for not only driving, but also observing the external world, events, and other road users. That means that Level 3 has no stated requirement for the driver to monitor or anticipate the behavior of other road users.

However, the human driver is responsible for noticing "evident" or "kinesthetically apparent" vehicle failures that impair the ability of the ADS to drive the vehicle safely, and for immediately operating the vehicle if that happens, even if the ADS does not issue an explicit request for intervention. This suggests a requirement to maintain situational awareness, but not a requirement to notice odd behavior by other road users.

Note: SAE J3016 uses the term "fallback-ready user" instead of "driver" while a Level 3 feature is engaged. I use "driver" here because I believe that being "receptive" is a task that requires the "fallback-ready" person to be performing an active monitoring rather than passive "receptivity" task, and thus is still a "driver." That having been said, precise standard terminology is "fallback-ready user."

Myth #9: A Level 3 or 4 feature can be mistakenly engaged outside its ODD

A Level 3 feature can only be engaged when within its ODD. If engagement is possible outside the ODD, it is not Level 3. Same for Level 4. (J3016 Table 2, entries for Levels 3 & 4.)

Any automation capability for which a driver can be blamed for operation outside the ODD is either Level 1 or 2 by definition.

Myth #10: Vehicles operating at Level 4 never need safety drivers

Vehicle features with a Level 4 "design intent" that are being tested with a safety driver are still Level 4. (J3016 8.2):

{Design Intent and Level 4}

Myth #11: Level 5 features have unlimited capability

While Level 5 features are said to have "unlimited" ODD, there are still two inherent limitations that designation:

In other words "unlimited ODD" (defined operationally in J3016 5.6 Note 1) at best means the ADS can drive on public roads anywhere a human can, which is not the same as drives anywhere a human can. ("On-road" is defined as publicly accessible roadways in J3016 Section 1 Scope.)

A further limitation is that if a Level 5 feature encounters a road that it is unable to operate on, it can still be called Level 5 (J3016 8.2).

Another limitation to Level 5 is that the ADS is not expected to perform "strategic" aspects of vehicle operation, such as route planning. (J3016 8.10, 8.11)

Another limitation is that some geo-fencing due to legal or business constraints still permits a Level 5 designation (J3016 8.8). It is unclear if this could be stretched to claim Level 5 within only a single city due to "business constraints."

The ANSI/UL 4600 autonomous product safety standard addresses safety beyond the scope of J3016 Level 5. ANSI/UL 4600 is not referenced by SAE J3016. For this reason, it is incorrect to say that Level 5 is sufficient to create a fully autonomous vehicle such as a robo-taxi. Drivers do more than just perform DDT and Fallback to keep a vehicle safe. All the things put out of scope by J3016 Level 5 -- but included in ANSI/UL 4600 -- still need to be dealt with for system-level safety.

Myth #12: Requiring a steering wheel prevents deployment of Level 4 & 5-capable vehicles

While it might be that some dedicated use vehicles such as Level 4-capable taxi services won't need a steering wheel, others will.

Consider a vehicle that can only operate at Level 4 on divided highways. A human driver might be responsible for driving the vehicle onto the highway and pressing the "Level 4 go" dispatching button. The human driver can then do other activities (since at Level 4 the ADS is responsible for both driving and Fallback). When it's time to exit the highway, the ADS would need to arrange a transition back to human driving, with contingencies in place if the human driver does not respond to any request. The concept of a Sub-Trip Feature (J3016 3.7.2, Figure 1) generally discusses this topic. A similar situation exists for Level 5-capable dual-mode vehicles (J3016 3.25 NOTE, 3.31.2 Example 3, 3.32 Note 3).

Additionally, there are likely to be situations in which it is impractical to expect a Level 4-capable or even Level 5-capable vehicle to be capable of managing its own maneuvers such as for example positioning itself onto a maintenance lift or parking in a hay field at a special event. It is likely that there will need to be at least an auxiliary controller for Level 4-capable and Level 5-capable vehicles, even if that is a plug-in controller or remote controller rather than a traditional steering wheel. (See J3016 3.32.3 Note 3)

Myth #13: A "Minimal Risk Condition" means the vehicle has been made safe

There is no safety analysis required by J3016. Section 3.16 Note 3 gives considerable flexibility in options, including expressly permitting a "stop within its current travel path." While this might generally be expected to be safer than continuing driving with a faulty vehicle, issues such as risk of being hit by passing high speed traffic or coming to a stop on top of the proverbial train tracks at a railroad crossing are not discussed. (Note: the 2021 edition of J3016 requires that the MRC result in a "stable stopped condition" in 3.16, while Note 3 also admits the possibility that the MRC might include automatically returning the vehicle to the dispatching facility before that stop is accomplished.)

J3016 8.5.iii states that "The minimal risk condition depends on both the vehicle condition and its operating environment at the time that fallback is triggered and could follow a degraded mode strategy that considers the relative risks associated with continuing operation, pulling off the road, or stopping in place." This is not to say that any particular vehicle would be unsafe, but rather simply emphasizes that J3016 is not a safety standard (see J3016 8.1), and that an MRC is more of an aspirational concept than a statement that any particular vehicle in any particular situation will actually achieve acceptable safety.

Myth #14: The human driver has 10 seconds warning in Level 3 operation

The length of warning for Level 3 is "at least several seconds" (J3016 3.12 Note 3; 3.17 Example 2). Nowhere in the standard is there a requirement that it be at least 10 seconds (or any other specific number).

Note: The ALKS standard does impose a 10 second minimum (ALKS 5.4.4.1.) However, there is no requirement in SAE J3016 that vehicles conform to that nor other aspects of ALKS that are not identical to what is in J3016.

Myth #15: Level 3 features ensure the human driver takes over when needed, and take over if the human driver does not

There is no requirement to ensure that the "at least several seconds" of delay (J3016 3.12 Note 3) is long enough for the human driver in any particular situation to safely regain operation. Rather, the requirement is that the fallback-ready user "is then expected to resume manual vehicle operation, or to achieve a minimal risk condition" In other words, there is no requirement that the "several seconds" be long enough to provide safety in realistic conditions. Nor is there any requirement that the vehicle ensure that the fallback-ready user is in fact fallback ready.

There is no requirement that the human driver actually be ready to intervene. Rather, the human driver "is expected" to intervene if necessary after a delay, if such a delay has been provided.

J3016 describes the possibility of a "failure mitigation strategy designed to bring the vehicle to a controlled stop wherever the vehicle happens to be." in the current lane of travel when exiting an ODD. This is an optional behavior (J3016 8.6) and is not necessarily implemented in any particular Level 3 feature.

Myth #16: A feature for which the driver should, but might not, perform Fallback is Level 3

The "should" situation is Level 4. This is true even though such vehicles are currently being marketed as being equipped with Level 3 features. (J3016 3.12 Note 4, 3.25 Note, 5.5 Note 2)

Myth #17: Automated Lane Keeping Systems (ALKS) are defined at Level 3

ECE/TRANS/WP.29/2020/81 ALKS does not reference J3016, and does not use the word "level" in a way that references any J3016 Level. An ALKS feature might or might not be operating at Level 3 (or even Level 4). While detailed interpretation of that standard might (or might not) necessarily result in a specific J3016 Level designation, the document itself does not do so. Detailed analysis to determine which J3016 Level - if any - is described by ALKS has not been performed.

Myth #18: There is such a thing as a "Level X vehicle" (e.g., "Level 4 vehicle")

There is no such thing as a "Level 2 vehicle" or any "Level X vehicle" per J3016 terminology.

Levels are associated with driving automation features, not with vehicles. (J3016 Section 1). Vehicles can have multiple features that operate at different levels. So the vehicle is operating at a particular Level at a particular time based on which feature has been activated, but that can change even during a single drive cycle (see J3016 3.7.3 and Figure 1).

The proper use of terminology is "Level [1 or 2] driving automation system-equipped vehicle" or "Level [3, 4, or 5] ADS-equipped vehicle" (J3016 7.2). (Strictly speaking an "ADS" supports only Levels 3-5, while "driving automation system" refers to technology that supports any driving level (J3016 3.2).) Related approved phrases are "Level [1 or 2] driving automation system-engaged vehicle" or "Level [3, 4, or 5] ADS-operated vehicle" (J3016 7.2).

Related terminology: an ADS-DV vehicle is always dispatched in driverless operation, but a dual-mode vehicle can be operated with or without a human driver. (J3016 3.7). It is acceptable to say "Level X ADS-DV" for levels 3, 4, 5 (e.g., "Level 4 ADS-DV") for vehicles which normally run full trips with Level 3 or higher features engaged. For example, a campus shuttle with no human operator might be called a "Level 4 ADS-DV."

In informal writing if I want to keep things compact and not use ADS-DV, I change "Level 4 vehicle" to "Level 4-capable vehicle" to respect the difference without being too wordy. This is not officially endorsed by J3016 but some might consider it an adequate workaround to avoid longer phrases, especially for vehicles that are equipped with multiple features at different Levels. Other handy informal phrasing approaches are "Level 4 operation" and "Level 4-engaged." Again, these are informal phrases that should not be used in regulations or technical specifications, but might be helpful in making writing more accessible for less rigorously technical purposes and to non-technical audiences.

Surprises, terminology misuse, and limitations of J3016

Potential surprises

There are extensive notes and examples in J3016 that need to be understood to fully appreciate the nuances of the standard. The Myths section above covers some of those. A few major potential surprises include:

SAE J3016 scope

There are some scope limitations of J3016 that are not always appreciated in interpreting what the standard says and doesn't say.

Terms that are not used in J3016

J3016 Section 7 contains details regarding deprecated terms (i.e., terms that should not be used in the context of J3016 Levels.) Some key terms include:

Misuse of ODD Terminology

A specific type of terminology misuse in practice is failing to distinguish between the real world and the ODD.

Thus, when the real world (OD) is outside the intended operational design environment (ODD), the vehicle has exited the ODD and is no longer in an environment it was designed to operate within.

Additionally, there is by definition only one ODD for a driving automation feature. However, there can be multiple driving features. Thus, it is incorrect for example to say that a Level 4 highway feature has "multiple ODDs of day and night with fair weather" when what should be said is that the feature has "an ODD that includes both day and night in fair weather." However, that same ADS might at the same time support a completely different feature of Level 3 highway operation with an all-weather daytime ODD. In other words, each driving feature is associated with its own, exactly one ODD. J3016 Section 6 and Figure 12 delve into this complexity in more detail. See also J3016 8.4. It's complicated.

Issues with J3016

While this is primarily and explanation of J3016 and not a critique, there are some problematic points that simply can't be ignored.

To be clear, the group that has spent many hours of discussion and debate to create the standard has done its best, especially in terms of reaching precise wordings of definitions. However, some overall approaches of the standard are problematic in terms of achieving safety (which is in fact a subject that is explicitly out of scope for J3016).

These points are the author's opinion, and not simply a purely factual interpretation of the standard:

  1. Clarity. The goal of providing clarity in communications (J3016 Rationale purpose 4) is arguably not met for anyone who is not deeply expert in the technology. This is demonstrated by the pervasive misapprehension of the document in the press, informal communications, and even occasional regulatory publications. The fact that this article even needs to be written speaks to this point. While J3016 might be useful for engineering and highly technical audiences, a different description other than the J3016 Levels should be used for public communication to ordinary drivers. (To be sure, the standard sets its audience to include media and public discourse on page 1.)
  2. Useful Framework. The goal of "providing a useful framework" for specifications and technical requirements (J3016 Rationale purpose 3) is only partially met. The human factors issues with Level 3 seem to be pushing a number of vehicle makers to offer vehicles which differ from the defined Level 3, especially in the area of guaranteeing that the ADS will perform Fallback if the driver does not respond within any set fixed time (which, technically, would seem to make them Level 4-capable vehicles). The apparent issue is the difference between a driver "need not" take over (Level 4), "should" take over (being sold as Level 3, but actually Level 4) vs. "shall" take over when necessary (Level 3). If the manufacturers are portraying Level 4 features as Level 3 in marketing communications, this brings into question the utility of the framework in practical application. Pervasive use of "Level 2+" and other prohibited fractional level notation in the media and other venues is additional evidence that there is a problem in this regard.
  3. Level 2. Level 2 is proving problematic due to the issue of driver complacency. While this issue is acknowledged in the current version of J3016, driver monitoring is still said to be optional and effective driver monitoring is not required. (J3016 3.18.1 defines user monitoring, but J3016 does not seem to require it. Moreover, J3016 8.2 gives a specific example of a situation in which a Level 3-engaged ADS is not expected to detect that a driver has fallen asleep.) It is still unclear whether even sophisticated driver monitoring will be sufficient to result in acceptably safe vehicle operation for large deployed cohorts of vehicles across the full demographic spread of licensed drivers.
  4. Level 3. Level 3 breeds significant confusion as to driver role due to the concept of the driver being "receptive" to "evident" or "kinesthetically apparent" "performance-relevant system failure" situations even if the ADS does not issue a warning. It is common to see Level 3 characterized as "eyes off road" (which seems unlikely to be true in practice). It is also common to see Level 3 incorrectly portrayed as always providing an ADS-issued takeover warning with a defined takeover grace period. Moreover, the topic of safely regaining or maintaining situational awareness to accomplish a takeover is not addressed for non-ADS failures that the driver is responsible for both noticing and handling with no ADS support. I believe that being "receptive" to failures and even ADS-generated notifications requires intentionally reserving cognitive capacity for alertness, and thus should properly be considered a "driving" task rather somehow being expected to happen "for free" regardless of secondary driver tasks, especially in individuals who might be prone to hyperfocus.
  5. Remote Fallback. An additional concern regarding Fallback requirements without an ADS failure is that J3016 3.31.3.2 specifically permits a remote fallback-ready user who is responsible for taking over operation if something goes wrong. It is unclear how a remote user could be "receptive" to "kinesthetically apparent" cues if not actually in the vehicle.
  6. Effective Warning. The required warning time for an ADS failure is not specified sufficiently to ensure safety. It seems unlikely that an unconstrained "several seconds" (e.g., 3 or 4 seconds) is sufficient. It should be noted that no warning time at all is required for an "evident" or "kinesthetically apparent" vehicle failure. (The ten seconds referred to in ALKS is likely to be insufficient for high speed vehicle operation.)
  7. Human deciding to engage ADS. For all levels, J3016 puts the burden of determining whether to engage the ADS on the human driver (J3016 Table 2 Role of User column). This could lead to ambiguity as to responsibility in the event of a mishap, such as whether there are factors beyond the ODD that are relevant to driving safety for which the user is responsible even in a Level 5-engaged vehicle. It is not the apparent intent of the standard to raise this issue, but it seems as written the issue could become a problem.
  8. Mode confusion. J3016 does not seem to raise the issue of potential mode confusion, such as when the driver's mental model does not match the current operational feature Level. In practice this is a significant contributing causal factor leading to mishaps.

In short, while J3016 provides definitions that are actionable by design engineers, it is asking a lot for ordinary drivers to understand the true implications of the Levels. Moreover, it seems likely that building Level 2 and 3 vehicles precisely as defined (and no more, for example without any driver monitoring since it is not required) could lead to unreasonably dangerous vehicles.

For these reasons I think it is a mistake to use the SAE Levels (especially Level 2 and Level 3) for regulatory purposes. (More on that here)

(The J3016 drafting committee has kindly put items from this list that they consider to be in scope for J3016 into the "parking lot" list for consideration in the next version of the standard. However the standard as currently issued does not, in my opinion, adequately address these issues.)

Notes

Acknowlegments

Thanks for reviews and suggestions to: Jackie Erickson, Gil Amid, Stefan Benz on content. Sebastian Holmqvist for a usability trick.

Disclaimers

The myths listed are not the fault of J3016 per se. Rather, they are the result of people writing and talking about J3016 without being fully aware of the contents of that standard.

SAE J3016 is so complex it is difficult to get everything right, even for me. If you see something that is incorrect please let me know, but please do tell me exactly where in the standard I'm conflicting. Conflicting with some other article would be no surprise, because so many of them buy in to one of the myths (or one I haven't run into yet).


Phil Koopman: koopman@cmu.edu
Updated September 4, 2021.