Link to this page: https://users.ece.cmu.edu/~koopman/SafeEnough/
If a link is dead, try it through the Wayback Machine here:https://archive.org/

Koopman, P., How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety, September 2022.
ISBN: 9798846251243 Trade Paperback
ISBN: 9798848273397 Hardcover

One hour video overview of the book: Slides | YouTube | Archive.org

• "Safe enough" resources
  • Blumenthal et al., Safe Enough, 2020:
    https://www.rand.org/pubs/research_reports/RRA569-1.html
    • Free download of e-book version with a different set of voices on the topic of ensuring acceptable AV safety.
  • European Commission, Ethics of connected and automated vehicles, June 2020:
    https://data.europa.eu/doi/10.2777/035239
    • An "AI safety" take on the topic emphasizing data ethics and responsibility.
  • UK Law Commission report on Automated Vehicles, 2022:
    https://www.lawcom.gov.uk/project/automated-vehicles/
  • UK Centre for Data Ethics and Innovation policy paper, 2022:
    https://www.gov.uk/government/publications/responsible-innovation-in-self-driving-vehicles/responsible-innovation-in-self-driving-vehicles
  • NHTSA Framework for Automated Driving System Safety, 2020:
    https://www.federalregister.gov/documents/2020/12/03/2020-25930/framework-for-automated-driving-system-safety
    • Also see the NHTSA Advanced Notice of Proposed Rulemaking (ANPRM) that would among other things require following industry standards for AVs. Public comments can be seen here:
      https://www.regulations.gov/docket/NHTSA-2020-0106/comments
• Educational resources
  • Author's lectures on AV safety and related materials:
    https://users.ece.cmu.edu/~koopman/lectures/index.html
    • Short course on Autonomous Vehicle Safety
    • Course material on dependable embedded systems
    • Various recorded keynote and other talks o Podcasts
    • AV safety overview from 2022:
      https://youtu.be/oE_2rBxNrfc
    • SPI lecture from 2022:
      https://youtu.be/mRXotHN0Z6I
    • Trust & Governance from 2022:
      https://youtu.be/hZQyFc9ETCE
    • Alternate source for lectures:
      https://archive.org/details/@pkoopman
  • IFIP WG 10.4 workshop on Intelligent Vehicle Dependability and Security (IVDS)
    https://ivds2021.dependability.org/program.html
    • o Talks including slides and video regarding AV safety.
  • NTSB hearing on the pedestrian fatality in Tempe AZ:
    https://youtu.be/mSC4Fr3wf0k
    • The first five minutes of this hearing should be required viewing for everyone working on AV technology. It emphasizes the need for a robust SMS and reminds us that companies do not have to wait for a fatality to get their safety house in order.
    • The crash investigation report is here: https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf
• Other resources
• Safe Autonomy Blog:
  https://safeautonomy.blogspot.com/
  • o Computer-based system safety essential reading list with many pointers to additional materials: https://safeautonomy.blogspot.com/p/safe-autonomy.html
  • State legislation case study materials from 2022:
    https://safeautonomy.blogspot.com/2022/02/kansas-av-regulation-bill-hearings.html
  • AV regulatory launch page:
    https://safeautonomy.blogspot.com/2021/12/automated-vehicle-regulator-launch-page.html
• Compendium of selected automotive software defects involving safety recalls:
  https://betterembsw.blogspot.com/p/potentially-deadly-automotive-software.html
• Web resources on assurance cases:
  • o City University of London:
    https://researchcentres.city.ac.uk/software-reliability/research/diversity
  • o John Rushby, SRI:
    http://www.csl.sri.com/users/rushby/assurance-cases.html
• ANSI/UL 4600 launch page:
  https://users.ece.cmu.edu/~koopman/ul4600/index.html
• Other general resources:
  • o System Engineering Body of Knowledge (SEBoK)
    https://sebokwiki.org/
• Topical sites and events:
  • SAFECOMP conference series: 2022 edition:
    https://safecomp22.iks.fraunhofer.de/
  • Fraunhofer IKS Institute for Cognitive Systems:
    https://www.iks.fraunhofer.de/
  • o University of York Institute for Safe Autonomy:
    https://www.york.ac.uk/safe-autonomy/
  • Safety-Critical Systems Club:
    https://scsc.uk/
  • Workshop on Artificial Intelligence Safety Engineering:
    https://www.waise.org/
  • International System Safety Society
    https://system-safety.org/

Footnote Pointers:

1. --
2. --
3. --
4. https://rosap.ntl.bts.gov/view/dot/38381
5. https://www.ri.cmu.edu/pub_files/pub2/thorpe_charles_1997_1/thorpe_charles_1997_1.pdf
6. https://path.berkeley.edu/sites/default/files/demo97foravs17v6.pdf
7. https://www.nrec.ri.cmu.edu/
8. https://youtu.be/DKHa7rxkvK8
9. https://users.ece.cmu.edu/~koopman/pubs/Koopman19_WAISE_UL4600.pdf
10. https://users.ece.cmu.edu/~koopman/ul4600/index.html
11. https://users.ece.cmu.edu/~koopman/lectures/index.html#av
12. --
13. --
14. --
15. --
16. https://users.ece.cmu.edu/~koopman/ul4600/index.html
17. --
18. --
19. --
20. https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/
21. --
22. --
23. --
24. --
25. https://unece.org/transport/documents/2021/03/standards/un-regulation-no-157-automated-lane-keeping-systems-alks
26. https://users.ece.cmu.edu/~koopman/ul4600/index.html
27. --
28. https://lizadixon.com/Autonowashing
29. https://www.sae.org/standards/content/j3016_202104/
30. https://youtu.be/Kykb75_41hY
31. --
32. --
33. https://users.ece.cmu.edu/~koopman/j3016/
34. --
35. --
36. --
37. --
38. --
39. --
40. --
41. --
42. --
43. --
44. --
45. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214
46. --
47. https://www.sae.org/blog/sae-j3016-update
48. https://www.sae.org/news/2020/12/rise-of-sae-level-2
49. https://www.consumerreports.org/autonomous-driving/cr-engineers-show-tesla-will-drive-with-no-one-in-drivers-seat/
50. https://journals.sagepub.com/doi/full/10.1177/0018720818788164
51. --
52. --
53. https://unece.org/transport/documents/2021/03/standards/un-regulation-no-157-automated-lane-keeping-systems-alks
54. --
55. --
56. --
57. https://www.southampton.ac.uk/news/2017/01/driverless-cars.page
58. --
59. --
60. --
61. --
62. --
63. --
64. --
65. --
66. --
67. --
68. --
69. --
70. --
71. --
72. --
73. --
74. https://en.wikipedia.org/wiki/Mode_(user_interface)#Mode_errors
75. --
76. --
77. https://usa.streetsblog.org/2022/02/15/the-brake-why-theres-no-such-thing-as-a-car-accident/
78. https://en.wikipedia.org/wiki/Sense_Plan_Act
79. --
80. --
81. https://en.wikipedia.org/wiki/Receiver_operating_characteristic
    https://en.wikipedia.org/wiki/F-score
82. https://en.wikipedia.org/wiki/Atomic_orbital
83. --
84. https://en.wikipedia.org/wiki/Confusion_matrix
85. https://en.wikipedia.org/wiki/Machine_learning
86. --
87. --
88. --
89. --
90. https://exchange.aaa.com/safety/aaa-school-safety-patrol/
91. https://www.adn.com/voices/article/recalling-alaskas-most-notorious-drunken-moose-street-smart-buzzwinkle/2013/09/10/
92. https://www.ntsb.gov/investigations/Pages/HWY22MH003.aspx
93. https://www.cnn.com/2019/10/28/us/bus-sinkhole-pittsburgh-trnd/index.html
94. https://www.usatoday.com/videos/news/nation/2012/12/20/1782791/
95. https://www.huffpost.com/entry/oregon-slime-eels-truck-crash_n_596845e1e4b03389bb164e65
    BONUS: The Great Vacaville Tomato Spill and Alfredo Sause is Everywhere -- You Can Smell It!
96. https://en.wikipedia.org/wiki/Heavy-tailed_distribution
97. https://youtu.be/VZpdUL5p4a4
98. --
99. https://www.washingtonpost.com/local/trafficandcommuting/a-traffic-incident-leads-to-robot-shuttles-being-ordered-to-stop-hauling-passengers/2020/02/29/db6bec64-5981-11ea-9b35-def5a027d470_story.html
100. --
101. --
102. --
103. --
104. https://en.wikipedia.org/wiki/Jagoff
105. --
106. For a practical illustration, see this short video: https://youtu.be/cOYRGZECuL0
107. https://en.wikipedia.org/wiki/Extreme_Pogo
108. https://99percentinvisible.org/article/red-white-sometimes-blue-how-safety-shaped-the-octagonal-stop-sign/
109. https://xkcd.com/1838/
110. --
111. --
112. --
113. --
114. https://www.cdc.gov/disasters/lightning/victimdata.html
115. --
116. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.455.4506&rep=rep1&type=pdf
117. https://en.wikipedia.org/wiki/ALARP
118. --
119. https://www.hse.gov.uk/managing/theory/alarpglance.htm
120. https://www.weather.gov/safety/lightning-fatalities20
121. https://www.cdc.gov/nchs/fastats/accidental-injury.htm
122. https://visionzeronetwork.org/
123. --
124. https://youtu.be/N9Yq_K0Y9gk
125. https://www.baua.de/EN/Topics/Safe-use-of-chemicals-and-products/Product-safety/Market-surveillance/Rapex.html
126. --
127. https://www.wolframalpha.com/input?i=15*350*30*10000
128. https://en.wikipedia.org/wiki/Automotive_Safety_Integrity_Level#Comparison_with_Other_Hazard_Level_Standards
129. https://youtu.be/gDsolNq2sEk?t=332
130. https://en.wikipedia.org/wiki/ISO_26262#Part_9:_Automotive_Safety_Integrity_Level_(ASIL)-oriented_and_safety-oriented_analysis
131. --
132. --
133. --
134. --
135. --
136. https://www.automotivesafetycouncil.org/wp-content/uploads/2020/12/2019-Fatality-Report-Overview-of-Motor-Vehicle-Crashes-in-2019.pdf
137. https://www.iihs.org/topics/fatality-statistics/detail/state-by-state%C2%A0
138. https://engrxiv.org/preprint/view/1973/3986
139. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/813266
140. https://usa.streetsblog.org/2016/05/31/3-graphs-that-explain-why-20-mph-should-be-the-limit-on-city-streets/
141. https://www.wired.com/story/self-driving-car-crashes-rear-endings-why-charts-statistics/
142. --
143. https://www.nytimes.com/2022/03/30/us/pennsylvania-crash-snow-squall.html
144. https://www.bts.gov/content/average-age-automobiles-and-trucks-operation-united-states
145. --
146. https://www.nhtsa.gov/press-releases/nhtsa-announces-update-historic-aeb-commitment-20-automakers
147. https://www.consumerreports.org/car-safety/automatic-emergency-braking-struggle-to-stop-for-pedestrians-a9924685047/
148. https://www.euroncap.com/en/vehicle-safety/the-ratings-explained/vulnerable-road-user-vru-protection/
149. https://www.iihs.org/news/detail/study-shows-front-crash-prevention-works-for-large-trucks-too
150. --
151. --
152. https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries
153. https://www.baltimoresun.com/news/bs-xpm-1996-11-07-1996312009-story.html
154. https://www.wired.com/story/criminologist-uber-crime-report-highly-alarming/
     https://www.washingtonpost.com/technology/2021/10/22/lyft-safety-report/
155. https://static.nhtsa.gov/odi/rcl/2022/RCLRPT-22V037-4462.PDF
156. https://www.npr.org/2022/01/18/1073857310/tesla-autopilot-crash-charges
157. https://www.wired.com/story/cruise-fire-truck-block-san-francisco-autonomous-vehicles/
158. https://www.rand.org/pubs/research_reports/RR2662.html
159. https://www.thestar.com/business/2014/04/07/why_ups_said_no_to_left_turns.html
     https://www.npr.org/2015/05/25/409531218/left-turns-cause-a-quarter-of-all-pedestrian-crashes-in-u-s
160. https://www.mobileye.com/responsibility-sensitive-safety/
161. https://thehill.com/policy/technology/542222-waymo-says-their-autonomous-car-ai-would-avoid-fatal-human-crashes/
162. https://www.nhtsa.gov/sites/nhtsa.gov/files/pre-crash_scenario_typology-final_pdf_version_5-2-07.pdf
163. https://blog.waymo.com/2021/03/replaying-real-life.html
164. https://www.rand.org/pubs/research_reports/RR1478.html
165. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/813266
166. https://reliabilityanalyticstoolkit.appspot.com/mtbf_test_calculator
167. --
168. --
169. --
170. --
171. --
172. https://en.wikipedia.org/wiki/Change_management_(engineering)
173. https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting
174. https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/disengagement-reports/
175. --
176. --
177. --
178. https://ecori.org/2019-8-2-little-roady-shuttle-reaches-milestone-hits-speedbumps/
179. --
180. --
181. https://www.abc.net.au/news/2017-06-24/driverless-cars-in-australia-face-challenge-of-roo-problem/8574816
182. https://www.anthrocon.org/
183. --
184. https://ftw.usatoday.com/2015/04/morgantown-bans-outdoor-furniture-west-virginia-fans-fires
185. https://en.wikipedia.org/wiki/ISO_26262
186. https://www.iso.org/standard/77490.html
187. https://users.ece.cmu.edu/~koopman/ul4600/index.html
188. https://www.sae.org/standards/content/j3018_202012/
189. https://de.wikipedia.org/wiki/ISO/SAE_21434
190. https://www.iso.org/standard/81920.html
191. https://standards.ieee.org/ieee/2846/10831/
192. https://www.sae.org/standards/content/avsc00007202107/
193. --
194. --
195. https://www.senatorlangerholc.com/2022/01/05/langerholc-introduces-legislation-to-create-a-roadmap-for-highly-automated-vehicles/
     https://youtu.be/ewULUNDBZO4
196. --
197. --
198. --
199. https://www.nhtsa.gov/automated-driving-systems/voluntary-safety-self-assessment
200. --
201. https://en.wikipedia.org/wiki/IEC_61508
202. https://webstore.ansi.org/Standards/DOD/MILSTD882E
203. --
204. --
205. https://www.dstan.mod.uk/StanMIS/indexes/DefenceStandards
206. --
207. --
208. --
209. --
210. https://www.nhtsa.gov/laws-regulations/fmvss
211. https://techcrunch.com/2022/03/10/nhtsa-first-autonomous-vehicle-occupant-safety-standards/
212. https://www.nhtsa.gov/ratings
213. https://en.wikipedia.org/wiki/Global_NCAP
214. https://www.marklines.com/en/report_all/rep1657_201712
215. https://en.wikipedia.org/wiki/Risk_management
216. --
217. --
218. https://hbr.org/2021/05/why-start-ups-fail
219. --
220. --
221. --
222. https://www.transportation.gov/office-policy/transportation-policy/revised-departmental-guidance-on-valuation-of-a-statistical-life-in-economic-analysis
223. https://www.beasleyallen.com/wp-content/uploads/toyota-sua-jury-verdict-form-schwarz.pdf
224. --
225. https://www.autosafety.org/wp-content/uploads/import/phpq3mJ7F_FordMemo.pdf
226. https://abcnews.go.com/Blotter/toyota-pay-12b-hiding-deadly-unintended-acceleration/story?id=22972214
     https://www.justice.gov/sites/default/files/opa/legacy/2014/03/19/toyota-def-pros-agr.pdf
227. https://en.wikipedia.org/wiki/General_Motors_ignition_switch_recalls
228. https://safeautonomy.blogspot.com/2022/01/comments-on-pa-sb-965-regulating.html
229. https://en.wikipedia.org/wiki/Captain_Phillips_(film)
230. https://www.ul.com/about/history
231. https://www.iihs.org/about-us
232. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812979
233. https://policyscout.com/auto-insurance/learn/motorcycle-insurance-vs-car-insurance
234. --
235. https://www.iihs.org/topics/advanced-driver-assistance/autonomous-vehicle-laws
236. --
237. https://www.cnbc.com/2017/03/16/anthony-levandowski-ubers-self-driving-car-engineer-safety-lawsuit.html
238. https://en.wikipedia.org/wiki/Safety_culture
239. https://airlines.iata.org/analysis/just-culture-can-improve-safety
240. https://youtu.be/5ythLOzQav0
241. --
242. https://insideevs.com/features/380532/how-to-rebot-tesla-model-3/
243. --
244. https://www.nytimes.com/1999/07/22/us/gm-admits-brake-flaws-after-inquiry.html
245. https://www.theverge.com/tldr/2020/11/3/21547392/ai-camera-operator-football-bald-head-soccer-mistakes
246. https://www.entrepreneur.com/slideshow/289621
247. https://spectrum.ieee.org/ai-failures
248. https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs
249. --
250. https://www.eetimes.com/autonomous-vehicle-myths-the-dirty-dozen/
251. https://ojoyoshidareport.com/u-s-lawyers-perpetuate-regulation-myths-at-the-autonomous-2/
252. --
253. https://www.nhtsa.gov/sites/nhtsa.gov/files/documents/13069a-ads2.0_090617_v9a_tag.pdf
254. https://safeautonomy.blogspot.com/2018/06/a-reality-check-on-94-percent-human.html
255. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812115
256. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/811059
257. https://crashstats.nhtsa.dot.gov/Api/Public/Publication/813060
258. https://usa.streetsblog.org/2022/01/31/it-aint-94-percent-ntsb-chair-jennifer-homendy-on-the-role-of-human-error-in-car-crashes/
259. https://www.law.cornell.edu/cfr/text/49/571.138
260. --
261. https://thehill.com/opinion/technology/394945-how-to-keep-self-driving-cars-safe-when-no-one-is-watching-for-dashboard/
262. https://www.sae.org/standards/content/j3018_202012/
263. --
264. https://www.nhtsa.gov/sites/nhtsa.gov/files/documents/ads_safety_principles_anprm_website_version.pdf
265. https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting
266. https://capitol.texas.gov/tlodocs/85R/billtext/pdf/SB02205F.pdf
267. https://safeautonomy.blogspot.com/2022/02/kansas-av-regulation-bill-hearings.html
268. https://rules.cityofnewyork.us/rule/autonomous-self-driving-vehicles/
269. https://www.mass.gov/doc/memorandum-of-agreement-to-test-automated-driving-systems-on-public-ways-in-massachusetts/download
270. https://www.cnn.com/2022/02/13/business/tesla-recall-boombox/index.html
271. --
272. --
273. https://static.nhtsa.gov/odi/inv/2021/INOA-PE21020-1893.PDF
274. https://www.nytimes.com/2014/09/12/business/air-bag-flaw-long-known-led-to-recalls.html
275. https://en.wikipedia.org/wiki/The_Treasure_of_the_Sierra_Madre_(film)#John_Huston's_screenplay
276. https://www.uniformlaws.org/committees/community-home?communitykey=4e70cf8e-a3f4-4c55-9d27-fb3e2ab241d6
277. https://www.uniformlaws.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=feaf72ec-473e-5a78-0621-a5119a10eec4
278. https://uclajolt.com/autonomous-vehicle-regulation-trust-the-impact-of-failures-to-comply-with-standards/
279. --
280. --
281. https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/disengagement-reports/
282. --
283. https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting-levels-driving-automation-2-5
284. --
285. https://www.rand.org/pubs/research_reports/RR2150.html
286. --
287. https://www.forbes.com/sites/niallmccarthy/2016/08/11/the-worst-countries-in-the-world-for-drunk-driving-infographic/
288. --
289. --
290. https://youtu.be/mSC4Fr3wf0k?t=178
291. --
292. https://en.wikipedia.org/wiki/Type_approval#Automotive_industry
293. https://safeautonomy.blogspot.com/p/safe-autonomy.html
294. --
295. https://transportation.house.gov/imo/media/doc/TI%20Preliminary%20Investigative%20Findings%20Boeing%20737%20MAX%20March%202020.pdf
296. https://doi.org/10.2777/035239
297. --
298. --
299. --
300. https://crashstats.nhtsa.dot.gov/Api/Public/Publication/813060
301. https://injuryfacts.nsc.org/motor-vehicle/overview/preliminary-estimates/
302. --
303. https://www.iihs.org/topics/fatality-statistics/detail/urban-rural-comparison
304. https://www.iihs.org/topics/fatality-statistics/detail/state-by-state#fatal-crash-totals
     https://www.fhwa.dot.gov/policyinformation/statistics/2019/fi30.cfm
305. https://web.archive.org/web/20210707072124/https://www.iihs.org/topics/fatality-statistics/detail/state-by-state
306. https://aaafoundation.org/rates-motor-vehicle-crashes-injuries-deaths-relation-driver-age-united-states-2014-2015/
307. --
308. --
309. https://morningconsult.com/2021/09/02/autonomous-vehicles-safety-consumer-interest-polling/
310. https://player.fm/series/no-parking-podcast/building-trust-and-reliability-in-autonomous-vehicles-with-dr-phillip-koopman
311. https://www.sciencedirect.com/science/article/pii/S0022437518304511
312. https://en.wikipedia.org/wiki/Algorithm_aversion
313. --
314. https://www.sciencedirect.com/science/article/abs/pii/S0968090X21000942
315. https://en.wikipedia.org/wiki/Risk#Dread_risk
316. https://data.ntsb.gov/Docket?ProjectID=96894
317. --
318. https://uclajolt.com/autonomous-vehicle-regulation-trust-the-impact-of-failures-to-comply-with-standards/
319. --
320. https://users.ece.cmu.edu/~koopman/lectures/ece649/19_dependability.pdf
321. --
322. https://www.rand.org/pubs/research_reports/RR2150.html
323. https://en.wikipedia.org/wiki/Factor_of_safety
     https://en.wikipedia.org/wiki/Derating
     https://en.wikipedia.org/wiki/N-version_programming
324. --
325. --
326. https://en.wikipedia.org/wiki/Goodhart%27s_law
327. https://users.ece.cmu.edu/~koopman/lectures/index.html#podcasts
328. https://www.rand.org/pubs/research_reports/RR2662.html
329. --
330. --
331. --
332. --
333. https://en.wikipedia.org/wiki/List_of_bridges_of_Pittsburgh
334. https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/disengagement-reports/
335. --
336. https://www.theverge.com/2020/2/26/21142685/california-dmv-self-driving-car-disengagement-report-data
337. --
338. --
339. https://www.theverge.com/2022/8/4/23288794/tusimple-self-driving-truck-crash-investigation
340. https://www.goodreads.com/quotes/98951-here-s-a-phrase-that-apparently-the-airlines-simply-made-up
341. --
342. --
343. --
344. https://betterembsw.blogspot.com/2014/09/go-beyond-system-functional-testing-to.html
345. --
346. https://en.wikipedia.org/wiki/All_models_are_wrong
347. --
348. https://arxiv.org/abs/2108.02403
349. https://www.mobileye.com/technology/responsibility-sensitive-safety/
350. https://www.nvidia.com/en-us/self-driving-cars/safety-force-field/
351. https://arxiv.org/abs/2005.09999
352. https://standards.ieee.org/ieee/2846/10831/
353. https://en.wikipedia.org/wiki/Newton's_laws_of_motion
354. https://arxiv.org/abs/1911.01207
355. --
356. --
357. --
358. --
359. https://hal.archives-ouvertes.fr/hal-01207152/document
360. --
361. --
362. https://www.pegasusprojekt.de/en/pegasus-method
363. https://www.asam.net/standards/detail/openscenario/
364. --
365. https://en.wikipedia.org/wiki/Receiver_operating_characteristic
366. https://en.wikipedia.org/wiki/Confusion_matrix
367. --
368. https://www.theverge.com/2018/11/22/18107885/china-facial-recognition-mistaken-jaywalker
369. https://www.hotcars.com/20-weirdest-things-people-painted-on-roads-to-slow-down-speeding-cars/
370. https://www.dispatch.com/story/news/2018/09/04/road-gator-can-bite/10845128007/
371. --
372. --
373. --
374. --
375. https://en.wikipedia.org/wiki/Object_permanence
376. https://calvinandhobbes.fandom.com/wiki/Transmogrifier
377. --
378. https://www.wesa.fm/development-transportation/2020-07-28/how-the-pittsburgh-left-became-embedded-in-city-driving
379. https://www.hpl.hp.com/techreports/tandem/TR-96.1.pdf
380. --
381. --
382. --
383. --
384. --
385. --
386. https://www.autoevolution.com/news/tesla-safety-score-beta-shows-its-impact-and-raises-doubts-a-few-hours-after-release-170309.html
387. https://xkcd.com/552
388. https://www.icao.int/nacc/documents/meetings/2014/sspsmsant/annex19.pdf
389. --
390. https://edmonton.ctvnews.ca/it-sounded-like-a-sonic-boom-couple-describes-lightning-strike-while-on-highway-1.1855556
391. --
392. --
393. https://www.sae.org/standards/content/avsc00006202103/
394. --
395. --
396. --
397. --
398. --
399. --
400. https://en.wikipedia.org/wiki/Normalization_of_deviance
401. https://www.dstan.mod.uk/StanMIS/
402. https://scsc.uk/gsn
403. https://youtu.be/iDLSNsIzDHo
404. --
405. --
406. --
407. https://en.wikipedia.org/wiki/Chewbacca_defense
408. --
409. https://scsc.uk/scsc-141C
410. https://www.omg.org/spec/SACM/
411. --
412. --
413. --
414. https://www4.icao.int/demo/SMI/9.5.4%20SM%20ICG%20Measuring%20Safety%20Performance%20Guidance%20for%20Service%20Providers.pdf
415. https://safeautonomy.blogspot.com/2021/06/a-more-precise-definition-of-ansiul.html
416. --
417. https://youtu.be/1hWYd2vizSQ
418. https://en.wikipedia.org/wiki/Performance_indicator
419. --
420. --
421. https://dl.acm.org/doi/10.1145/390016.808467
422. --
423. https://en.wikipedia.org/wiki/Falsifiability
424. --
425. https://en.wikipedia.org/wiki/Abductive_reasoning
426. https://en.wikipedia.org/wiki/Defeasible_reasoning
427. https://www.researchgate.net/publication/272678149_Eliminative_Argumentation_A_Basis_for_Arguing_Confidence_in_System_Properties
428. https://plato.stanford.edu/entries/reasoning-defeasible/
429. http://sunnyday.mit.edu/safer-world/safety-case.doc
430. https://scsc.uk/r159:1
431. --
432. --
433. --
434. https://arxiv.org/pdf/1902.11097.pdf
435. --
436. --
437. --
438. --
439. --
440. --
441. --
442. --
443. --
444. --
445. https://en.wikipedia.org/wiki/Software_quality_assurance
446. https://www.misra.org.uk/misra-c/
447. --
448. --
449. https://www4.icao.int/demo/SMI/9.5.4%20SM%20ICG%20Measuring%20Safety%20Performance%20Guidance%20for%20Service%20Providers.pdf
450. --
451. --
452. --
453. https://youtu.be/_wxlgbl4s-g
454. --
455. --
456. https://betterembsw.blogspot.com/2014/05/proper-watchdog-timer-use.html
457. --
458. --
459. https://www.youtube.com/results?search_query=car+sliding+hill+ice
460. https://en.wikipedia.org/wiki/Metric_temporal_logic
461. https://users.ece.cmu.edu/~koopman/pubs/kane15_monitoring.pdf
462. https://en.wikipedia.org/wiki/Synchronization_(computer_science)
463. https://en.wikipedia.org/wiki/Clock_synchronization
     https://www.microsoft.com/en-us/research/video/the-time-triggered-architecture/
464. https://users.ece.cmu.edu/~koopman/pubs/kane14_dsn_cpsmonitor.pdf
     https://users.ece.cmu.edu/~koopman/thesis/kane.pdf
465. https://en.wikipedia.org/wiki/Root_cause_analysis#General_principles
466. --
467. --
468. --
469. --
470. --
471. --
472. https://l7.curtisnorthcutt.com/label-errors
473. --
474. --
475. --
476. --
477. --
478. --
479. --
480. --
481. --
482. --
483. --
484. --
485. --
486. https://en.wikipedia.org/wiki/Five_whys
     https://en.wikipedia.org/wiki/Why%E2%80%93because_analysis
487. https://en.wikipedia.org/wiki/Root_cause_analysis
488. --
489. https://en.wikipedia.org/wiki/Bootstrapping
490. https://arxiv.org/abs/2110.10718
491. --
492. --
493. --
494. https://reliabilityanalyticstoolkit.appspot.com/mtbf_test_calculator
495. https://en.wikipedia.org/wiki/Normalization_of_deviance
496. --
497. https://en.wikipedia.org/wiki/The_Black_Swan:_The_Impact_of_the_Highly_Improbable
498. --
499. https://en.wikipedia.org/wiki/Russian_roulette
500. https://www.wolframalpha.com/input?i=1%2F%28%281-%281%2F6%29%29**100%29
501. https://www.wolframalpha.com/input?i=%281-%281%2F60%29%29**100
502. --
503. --
504. --
505. --
506. --
507. https://en.wikipedia.org/wiki/Bayesian_probability
508. --
509. --
510. --
511. --
512. --
513. --
514. https://users.ece.cmu.edu/~koopman/thesis/hutchison.pdf
515. https://youtu.be/g9wOyuiPuBI
516. https://www.youtube.com/watch?v=HSxSgbNQi-g
517. --
518. --
519. --
520. --
521. https://en.wikipedia.org/wiki/Reinsurance
522. https://en.wikipedia.org/wiki/Probabilistic_risk_assessment
523. https://en.wikipedia.org/wiki/Fukushima_nuclear_disaster
524. https://en.wikipedia.org/wiki/Alea_iacta_est
525. https://en.wikipedia.org/wiki/Uncertainty_quantification
526. https://commons.wikimedia.org/wiki/Dice_by_number_of_sides
527. https://www.npr.org/2021/06/09/1004684932/fastly-tuesday-internet-outage-down-was-caused-by-one-customer-changing-setting
528. --
529. --
530. https://en.wikipedia.org/wiki/Turtles_all_the_way_down
531. --
532. --
533. --
534. --
535. --
536. https://en.wikipedia.org/wiki/Change_impact_analysis
537. https://en.wikipedia.org/wiki/Configuration_management
     https://www.sae.org/standards/content/eia649c
538. --
539. https://www.ntsb.gov/Advocacy/safety-topics/Pages/automated-vehicles-investigative-outcomes.aspx
540. https://users.ece.cmu.edu/~koopman/pubs/koopman19_TestingSafetyCase_SAEWCX.pdf
541. https://www.eetimes.com/busting-myths-of-driver-monitoring-systems/
     https://www.embedded.com/automotive-focus-shifts-to-driver-monitoring-systems/
542. https://www.sae.org/standards/content/j3018_202012/
     https://www.sae.org/standards/content/avsc00001201911/
543. https://www.sae.org/standards/content/avsc00007202107/
544. --
545. https://www.youtube.com/watch?v=0cT5cXuyiYY
546. https://www.postindependent.com/news/local/the-goof-behind-losing-911-service-in-mays-big-outage/
547. --
548. --
549. https://en.wikipedia.org/wiki/Kill_switch
550. --
551. https://www.wsj.com/articles/self-driving-truck-accident-draws-attention-to-safety-at-tusimple-11659346202
552. --
553. https://www.ntsb.gov/investigations/AccidentReports/Reports/HAB1906.pdf
554. --
555. https://www.edmunds.com/car-news/the-wrong-thing-at-the-wrong-time-tesla-full-self-driving-disclaimer.html
556. --
557. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3931341
558. https://techcrunch.com/2022/06/30/cruise-robotaxis-blocked-traffic-for-hours-on-this-san-francisco-street/
559. --
560. --
561. https://www.smbc-comics.com/comic/self-driving-car-ethics
562. https://ieeexplore.ieee.org/document/9622307
     https://en.wikipedia.org/wiki/Trolley_problem#Original_dilemma
563. --
564. --
565. https://www.bmvi.de/SharedDocs/EN/publications/report-ethics-commission.html
566. --
567. --
568. https://www.theverge.com/2022/8/20/23314117/youtube-tesla-removes-video-full-self-driving-beta-real-kids
569. --
570. --
571. --
572. --
573. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236
574. --
575. --
576. https://users.ece.cmu.edu/~koopman/pubs/koopman18_safecomp.pdf
577. https://www.bbc.com/news/magazine-26073797
578. https://www.wired.com/story/uber-self-driving-car-fatal-crash/
579. https://www.npr.org/2022/01/18/1073857310/tesla-autopilot-crash-charges
580. --
581. --
582. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3969214
583. --
584. --
585. https://idioms.thefreedictionary.com/no+harm%2c+no+foul
586. --
587. --
588. --
589. https://onlineethics.org/cases/telecon-meeting
590. https://usa.streetsblog.org/2022/04/07/road-deaths-rise-again-as-post-pandemic-period-proves-particularly-perilous-to-pedestrians/
591. https://static.nhtsa.gov/odi/inv/2021/INOA-PE21020-1893.PDF
592. https://en.wikipedia.org/wiki/I_Can't_Drive_55
593. https://usa.streetsblog.org/2017/07/27/ntsb-speed-kills-and-were-not-doing-enough-to-stop-it/
594. --
595. --
596. https://static.nhtsa.gov/odi/rcl/2022/RCLRPT-22V037-4462.PDF
597. https://www.jurist.org/commentary/2021/09/william-widen-philip-koopman-autonomous-vehicles/
598. https://en.wikipedia.org/wiki/Whataboutism
599. https://en.wikipedia.org/wiki/Whac-A-Mole
600. https://www.npr.org/2022/02/01/1077274384/tesla-recalls-autos-over-software-that-allows-them-to-roll-through-stop-signs
601. https://www.youtube.com/watch?v=ogGBwrrkKY4
602. https://www.insideedition.com/how-drive-through-fire-48422
603. --
604. https://www.thedrive.com/news/39184/americas-rarest-drivers-license-lets-14-year-olds-hit-the-road-legally
605. --
606. https://guinnessworldrecords.com/world-records/most-people-crammed-into-an-old-vw-beetle/
607. --
608. https://www.motortrend.com/reviews/2022-rivian-r1t-secret-software-mode-yearlong-review/
609. https://darwinawards.com/darwin/
610. https://www.transportation.gov/office-policy/transportation-policy/revised-departmental-guidance-on-valuation-of-a-statistical-life-in-economic-analysis
611. https://carbuzz.com/news/gm-loses-a-staggering-5-5-million-per-day-on-robotaxis
612. --
613. https://sma.nasa.gov/docs/default-source/safety-messages/safetymessage-2008-10-01-lossofpetrobrasp36-vits.pdf
614. https://safeautonomy.blogspot.com/2022/02/kansas-av-regulation-bill-hearings.html
615. --
616. --
617. --
618. https://insurance.kansas.gov/auto-insurance/
619. --
620. https://qz.com/798092/a-self-driving-uber-car-went-the-wrong-way-on-a-one-way-street-in-pittsburgh/
621. https://usa.streetsblog.org/2014/12/19/heres-how-much-safer-transit-is-compared-to-driving/
622. https://usa.streetsblog.org/2020/03/02/the-broken-promises-of-the-ride-share-revolution/ --
623. --
624. https://www.nsc.org/road/future-of-mobility
625. --
626. https://www.nhtsa.gov/document/automated-driving-systems-20-voluntary-guidance
627. https://safeautonomy.blogspot.com/2022/02/five-principles-for-regulation-of.html
628. --
629. https://www.transportation.gov/office-policy/transportation-policy/revised-departmental-guidance-on-valuation-of-a-statistical-life-in-economic-analysis
630. --
631. https://ethicsinaction.ieee.org/p7000/
632. https://www.lawcom.gov.uk/project/automated-vehicles/
633. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai
634. https://www.whitehouse.gov/environmentaljustice/justice40/
635. https://en.wikiquote.org/wiki/F._E._Smith,_1st_Earl_of_Birkenhead

Last updated: 25 August 2022