Do AVs really drive safer than humans?

Assume, for a moment, that autonomous vehicles will someday deliver on their potential to travel more safely than their human-driven counterparts.

How would anyone measure the difference?

Aspiring to a better safety record is one thing, but proving that autonomous vehicles outperform humans is another matter -- and there's no agreed-upon way to keep score.

A patent application published last month by the U.S. Patent and Trademark Office offers a possibility. The application details a way to determine the safety-related competence of autonomous vehicles, then benchmarks their performance against that of human-driven autos on the same stretches of road.

"Autonomous vehicles are going to be deployed slowly in gradual ways that are sensitive to operational design domains, and they'll be mixed in with human drivers," Amitai Bin-Nun, the prospective patent holder, told Automotive News Europe sister publication Automotive News. "So there's value in a method that helps you tell how well an AV does compared to a human driver on a given route. So long as we're operating in a system like that, it's a valuable capability."

Far from relying on information provided by sensors such as radar and lidar -- those that make self-driving possible -- the proposed system would use data available from today's vehicles.

Borrowing from the insurance industry, the system would capture events such as hard braking, sharp turns and sudden acceleration, so it could make apples-to-apples comparisons. Of course, crashes would be counted, too. But because self-driving vehicles will need to drive tens of millions of miles before statistically significant amounts of crash data exist, the hope is that other metrics can help.

"Waiting for AVs to crash, it's really hard to accumulate the data needed to calculate whether they crash at rates equal to or less than human drivers," Bin-Nun said. "So if we begin to think about noncrash events that tell us about the quality of driving, we'll have a way to benchmark. … It comes from the idea that we already understand in the context of human driving -- you can ride in [a] car with someone and you realize the person is not a great driver. There are behaviors to tell us that, and we intuitively understand."

Bin-Nun is vice president of autonomous vehicles and mobility innovation at Securing America's Future Energy, a nonpartisan Washington think tank that backs transportation policies that curb energy dependence on oil. A white paper by the organization in June 2018 found that automated vehicles could bring nearly $800 billion in economic benefits by 2050, "mostly from reducing the toll of vehicle crashes."

Bin-Nun, who said his work on the patent is separate from his role at Securing America's Future Energy, filed the application in November 2017 and then expanded on it last November. The patent and trademark office published the application in late May.

Other research on the subject helped shape the idea. In October, a Rand Corp. paper, "Measuring Automated Vehicle Safety: Forging A Framework," suggested that AVs would need leading indicators for measuring safety, not lagging ones such as crashes. It also proposed the idea of "roadmanship," a concept centered on measuring whether a vehicle "plays well with others" in traffic.

Research by Purdue University in 2006 explored the idea of developing these leading indicators for crashes involving human drivers. By measuring the distance between vehicles passing through intersections in Indianapolis, researchers could apply statistics and determine how often a crash would likely occur.

Pairing the broad idea of measuring self-driving system competence with leading metrics, Bin-Nun believes there's a method for identifying correct performance metrics and deriving statistically significant comparisons — whether between self-driving systems and human drivers or among AVs.

Early-stage conventional thinking -- often spouted by regulators -- suggests automakers and tech companies should borrow from the aviation industry and avoid competing on safety-related issues in the autonomous realm. But the advent of a method for making specific comparisons might indirectly invite safety competition. Can a system deliver a smooth ride to passengers without hard braking and herky-jerky motions? Such a system might be considered more mature than its counterparts.

"You're competing on ride quality, and whether the vehicle can accomplish its mission," Bin-Nun said.

"How do you know when you've reached the right point where it is safe enough to deploy? When you can robustly serve the routes in your market of choice -- and safety will be one of the dimensions you need to deliver results on."