California's Department of Motor Vehicles released on Wednesday its annual cache of records related to the testing of self-driving technology on the state's public roads.
The importance of these records is open to interpretation.
Submitted by most of the companies testing autonomous technology, the records contain information on the total miles they have logged on public roads in autonomous mode and the number of times that human safety drivers have needed to intervene.
Yet there are variables that make it difficult to draw meaningful insights.
The records don't make it clear whether companies conduct their testing in complex city environments or on empty highways. They don't delineate whether the companies are logging miles or seeking to test particular aspects of their technology. Nor do they detail exactly what constitutes a disengagement — in the past, companies have interpreted provisions differently. Simply, they can lack context.
"We appreciate what the California DMV was trying to do when creating this requirement, but the disengagement metric does not provide relevant insights into the capabilities of the Waymo Driver or distinguish its performance from others in the self-driving space," Waymo said in a written statement Wednesday.
Waymo drove 1.45 million miles with 153 vehicles on California public roads in 2019. It had a rate of one disengagement per 13,219 miles.
The disengagement reports provide a snapshot glance of certain testing efforts. But they cover only testing that occurs on public roads in California. They don't, for example, offer any information on the bulk of Waymo's testing efforts, which occur in the metro Phoenix area.
Nor do they serve as a barometer for self-driving progress that occurs in simulated environments. Aurora Innovation saw the number of miles logged on California's public roads approximately fall to half of its 2018 number. That may be a cause for alarm, until observers realize the company has placed an emphasis on testing its software in virtual environments rather than maintaining and deploying a large fleet.
"This data has historically been viewed as a report card for self-driving companies, but as we have repeatedly said, using these statistics as a measuring stick is misguided," wrote Chris Urmson, CEO of Aurora. "The self-driving teams I've led over the past 17 years have championed various attitudes toward on-road testing. While huge numbers of on-road miles may initially seem impressive, experience has taught me which approaches merely look like progress and which can actually move the needle toward real progress."
Aurora drove 39,729 miles on California's public roads, and 26,300 of those came in manual mode. Its rate of disengagement was 10.6 per 1,000 miles driven. The company says it increased its virtual testing more than a hundredfold in 2019. It highlights the value of those simulated worlds in the fact it attempted 2.27 million unprotected left turns virtually before it ever tried one in the real world, a milepost it reached in September.
The company says its on-road miles occurred mainly during "targeted data collection in increasingly complex environments."
Which is all not to entirely dismiss the importance of the reports. California is the only state with a requirement that the industry submit such information. From that, industry observers have learned, for instance, that Apple used 70 vehicles in its 2019 testing.
Consider the information in the reports one piece of a complicated puzzle that might provide insights into the state of the industry. In some cases, companies themselves can provide the context needed to make apples-to-apples comparisons.
Cruise says that all of its 233 cars on California's public roads have been testing within San Francisco, which the company expects to be its first commercial market.
Cruise covered nearly twice as many miles in 2019 as it did the previous year, and it saw a 26 percent reduction in reported disengagements. In the first half of 2019, the company saw a rate of one disengagement per 7,635 miles traveled. Over the second half, that increased to one disengagement per 20,110 miles traveled.
Overall, the companies required to file disengagement reports with the California regulators collectively drove nearly 2.9 million miles on the state's public roads. That's an increase from the 2.05 million driven the previous year.
