Last year in California, safety drivers in Waymo’s autonomous vehicles took over the wheel once every 13,000 miles. Baidu, on the other hand, recorded human intervention in its autonomous cars only once every 18,000 miles. That’s according to annual figures the companies submitted to the state’s Department of Motor Vehicles.

In China, some media outlets were quick to latch on to the apparent triumph. Baidu, the country’s self-driving pioneer, has long been seen as lagging behind Waymo -- the Google spinoff that’s often considered a global leader in autonomous driving. The disengagement rate is “one of the best measurements of how advanced a company’s self-driving program is,” claimed one Chinese news site.

But industry experts say it’s not. 

Baidu launched its robotaxi services in Changsha, capital of Hunan province, with a fleet of 45 cars on  September 26, 2019. (Picture: Baidu)

“The disengagement metric is essentially worthless and does not really tell anything about the performance of an automated driving system,” Sam Abuelsamid, an analyst at consultancy Navigant, told us. “While Baidu's results are indeed impressive this year, they really don't mean very much.”

For one, there is no standard for when a driver should disengage: Each company is free to decide on its own. The rate of disengagement also depends largely on where the vehicle was traveling.

“If the company does a lot of testing in areas with little traffic or other obstacles, there may be fewer disengagements than a company testing in a dense urban environment like San Francisco,” Abuelsamid explained.

It’s a challenge that researchers understand well. When startup Pony.ai moved its headquarters from Silicon Valley to southern China, co-founder James Peng said the company had to contend with new challenges like rain and jaywalkers.

A Waymo self-driving car cruising on a street in Silicon Valley. (Picture: Shutterstock)

Waymo also cautions against using disengagement numbers to compare its performance with rivals, saying that most of its real-world driving wasn’t done in California, but in Phoenix, Arizona, where it’s been trialling a robotaxi service.

Baidu’s self-driving arm Apollo agrees that using disengagement rates for comparison across companies is “not that meaningful.” It says the data is better used for examining a company’s performance over time. Baidu has gradually improved its testing scale and capability without any reported accidents, Apollo said.

The DMV’s figures reveal that Baidu drove some 108,000 miles in California last year, a considerable jump from 18,000 miles in 2018 -- though it’s still far less than Waymo’s 1.45 million miles in 2019. But the California tests also appear to be just a small portion of Baidu’s overall automated driving experience. Chinese reports say the company has driven more than 2 million kilometers (1.24 million miles) to date in 13 cities.

Companies’ performances aren’t comparable across borders since testing criteria vary in different countries.

“In China, autonomous vehicle companies actually have to pass on-track performance evaluations before they can get a permit for on-road testing, something that does not happen in the US,” said Abuelsamid. 

Last year, Beijing granted Baidu official permits to test-drive 40 autonomous vehicles with passengers in a designated area. All the cars had logged at least 62,000 miles. The Apollo team tells us they’re now preparing for a large-scale implementation of autonomous vehicles without safety drivers.

Even California’s DMV advises against treating its annual report as a scoreboard.

“[The reports] are not intended to compare one company with another or reach broad conclusions on technological capabilities,” said Marty Greenstein, a DMV spokesman. He added that companies aren’t required to report testing done on private roads, test tracks, out-of-state locations or virtual simulations.