Advertisement
Advertisement
Self-driving cars and autonomous vehicles
Get more with myNEWS
A personalised news feed of stories that matter to you
Learn more
Baidu launched its robotaxi services in Changsha, capital of Hunan province, with a fleet of 45 cars on  September 26, 2019. (Picture: Baidu)

Did Baidu really do better than Google’s Waymo in California’s self-driving report?

Chinese self-driving pioneer has a lower disengagement rate than the Google spinoff, but experts say the numbers don’t mean much

This article originally appeared on ABACUS
Last year in California, safety drivers in Waymo’s autonomous vehicles took over the wheel once every 13,000 miles. Baidu, on the other hand, recorded human intervention in its autonomous cars only once every 18,000 miles. That’s according to annual figures the companies submitted to the state’s Department of Motor Vehicles.
In China, some media outlets were quick to latch on to the apparent triumph. Baidu, the country’s self-driving pioneer, has long been seen as lagging behind Waymo – the Google spinoff that’s often considered a global leader in autonomous driving. The disengagement rate is “one of the best measurements of how advanced a company’s self-driving program is,” claimed one Chinese news site.

But industry experts say it’s not. 

Baidu launched its robotaxi services in Changsha, capital of Hunan province, with a fleet of 45 cars on  September 26, 2019. (Picture: Baidu)

“The disengagement metric is essentially worthless and does not really tell anything about the performance of an automated driving system,” Sam Abuelsamid, an analyst at consultancy Navigant, told us. “While Baidu's results are indeed impressive this year, they really don't mean very much.”

For one, there is no standard for when a driver should disengage: Each company is free to decide on its own. The rate of disengagement also depends largely on where the vehicle was travelling.

“If the company does a lot of testing in areas with little traffic or other obstacles, there may be fewer disengagements than a company testing in a dense urban environment like San Francisco,” Abuelsamid explained.

It’s a challenge that researchers understand well. When startup Pony.ai moved its headquarters from Silicon Valley to southern China, co-founder James Peng said the company had to contend with new challenges like rain and jaywalkers.
A Waymo self-driving car cruising on a street in Silicon Valley. (Picture: Shutterstock)
Waymo also cautions against using disengagement numbers to compare its performance with rivals, saying that most of its real-world driving wasn’t done in California, but in Phoenix, Arizona, where it’s been trialling a robotaxi service.

Baidu’s self-driving arm, Apollo, agrees that using disengagement rates for comparison across companies is “not that meaningful”. It says the data is better used for examining a company’s performance over time. Baidu has gradually improved its testing scale and capability without any reported accidents, Apollo said.

The DMV’s figures reveal that Baidu drove some 108,000 miles (174,000 km) in California last year, a considerable jump from 18,000 miles (29,000km) in 2018 – though it’s still far less than Waymo’s 1.45 million miles (2.3 million km) in 2019. But the California tests also appear to be just a small portion of Baidu’s overall automated driving experience. Chinese reports say the company has driven more than 2 million kilometres (1.24 million miles) to date in 13 cities.

China's robotaxis are high-tech but overly 'considerate'

Companies’ performances aren’t comparable across borders since testing criteria vary in different countries.

“In China, autonomous vehicle companies actually have to pass on-track performance evaluations before they can get a permit for on-road testing, something that does not happen in the US,” said Abuelsamid. 

Last year, Beijing granted Baidu official permits to test-drive 40 autonomous vehicles with passengers in a designated area. All the cars had logged at least 62,000 miles (100.000km). The Apollo team tells us they’re now preparing for a large-scale implementation of autonomous vehicles without safety drivers.

Even California’s DMV advises against treating its annual report as a scoreboard.

“[The reports] are not intended to compare one company with another or reach broad conclusions on technological capabilities,” said Marty Greenstein, a DMV spokesman. He added that companies aren’t required to report testing done on private roads, test tracks, out-of-state locations or virtual simulations.

Purchase the China AI Report 2020 brought to you by SCMP Research and enjoy a 20% discount (original price US$400). This 60-page all new intelligence report gives you first-hand insights and analysis into the latest industry developments and intelligence about China AI. Get exclusive access to our webinars for continuous learning, and interact with China AI executives in live Q&A. Offer valid until 31 March 2020.

Post