Cars That Watch Their Drivers Could Reteach the World to Drive

To get their cars to stay in lane or follow traffic by themselves, automakers are adding more sophisticated and pricey sensors and software to their vehicles. But not all of them are pointed at the road.

Systems like Tesla Autopilot, GM’s Super Cruise, and Nissan’s ProPilot Assist can automatically follow traffic and even drive hands-free on highways but need drivers to be ready to take over from the supposedly smart software at any moment. To guarantee drivers aren’t napping, Snapchatting, or Candy Crushing when they’re using these advanced driver-assistance features, automakers have rolled out a second layer of technology called driver-monitoring systems. Their job is to look for signs that a human—notoriously unpredictable as they are—is indeed ready to take over when those limited automated-driving features get out of their depth.

“Humans are very poor in general at constantly supervising a process where they only have to intervene very periodically,” says Matt Lum, an automotive technical engineer with the American Automobile Association who studies these partially automated systems.

As more of those driver-facing sensors make their way inside vehicles, industry experts expect the implications to go well beyond their role in automated driving features. Through alerts, alarms, and nudges they could retrain the fatigued, smartphone-fiddling, infotainment system-scrolling drivers of 21st century highways to make them safer—or at least force them to drive differently to avoid nagging from a digital overseer.

Today, these systems are relatively simple, but researchers say they may eventually combine information from internal and external cameras, radars, and lidars to decide when drivers need extra nannying. Driving, in other words, could become almost a different task entirely—a collaboration with a complex machine that expects you to behave in certain ways and isn’t shy to tell you when you’re not playing your part.

Driver-monitoring systems are making their way into vehicles not just because automakers need them to enable automated wizardry but because they’re facing increasing pressure from regulators and safety groups in Europe and the US.

All new car models sold in Europe will have to have driver-monitoring systems by mid-decade, even if they don’t come with partially automated features. And new vehicles sold in Europe with advanced driver-assistance features already have to include driver-monitoring systems to receive a coveted five-star safety rating from the European New Car Assessment Programme, a government-funded vehicle safety organization. “We want to get those systems into all the vehicles in Europe quicker than regulation will do,” says Richard Schram, the technical director of the group. The US infrastructure bill, passed in 2021, requires the Department of Transportation to conduct research on the effectiveness of driver-monitoring systems, potentially a first step toward requiring them in new vehicles.

In the US, federal data shows that distracted and drowsy driving was responsible for less than 10 percent of the 38,824 highway deaths in 2020, the last year with complete data. But safety groups argue that the share caused by distraction is much higher, because federal crash data often depends on observations at the scenes of crashes.

In most cases, driver-monitoring systems use an infrared camera or cameras on a car’s dash, steering wheel, or rearview mirror. Instead of streaming imagery to the cloud like a cellphone, they process imagery using software inside the car to track head, eye, or hand movements. Those systems have been trained on millions of images of real and synthetic people behind the wheel, their makers say, and can determine whether a driver is looking at the road ahead—or at the cellphone on their lap.

Depending on the automaker, a system that senses that the driver isn’t paying enough attention when using an automated driving feature might flash a warning, change the color of internal lights on the dashboard or steering wheel, sound an alert,  buzz the steering wheel, or some combination of all those tactics. If the driver doesn’t respond, the system will trigger the car to slowly come to a stop.

Gabi Zijderveld, the chief marketing officer of Smart Eye, a Swedish company that sells eye-tracking systems to automotive suppliers, calls the driver monitoring in cars today “fairly simplistic.” The company says its driver-monitoring products are in more than 1 million vehicles on the road and will be in Volvo’s upcoming luxury electric Polestar 3. But in the future, a car equipped with a driver-monitoring system might be able to pull off more nuanced feats, Zijderveld says.

Data from cars could be used to teach more sophisticated algorithms to recognize when a person is, say, safely scrolling through the car’s infotainment system during an easy drive on a sunny four-lane highway—or riskily fiddling with a playlist while navigating city streets in the snow. Alerts might sound or flash or buzz in the latter situation but not the former.

When a driver is trying to multitask, researchers who study the psychology and mechanics of driving tend to evaluate their distraction based on whether their eyes are coming back to the road often, and for long enough, to reestablish a sense for where their car and other vehicles, cyclists, and pedestrians are in space. Driver-monitoring systems may eventually be able to combine information from the car’s many sensors to, for example, determine that a driver isn’t sufficiently paying attention when their vehicle is about to be T-boned, and tighten their seat belt.

Drivers who have already gotten on the wrong side of existing driver-monitoring systems know that their warnings and wailing can be annoying, and they sometimes cry wolf. Automotive engineers choosing when the systems buzz or beep, and how, have to strike a tricky balance.

Experts say the key to building great driver-monitoring systems is to create software that doesn’t just tell a driver when they’re doing something wrong but supports their attention. “It's about defensive driving and avoiding conflict altogether versus avoiding crashes once you're in a conflict,” says Greg Fitch, head of safety research at Android Auto, Google’s in-vehicle app. That could mean sounding quiet but escalating tones, not a high-pitched beep, when it sees you looking off to the side—when you may be watching for pedestrians. Perhaps the system doesn’t totally disengage automatic lane-keeping when you use the wheel to hug the side of a lane but instead shares control. 

Driver-monitoring systems are too new for there to be exhaustively proven rules for what work. How best to support flawed, easily distractible people while they’re driving is still being debated by regulators, automakers, and academics. The answers may vary for different people, companies, or cultures. “Especially in Asian countries, they're a bit reluctant to deploy loud notifications, because you might be driving other people in the car,” says Zijderveld of Smart Eye. “If an alarm goes off, other people will hear that, and they will assume you're a bad driver.” 

Some safety experts argue that much of what automakers are putting out right now isn’t good enough. Last year, the US safety group Consumer Reports began to give extra safety points to vehicles with partially automated features that also have effective driver-monitoring systems.

CR’s testing found that some automakers’ systems allow vehicles to travel for up to 30 seconds—half a mile at 60 mph—without the driver’s hand touching the wheel. (Its most recent testing gave top honors to Ford’s BlueCruise, which the company says can change lanes, reposition a car within a lane, and drive hands-free on 130,000 miles of North American highway—and includes an eye-tracking camera that alerts when a driver stops paying attention to the road for just a few seconds.)

The US-based Insurance Institute for Highway Safety last year created a “safeguard list” of technology to include in driver-monitoring systems, including monitoring both driver gaze and hand position and using quickly escalating alerts to regain drivers’ attention if they don’t respond to an initial warning. (The group has yet to release its first set of ratings.) How a system interacts with a driver can be more crucial—and difficult—than how it monitors them. “You have the vehicle monitoring the driver and verifying that they're doing what they’re supposed to be doing. But so what?” says Alexandra Mueller, a senior research scientist with the group. “It’s about how the vehicle responds to the information.”

Still, people who study how new car technology interacts with humans say those little driver-facing cameras have big potential and could help make all drivers safer. “A lot of people don’t know they’re bad drivers, unless maybe they get honked at or get a ticket or get in a car crash,” says Greg Neiswander, research and testing lead for Android Auto. If driver-monitoring technology is able to give people the right sort of feedback at the right time—a momentous task, it turns out—“you can theoretically make people more aware drivers, better drivers,” he says.

In the future, “the question is not, can we monitor the driver? but can we support them more efficiently?” says Bryan Reimer, a research scientist at MIT who leads the Advanced Vehicle Technology consortium, which brings together industry experts and academics to study how people interact with automated driving features. Implemented the right way, the systems could not just stop people from doing bad stuff behind the wheel but help them to be actively good.