Bio-inspired camera system uses AI to help drivers detect pedestrians and obstacles 100x faster

Imagine driving and suddenly, a pedestrian appears in front of your car seemingly out of nowhere. With only a split second to react, the outcome can be disastrous. Modern cars are equipped with camera systems designed to warn drivers or initiate emergency braking in such scenarios.

However, these systems are not yet quick or reliable enough to prevent all accidents, especially in autonomous vehicles where human intervention is not an option.

Researchers Daniel Gehrig and Davide Scaramuzza from the Department of Informatics at the University of Zurich (UZH) have developed a groundbreaking system that detects obstacles faster and with less computational power. Their study, recently published in Nature, details a new approach combining a bio-inspired camera with artificial intelligence (AI) to enhance vehicle safety.

Traditional car cameras are frame-based, capturing snapshots at regular intervals—usually between 30 to 50 frames per second. These cameras rely on artificial neural networks trained to recognize objects such as pedestrians, cyclists, and other vehicles.

However, as Daniel Gehrig explains, "If something happens during the 20 or 30 milliseconds between two snapshots, the camera may see it too late." Increasing the frame rate could solve this issue but would require more data processing and computational power.

Event cameras represent a new technology that operates on a different principle. Instead of capturing constant frame rates, they have smart pixels that record information whenever they detect rapid movement.

"This way, they have no blind spot between frames, allowing them to detect obstacles more quickly," says Davide Scaramuzza, head of the Robotics and Perception Group. These neuromorphic cameras mimic human vision, but they struggle with slow-moving objects and their data isn't easily compatible with AI training algorithms.

Related Stories

To address these challenges, Gehrig and Scaramuzza developed a hybrid system. It uses a standard camera that captures 20 frames per second, which is lower than the typical rate but adequate for their purposes. The images are processed by a convolutional neural network trained to identify cars and pedestrians.

The event camera data feeds into an asynchronous graph neural network, adept at analyzing dynamic 3-D data. This combination allows the event camera to predict and enhance the standard camera's performance. "The result is a visual detector that can detect objects as quickly as a standard camera taking 5,000 images per second but requires the same bandwidth as a 50-frame-per-second camera," explains Gehrig.

The researchers tested their hybrid system against the best automotive cameras and visual algorithms available. Their system proved to be 100 times faster in detection while significantly reducing data transmission and computational power needs without compromising accuracy. The ability to detect cars and pedestrians appearing between frames of the standard camera adds a layer of safety crucial at high speeds.

Looking ahead, the team believes their method can be further enhanced by integrating it with LiDAR sensors, commonly used in self-driving cars. "Hybrid systems like this could be crucial for enabling autonomous driving, ensuring safety without a significant increase in data and computational demands," says Scaramuzza.

By drawing inspiration from human vision and combining two advanced camera types with sophisticated AI, they have created a system that could revolutionize how autonomous vehicles detect and respond to obstacles, potentially saving countless lives on the road.

Note: Materials provided above by Ohio State University. Content may be edited for style and length.

Like these kind of feel good stories? Get the Brighter Side of News' newsletter.