Improving Driving Experience and Safety Through Human-Car Interaction with Cognitive AI

Improving Driving Experience and Safety Through Human-Car Interaction with Cognitive AI

3.8.2020 | Author: Martti Perälä, Senior Manager, Bittium

“Welcome Mary" flashes on the IVI display as Mary enters the vehicle. The car's face detection system has recognized her in real time and the vehicle system settings are automatically aligned with her preferences. The seat and steering wheel are adjusted, the cluster panel view and in-vehicle infotainment systems (IVIs) are personalized, showing her favorite features on the front page and her personal accounts are enabled while data privacy is ensured through a secured backend connection.

What might have sounded like science fiction only a few years ago has already become partially reality with biometrical driver recognition systems. The opportunities for additional applications however open a growth market with a huge potential for car system manufacturers and OEMs.

High Market Potential Comes with New Challenges for OEMs

However, this also presents new challenges for OEMs. The development of suitable hardware and software is time-consuming and costly. At the same time, not every system manufacturer can develop skills in all areas necessary for the increasingly interconnected systems. This includes tamper-proof hardware as well as certified security solutions, device and application management, cloud backend, and secure VPN on the software side. In addition, there are specialized applications such as tools for biometric identification, face analysis or video evaluation.

Manufacturers should therefore carefully consider which area they are concentrating on within their core competencies. Working with development partners who have specialized knowledge in areas such as secure connectivity, application management or offer preconfigured toolkits that are optimized for different platforms, can accelerate the development of new solutions enormously and save costs.

IVI Reference Design made by Bittium

At Bittium we provide intelligent and secure software solutions and professional engineering services for the Automotive market. During CAR-HMI Europe 2020 I presented a reference system design featuring driver recognition and emotion tracking which was built to provide an adaptive user experience (the presentation is available here). With driver monitoring and adaptive UI (user interface), the car IVI and cluster systems adapt according to the driver´s behavior and needs to reduce unnecessary distractions and to improve road safety.

Bittium reference design utilizes a Face Analysis Toolkit by Valossa, which combines state-of-the-art face recognition, real-time person identity training and facial analysis into a simple-to-use video analysis package. The system monitors the driver´s emotional state as well as the person´s viewing direction.  

Situation-Based Ad Hoc Interaction

Facial analysis enables the system to recognize when a driver is distracted or stressed by a traffic situation. If the driver is distracted, the system can redirect his attention to important information. If, on the other hand, the driver is in a critical traffic situation that requires his full attention, the infotainment system (IVI) can be switched to a distraction-free "focus mode". The system then only shows the information that is absolutely necessary and relevant to the situation. When the situation – and the driver's face – have relaxed, the system switches back to the personalized display selection.

Since infotainment systems now display more and more information – from radio stations, song titles and artists to the most used apps, the current weather, etc. – they could easily become a source of distraction by themselves in certain situations. Cognitive systems counteract this danger by implementing situation-dependent modes as described. Systems can also switch to an energy-saving "sleep mode" when the driver is not looking at the display. The system recognizes when the driver is paying attention to the display by tracking his line of vision and immediately reactivates all the information to be displayed without the driver having to take his hands off the steering wheel.

Sounds interesting? Take a look at this video showing examples of the reference system in different situations and of the interface in active and focus mode. It is of course fully customizable.

How Does This Work?

Bittium used Rightware's Kanzi UI tools to design the digital cluster and infotainment reference design system which runs on Android™ Automotive OS. The system was enhanced by utilizing an AI-based face analysis toolkit by Valossa and by developing an adaptive HMI design to support the functionality.

Bittium integrated the toolkit into the reference design and added an adaptive HMI design to support the functionality. Communication works with the toolkit software through the Android VHAL (Vehicle Hardware Abstraction Layer).

The Valossa solution uses deep learning software which uses camera image input and then detects and tracks all individual faces. It recognizes persons, their gender and can analyze their mood and behavior. It also provides a feature for real-time training of facial identities. The toolkit can be set to remember any person appearing in the camera without a separate training process, which makes it suitable for dynamic personalization tasks. The key benefit however is that it provides simple-to-use functionality to teach, recognize and interpret facial-based behavior.

Learn more about our cooperation with Rightware and how OEMs can profit from the joint expertise in previous blog post.