Video capturing device for predicting special driving situations

• US 10,296,796 B2
• Filed: 04/04/2017
• Issued: 05/21/2019
• Est. Priority Date: 04/06/2016
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Cosatto, Eric, Melvin, Iain, Graf, Hans Peter
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US15/478,886
• Publication Number:
  US 20170293815A1
• Time in Patent Office: 777 Days
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A video device for predicting driving situations while a person drives a car is presented. The video device includes multi-modal sensors and knowledge data for extracting feature maps, a deep neural network trained with training data to recognize real-time traffic scenes (TSs) from a viewpoint of the car, and a user interface (UI) for displaying the real-time TSs. The real-time TSs are compared to predetermined TSs to predict the driving situations. The video device can be a video camera. The video camera can be mounted to a windshield of the car. Alternatively, the video camera can be incorporated into the dashboard or console area of the car. The video camera can calculate speed, velocity, type, and/or position information related to other cars within the real-time TS. The video camera can also include warning indicators, such as light emitting diodes (LEDs) that emit different colors for the different driving situations.

Claims

1. A video device for predicting driving situations while a person drives a car, the video device comprising:

multi-modal sensors and knowledge data for extracting feature maps;

a deep convolutional neural network trained with training data to recognize real-time traffic scenes (TSs) from a viewpoint of the car; and

a user interface (UI) for displaying the real-time TSs and to warn of possible danger, wherein the real-time TSs are compared to predetermined TSs to predict the driving situations,

wherein the training data is labeled semi-automatically by defining a set of constraints on sensory variables for each label, encoding each label into a set of rules, and employing the multi-modal sensors for which all rules are verified and assigned to a corresponding label.

2. The device of claim 1, wherein the video device includes a video camera.

3. The device of claim 2, wherein the video camera mounts to a dashboard of the car.

4. The device of claim 1, wherein the video device is incorporated into the dashboard or console area of the car.

5. The device of claim 1, wherein the UI of the video device displays other cars driving in the real-time traffic scene (TS).

6. The device of claim 5, wherein the video device calculates speed, velocity and/or position information related to the other cars.

7. The device of claim 1, wherein the UI of the video device includes warning indicators for notifying the person of the driving situations.

8. The device of claim 7, wherein the warning indicators include light emitting diodes (LEDs) that emit different colors for the different driving situations.

9. A method for predicting driving situations while a person drives a car, the method comprising:

extracting feature maps from multi-modal sensors and knowledge data;

training a deep convolutional neural network, with training data, to recognize real-time traffic scenes (TSs) from a viewpoint of the car;

displaying the real-time TSs on a user interface (UI) to warn of possible dangers; and

comparing the real-time TSs to predetermined TSs to predict the driving situations,

wherein the training data is labeled semi-automatically by defining a set of constraints on sensory variables for each label, encoding each label into a set of rules, and using the multi-modal sensors for which all rules are verified and assigned to a corresponding label.

10. The method of claim 9, wherein the video device includes a video camera.

11. The method of claim 10, wherein the video camera mounts to a dashboard of the car.

12. The method of claim 9, wherein the video device is incorporated into the dashboard or console area of the car.

13. The method of claim 9, wherein the UI of the video device displays other cars driving in the real-time traffic scene (TS).

14. The method of claim 13, wherein the video device calculates speed, velocity, and/or position information related to the other cars.

15. The method of claim 9, wherein the UI of the video device includes warning indicators for notifying the person of the driving situations.

16. The method of claim 15, wherein the warning indicators include light emitting diodes (LEDs) that emit different colors for the different driving situations.

17. A non-transitory computer-readable storage medium comprising a computer-readable program for predicting, by employing a video device, driving situations while a person drives a car, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:

extracting feature maps from multi-modal sensors and knowledge data;

training a deep convolutional neural network, with training data, to recognize real-time traffic scenes (TSs) from a viewpoint of the car;

displaying the real-time TSs on a user interface (UI) to warn of possible dangers; and

comparing the real-time TSs to predetermined TSs to predict the driving situations,

wherein the training data is labeled semi-automatically by defining a set of constraints on sensory variables for each label, encoding each label into a set of rules, and employing the multi-modal sensors for which all rules are verified and assigned to a corresponding label.