System and method for contextualized vehicle operation determination

• US 10,430,695 B2
• Filed: 06/18/2018
• Issued: 10/01/2019
• Est. Priority Date: 06/16/2017
• Status: Active Grant

• Current Assignee: Nauto, Inc.
• Original Assignee: Nauto, Inc.
• Inventors: Sathyanarayana, Suchitra, Satzoda, Ravi Kumar, Thompson, Alex, Gleeson-May, Michael, Levkova, Ludmilla
• Primary Examiner(s): Anwari, Maceeh
• Application Number:
  US16/011,437
• Publication Number:
  US 20180365533A1
• Time in Patent Office: 470 Days
• Field of Search: 701301
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/043 Identity of occupants
• CPC Class Codes: B60W 2540/049 Number of occupants
• CPC Class Codes: B60W 2540/18 Steering angle
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/223 Posture, e.g. hand, foot, or seat position, turned or inclined
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/60 Traversable objects, e.g. speed bumps or curbs
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2556/05 Big data
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 30/095 : Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 40/08 : related to drivers or passengers
• CPC Class Codes: B60W 60/001 : Planning or execution of driving tasks
• CPC Class Codes: B60W 60/0055 : only part of driving tasks shifted to occupants
• CPC Class Codes: G05D 1/0214 : in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/0221 : involving a learning process
• CPC Class Codes: G05D 1/0251 : extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G06F 18/2148 : characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/24 : Classification techniques
• CPC Class Codes: G06F 18/251 : of input or preprocessed data
• CPC Class Codes: G06F 18/256 : of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/04 : Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 10/00 : Administration; Management
• CPC Class Codes: G06Q 10/20 : Administration of product repair or maintenance
• CPC Class Codes: G06T 2207/20076 : Probabilistic image processing
• CPC Class Codes: G06T 7/50 : Depth or shape recovery
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/811 : the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G07C 5/0866 : the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0145 : for active traffic flow control
• CPC Class Codes: G08G 1/04 : using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method for determining event data including: sampling a first data stream within a first time window at a first sensor of an onboard vehicle system coupled to a vehicle, extracting interior activity data from the first data stream; determining an interior event based on the interior activity data; sampling a second data stream within a second time window at a second sensor of the onboard vehicle system; extracting exterior activity data from the second image stream; determining an exterior event based on the exterior activity data; correlating the exterior event and the interior event to generate combined event data; automatically classifying the combined event data to generate an event label; and automatically labeling the first time window of the first data stream and the second time window of the second data stream with the combined event label to generate labeled event data.

Claims

1. A method for determining event data comprising:

sampling a first image stream within a first time window at an interior-facing camera of an onboard vehicle system, wherein the onboard vehicle system is integrated into a mountable unit coupled to a vehicle at a single interior location;

extracting interior activity data, comprising a driver gaze direction, from the first image stream;

sampling a second image stream within a second time window at an exterior-facing camera of the onboard vehicle system, wherein the first and second time window are coextensive;

extracting exterior activity data from the second image stream;

determining an interior event based on the interior activity data, comprising mapping the driver gaze direction relative to a first region of the second image stream based on a relative orientation between the interior-facing camera and the exterior-facing camera of the onboard vehicle system;

determining an exterior event based on the exterior activity data, comprising determining that a distance between the vehicle and an object depicted in the first region of the second image stream has fallen below a threshold distance;

correlating the interior event with the exterior event to generate combined event data, comprising determining that the driver gaze direction overlaps with the first region of the second image stream at a time point within the first time window;

automatically classifying the combined event data to generate an event label; and

automatically labeling the first image stream within the first time window and the second image stream within the second time window with the event label to generate labeled training data;

transmitting the labeled training data to a remote computing system; and

aggregating the labeled training data at the remote computing system with a corpus of labeled training data, wherein the corpus of labeled training data is received from a plurality of onboard vehicle systems operating in a plurality of vehicles.

2. The method of claim 1, further comprising:

training a vehicle control model, at the remote computing system, using the corpus of labeled training data;

transmitting a copy of the vehicle control model to the vehicle; and

controlling the vehicle using the copy of the vehicle control model.

3. The method of claim 1, wherein the interior event comprises a driver distraction event.

4. The method of claim 3, wherein correlating the interior event with the exterior event comprises:

determining a distraction weight based on the driver distraction event;

determining a severity of the exterior event; and

weighting the severity of the exterior event by the distraction weight, wherein the combined event data comprises a distraction-weighted exterior event.

5. The method of claim 3, wherein extracting the exterior activity data comprises determining a relative position between the vehicle and the object depicted in the second image stream within the second time window.

6. The method of claim 5, wherein the object is a secondary vehicle, and wherein determining the exterior event comprises determining that the relative position between the vehicle and the secondary vehicle has fallen below a threshold distance.

7. The method of claim 1, further comprising determining a near-miss event, wherein the near-miss event occurs within the first time window, and wherein correlating the interior event with the exterior event comprises:

associating the interior event with the near-miss event; and

associating the exterior event with the near-miss event, wherein the combined event data comprises the near-miss event, the associated interior event, and the associated exterior event.

8. A method for determining event data comprising:

sampling a first data stream within a first time window at a first sensor of an onboard vehicle system coupled to a vehicle,

extracting interior activity data, comprising a driver gaze direction, from the first data stream;

sampling a second data stream within a second time window at a second sensor of the onboard vehicle system;

extracting exterior activity data from the second image stream;

determining an interior event based on the interior activity data, comprising mapping the driver gaze direction relative to a first region of the second image stream based on a relative orientation between the interior-facing camera and the exterior-facing camera of the onboard vehicle system;

determining an exterior event based on the exterior activity data, comprising determining that a distance between the vehicle and an object depicted in the first region of the second image stream has fallen below a threshold distance;

correlating the exterior event and the interior event to generate combined event data, comprising determining that the driver gaze direction overlaps with the first region of the second image stream at a time point within the first time window;

automatically classifying the combined event data to generate an event label; and

automatically labeling the first time window of the first data stream and the second ti me window of the second data stream with the combined event label to generate labeled event data; and

transmitting the labeled event data to a remote computing system.

9. The method of claim 8, further comprising:

training a vehicle control model, at the remote computing system, using the labeled event data.

10. The method of claim 9, further comprising aggregating the labeled event data at the remote computing system with a corpus of labeled event data, wherein the corpus of labeled training data is received from a plurality of onboard vehicle systems operating in a plurality of vehicles, and training the vehicle control module using the corpus of labeled event data.

11. The method of claim 8, wherein determining the exterior event further comprises determining a near-miss event, and wherein correlating the interior event with the exterior event comprises:

associating the interior event with the near-miss event; and

calculating a probability that the interior event caused the near-miss event, wherein the combined event data comprises the near-miss event, the associated interior event, and the probability.

12. The method of claim 8, wherein the first data stream comprises gyroscope data.

13. The method of claim 12, wherein extracting interior activity data further comprises determining a mean time interval between steering inputs based on the gyroscope data, and wherein determining the interior event comprises determining a driver distraction event based on the mean time interval exceeding a threshold duration.

14. The method of claim 13, wherein correlating the interior event with the exterior event comprises:

determining a distraction weight based on the driver distraction event;

determining a severity of the exterior event; and

weighting the severity of the exterior event by the distraction weight, wherein the combined event data comprises a distraction-weighted exterior event.

15. The method of claim 8, wherein the onboard vehicle system is integrated into a mountable unit coupled to the vehicle at a single interior location.

16. The method of claim 15, further comprising correlating the interior event with the exterior event based on relative geometric arrangement between the first sensor and the second sensor.

17. The method of claim 16, wherein the first data stream comprises a first image stream and the first sensor comprises a first camera of the onboard vehicle system, and wherein the second data stream comprises a second image stream and the second sensor comprises a second camera of the onboard vehicle system.

18. The method of claim 8, wherein the first time window is coextensive with the second time window.