System and method for driver distraction determination

• US 10,703,268 B2
• Filed: 01/03/2019
• Issued: 07/07/2020
• Est. Priority Date: 11/07/2016
• Status: Active Grant

• Current Assignee: Nauto, Inc.
• Original Assignee: Nauto, Inc.
• Inventors: Levkova, Ludmila, Heck, Stefan, Alpert, Benjamin O., Satzoda, Ravi Kumar, Sathyanarayana, Suchitra, Sekar, Vivek
• Primary Examiner(s): Sherwin, Ryan W
• Application Number:
  US16/239,326
• Publication Number:
  US 20190152390A1
• Time in Patent Office: 551 Days
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

A method for determining distraction of a driver of a vehicle, including sampling sensor measurements at an onboard system of the vehicle; generating an output indicative of a distracted state; determining that the driver of the vehicle is characterized by the distracted state, based on the output; generating, at a second distraction detection module of a remote computing system, a second output indicative that the driver is characterized by the distracted state, based on the sensor measurements; computing a distraction score, at a scoring module of the remote computing system, in response to generating the second output and based on the sensor measurements and the distracted state.

Claims

1. A method for determining distraction of a driver of vehicle, comprising:

sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements comprise an original image of the driver, wherein the original image has a first resolution;

receiving the original image at a face detection module of the onboard system, and transforming the original image into a first sub-image and a second sub-image, wherein the first sub-image comprises a portion of the original image defining a face of the driver, wherein the first sub-image has the first resolution, and wherein the second sub-image comprises the original image transformed to have a second resolution less than the first resolution;

providing the first sub-image and the second sub-image to a driver distraction module of the onboard system; and

determining, at the driver distraction module, that the driver is characterized by a distracted state based on a combination of the first sub-image and second sub-image.

2. The method of claim 1, further comprising determining a context of the distracted state based on the sensor measurements, and notifying the driver based on the distracted state in combination with the context of the distracted state.

3. The method of claim 2, wherein the sensor measurements further comprise a forward image of a region in front of the vehicle, and wherein determining the context of the distracted state comprises determining a following distance between a leading vehicle depicted in the forward image and the vehicle.

4. The method of claim 2, wherein determining the context of the distracted state comprises determining a score of the distracted state, and wherein notifying the driver based on the distracted state is performed in response to the score exceeding a threshold score.

5. The method of claim 2, wherein the distraction determination module comprises a convolutional neural network (CNN), and wherein determining that the driver is characterized by the distracted state comprises:

classifying a pose of the driver based on the combination of the first sub-image and the second sub-image at the CNN; and

determining that the driver is characterized by the distracted state based on the pose of the driver in combination with the context.

6. The method of claim 5, wherein classifying the pose of the driver comprises classifying the pose as corresponding to a class selected from:

looking-down, looking-up, looking-left, looking-right, holding-an-object, and no-face-visible.

7. The method of claim 5, wherein the CNN of the distraction determination module has a set of processing layers that each:

receive an input based on the first sub-image and the second sub-image,

generate an output based on the first sub-image and the second sub-image, and

wherein the combination of the first sub-image and the second sub-image comprises an addition of a first portion of the output of each processing layer and a second portion of the output of each processing layer.

8. The method of claim 1, wherein the original image is recorded at a first time point, wherein the sensor measurements further comprise a second image recorded at a second time point subsequent to the first time point, and further comprising transforming the second image into a third sub-image and a fourth sub-image, wherein the third sub-image comprises a portion of the second image defining the face of the driver, wherein the third sub-image has the first resolution, and wherein the fourth sub-image comprises the second image transformed to have the second resolution, and wherein determining that the driver is characterized by the distracted state is based on a comparison between the combination of the first sub-image and second sub-image and a second combination of the third sub-image and the fourth sub-image.

9. A method for determining distraction of a driver of a vehicle, comprising:

sampling sensor measurements at an onboard system of the vehicle during a driving session, wherein the sensor measurements comprise a first image of the driver recorded at a first time point during the driving session and a second image of the driver recorded at a second time point during the driving session, wherein the second time point is subsequent to the first time point;

processing the first image at a convolutional neural network (CNN) of a driver distraction module of the onboard system to generate a set of network weights and a first output;

processing the second image at the CNN using the set of network weights to generate a second output; and

determining that the driver of the vehicle is characterized by the distracted state based on a comparison between the first output and the second output.

10. The method of claim 9, wherein determining that the driver of the vehicle is characterized by the distracted state based on the comparison between the first output and the second output comprises:

determining that the driver of the vehicle is characterized by an undistracted state based on the first output;

comparing the second output to the first output to determine a deviation between the second output and the first output, and determining that the driver of the vehicle is characterized by the distracted state based upon the deviation.

11. The method of claim 9, further comprising determining a context of the distracted state, at the onboard system of the vehicle, based on the sensor measurements, and wherein determining that the driver is characterized by the distracted state is based on the context.

12. The method of claim 11, wherein the sensor measurements further comprise a third image of a region in front of the vehicle, and wherein determining the context of the distracted state comprises determining a following distance between a leading vehicle depicted in the third image and the vehicle.

13. The method of claim 11, wherein determining the context of the distracted state comprises determining a score of the distracted state, and notifying the driver based on the distracted state in response to the score exceeding a threshold score.

14. The method of claim 9, wherein the first image has a first resolution, and wherein processing the first image at the CNN comprises:

receiving the first image at a face detection module of the onboard system, and transforming the first image into a first sub-image and a second sub-image, wherein the first sub-image comprises a portion of the first image defining a face of the driver, wherein the first sub-image has the first resolution, and wherein the second sub-image comprises the first image transformed to have a second resolution less than the first resolution; and

providing the first sub-image and the second sub-image to the CNN to generate the set of network weights.

15. The method of claim 14, wherein the second image has the first resolution, and wherein processing the second image at the CNN comprises:

receiving the second image at the face detection module, and transforming the second image into a third sub-image and a fourth sub-image, wherein the third sub-image comprises a portion of the second image defining the face of the driver, wherein the third sub-image has the first resolution, and wherein the fourth sub-image comprises the second image transformed to have the second resolution;

processing the third sub-image and the fourth sub-image at the CNN with the set of network weights to generate the second output.

16. An onboard system for determining distraction of a driver of a vehicle, comprising:

a housing, rigidly mounted to the vehicle proximal a driver'"'"'s seat of the vehicle;

a first image sensor, coupled to and retained by the housing, arranged to record a first image of the driver at a first resolution;

a processor, coupled to and retained by the housing, that executes;

a face detection module that transforms the first image into a first sub-image and a second sub-image, wherein the first sub-image comprises a portion of the first image defining a face of the driver, wherein the first sub-image has the first resolution, and wherein the second sub-image comprises the first image transformed to have a second resolution less than the first resolution, and

a driver distraction module that characterizes the driver as in a distracted state based on a combination of the first sub-image and the second sub-image; and

an output mechanism, coupled to and retained by the housing, that generates an output signal in response to the driver distraction module characterizing the driver in the distracted state.

17. The onboard system of claim 16, wherein the sensor measurements further comprise a forward image of a region in front of the vehicle, wherein the driver distraction module generates a score of the distracted state based on a following distance between a leading vehicle depicted in the forward image and the vehicle, extracted from the forward image, and wherein the output mechanism generates the output signal based on the score exceeding a threshold score.

18. The onboard system of claim 16, wherein the distraction determination module comprises a convolutional neural network (CNN), and wherein the distraction determination module further:

classifies a pose of the driver based on the combination of the first sub-image and the second sub-image at the CNN; and

determines that the driver is characterized by the distracted state based on the pose of the driver.

19. The onboard system of claim 18, wherein the distraction determination module classifies the pose as corresponding to a class selected from:

looking-down, looking-up, looking-left, looking-right, holding-an-object, and no-face-visible.

20. The onboard system of claim 18, wherein the CNN of the distraction determination module has a set of processing layers that each:

receive an input based on the first sub-image and the second sub-image,

generate an output based on the first sub-image and the second sub-image, and

wherein the combination of the first sub-image and the second sub-image comprises an addition of a first portion of the output of each processing layer and a second portion of the output of each processing layer.