Method and system for monitoring the status of the driver of a vehicle

• US 10,318,831 B2
• Filed: 07/21/2016
• Issued: 06/11/2019
• Est. Priority Date: 07/21/2016
• Status: Active Grant

• Current Assignee: Gestigon GmbH (Valeo SE)
• Original Assignee: Gestigon GmbH (Valeo SE)
• Inventors: Klement, Sascha, Low, Sebastian, Munster, Dennis
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Tarko, Asmamaw G
• Application Number:
  US15/216,177
• Publication Number:
  US 20180025240A1
• Time in Patent Office: 1,055 Days
• Field of Search: 348 47
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

A system and method are disclosed relating to a computer system that estimates the status of the driver of a vehicle. The system comprises the following steps: Acquisition of an image from a depth sensor containing depth and optional an IR intensity image and an RGB color image; identification of pixels that belong to the drivers head; creation of a 3D model of the head including an intensity model and a variability estimate for depth, grayscale and color information; estimation of the principal head pose and the neutral facial expression; estimation of the current relative head pose with respect to the principal head pose; identification of pixels that do not match the neutral face model with respect to depth, grayscale or color information or any combination thereof; clustering of the pixels with identified deviations; classification of spatial and temporal patterns to identify driver status and distraction events.

Claims

1. A method for real-time driver state monitoring comprising:

operating an image processing system to receive a sequence of depth images from a depth sensor, each depth image comprising image data representing an image of a scene, wherein said image data comprises a depth value indicating distance from the depth sensor;

the image processing system identifying a human head in the scene using the sequence of images;

the image processing system using the depth images to track position and orientation of the human head with respect to a principal position and orientation where the human head is modeled as a 3D surface to define a head model and continuously adapted over time to obtain an average neutral face model where the neutral face is stored in a database including history of changes;

the image processing system identifying depth deviations from the neutral face; and

the image processing system classifying such depth deviations as different driver states, where depth variations are compared to the history of changes that has been stored, for a specific driver, in the database and where the comparison is used to extract features for driver-state detection.

2. The method as claimed in claim 1 where supplementary images comprising gray-scale or color images are obtained from the depth sensor or an additional sensor that has been pixel-wise registered to the depth image.

3. The method as claimed in claim 1 where all but head pixels are excluded according to a CAD model of an interior of a car.

4. The method as claimed in claim 1 where the human head is identified by applying a random decision forest.

5. The method as claimed in claim 1 where the human head is identified by skin color.

6. The method as claimed in claim 1 where the human head is identified by a face detection method.

7. The method as claimed in claim 1 where the head model is obtained by an iterative closest point match.

8. A system for real-time driver state monitoring, comprising:

a depth sensor to capture depth images of a scene and generate image data representing the image of the scene, wherein said image data comprises a depth value indicating distance from the depth sensor; and

an image processing system operatively connected to said sensor to receive a sequence of depth images from the sensor, the image processing system being programmed to identify a human head in the scene using the sequence of images, using the depth images to track position and orientation of the human head with respect to a principal position and orientation where the human head is modeled as a 3D surface to define a head model and continuously adapted over time to obtain an average neutral face model, identifying depth deviations from the neutral face, and classifying such depth deviations as different driver states, and wherein the image processing system is programmed whereby the neutral face is stored in a database including history of changes and whereby depth variations are compared to the history of changes that has been stored, for a specific driver, in the database and where the comparison is used to extract features for driver-state detection.

9. The system as claimed in claim 8 where the image processing system receives supplementary images comprising gray-scale or color images from the depth sensor or an additional sensor that has been pixel-wise registered to the depth image.

10. The system as claimed in claim 8 where the image processing system is programmed whereby all but head pixels are excluded according to a CAD model of an interior of a car.

11. The system as claimed in claim 8 where the image processing system is programmed whereby the human head is identified by applying a random decision forest.

12. The system as claimed in claim 8 where the image processing system is programmed whereby the human head is identified by skin color.

13. The system as claimed in claim 8 where the image processing system is programmed whereby the human head is identified by a face detection method.

14. The system as claimed in claim 8 where the image processing system is programmed whereby the head model is obtained by an iterative closest point match.