System and method for detecting distraction and a downward vertical head pose in a vehicle

• US 10,089,543 B2
• Filed: 07/29/2016
• Issued: 10/02/2018
• Est. Priority Date: 07/29/2016
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Fung, Kin C., Chpak, Peter, Dick, Timothy J., Palande, Dhanashree
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Mangialaschi, Tracy
• Application Number:
  US15/224,267
• Publication Number:
  US 20180032825A1
• Time in Patent Office: 795 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A computer-implemented method for detecting a head pose in a vehicle including receiving images of a vehicle occupant located in the vehicle from an imaging device and selecting facial feature points from a plurality of facial feature points extracted from the images. The method includes calculating a head pose point based on normalizing the selected facial feature points, determining the head pose based on a change in position of the head pose point over a period of time T, and controlling one or more vehicle systems of the vehicle based on the head pose.

Claims

1. A computer-implemented method for detecting a head pose in a vehicle, comprising:

receiving images of a vehicle occupant located in the vehicle from an imaging device;

selecting facial feature points from a plurality of facial feature points extracted from the images;

calculating a head pose point by averaging vertical position components of the selected facial feature points;

determining the head pose based on a change in position of the head pose point over a period of time T, wherein determining the head pose further includes calculating a difference between the head pose point and a reference point, the reference point being a value indicating a position on a plane of the image and being a learned variant of the head pose point; and

controlling one or more vehicle systems of the vehicle based on the head pose.

2. The computer-implemented method of claim 1, wherein selecting the facial feature points includes selecting fixed facial feature points from the plurality of facial feature points, the fixed facial feature points having positions unaffected by movement of other facial feature points.

3. The computer-implemented method of claim 1, wherein calculating the head pose point by normalizing the selected facial feature points further includes averaging positions of the selected facial feature points.

4. The computer-implemented method of claim 3, wherein calculating the head pose point by normalizing the selected facial feature points further includes weighting the selected facial feature points based on a confidence level assigned to each of the selected facial feature points.

5. The computer-implemented method of claim 4, further including determining a change in position of one or more nose facial feature points over the period of time T.

6. The computer-implemented method of claim 5, wherein upon determining the change in position of the one or more nose facial feature points increases over the period of time T, determining the head pose further includes determining the head pose is oriented in a downward direction.

7. The computer-implemented method of claim 1, further including updating the reference point based on a variance of the head pose point over a second period of time T.

8. The computer-implemented method of claim 7, where upon determining the variance of the head pose point is within a predetermined standard deviation, the method includes updating the reference point to the mean of the head pose point over the second period of time T.

9. The computer-implemented method of claim 1, wherein upon determining a change in position of the head pose point over the period of time T has a periodic pattern, determining the head pose further includes determining the head pose is oriented in a downward direction.

10. The computer-implemented method of claim 1, further comprising:

comparing the difference between the head pose point and the reference point to a pose variation threshold;

when the difference is determined to be greater than the pose variation threshold, determining a change in a distance between two nose facial feature points over the period of time T, and upon determining the change in the distance increases, determining the head pose is oriented in a downward direction;

when the difference is determined to be not greater than the pose variation threshold, tracking the head pose point for at least two consecutive time periods, the at least two consecutive time periods equal to the period of time T, and determining a movement pattern from a first position to a second position for each of the at least two consecutive time periods, and upon determining the movement pattern of each of the at least two consecutive time periods are approximately the same, determining the head pose is oriented in a downward direction.

11. A system for detecting a head pose in a vehicle, comprising:

an imaging device operably connected for computer communication with the vehicle, the imaging device capturing images of a face of a vehicle occupant located in the vehicle;

a processor operably connected for computer communication with the imaging device and the vehicle, the processor extracts a plurality of facial feature points from the images and selects facial feature points from the plurality of facial feature points, wherein each of the selected facial feature points have a position on a plane of the images, wherein the processor calculates a head pose point by normalizing vertical components of the positions of the selected facial feature points, and determines the head pose based on a change in position of the head pose point over a period of time T, wherein the processor calculates the change in position of the head pose point as a change in position between the head pose point and a reference point over the period of time T, the reference point being a value indicating a position on a plane of the image and being a learned value based on a mean value of the head pose point over time; and

one or more vehicle systems operably connected for computer communication to the processor and the vehicle, wherein the processor controls the one or more vehicle systems based on the head pose.

12. The system of claim 11, wherein the selected facial feature points maintain a stable position when exposed to movement from facial expressions.

13. The system of claim 11, wherein the processor calculates the head pose point as a vertical normalization of the positions of the selected facial feature points by averaging the positions of the selected facial feature points.

14. The system of claim 11, wherein the processor updates the reference point based on a variance and a mean of the head pose point over the period of time T.

15. The system of claim 11, wherein the processor determines a change in a distance between two nose facial feature points over the period of time T.

16. The system of claim 15, wherein upon determining the change in the distance between the two nose facial feature points increases over the period of time T, the processor determines the head pose is oriented in a downward direction.

17. The system of claim 11, wherein the imaging device tracks the head pose point for at least two consecutive time periods, the at least two consecutive time periods equal to the period of time T, and the processor determines a movement pattern from a first position to a second position for each of the at least two consecutive time periods.

18. The system of claim 17, wherein upon determining the movement pattern of each of the at least two consecutive time periods are approximately the same, the processor determines the head pose is oriented in a downward direction.

19. The system of claim 11, wherein the imaging device tracks an eye gaze of the vehicle occupant.

20. The system of claim 19, wherein determining the head pose further includes the processor verifying the eye gaze is oriented in a direction of the head pose.

21. A non-transitory computer-readable medium for detecting a head pose in a vehicle including instructions stored thereon, that when executed by a processor perform a method, the method comprising:

receiving images of a vehicle occupant located in the vehicle from an imaging device;

calculating a head pose point by averaging vertical position components of facial feature points selectively identified from the images;

determining the head pose based on a change in position of the head pose point over a period of time T, wherein determining the head pose further includes calculating a difference between the head pose point and a reference point, the reference point being a value indicating a position on a plane of the image and being a learned variant of the head pose point; and

controlling a function of one or more vehicle systems of the vehicle based on the head pose.

22. The non-transitory computer-readable medium of claim 21, wherein the facial feature points are selectively identified based on whether the facial feature points maintain a stable position when the facial feature points are exposed to movement of other facial feature points.

23. The non-transitory computer-readable medium of claim 21, further including upon determining the difference between the head pose point and the reference point meets a head down threshold, determining a change in displacement between one or more nose facial feature points over the period of time T.

24. The non-transitory computer-readable medium of claim 23, wherein upon determining the change in displacement between the one or more nose facial feature points increases over the period of time T, determining the head pose includes determining the head pose is oriented in a downward direction.