Head pose assessment methods and systems

• US 7,844,086 B2
• Filed: 06/20/2008
• Issued: 11/30/2010
• Est. Priority Date: 05/30/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hu, Yuxiao, Li, Mingjing, Zhang, Lei, Zhang, Hong-Jiang
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US12/143,717
• Publication Number:
  US 20080298637A1
• Time in Patent Office: 893 Days
• Field of Search: 382/103, 382/106, 382/107, 382/117, 382/118, 382/119, 382/181, 382/189, 382/190, 382/195, 382/201, 382/209, 382/222, 382/5, 348/169, 345/619, 345156-158, 345/175, 345/177, 345/181, 345/204, 345/207
• US Class Current: 382/118
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

Improvements are provided to effectively assess a user'"'"'s face and head pose such that a computer or like device can track the user'"'"'s attention towards a display device(s). Then the region of the display or graphical user interface that the user is turned towards can be automatically selected without requiring the user to provide further inputs. A frontal face detector is applied to detect the user'"'"'s frontal face and then key facial points such as left/right eye center, left/right mouth corner, nose tip, etc., are detected by component detectors. The system then tracks the user'"'"'s head by an image tracker and determines yaw, tilt and roll angle and other pose information of the user'"'"'s head through a coarse to fine process according to key facial points and/or confidence outputs by pose estimator.

Claims

1. An apparatus for use with at least one display device and at least one image capturing device, the apparatus comprising:

a processor;

one or more memories coupled to the processor, the one or more memories having stored instructions that configure the apparatus to implement;

display logic configurable to output at least one signal suitable for causing a display device to display at least two different selectable regions;

interface logic configurable to receive image data from an image capturing device;

pose estimation logic operatively coupled to said display logic and said interface logic and configured to determine a first head pose based on a first image and at least a second head pose based on a second image temporally subsequent to said first image, and automatically operatively switch an operative user input focus between said at least two selectable regions based on at least one difference between said first head pose and at least said second head pose; and

programmable filter logic operatively coupled to said display logic and said interface logic and configured to filter the second image with respect to the first image, the second image being logically weighted differently than the first image based on an amount of time between a capture of the first image and a capture of the second image.

2. The apparatus as recited in claim 1, wherein said pose estimation logic is further configured to classify each of a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region associated with at least one portion of a face of a user, wherein said face of said user is captured by said first image.

3. An apparatus for use with at least one display device and at least one image capturing device, the apparatus comprising:

a processor;

one or more memories coupled to the processor, the one or more memories having stored instructions that configure the apparatus to implement;

display logic configurable to output at least one signal suitable for causing a display device to display at least two different selectable regions;

interface logic configurable to receive image data from an image capturing device;

pose estimation logic operatively coupled to said display logic and said interface logic and configured to determine a first head pose based on a first image and at least a second head pose based on a second image temporally subsequent to said first image, and automatically operatively switch an operative user input focus between said at least two selectable regions based on at least one difference between said first head pose and at least said second head pose; and

memory operatively coupled to at least said pose estimation logic, said memory being configurable to store said first image and said second image, wherein said first image captures at least a first portion of a face of a user at a first time and said second image captures at least a second portion of said face of said user at a second subsequent time;

wherein said pose estimation logic is further configured to;

access said first image, detect at least said first portion of said face within said first image, detect at least one point within said detected first portion of said face, and store first tracking information associated with each of said at least one point within said detected first portion to said memory, and

access said second image, track at least said second portion of said face within said second image, detect said at least one point within said detected second portion of said face, and store second tracking information associated with each of said at least one point within said detected second portion to said memory, and

classify each of a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region associated with said first portion of said face.

4. The apparatus as recited in claim 3, wherein said pose estimation logic is further configured to compare at least said first tracking information and said second tracking information to determine if an assessed display device view associated with said user has changed between said first time to said second time, and if said assessed display device view associated with said user has changed between said first time to said second time, then switch said operative user input focus.

5. The apparatus as recited in claim 3, said pose estimation logic is further configured to image patch track said facial region associated with said first portion of said face in a corresponding portion of said second image to identify at least one detected face area.

6. The apparatus as recited in claim 5, wherein said pose estimation logic includes a sum-of-square difference (SSD) image patch tracker to identify said at least one detected face area.

7. The apparatus as recited in claim 5, wherein said image patch tracker compares at least a portion of said resulting detected face area with at least one alert threshold parameter to determine if an associated system initialization process is required.

8. The apparatus as recited in claim 5, wherein said pose estimation logic is configured to detect at least one key facial component within said at least one detected face area to conduct a coarse pose estimation.

9. The apparatus as recited in claim 8, wherein said pose estimation logic is configured to determine a fine head pose of said user based on said detected key facial components.

10. The apparatus as recited in claim 9, wherein said pose estimation logic is configured to determine said fine head pose using by combining the results of at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

11. The apparatus as recited in claim 9, wherein said pose estimation logic is configured to determine said fine head pose using by combining the results of at least one view-based pose estimation technique.

12. The apparatus as recited in claim 8, wherein said pose estimation logic is configured to determine the coarse head pose of said user based on confidence information associated with detecting said at least one key facial component within said at least one detected face area.

13. The apparatus as recited in claim 12, wherein said pose estimation logic is configured to determine a fine head pose of said user based on said detected key facial components and said confidence information.

14. The apparatus as recited in claim 12, wherein said coarse head pose is associated with at least one head pose parameter selected from a group of head pose parameter comprising a yaw angle, a tilt angle, a roll angle, an x translation, a y translation, and a scale factor.

15. A system comprising:

at least one display device;

at least one image capturing device; and

a computing device operatively coupled to said display device and said image capturing device and having;

display logic configured to output at least one signal suitable for causing a display device to display at least two different selectable regions,

interface logic configured to receive image data from an image capturing device, and

pose estimation logic configured to determine a first head pose based on a first image and at least a second head pose based on a second image temporally subsequent to said first image, and automatically operatively switch an operative user input focus between said at least two selectable regions based on at least one difference between said first head pose and at least said second head pose,

wherein said pose estimation logic is further configured to classify each of a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region associated with at least one portion of a face of a user, wherein said face of said user is captured by said first image.

16. The system according to claim 15, further comprising memory operatively coupled to at least said pose estimation logic, said memory being configurable to store said first image and said second, wherein said first image captures at least a first portion of a face of a user at a first time and said second image captures at least a second portion of said face of said user at a second subsequent time.

17. The system according to claim 15, wherein the pose estimation logic is further configured to detect at least one point within said first image and to detect at least one point within said second image.

18. The system according to claim 17, wherein the at least one point within said second image corresponds to the at least one point within said first image.

19. The system according to claim 15, wherein the pose estimation logic is further configured to detect at least two points within said first image and to detect at least two points within said second image.

20. The system according to claim 15, wherein the pose estimation logic is further configured to track information associated with said first image and track information associated said second image, the information tracked from said first and second images used to determine said at least one difference between said first head pose and said second head pose.