Head pose assessment methods and systems

• US 8,135,183 B2
• Filed: 11/05/2010
• Issued: 03/13/2012
• Est. Priority Date: 05/30/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hu, Yuxiao, Zhang, Lei, Li, Mingjing, Zhang, Hong-Jiang
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US12/940,408
• Publication Number:
  US 20110050568A1
• Time in Patent Office: 494 Days
• Field of Search: 382/100, 382/103, 382/106, 382/107, 382/117, 382/118, 382/119, 382/181, 382/189, 382/190, 382/195, 382/201, 382/209, 382/225, 348/169, 345/619, 345/156, 345/157, 345/158, 345/175, 345/177, 345/181, 345/204, 345/207
• US Class Current: 382/118
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

Improvements are provided to effectively assess a user'"'"'s face and head pose such that a computer or like device can track the user'"'"'s attention towards a display device(s). Then the region of the display or graphical user interface that the user is turned towards can be automatically selected without requiring the user to provide further inputs. A frontal face detector is applied to detect the user'"'"'s frontal face and then key facial points such as left/right eye center, left/right mouth corner, nose tip, etc., are detected by component detectors. The system then tracks the user'"'"'s head by an image tracker and determines yaw, tilt and roll angle and other pose information of the user'"'"'s head through a coarse to fine process according to key facial points and/or confidence outputs by pose estimator.

Claims

1. An apparatus for use with at least one display device and at least one image capturing device, the apparatus comprising:

a processor;

one or more memories coupled to the processor, the one or more memories having stored instructions that configure the apparatus to implement;

display logic configurable to output at least one signal suitable for causing a display device to display at least two different selectable regions; and

pose estimation logic operatively coupled to the display logic and the interface logic and configured to determine a first head pose based on a first image and at least a second head pose based on a second image temporally subsequent to the first image, and automatically switch an operative user input focus between the at least two selectable regions based on at least one difference between the first head pose and at least the second head pose;

the pose estimation logic being configured to;

image patch track at least one facial region associated with a portion of a face in the first image to a corresponding portion of the second image, detect at least one face area comprising at least one key facial component based on the image patch tracking, and produce at least one of a coarse pose estimation based at least in part on the at least one detected face area and a fine pose estimation based at least in part on the at least one key facial component.

2. The apparatus as recited in claim 1, wherein the pose estimation logic is configured to classify each of a plurality of portions of image data associated with the first image based on at least one classifying parameter to determine the at least one facial region associated with the portion of the face.

3. The apparatus as recited in claim 1, wherein the pose estimation logic includes a sum-of-square difference (SSD) image patch tracker.

4. The apparatus as recited in claim 1, wherein the image patch tracker compares at least a portion of the resulting detected face area with at least one alert threshold parameter to determine if an associated system initialization process is required.

5. The apparatus as recited in claim 1, wherein the coarse head pose is associated with at least one head pose parameter selected from a group of head pose parameters comprising a yaw angle, a tilt angle, a roll angle, an x translation, a y translation, and a scale factor.

6. The apparatus as recited in claim 1, wherein the pose estimation logic is configured to determine the fine head pose by combining results of at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

7. The apparatus as recited in claim 1, further comprising programmable filter logic operatively coupled to the display logic and the interface logic and configured to filter the second image with respect to the first image, the second image being logically weighted differently than the first image based on an amount of time between a capture of the first image and a capture of the second image.

8. A system comprising:

a display device;

an image capturing device; and

a computing device operatively coupled to the display device and the image capturing device, and including;

a display module configured to output at least one signal suitable for causing the display device to display at least two different selectable regions; and

an iterated pose estimation module configured to determine a first head pose based on a first image and at least a second head pose based on a second image temporally subsequent to the first image, and automatically switch an operative user input focus between the at least two selectable regions based on at least one difference between the first head pose and at least the second head pose,

the iterated pose estimation module being configured to estimate a configuration for a plurality of key facial points associated with at least one of the first image or the second image, and iteratively optimize one or more pose parameters to minimize a distance between a projection of the estimated configuration for the plurality of key facial points and a corresponding actual configuration of the plurality of key facial points.

9. The system as recited in claim 8, wherein the pose parameters correspond to at least one of the first image or the second image and comprise at least one of:

a yaw angle, a tilt angle, a roll angle, and a scale.

10. The system as recited in claim 8, wherein the iterated pose estimation module is configured to classify each of a plurality of portions of image data associated with the first image based on at least one classifying parameter to determine at least one facial region associated with at least a portion of a face of a user, wherein the face of the user is captured by the first image.

11. The system as recited in claim 8, wherein the iterated pose estimation module is configured to image patch track at least one facial region associated with a portion of a face in the first image to a corresponding portion of the second image, detect at least one face area comprising at least one of the plurality of key facial points based on the image patch tracking, and produce at least one of a coarse pose estimation based at least in part on the at least one detected face area and a fine pose estimation based at least in part on the at least one of the plurality of key facial points.

12. The system as recited in claim 8, further comprising a programmable filter module operatively coupled to the display module and the iterated pose estimation module and configured to filter the second image with respect to the first image, the second image being logically weighted differently than the first image based on an amount of time between a capture of the first image and a capture of the second image.

13. A computer-implemented method operable on a processor, the method comprising:

receiving a first image and at least a second image from an image capturing device, the second image temporally subsequent to the first image;

determining, at the processor, a first head pose based on the first image;

determining, at the processor, at least a second head pose based on the second image;

switching an operative user input focus between at least two selectable regions of a display device based on at least one difference between the first head pose and at least the second head pose, the switching including storing a present work status associated with the first head pose and restoring a previously stored work status associated with the second head pose.

14. The method of claim 13, wherein the restoring includes activating a document operative when the previously stored work status was stored, and locating a pointing indicator on the display device at a location that the pointer was located when the previously stored work status was stored.

15. The method of claim 13, further comprising tracking information associated with the first image and tracking information associated the second image, and determining the at least one difference between the first head pose and the second head pose based at least in part on the information tracked from the first and second images.

16. The method of claim 13, further comprising filtering the second image with respect to the first image, the second image being logically weighted differently than the first image based on an amount of time between a capture of the first image and a capture of the second image.

17. The method of claim 13, further comprising classifying each of a plurality of portions of image data associated with the first image based on at least one classifying parameter to determine at least one facial region associated with at least one portion of a face of a user, wherein the face of the user is captured by the first image.

18. The method of claim 16, further comprising detecting at least one key facial component within the at least one facial region to conduct a coarse pose estimation and determining a fine head pose of the user based on the at least one detected key facial component.

19. The method of claim 13, further comprising determining a fine head pose by combining results of at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

20. The method of claim 13, further comprising estimating a configuration for a plurality of key facial components associated with at least one of the first image or the second image, and iteratively optimizing one or more pose parameters to minimize a distance between a projection of the estimated configuration for the plurality of key facial components and a corresponding actual configuration of the plurality of key facial components.