Head pose assessment methods and systems

• US 20040240708A1
• Filed: 05/30/2003
• Published: 12/02/2004
• Est. Priority Date: 05/30/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Zhang, Lei, Zhang, Hong-Jiang, Hu, Yuxiao
• Granted Patent: US 7,391,888 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

Improvements are provided to effectively assess a user'"'"'s face and head pose such that a computer or like device can track the user'"'"'s attention towards a display device(s). Then the region of the display or graphical user interface that the user is turned towards can be automatically selected without requiring the user to provide further inputs. A frontal face detector is applied to detect the user'"'"'s frontal face and then key facial points such as left/right eye center, left/right mouth corner, nose tip, etc., are detected by component detectors. The system then tracks the user'"'"'s head by an image tracker and determines yaw, tilt and roll angle and other pose information of the user'"'"'s head through a coarse to fine process according to key facial points and/or confidence outputs by pose estimator.

Claims

1. A method for use with a computing device comprising:

determining a first head pose for a user of the computing device at a first time;

determining at least a second head pose for said user at a second time that is subsequent to said first time; and

automatically operatively switching between at least two selectable regions of at least one display device of the computing device based on at least one difference between said first head pose and at least said second head pose.

2. The method as recited in claim 1, wherein:

determining said first head pose further includes;

at said first time, generating a first image showing at least a first portion of a face of said user,

detecting at least said first portion of said face within said first image, and

detecting at least one point within said detected first portion of said face and storing first tracking information associated with said at least one point; and

determining said second head pose further includes;

at said second time, generating a second image showing at least a second portion of said face of said user,

detecting at least said second portion of said face within said second image, and

detecting said at least one point within said detected second portion of said face and storing second tracking information associated with said at least one point.

3. The method as recited in claim 2, wherein automatically operatively switching between said at least two selectable regions further includes:

comparing at least said first tracking information and said second tracking information to determine if an assessed display device view associated with said user has changed between said first time to said second time; and

if said assessed display device view associated with said user has changed between said first time to said second time, then selectively operatively switching a user input interface focus presented on said display device to a selectable region associated with said assessed display device view.

4. The method as recited in claim 2, wherein generating said first image further includes causing at least one camera device to output said first image.

5. The method as recited in claim 4, wherein said at lest one camera device includes at least one video camera device and said first image includes a frame of video captured by said video camera device.

6. The method as recited in claim 4, wherein said at lest one camera device includes at least one digital still camera device and said first image includes a digital still image captured by said digital still camera device.

7. The method as recited in claim 2, wherein detecting said first portion of said face within said first image further includes:

classifying each of a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region associated with said first portion of said face.

8. The method as recited in claim 7, further comprising:

image patch tracking said facial region associated with said first portion of said face in a corresponding portion of said second image to identify at least one detected face area.

9. The method as recited in claim 8, wherein image patch tracking further includes:

using a sum-of-square difference (SSD) image patch tracker to identify said at least one detected face area.

10. The method as recited in claim 8, wherein image patch tracking further includes:

comparing at least a portion of said resulting detected face area with at least on alert threshold parameter to determine if associated system initialization process is required.

11. The method as recited in claim 8, wherein detecting said at least one point within said detected second portion of said face further includes:

detecting at least one key facial component within said at least one detected face area.

12. The method as recited in claim 11, further comprising:

determining a fine head pose of said user based on said at least three detected key facial components.

13. The method as recited in claim 11, further comprising:

determining a coarse head pose of said user based on confidence information associated with detecting said at least one key facial component within said at least one detected face area.

14. The method as recited in claim 13, further comprising:

determining a fine head pose of said user based on said detected key facial components and said confidence information.

15. The method as recited in claim 13, wherein said coarse head pose is associated with at least one head pose parameter selected from a group of head pose parameters comprising a yaw angle, a tilt angle, a roll angle, an x translation, a y translation, and a scale factor.

16. The method as recited in claim 12, wherein determining said fine head pose further includes:

determining said fine head pose using at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

17. The method as recited in claim 12, wherein determining said fine head pose further includes:

determining said fine head pose using at least one view-based pose estimation technique.

18. The method as recited in claim 1, wherein at least one of said at least two selectable regions of said at least one display includes at least one operatively selectable region within a graphical user interface (GUI).

19. The method as recited in claim 1, wherein each of said at least two selectable regions is associated with different computer logic or different computer data.

20. The method as recited in claim 1, wherein automatically operatively switching between said at least two selectable regions of said at least one display device further includes:

selectively causing a graphical pointing device to move on said display device.

21. The method as recited in claim 1, wherein automatically operatively switching between said at least two selectable regions of said at least one display device further includes:

selectively delaying automatically operatively switching between said at least two selectable regions of said at least one display device.

22. A computer-readable medium having computer-implementable instructions for causing at least one processor to perform acts comprising:

assessing a first head pose for a user of the computing device at a first time and at least a second head pose for said user at a second time that is subsequent to said first time; and

automatically operatively switching between at least two selectable regions of at least one display device of the computing device based on at least one difference between said first head pose and at least said second head pose.

23. The computer-readable medium as recited in claim 22, wherein:

assessing said first head pose further includes generating a first image showing at least a first portion of a face of said user, detecting at least said first portion of said face within said first image, and detecting at least one point within said detected first portion of said face and storing first tracking information associated with each of said at least one point; and

assessing said second head pose further includes generating a second image showing at least a second portion of said face of said user, tracking/detecting at least said second portion of said face within said second image, and detecting said at least one point within said detected second portion of said face and storing second tracking information associated with each of said at least one point.

24. The computer-readable medium as recited in claim 23, wherein automatically operatively switching between said at least two selectable regions further includes:

comparing at least said first tracking information and said second tracking information to determine if an assessed display device view associated with said user has changed between said first time to said second time, and if said assessed display device view associated with said user has changed between said first time to said second time, then selectively operatively switching a user input interface focus presented on said display device to a selectable region associated with said assessed display device view.

25. The computer-readable medium as recited in claim 23, wherein generating said first image further includes causing at least one camera device to output said first image.

26. The computer-readable medium as recited in claim 25, wherein said at least one camera device includes at least one video camera device and said first image includes a frame of video captured by said video camera device.

27. The computer-readable medium as recited in claim 23, wherein detecting said first portion of said face within said first image further includes:

classifying each of a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region associated with said first portion of said face.

28. The computer-readable medium as recited in claim 27, further comprising:

image patch tracking said facial region associated with said first portion of said face in a corresponding portion of said second image to identify at least one detected face area.

29. The computer-readable medium as recited in claim 28, wherein image patch tracking further includes:

using a sum-of-square difference (SSD) image patch tracker to identify said at least one detected face area.

30. The computer-readable medium as recited in claim 28, wherein image patch tracking further includes:

comparing at least a portion of said resulting detected face area with at least on alert threshold parameter to determine if associated system initialization process is required.

31. The computer-readable medium as recited in claim 28, wherein detecting said at least one point within said detected second portion of said face further includes:

detecting at least one key facial components within said at least one detected face area.

32. The computer-readable medium as recited in claim 31, further comprising:

determining a fine head pose of said user based on said detected key facial components.

33. The computer-readable medium as recited in claim 31, further comprising:

determining a coarse head pose of said user based on confidence information associated with detecting said at least one key facial component within said at least one detected face area.

34. The computer-readable medium as recited in claim 33, further comprising:

determining a fine head pose of said user based on said detected key facial components and said confidence information.

35. The computer-readable medium as recited in claim 33, wherein said coarse head pose is associated with at least one head pose parameter selected from a group of head pose parameter comprising a yaw angle, a tilt angle, a roll angle, an x translation, a y translation, and a scale factor.

36. The computer-readable medium as recited in claim 32, wherein determining said fine head pose further includes:

determining said fine head pose using at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

37. The computer-readable medium as recited in claim 32, wherein determining said fine head pose further includes:

determining said fine head pose using at least one view-based pose estimation technique.

38. The computer-readable medium as recited in claim 22, wherein at least one of said at least two selectable regions of said at least one display includes at least one operatively selectable region within a graphical user interface (GUI).

39. The computer-readable medium as recited in claim 22, wherein each of said at least two selectable regions is associated with different computer logic or different computer data.

40. The computer-readable medium as recited in claim 22, wherein automatically operatively switching between said at least two selectable regions of said at least one display device further includes:

selectively causing a graphical pointing device to move on said display device.

41. The computer-readable medium as recited in claim 22, wherein automatically operatively switching between said at least two selectable regions of said at least one display device further includes:

selectively delaying automatically operatively switching between said at least two selectable regions of said at least one display device.

42. An apparatus for use with at least one display device and at least one image capturing device, the apparatus comprising:

display logic configurable to output at least one signal suitable for causing a display device to display at least two different selectable regions;

interface logic configurable to receive image data from an image capturing device; and

pose estimation logic operatively coupled to said display logic and said interface logic and configured to determine a first head pose based on a first image and at least a second head pose based on a second image temporally subsequent to said first image, and automatically operatively switch an operative user input focus between said at least two selectable regions based on at least one difference between said first head pose and at least said second head pose.

43. The apparatus as recited in claim 42, further comprising:

memory operatively coupled to at least said pose estimation logic, said memory being configurable to store said first image and said second, wherein said first image captures at least a first portion of a face of a user at first time and said second image captures at least a second portion of said face of said user at a second subsequent time; and

wherein said pose estimation logic is further configured to;

access said first image, detect at least said first portion of said face within said first image, detect at least one point within said detected first portion of said face, and store first tracking information associated with each of said at least one point within said detected first portion to said memory, and

access said second image, track at least said second portion of said face within said second image, detect said at least one point within said detected second portion of said face, and store second tracking information associated with each of said at least one point within said detected second portion to said memory.

44. The apparatus as recited in claim 43, wherein said pose estimation logic is further configured to compare at least said first tracking information and said second tracking information to determine if an assessed display device view associated with said user has changed between said first time to said second time, and if said assessed display device view associated with said user has changed between said first time to said second time, then switch said operative user input focus.

45. The apparatus as recited in claim 43, wherein said pose estimation logic is further configured to classify each of a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region associated with said first portion of said face.

46. The apparatus as recited in claim 45, said pose estimation logic is further configured to image patch track said facial region associated with said first portion of said face in a corresponding portion of said second image to identify at least one detected face area.

47. The apparatus as recited in claim 46, wherein said pose estimation logic includes a sum-of-square difference (SSD) image patch tracker to identify said at least one detected face area.

48. The apparatus as recited in claim 46, wherein said image patch tracker compares at least a portion of said resulting detected face area with at least on alert threshold parameter to determine if an associated system initialization process is required.

49. The apparatus as recited in claim 45, wherein said pose estimation logic is configured to detect at least one key facial component within said at least one detected face area to conduct the coarse pose estimation.

50. The apparatus as recited in claim 49, wherein said pose estimation logic is configured to determine a fine head pose of said user based on said detected key facial components.

51. The apparatus as recited in claim 49, wherein said pose estimation logic is configured to determine a coarse head pose of said user based on confidence information associated with detecting said at least one key facial component within said at least one detected face area.

52. The apparatus as recited in claim 51, wherein said pose estimation logic is configured to determine a fine head pose of said user based on said detected key facial components and said confidence information.

53. The apparatus as recited in claim 51, wherein said coarse head pose is associated with at least one head pose parameter selected from a group of head pose parameter comprising a yaw angle, a tilt angle, a roll angle, an x translation, a y translation, and a scale factor.

54. The apparatus as recited in claim 50, wherein said pose estimation logic is configured to determine said fine head pose using by combining the results of at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

55. The apparatus as recited in claim 50, wherein said pose estimation logic is configured to determine said fine head pose using by combining the results of at least one view-based pose estimation technique.

56. A system comprising:

at least one display device;

at least one image capturing device; and

a computing device operatively coupled to said display device and said image capturing device and having;

display logic configured to output at least one signal suitable for causing a display device to display at least two different selectable regions,

interface logic configured to receive image data from an image capturing device, and −

pose estimation logic configured to determine a first head pose based on a first image and at least a second head pose based on a second image temporally subsequent to said first image, and automatically operatively switch an operative user input focus between said at least two selectable regions based on at least one difference between said first head pose and at least said second head pose.