Head pose assessment methods and systems

• US 7,391,888 B2
• Filed: 05/30/2003
• Issued: 06/24/2008
• Est. Priority Date: 05/30/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Zhang, Lei, Zhang, Hong-Jiang, Hu, Yuxiao
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US10/452,783
• Publication Number:
  US 20040240708A1
• Time in Patent Office: 1,852 Days
• Field of Search: 382/100, 382/103, 382/106, 382/107, 382/190, 382/195, 382/117, 382/118, 382/181, 382/189, 382/201, 382/209, 382/225, 345/619, 345/156, 345/157, 345/158, 345/175, 345/177, 345/181, 345/204, 345/207, 348/169
• US Class Current: 382/118
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

Improvements are provided to effectively assess a user'"'"'s face and head pose such that a computer or like device can track the user'"'"'s attention towards a display device(s). Then the region of the display or graphical user interface that the user is turned towards can be automatically selected without requiring the user to provide further inputs. A frontal face detector is applied to detect the user'"'"'s frontal face and then key facial points such as left/right eye center, left/right mouth corner, nose tip, etc., are detected by component detectors. The system then tracks the user'"'"'s head by an image tracker and determines yaw, tilt and roll angle and other pose information of the user'"'"'s head through a coarse to fine process according to key facial points and/or confidence outputs by pose estimator.

Claims

1. A method for use with a computing device comprising:

determining a first head pose for said a user of said computing device at a first time, said determining of said first pose including generating a first image showing a portion of said first head pose and detecting at least two different points within said first image showing said portion of said first head pose;

determining at least a second head pose for said user at a second time that is subsequent to said first time, said determining of said second pose including generating a second image showing a portion of said second head pose and detecting at least two different points within said second image showing said portion of said second head pose, said at least two different points within said second image corresponding to said at least two different points detected within said first image;

automatically operatively switching between at least two selectable regions of at least one display device of said computing device based on at least one difference between said first head pose and at least said second head pose, said at least two selectable regions each being a separate graphical user interface (GUI) associated with one or more programs or computer logic executed by said computing device; and

storing work status information related to at least one of the at least two selectable regions, the work status information retrieved when automatically operatively switching between the at least two selectable regions.

2. The method as recited in claim 1, wherein said determining said first head pose includes:

storing first tracking information associated with at least one portion of said at least two different points within said first image showing a portion of said first head pose; and

storing second tracking information associated with at least one point of said at least two different points within said second image showing a portion of said second head pose.

3. The method as recited in claim 2, wherein said automatically operatively switching between said at least two selectable regions includes:

comparing at least said first tracking information and said second tracking information to determine if an assessed display device view associated with said user has changed between said first time to said second time; and

selectively operatively switching a user input interface focus presented on said display device to a selectable region associated with said assessed display device view.

4. The method as recited in claim 1, wherein said generating said first image includes causing at least one camera device to output said first image.

5. The method as recited in claim 4, wherein said at least one camera device includes at least one video camera device and said first image includes a frame of video captured by said video camera device.

6. The method as recited in claim 4, wherein said at least one camera device includes at least one digital still camera device and said first image includes a digital still image captured by said digital still camera device.

7. The method as recited in claim 1, further comprising classifying a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region.

8. The method as recited in claim 7, further comprising:

image patch tracking said facial region in a corresponding portion of said second image to identify at least one detected face area.

9. The method as recited in claim 8, wherein said image patch tracking includes:

using a sum-of-square difference (SSD) image patch tracker to identify said at least one detected face area.

10. The method as recited in claim 8, wherein said image patch tracking includes:

comparing at least a portion of said resulting detected face area with at least one alert threshold parameter to determine if associated system initialization process is required.

11. The method as recited in claim 8, further comprising:

determining a coarse head pose of said user based on confidence information associated with detecting at least one key facial component within said at least one detected face area.

12. The method as recited in claim 11, further comprising:

determining a fine head pose of said user based on said detected key facial component and said confidence information.

13. The method as recited in claim 11, wherein said coarse head pose is associated with at least one head pose parameter selected from a group of head pose parameters comprising a yaw angle, a tilt angle, a roll angle, an x translation, a y translation, and a scale factor.

14. The method as recited in claim 1, further comprising:

determining a fine head pose of said user based on at least three detected key facial components.

15. The method as recited in claim 14, wherein said determining said fine head pose further includes:

determining said fine head pose using at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

16. The method as recited in claim 14, wherein said determining said fine head pose includes:

determining said fine head pose using at least one view-based pose estimation technique.

17. The method as recited in claim 1, wherein at least one of said at least two selectable regions of said at least one display includes at least one operatively selectable graphical user interface (GUI) region within a GUI.

18. The method as recited in claim 1, wherein each of said at least two selectable regions is associated with a different program, different computer logic, or different computer data.

19. The method as recited in claim 1, wherein said automatically operatively switching between said at least two selectable regions of said at least one display device includes:

selectively causing a graphical pointing device to move on said display device.

20. The method as recited in claim 1, wherein said automatically operatively switching between said at least two selectable regions of said at least one display device includes:

selectively delaying automatically operatively switching between said at least two selectable regions of said at least one display device.

21. The method as recited in claim 1, wherein the work status information includes at least one of an activated selectable region, a pointer position, and an insert cursor.

22. A computer-readable media storing computer-implementable instructions for causing at least one processor to perform acts comprising:

determining a first head pose at a first time, said determining of said first pose including generating a first image showing a portion of said first head pose and detecting at least two different points within said first image showing said portion of said first head pose;

determining at least a second head pose at a second time that is subsequent to said first time, said determining of said second pose including generating a second image showing a portion of said second head pose and detecting at least two different points within said second image showing said portion of said second head pose, said at least two different points within said second image corresponding to said at least two different points detected within said first image;

assessing said first head pose and said second head pose;

automatically operatively switching between at least two selectable regions of at least one display device of said computing device based on at least one difference between said first head pose and at least said second head pose, said at least two selectable regions each being a separate graphical user interface (GUI) associated with one or more programs or computer logic executed by said computing device; and

storing work status information related to at least one of the at least two selectable regions, the work status information retrieved when automatically operatively switching between the at least two selectable regions.

23. The computer-readable media as recited in claim 22, wherein:

said assessing said first head pose includes storing first tracking information associated with each of said at least two different points within said first image showing said portion of said first head pose; and

said assessing said at least second head pose includes storing second tracking information associated with each of said at least two different points within said second image showing said portion of said second head pose.

24. The computer-readable media as recited in claim 23, wherein said automatically operatively switching between said at least two selectable regions includes:

comparing at least said first tracking information and said second tracking information to determine if an assessed display device view has changed between said first time to said second time, and selectively operatively switching a user input interface focus presented on said display device to a selectable region associated with said assessed display device view.

25. The computer-readable media as recited in claim 23, wherein said generating said first image includes causing at least one camera device to output said first image.

26. The computer-readable media as recited in claim 25, wherein said at least one camera device includes at least one video camera device and said first image includes a frame of video captured by said video camera device.

27. The computer-readable media as recited in claim 22, further comprising classifying each of a plurality of portions of image data associated with said first image based on at least one classifying parameter to determine at least one facial region associated with a first portion of a face.

28. The computer-readable media as recited in claim 27, wherein the computer-implementable instructions further comprise:

image patch tracking said facial region associated with said first portion of said face in a corresponding portion of said second image to identify at least one detected face area.

29. The computer-readable media as recited in claim 28, wherein said image patch tracking includes:

using a sum-of-square difference (SSD) image patch tracker to identify said at least one detected face area.

30. The computer-readable media as recited in claim 28, wherein said image patch tracking includes:

comparing at least a portion of said resulting detected face area with at least on alert threshold parameter to determine if associated system initialization process is required.

31. The computer-readable media as recited in claim 28, wherein the computer-implementable instructions further comprise:

determining a coarse head pose of said user based on confidence information associated with detecting at least one key facial component within said at least one detected face area.

32. The computer-readable media as recited in claim 31, wherein the computer-implementable instructions further comprise:

determining a fine head pose of said user based on said detected key facial component and said confidence information.

33. The computer-readable media as recited in claim 31, wherein said coarse head pose is associated with at least one head pose parameter selected from a group of head pose parameter comprising a yaw angle, a tilt angle, a roll angle, an x translation, a y translation, and a scale factor.

34. The computer-readable media as recited in claim 22, wherein the computer-implementable instructions further comprise:

determining a fine head pose of said user based on detected key facial components.

35. The computer-readable media as recited in claim 34, wherein said determining said fine head pose includes:

determining said fine head pose using at least one estimator technique selected from a group of estimating techniques comprising an ellipse estimator technique and an iterated estimator technique.

36. The computer-readable media as recited in claim 34, wherein said determining said fine head pose includes:

determining said fine head pose using at least one view-based pose estimation technique.

37. The computer-readable media as recited in claim 22, wherein at least one of said at least two selectable regions of said at least one display includes at least one operatively selectable graphical user interface (GUI) region within a graphical GUI.

38. The computer-readable media as recited in claim 22, wherein each of said at least two selectable regions is associated with a different program, different computer logic, or different computer data.

39. The computer-readable media as recited in claim 22, wherein said automatically operatively switching between said at least two selectable regions of said at least one display device includes:

selectively causing a graphical pointing device to move on said display device.

40. The computer-readable media as recited in claim 22, wherein said automatically operatively switching between said at least two selectable regions of said at least one display device includes:

selectively delaying automatically operatively switching between said at least two selectable regions of said at least one display device.

41. The computer-readable media as recited in claim 22, wherein the work status information includes at least one of an activated selectable region, a pointer position, and an insert cursor.