Method and system for head tracking and pose estimation

• US 20140320691A1
• Filed: 07/11/2014
• Published: 10/30/2014
• Est. Priority Date: 01/05/2011
• Status: Active Grant

• Current Assignee: AiLive, Inc.
• Original Assignee: AiLive, Inc.
• Inventors: Reynolds, Stuart, Kay, Robert, Wilkinson, Dana, Zhu, Youding, Musick, JR., Charles, Powers, III, William Robert
• Granted Patent: US 9,292,734 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques for performing accurate and automatic head pose estimation are disclosed. According to one aspect of the techniques, head pose estimation is integrated with a scale-invariant head tracking method along with facial features detected from a located head in images. Thus the head pose estimation works efficiently even when there are large translational movements resulting from the head motion. Various computation techniques are used to optimize the process of estimation so that the head pose estimation can be applied to control one or more objects in a virtual environment and virtual character gaze control.

Claims

1. A method for determining pose of a head, the method comprising:

generating a sequence of images from a camera provided to look at a user, wherein the user is not required to wear any marker or any special device;

estimating a position and a size of the head of the user in at least one image from a plurality of images generated from the camera, wherein said estimating a position and a size of the head of the user is performed using a scale-invariant head tracking technique designed to scan at least one image regardless of whatever scale the at least one image is in; and

determining the pose of the head from the position and size of the head, wherein said determining the pose of the head from the position and size of the head is performed based on tracked facial features in the at least one image and projected corresponding features obtained from a 3D head model.

2. The method as recited in claim 1, wherein said estimating a position and a size of the head of the user in at least one image comprises:

tracking in the at least one image to determine whether the head is present or not in the at least one image;

defining a region of interest when the head is detected in the at least one image;

locating the position of the head in the at least one image; and

measuring the size of the head in the region of interest.

3. The method as recited in claim 2, wherein said tracking in the at least one image comprises:

tracking the head under partial occlusion;

tracking the head being surrounded with other heads without being distracted;

or

tracking a temporarily occluded head.

4. The method as recited in claim 3, wherein a color histogram and a gradient histogram are computed for the tracking in the at least one image to facilitate the tracking of the head.

5. The method as recited in claim 2, wherein said tracking in the at least one image is performed on multiple scales of the tracking in the at least one image without degrading tracking results.

6. The method as recited in claim 2, further comprising:

detecting the head in the region of interest;

checking a position and a size of the head in the region of interest; and

updating the position and size of the head obtained from said estimating a position and a size of the head of the user in at least one image when the position and the size of the head in the region of interest are so different from the position and size of the head previously obtained.

7. The method as recited in claim 2, wherein the position of the head and the size of the head in the region of interest are further provided to a head pose initialization unit to get frontal facial features of the head or a head pose tracker unit to define the region of interest.

8. The method as recited in claim 7, wherein the frontal facial features include eye centers, nose, and mouth corners of the head, and a head pose tracker unit is configured to work with large and fast translation movements of the head.

9. The method as recited in claim 1, wherein said determining the pose of the head from the position and size of the head is based on a generic model provided to be imposed with the detected facial features from the images.

10. The method as recited in claim 1, wherein said determining the pose of the head from the position and size of the head uses non-linear optimization to minimize a sum of squared distances between the tracked facial features and the projected corresponding features obtained from the 3D head model.

11. A device for determining pose of a head, the device comprising:

an interface, coupled to a camera, to receive a sequence of images from the camera disposed to look at a user, wherein the user is not required to wear any marker or any special device;

a memory space for storing code;

a processor, coupled to the memory space, executing the code to perform operations of;

estimating a position and a size of the head of the user in at least one image from a plurality of images generated from the camera, wherein said estimating a position and a size of the head of the user is performed using a scale-invariant head tracking technique designed to scan at least one image regardless of whatever scale the at least one image is in; and

determining the pose of the head from the position and size of the head, wherein said determining the pose of the head from the position and size of the head is performed based on tracked facial features in the at least one image and projected corresponding features obtained from a 3D head model.

12. The device as recited in claim 11, wherein said estimating a position and a size of the head of the user comprises:

tracking in the at least one image to determine whether the head is present or not in the at least one image;

defining a region of interest when the head is detected in the at least one image;

locating the position of the head in the at least one image; and

measuring the size of the head in the region of interest.

13. The device as recited in claim 12, wherein said tracking in the at least one image comprises:

tracking the head under partial occlusion;

tracking the head being surrounded with other heads without being distracted;

or

tracking a temporarily occluded head.

14. The device as recited in claim 13, wherein a color histogram and a gradient histogram are computed for the tracking in the at least one image to facilitate the tracking of the head.

15. The device as recited in claim 12, wherein said tracking in the at least one image is performed on multiple scales of the tracking in the at least one image without degrading tracking results.

16. The device as recited in claim 12, wherein the operations further comprise:

detecting the head in the region of interest;

checking a position and a size of the head in the region of interest; and

updating the position and size of the head obtained from said estimating a position and a size of the head of the user in at least one image when the position and the size of the head in the region of interest are so different from the position and size of the head previously obtained.

17. The device as recited in claim 12, wherein the position of the head and the size of the head in the region of interest are further provided to a head pose initialization unit to get frontal facial features of the head or a head pose tracker unit to define the region of interest.

18. The device as recited in claim 17, wherein the frontal facial features include eye centers, nose, and mouth corners of the head, and a head pose tracker unit is configured to work with large and fast translation movements of the head.

19. The device as recited in claim 11, wherein said determining the pose of the head from the position and size of the head is based on a generic model provided to be imposed with the detected facial features from the images.

20. The device as recited in claim 11, wherein said determining the pose of the head from the position and size of the head uses non-linear optimization to minimize a sum of squared distances between the tracked facial features and the projected corresponding features obtained from the 3D head model.