Method and system for head tracking and pose estimation

• US 20120169887A1
• Filed: 01/05/2011
• Published: 07/05/2012
• Est. Priority Date: 01/05/2011
• Status: Active Grant

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: Reynolds, Stuart, Kay, Robert, Wilkinson, Dana, Zhu, Youding, Musick, JR., Charles, Powers, III, William Robert
• Granted Patent: US 8,781,162 B2
• Time in Patent Office: Days
• US Class Current: 348/207.1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques for performing accurate and automatic head pose estimation are disclosed. According to one aspect of the techniques, head pose estimation is integrated with a scale-invariant head tracking method along with facial features detected from a located head in images. Thus the head pose estimation works efficiently even when there are large translational movements resulting from the head motion. Various computation techniques are used to optimize the process of estimation so that the head pose estimation can be applied to control one or more objects in a virtual environment and virtual character gaze control.

Claims

1. A method for determining motion of a head, the method comprising:

generating a sequence of images from a camera looking at a user;

estimating a position and a size of a head of the user in each of the images using a scale-invariant head tracking technique designed to scan the each of the images at all scales; and

determining pose of the head from the position and size of the head.

2. The method as recited in claim 1, wherein said estimating a position and a size of the head in each of the images using a scale-invariant head tracking technique comprises:

tracking in each of the images to determine whether there is a head or not;

defining a region of interest when the head is detected in the each of the images;

locating the position of the head in the region of interest; and

measuring the size of the head in the region of interest.

3. The method as recited in claim 2, wherein said tracking in each of the images is capable of:

tracking the head under partial occlusion;

tracking the head with surrounding other heads without being distracted;

or

tracking a temporarily occluded head.

4. The method as recited in claim 3, wherein a color histogram and a gradient histogram are computed for the each of the images to facilitate the tracking of the head.

5. The method as recited in claim 2, wherein said tracking in each of the images is performed at multi-scale of the one of the images without degrading tracking results.

6. The method as recited in claim 2, further comprising:

detecting the head in the region of interest;

checking a position and a size of the head in the region of interest; and

updating the position and size of the head obtained from said estimating a position and a size of a head of the user in each of the images when the position and the size of the head in the region of interest are so different from the position and size of the head previously obtained.

7. The method as recited in claim 1, wherein said determining pose of the head from the position and size of the head operates on non-linear optimization to minimize a sum of squared distances between tracked facial features and projected corresponding features obtained from a 3D head model.

8. The method as recited in claim 7, wherein analytical Jacobian computation is utilized to speed up the non-linear optimization with second-order Gaussian-Newton method, and use residual errors to confirm convergence of the non-linear optimization.

9. The method as recited in claim 1, wherein the position of the head and the size of the head in the region of interest are further provided to a head pose initialization unit to get frontal facial features of the head or a head pose tracker unit to define the region of interest.

10. The method as recited in claim 9, wherein the frontal facial features include eye centers, nose, and mouth corners of the head, and the head pose tracker unit works with large and fast translation movements of the head.

11. The method as recited in claim 9, wherein the head pose tracker defines the region of interest at two successive frames.

12. The method as recited in claim 11, wherein the head pose tracker utilizes an optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking with reference frames.

13. The method as recited in claim 12, wherein the optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking compensate each other to facilitate better pose estimation of the head in the images.

14. The method as recited in claim 13, wherein the pose estimation is based on a generic model provided to be imposed with detected facial features from the images.

15. The method as recited in claim 13, wherein the pose estimation is performed by minimizing a sum of squared distances between detected facial features and projected model vertices.

16. The method as recited in claim 11, wherein the head pose tracker possesses features of feature point tracking with large head movement compensation based on scale-invariant head tracking, model-based head pose estimation with sparse optical flow, model-based head pose estimation with reference frames, and head pose estimation result integrated with Kalman filtering.

17. The method as recited in claim 1, wherein the pose of the head provides a natural and intuitive motion control interface to control one or more objects in a virtual environment.

18. The method as recited in claim 17, wherein the virtual environment is part of a video game being displayed on a display unit.

19. A device for determining motion of a head, the device comprising:

an interface, coupled to a camera, to receive a sequence of images from the camera disposed to look at a user;

a memory space for storing code;

a processor, coupled to the memory space, executing the code to perform operations of;

estimating a position and a size of a head of the user in each of the images using a scale-invariant head tracking technique designed to scan the each of the images at all scales; and

determining pose of the head from the position and size of the head.

20. The device as recited in claim 19, wherein said estimating a position and a size of the head in each of the images using a scale-invariant head tracking technique comprises:

tracking in each of the images to determine whether there is a head or not;

defining a region of interest when the head is detected in the each of the images;

locating the position of the head in the region of interest; and

measuring the size of the head in the region of interest.

21. The device as recited in claim 20, wherein said tracking in each of the images is capable of:

tracking the head under partial occlusion;

tracking the head with surrounding other heads without being distracted;

or

tracking a temporarily occluded head.

22. The device as recited in claim 21, wherein a color histogram and a gradient histogram are computed for the each of the images to facilitate the tracking of the head.

23. The device as recited in claim 20, wherein said tracking in each of the images is performed at multi-scale of the one of the images without degrading tracking results.

24. The device as recited in claim 20, wherein the operations further comprises:

detecting the head in the region of interest;

checking a position and a size of the head in the region of interest; and

updating the position and size of the head obtained from said estimating a position and a size of a head of the user in each of the images when the position and the size of the head in the region of interest are so different from the position and size of the head previously obtained.

25. The device as recited in claim 19, wherein said determining pose of the head from the position and size of the head operates on non-linear optimization to minimize a sum of squared distances between tracked facial features and projected corresponding features obtained from a 3D head model.

26. The device as recited in claim 25, wherein analytical Jacobian computation is utilized to speed up the non-linear optimization with second-order Gaussian-Newton device, and use residual errors to confirm convergence of the non-linear optimization.

27. The device as recited in claim 19, wherein the position of the head and the size of the head in the region of interest are further provided to a head pose initialization unit to get frontal facial features of the head or a head pose tracker unit to define the region of interest.

28. The device as recited in claim 27, wherein the frontal facial features include eye centers, nose, and mouth corners of the head, and the head pose tracker unit works with large and fast translation movements of the head.

29. The device as recited in claim 27, wherein the head pose tracker defines the region of interest at two successive frames.

30. The device as recited in claim 29, wherein the head pose tracker utilizes an optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking with reference frames.

31. The device as recited in claim 20, wherein the optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking compensate each other to facilitate better pose estimation of the head in the images.

32. The device as recited in claim 31, wherein the pose estimation is based on a generic model provided to be imposed with detected facial features from the images.

33. The device as recited in claim 31, wherein the pose estimation is performed by minimizing a sum of squared distances between detected facial features and projected model vertices.

34. The device as recited in claim 29, wherein the head pose tracker possesses features of feature point tracking with large head movement compensation based on scale-invariant head tracking, model-based head pose estimation with sparse optical flow, model-based head pose estimation with reference frames, and head pose estimation result integrated with Kalman filtering.

35. The device as recited in claim 19, wherein the pose of the head provides a natural and intuitive motion control interface to control one or more objects in a virtual environment.

36. The device as recited in claim 35, wherein the virtual environment is part of a video game being displayed on a display unit.