Method and system for head tracking and pose estimation

• US 8,781,162 B2
• Filed: 01/05/2011
• Issued: 07/15/2014
• Est. Priority Date: 01/05/2011
• Status: Active Grant

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: Powers, III, William Robert, Zhu, Youding, Wilkinson, Dana, Musick, JR., Charles, Reynolds, Stuart, Kay, Robert
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): HEIDEMANN, JASON E
• Application Number:
  US12/984,606
• Publication Number:
  US 20120169887A1
• Time in Patent Office: 1,287 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques for performing accurate and automatic head pose estimation are disclosed. According to one aspect of the techniques, head pose estimation is integrated with a scale-invariant head tracking method along with facial features detected from a located head in images. Thus the head pose estimation works efficiently even when there are large translational movements resulting from the head motion. Various computation techniques are used to optimize the process of estimation so that the head pose estimation can be applied to control one or more objects in a virtual environment and virtual character gaze control.

Claims

1. A method for determining motion of a head, the method comprising:

generating a sequence of images from a camera looking at a user;

estimating a position and a size of a head of the user in each of the images using a scale-invariant head tracking technique designed to scan the each of the images at all scales; and

determining pose of the head from the position and size of the head, wherein said determining pose of the head from the position and size of the head uses non-linear optimization to minimize a sum of squared distances between tracked facial features and projected corresponding features obtained from a 3D head model.

2. The method as recited in claim 1, wherein said estimating a position and a size of the head in each of the images using a scale-invariant head tracking technique comprises:

tracking in each of the images to determine whether the head is present or not in the each of the images;

defining a region of interest when the head is detected in the each of the images;

locating the position of the head in the region of interest; and

measuring the size of the head in the region of interest.

3. The method as recited in claim 2, wherein said tracking in each of the images comprises:

tracking the head under partial occlusion;

tracking the head with surrounded with other heads without being distracted;

or

tracking a temporarily occluded head.

4. The method as recited in claim 3, wherein a color histogram and a gradient histogram are computed for the each of the images to facilitate the tracking of the head.

5. The method as recited in claim 2, wherein said tracking in each of the images is performed on multiple scales of the one of the images without degrading tracking results.

6. The method as recited in claim 2, further comprising:

detecting the head in the region of interest;

checking a position and a size of the head in the region of interest; and

updating the position and size of the head obtained from said estimating a position and a size of a head of the user in each of the images when the position and the size of the head in the region of interest are so different from the position and size of the head previously obtained.

7. The method as recited in claim 1, wherein analytical Jacobian computation is utilized to speed up the non-linear optimization with second-order Gaussian-Newton method, and use residual errors to confirm convergence of the non-linear optimization.

8. The method as recited in claim 1, wherein the position of the head and the size of the head in the region of interest are further provided to a head pose initialization unit to get frontal facial features of the head or a head pose tracker unit to define the region of interest.

9. The method as recited in claim 8, wherein the frontal facial features include eye centers, nose, and mouth corners of the head, and the head pose tracker unit works with large and fast translation movements of the head.

10. The method as recited in claim 8, wherein the head pose tracker defines the region of interest at two successive frames.

11. The method as recited in claim 10, wherein the head pose tracker utilizes an optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking with reference frames.

12. The method as recited in claim 11, wherein the optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking compensate each other to facilitate better pose estimation of the head in the images.

13. The method as recited in claim 12, wherein the pose estimation is based on a generic model provided to be imposed with detected facial features from the images.

14. The method as recited in claim 12, wherein the pose estimation is performed by minimizing a sum of squared distances between detected facial features and projected model vertices.

15. The method as recited in claim 10, wherein the head pose tracker possesses features of feature point tracking with large head movement compensation based on scale-invariant head tracking, model-based head pose estimation with sparse optical flow, model-based head pose estimation with reference frames, and head pose estimation result integrated with Kalman filtering.

16. The method as recited in claim 1, wherein the pose of the head provides a natural and intuitive motion control interface to control one or more objects in a virtual environment.

17. The method as recited in claim 16, wherein the virtual environment is part of a video game being displayed on a display unit.

18. A device for determining motion of a head, the device comprising:

an interface, coupled to a camera, to receive a sequence of images from the camera disposed to look at a user;

a memory space for storing code;

a processor, coupled to the memory space, executing the code to perform operations of;

estimating a position and a size of a head of the user in each of the images using a scale-invariant head tracking technique designed to scan the each of the images at all scales; and

determining pose of the head from the position and size of the head, wherein said determining pose of the head from the position and size of the head uses non-linear optimization to minimize a sum of squared distances between tracked facial features and projected corresponding feature obtained from a 3D head model.

19. The device as recited in claim 18, wherein said estimating a position and a size of the head in each of the images using a scale-invariant head tracking technique comprises:

tracking in each of the images to determine whether there is a head or not in the each of the images;

defining a region of interest when the head is detected in the each of the images;

locating the position of the head in the region of interest; and

measuring the size of the head in the region of interest.

20. The device as recited in claim 19, wherein said tracking in each of the images comprises:

tracking the head under partial occlusion;

tracking the head with surrounding other heads without being distracted;

or

tracking a temporarily occluded head.

21. The device as recited in claim 20, wherein a color histogram and a gradient histogram are computed for the each of the images to facilitate the tracking of the head.

22. The device as recited in claim 19, wherein said tracking in each of the images is performed on multiple scales of the one of the images without degrading tracking results.

23. The device as recited in claim 18, wherein the operations further comprises:

detecting the head in the region of interest;

checking a position and a size of the head in the region of interest; and

updating the position and size of the head obtained from said estimating a position and a size of a head of the user in each of the images when the position and the size of the head in the region of interest are so different from the position and size of the head previously obtained.

24. The device as recited in claim 18, wherein analytical Jacobian computation is utilized to speed up the non-linear optimization with second-order Gaussian-Newton device, and use residual errors to confirm convergence of the non-linear optimization.

25. The device as recited in claim 18, wherein the position of the head and the size of the head in the region of interest are further provided to a head pose initialization unit to get frontal facial features of the head or a head pose tracker unit to define the region of interest.

26. The device as recited in claim 25, wherein the frontal facial features include eye centers, nose, and mouth corners of the head, and the head pose tracker unit works with large and fast translation movements of the head.

27. The device as recited in claim 25, wherein the head pose tracker defines the region of interest at two successive frames.

28. The device as recited in claim 27, wherein the head pose tracker utilizes an optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking with reference frames.

29. The device as recited in claim 28, wherein the optical flow based feature tracking and Speeded up robust features (SURF) based feature tracking compensate each other to facilitate better pose estimation of the head in the images.

30. The device as recited in claim 29, wherein the pose estimation is based on a generic model provided to be imposed with detected facial features from the images.

31. The device as recited in claim 29, wherein the pose estimation is performed by minimizing a sum of squared distances between detected facial features and projected model vertices.

32. The device as recited in claim 27, wherein the head pose tracker possesses features of feature point tracking with large head movement compensation based on scale-invariant head tracking, model-based head pose estimation with sparse optical flow, model-based head pose estimation with reference frames, and head pose estimation result integrated with Kalman filtering.

33. The device as recited in claim 18, wherein the pose of the head provides a natural and intuitive motion control interface to control one or more objects in a virtual environment.

34. The device as recited in claim 33, wherein the virtual environment is part of a video game being displayed on a display unit.