Head pose estimation using RGBD camera

• US 9,582,707 B2
• Filed: 04/25/2012
• Issued: 02/28/2017
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Sharma, Piyush, Swaminathan, Ashwin, Rezaiifar, Ramin, Xue, Qi
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): WALKER, JARED T
• Application Number:
  US13/456,061
• Publication Number:
  US 20120293635A1
• Time in Patent Office: 1,770 Days
• Field of Search: 348143-147, 348/135, 348/137, 348/139
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

A three-dimensional pose of the head of a subject is determined based on depth data captured in multiple images. The multiple images of the head are captured, e.g., by an RGBD camera. A rotation matrix and translation vector of the pose of the head relative to a reference pose is determined using the depth data. For example, arbitrary feature points on the head may be extracted in each of the multiple images and provided along with corresponding depth data to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with the reference pose for the head and a current orientation and a current position. The three-dimensional pose of the head with respect to the reference pose is then determined based on the rotation matrix and the translation vector.

Claims

1. A method comprising:

capturing a series of images with depth data of a head of a subject;

obtaining a reference pose for the head of the subject from one image in the series of images and using the reference pose to define a reference coordinate frame for the series of images;

determining a rotation matrix and a translation vector associated with a pose of the head in each image in the series of images relative to the reference coordinate frame using the depth data;

extracting a face on the head from a background in the series of images using the depth data, wherein extracting the face from the background comprises;

calculating a depth of the face in each image using the depth data; and

segmenting out the face from the background using a threshold and the depth of the face; and

tracking the face in the series of images after extracting the face from the background.

2. The method of claim 1, wherein determining the rotation matrix and the translation vector associated with the pose of the head comprises:

extracting arbitrary feature points on the head in each image in the series of images; and

using the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

3. The method of claim 2, further comprising:

generating an edge map of the head using the depth data; and

discarding the arbitrary feature points on edges of the edge map.

4. The method of claim 2, wherein determining the rotation matrix and the translation vector associated with the pose of the head comprises:

providing image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with the reference pose for the head and a current orientation and a current position; and

determining the rotation matrix and the translation vector using the Extended Kalman filter.

5. An apparatus comprising:

a Red, Green, Blue, Distance (RGBD) camera to capture images with depth data of a head of a subject; and

a processor coupled to the RGBD camera to receive a series of images with the depth data of the head of the subject, the processor being configured to obtain a reference pose for the head of the subject from one image in the series of images and use the reference pose to define a reference coordinate frame for the series of images;

determine a rotation matrix and a translation vector associated with a pose of the head in each image in the series of images relative to the reference coordinate frame using the depth data, wherein the processor is further configured to extract a face on the head from a background in the series of images using the depth data; and

track the face in the series of images after extracting the face from the background, wherein the processor is configured to extract the face from the background by being configured to calculate a depth of the face in each image using the depth data; and

segment out the face from the background with a threshold and the depth of the face.

6. The apparatus of claim 5, wherein the processor is configured to determine the rotation matrix and the translation vector associated with the pose of the head by being configured to:

extract arbitrary feature points on the head in each image in the series of images; and

use the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

7. The apparatus of claim 6, wherein the processor is further configured to:

generate an edge map of the head using the depth data; and

discard arbitrary feature points on edges of the edge map.

8. The apparatus of claim 6, wherein the processor is configured to determine the rotation matrix and the translation vector associated with the pose of the head by being configured to:

provide image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with the reference pose for the head and a current orientation and a current position; and

determine the rotation matrix and the translation vector with the Extended Kalman filter.

9. An apparatus comprising:

means for capturing a series of images with depth data of a head of a subject;

means for obtaining a reference pose for the head of the subject from one image in the series of images and using the reference pose to define a reference coordinate frame for the series of images;

means for determining a rotation matrix and a translation vector associated with a pose of the head in each image in the series of images relative to the reference coordinate frame using the depth data;

means for extracting a face on the head from a background in the series of images using the depth data, wherein the means for extracting the face from the background comprises;

means for calculating a depth of the face in each image using the depth data; and

means for segmenting out the face from the background using a threshold and the depth of the face; and

means for tracking the face in the series of images after extracting the face from the background.

10. The apparatus of claim 9, wherein the means for determining the rotation matrix and the translation vector associated with the pose of the head comprises:

means for extracting arbitrary feature points on the head in each image in the series of images; and

means for using the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

11. The apparatus of claim 10, further comprising:

means for generating an edge map of the head using the depth data; and

means for discarding arbitrary feature points on edges of the edge map.

12. The apparatus of claim 10, wherein the means for determining the rotation matrix and the translation vector associated with the pose of the head comprises:

means for providing image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with the reference pose for the head and a current orientation and a current position; and

means for determining the rotation matrix and the translation vector using the Extended Kalman filter.

13. A non-transitory computer-readable medium including program code executable by one or more processors stored thereon, comprising:

program code to receive a series of images with depth data of a head of a subject;

program code to obtain a reference pose for the head of the subject from one image in the series of images and use the reference pose to define a reference coordinate frame for the series of images;

program code to determine a rotation matrix and a translation vector associated with a pose of the head in each image in the series of images relative to the reference coordinate frame using the depth data;

program code to extract a face on the head from a background in the series of images using the depth data, wherein the program code to extract the face from the background comprises;

program code to calculate a depth of the face in each image using the depth data; and

program code to segment out the face from the background using a threshold and the depth of the face; and

program code to track the face in the series of images after extracting the face from the background.

14. The non-transitory computer-readable medium of claim 13, wherein the program code to determine the rotation matrix and the translation vector associated with the pose of the head comprises:

program code to extract arbitrary feature points on the head in each image in the series of images; and

program code to use the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

15. The non-transitory computer-readable medium of claim 14, further comprising:

program code to generate an edge map of the head using the depth data; and

program code to discard arbitrary feature points on edges of the edge map.

16. The non-transitory computer-readable medium of claim 14, wherein the program code to determine the rotation matrix and the translation vector associated with the pose of the head further comprises:

program code to provide image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with the reference pose for the head and a current orientation and a current position; and

program code to determine the rotation matrix and the translation vector using the Extended Kalman filter.