HEAD POSE ESTIMATION USING RGBD CAMERA

• US 20120293635A1
• Filed: 04/25/2012
• Published: 11/22/2012
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Sharma, Piyush, Swaminathan, Ashwin, Rezaiifar, Ramin, Xue, Qi
• Granted Patent: US 9,582,707 B2
• Time in Patent Office: Days
• US Class Current: 348/50
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

A three-dimensional pose of the head of a subject is determined based on depth data captured in multiple images. The multiple images of the head are captured, e.g., by an RGBD camera. A rotation matrix and translation vector of the pose of the head relative to a reference pose is determined using the depth data. For example, arbitrary feature points on the head may be extracted in each of the multiple images and provided along with corresponding depth data to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with the reference pose for the head and a current orientation and a current position. The three-dimensional pose of the head with respect to the reference pose is then determined based on the rotation matrix and the translation vector.

Claims

1. A method comprising:

capturing multiple images with depth data of a head of a subject; and

determining a rotation matrix and a translation vector associated with a pose of the head in the multiple images relative to a reference coordinate frame using the depth data.

2. The method of claim 1, wherein determining the rotation matrix and the translation vector associated with the pose of the head comprises:

extracting arbitrary feature points on the head in each of the multiple images; and

using the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

3. The method of claim 2, further comprising:

generating an edge map of the head using the depth data; and

discarding the arbitrary feature points on edges of the edge map.

4. The method of claim 2, wherein determining the rotation matrix and the translation vector associated with the pose of the head comprises:

providing image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with a reference pose for the head and a current orientation and a current position; and

determining the rotation matrix and the translation vector using the Extended Kalman filter.

5. The method of claim 1, further comprising:

extracting a face on the head from a background in the multiple images using the depth data; and

tracking the face in the multiple images after extracting the face from the background.

6. The method of claim 5, wherein extracting the face from the background comprises:

calculating a depth of the face in each image using the depth data; and

segmenting out the face from the background using a threshold and the depth of the face.

7. An apparatus comprising:

a Red, Green, Blue, Distance (RGBD) camera to capture images with depth data of a head of a subject; and

a processor coupled to the RGBD camera to receive multiple images with the depth data of the head of the subject, the processor being configured to determine a rotation matrix and a translation vector associated with a pose of the head in the multiple images relative to a reference coordinate frame using the depth data.

8. The apparatus of claim 7, wherein the processor is configured to determine the rotation matrix and the translation vector associated with the pose of the head by being configured to:

extract arbitrary feature points on the head in each of the multiple images; and

use the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

9. The apparatus of claim 8, wherein the processor is further configured to:

generate an edge map of the head using the depth data; and

discard arbitrary feature points on edges of the edge map.

10. The apparatus of claim 8, wherein the processor is configured to determine the rotation matrix and the translation vector associated with the pose of the head by being configured to:

provide image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with a reference pose for the head and a current orientation and a current position; and

determine the rotation matrix and the translation vector with the Extended Kalman filter.

11. The apparatus of claim 7, wherein the processor is further configured to:

extract a face on the head from a background in the multiple images using the depth data; and

track the face in the multiple images after extracting the face from the background.

12. The apparatus of claim 11, wherein the processor is configured to extract the face from the background by being configured to:

calculate a depth of the face in each image using the depth data; and

segment out the face from the background with a threshold and the depth of the face.

13. An apparatus comprising:

means for capturing multiple images with depth data of a head of a subject; and

means for determining a rotation matrix and a translation vector associated with a pose of the head in the multiple images relative to a reference coordinate frame using the depth data.

14. The apparatus of claim 13, wherein the means for determining the rotation matrix and the translation vector associated with the pose of the head comprises:

means for extracting arbitrary feature points on the head in each of the multiple images; and

means for using the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

15. The apparatus of claim 14, further comprising:

means for generating an edge map of the head using the depth data; and

means for discarding arbitrary feature points on edges of the edge map.

16. The apparatus of claim 14, wherein determining the rotation matrix and the translation vector associated with the pose of the head comprises:

means for providing image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with a reference pose for the head and a current orientation and a current position; and

means for determining the rotation matrix and the translation vector using the Extended Kalman filter.

17. The apparatus of claim 13, further comprising:

means for extracting a face on the head from a background in the multiple images using the depth data; and

means for tracking the face in the multiple images after extracting the face from the background.

18. The apparatus of claim 17, wherein the means for extracting the face from the background comprises:

means for calculating a depth of the face in each image using the depth data; and

means for segmenting out the face from the background using a threshold and the depth of the face.

19. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to receive multiple images with the depth data of a head of a subject;

program code to determine a rotation matrix and a translation vector associated with a pose of the head in the multiple images relative to a reference coordinate frame using the depth data.

20. The non-transitory computer-readable medium of claim 19, wherein the program code to determine the rotation matrix and the translation vector associated with the pose of the head comprises:

program code to extract arbitrary feature points on the head in each of the multiple images; and

program code to use the depth data associated with the arbitrary feature points to determine the rotation matrix and the translation vector of the pose of the head.

21. The non-transitory computer-readable medium of claim 20, further comprising:

program code to generate an edge map of the head using the depth data; and

program code to discard arbitrary feature points on edges of the edge map.

22. The non-transitory computer-readable medium of claim 20, wherein the program code to determine the rotation matrix and the translation vector associated with the pose of the head further comprises:

program code to provide image coordinates of the arbitrary feature points and the depth data for corresponding arbitrary feature points to an Extended Kalman filter with states including a rotation matrix and a translation vector associated with a reference pose for the head and a current orientation and a current position; and

program code to determine the rotation matrix and the translation vector using the Extended Kalman filter.

23. The non-transitory computer-readable medium of claim 19, further comprising:

program code to extract a face on the head from a background in the multiple images using the depth data; and

program code to track the face in the multiple images after extracting the face from the background.

24. The non-transitory computer-readable medium of claim 23, wherein the program code to extract the face from the background comprises:

program code to calculate a depth of the face in each image using the depth data; and

program code to segment out the face from the background using a threshold and the depth of the face.