Real time head pose estimation

• US 8,687,880 B2
• Filed: 03/20/2012
• Issued: 04/01/2014
• Est. Priority Date: 03/20/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wei, Yichen, Wen, Fang, Sun, Jian, Leyvand, Tommer, Li, Jinyu, Meekhof, Casey, Keosababian, Tim
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): LE, TOTAM HA
• Application Number:
  US13/425,188
• Publication Number:
  US 20130251244A1
• Time in Patent Office: 742 Days
• Field of Search: 382/159
• US Class Current: 382/159
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Methods are provided for generating a low dimension pose space and using the pose space to estimate one or more head rotation angles of a user head. In one example, training image frames including a test subject head are captured under a plurality of conditions. For each frame an actual head rotation angle about a rotation axis is recorded. In each frame a face image is detected and converted to an LBP feature vector. Using principal component analysis a PCA feature vector is generated. Pose classes related to rotation angles about a rotation axis are defined. The PCA feature vectors are grouped into a pose class that corresponds to the actual rotation angle associated with the PCA feature vector. Linear discriminant analysis is applied to the pose classes to generate the low dimension pose space.

Claims

1. A method for generating a low dimension pose space for estimating one or more head rotation angles, comprising:

capturing a plurality of training image frames that each includes a test subject head under a plurality of conditions;

for each of the training image frames, recording an actual rotation angle of the test subject head about a rotation axis;

sampling a portion of the plurality of training image frames to create a training model subset of the training image frames;

for each of the training image frames in the training model subset;

detecting a test subject face image in the training image frame;

converting the test subject face image to a test subject LBP (Local Binary Patterns) feature vector based on local binary patterns; and

applying principal component analysis to the test subject LBP feature vector to generate a test subject PCA (Principal Component Analysis) feature vector;

defining a plurality of pose classes related to rotation angles about the rotation axis;

for each of the test subject PCA feature vectors, grouping the test subject PCA feature vector into one of the plurality of pose classes that corresponds to the actual rotation angle associated with the test subject PCA feature vector; and

applying linear discriminant analysis to each of the plurality of pose classes to generate the low dimension pose space.

2. The method of claim 1, further comprising projecting the test subject PCA feature vectors into the low dimension pose space.

3. The method of claim 1, wherein the plurality of training image frames includes one or more of color image data, monochrome image data, and depth image data.

4. The method of claim 1, wherein capturing a plurality of training image frames of a test subject head under a plurality of conditions further comprises capturing a plurality of training image frames of test subject heads of at least approximately 200 test subjects under a plurality of conditions.

5. The method of claim 1, wherein the plurality of conditions includes one or more of a lighting condition, a distance from a camera, and a test subject expression.

6. The method of claim 5, wherein the lighting condition includes at least a bright lighting condition and a dark lighting condition that is darker than the bright lighting condition, the distance from the camera includes at least a near distance from the camera and a far distance from the camera that is larger than the near distance, and the test subject expression includes at least a neutral mouth expression, a mouth open expression, and a smiling expression.

7. The method of claim 1, wherein recording an actual rotation angle of the test subject head about a rotation axis further comprises recording an actual rotation angle of the test subject head about each of a yaw, pitch and roll axis.

8. The method of claim 1, wherein each pose class of the plurality of pose classes represents a range of head rotation angles.

9. The method of claim 1, wherein recording an actual rotation angle of the test subject head about a rotation axis further comprises receiving data from a motion sensor associated with the test subject head.

10. The method of claim 1, further comprising, for each pose class of the plurality of pose classes:

determining a pose class center point; and

discarding all points from the pose class except for the pose class center point.

11. A method for estimating a head rotation angle of a user head, comprising:

capturing a user image frame that includes the user head;

detecting a user face image in the user image frame;

converting the user face image to a user LBP (Local Binary Patterns) feature vector based on local binary patterns;

projecting the user LBP feature vector into a low dimension pose space using a training model to obtain a low dimension pose space user vector, wherein the training model comprises applying principal component analysis and linear discriminant analysis to generate the low dimension pose space user vector, and wherein the low dimension pose space includes a plurality of pose classes related to rotation angles about a rotation axis, the pose classes obtained using the training model; and

comparing the low dimension pose space user vector to other vectors in the low dimension pose space to estimate the head rotation angle of the user head about the rotation axis.

12. The method of claim 11, wherein comparing the low dimension pose space user vector to other vectors in the low dimension pose space further comprises using a k-nearest-neighbor regression.

13. The method of claim 12, further comprising finding k nearest neighbors to the low dimension pose space user vector in the low dimension pose space using distances from pre-determined center points of k nearest pose classes, wherein k is a positive integer.

14. The method of claim 13, wherein using a k-nearest-neighbor regression further comprises computing a weighted average of rotation angles related to the k nearest pose classes.

15. The method of claim 11, wherein the low dimension pose space is a first low dimension pose space, the rotation axis is a first rotation axis, the training model is a first training model, the low dimension pose space user vector is a first low dimension pose space user vector, and the pose classes are first pose classes and further comprising:

projecting the user LBP feature vector into a second low dimension pose space that corresponds to a second rotation axis using a second training model to obtain a second low dimension pose space user vector, wherein the second training model comprises applying principal component analysis and linear discriminant analysis to generate the second low dimension pose space user vector, and wherein the second low dimension pose space includes a plurality of second pose classes related to rotation angles about the second rotation axis, the second pose classes obtained using the second training model; and

comparing the second low dimension pose space user vector to other vectors in the second low dimension pose space to estimate the head rotation angle of the user head about the second rotation axis.

16. The method of claim 15, further comprising:

projecting the user LBP feature vector into a third low dimension pose space that corresponds to a third rotation axis using a third training model to obtain a third low dimension pose space user vector, wherein the third training model comprises applying principal component analysis and linear discriminant analysis to generate the third low dimension pose space user vector, and wherein the third low dimension pose space includes a plurality of third pose classes related to rotation angles about the third rotation axis, the third pose classes obtained using the third training model; and

comparing the third low dimension pose space user vector to other vectors in the third low dimension pose space to estimate the head rotation angle about the third rotation axis of the user head.

17. The method of claim 11, wherein the rotation axis is a pitch rotation axis, further comprising:

capturing a plurality of user image frames that each includes the user head;

for each of the plurality of user image frames, estimating the head rotation angle of the user head about the pitch rotation axis and about a yaw rotation axis; and

analyzing the head rotation angles of the plurality of user image frames to detect either a yes response from the user head or a no response from the user head.

18. The method of claim 11, wherein the user image frame is a first user image frame, the user face image is a first user face image, and the head rotation angle is a first head rotation angle, further comprising:

storing the first user face image and the first head rotation angle of the user head;

capturing a second user image frame;

detecting a second user face image in the second user image frame;

estimating a second head rotation angle based on the second user face image; and

if the second head rotation angle corresponds to the first head rotation angle, then authenticating an identity of the second user face image by comparing the second user face image to the first user face image.

19. A method for generating a low dimension pose space and estimating a head rotation angle of a user head, comprising:

in a training phase at a training computing device;

capturing a plurality of training image frames that each includes a test subject head under a plurality of conditions;

for each of the training image frames, recording an actual rotation angle of the test subject head about a rotation axis;

sampling a portion of the plurality of training image frames to create a training model subset of the training image frames;

for each of the training image frames in the training model subset;

detecting a test subject face image in the training image frame;

converting the test subject face image to a test subject LBP (Local Binary Patterns feature vector based on local binary patterns; and

applying principal component analysis to the test subject LBP feature vector to generate a test subject PCA (Principal Component Analysis) feature vector;

defining a plurality of pose classes related to rotation angles about the rotation axis;

for each of the test subject PCA feature vectors, grouping the test subject PCA feature vector into one of the plurality of pose classes that corresponds to the actual rotation angle associated with the test subject PCA feature vector; and

applying linear discriminant analysis to each of the plurality of pose classes to generate the low dimension pose space;

providing a program to be executed in a run-time phase at a run-time computing device for estimating the head rotation angle of the user head, the program causing the run-time computing device to perform the steps of;

capturing a user image frame that includes the user head;

detecting a user face image in the user image frame;

converting the user face image to a user LBP feature vector based on local binary patterns;

projecting the user LBP feature vector into the low dimension pose space to obtain a low dimension pose space user vector; and

comparing the low dimension pose space user vector to other vectors in the low dimension pose space to estimate the head rotation angle of the user head about the rotation axis.

20. The method of claim 19, wherein the low dimension pose space is a first low dimension pose space, the rotation axis is a first rotation axis, and the low dimension pose space user vector is a first low dimension pose space user vector, and wherein the program causes the run-time computing device to perform the additional steps of:

projecting the user LBP feature vector into a second low dimension pose space that corresponds to a second rotation axis to obtain a second low dimension pose space user vector; and

comparing the second low dimension pose space user vector to other vectors in the second low dimension pose space to estimate the head rotation angle of the user head about the second rotation axis.