Head pose tracking using a depth camera

• US 9,529,426 B2
• Filed: 02/08/2012
• Issued: 12/27/2016
• Est. Priority Date: 02/08/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, Zicheng, Zhang, Zhengyou, Li, Zhenning
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Joisil, Berteau
• Application Number:
  US13/369,168
• Publication Number:
  US 20130201291A1
• Time in Patent Office: 1,784 Days
• Field of Search: 348/47
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Head pose tracking technique embodiments are presented that use a group of sensors configured so as to be disposed on a user'"'"'s head. This group of sensors includes a depth sensor apparatus used to identify the three dimensional locations of features within a scene, and at least one other type of sensor. Data output by each sensor in the group of sensors is periodically input, and each time the data is input it is used to compute a transformation matrix that when applied to a previously determined head pose location and orientation established when the first sensor data was input identifies a current head pose location and orientation. This transformation matrix is then applied to the previously determined head pose location and orientation to identify a current head pose location and orientation.

Claims

1. A system for head pose tracking, comprising:

a group of head pose tracking sensors configured such that all the sensors in the group are disposed on a user'"'"'s head, said group of sensors comprising a depth sensor apparatus used to identify the three dimensional locations of points within a scene sensed by the group of sensors, and a color video camera, said depth sensor apparatus and the color video camera being synchronized so as to periodically produce contemporaneous scene data in the form of a depth frame and a color image frame, and calibrated so as to map each pixel in each color image frame to a corresponding three dimensional scene location in the contemporaneously-produced depth frame;

a computing device; and

a computer program comprising program modules executed by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input each contemporaneously-produced depth frame and color image frame,

for each contemporaneously-produced depth frame and color image frame pair input after the first,

identify matching features between the last-input color image frame and a color image frame produced immediately preceding the last-input color image frame,

estimate a first transformation matrix using the identified matching features and the corresponding three dimensional locations of the matching features in both the last-input color image frame and the color image frame produced immediately preceding the last-input color image frame, said transformation matrix defining the translation and rotation of points to the last-input color image frame from the color image frame produced immediately preceding the last-input color image frame,

estimate a final transformation matrix that defines the translation and rotation of points from the first color image frame input to the last-input color image frame by accumulating a previously-computed transformation matrix, which defines the translation and rotation of points from the first color image frame to the color image frame input immediately preceding the last-input color image frame, with the first transformation matrix defining the translation and rotation of points to the last-input color image frame from the immediately preceding color image frame, and

apply the final transformation matrix to a previously determined head pose location and orientation within the scene depicted in the first color image frame input to identify the current head pose location and orientation in the scene depicted in the last-input color image frame.

2. The system of claim 1, wherein the program module for identifying matching features comprises a sub-module for identifying matching points between the last-input color image frame and a color image frame produced immediately preceding the last-input color image frame using an optical flow technique.

3. The system of claim 2, further comprising a depth correcting program module that is executed prior to executing the program module for estimating the transformation matrix, said depth correcting program module comprising sub-modules for:

defining a window around each identified matching point;

identifying the closest point to the camera within each window; and

designating the identified closest point within each window as a matching point in lieu of the previously identified matching point associated with the window.

4. The system of claim 2, further comprising a matching point outlier de-designation program module that is executed prior to executing the program module for estimating the transformation matrix, said matching point outlier de-designation program module comprising de-designating as matching points each pair of identified matching points having a matching quality level that falls below a prescribed minimum level.

5. The system of claim 2, further comprising a matching point outlier de-designation program module that is executed after executing the program module for estimating the transformation matrix and before executing the program module for estimating the final transformation matrix, said matching point outlier de-designation program module comprising sub-modules for:

(a) applying the last-estimated transformation matrix to the matching points identified in the last-input color image frame to ascertain matching points in the color image frame produced immediately preceding the last-input color image frame that correspond to matching points in the last-input color image frame;

(b) for each previously identified matching point in the color image frame produced immediately preceding the last-input color image frame, computing a difference measure between the coordinates of the previously identified matching point and the coordinates of that matching point ascertained by applying the last-estimated transformation matrix to the matching point in the last-input color image frame that corresponds to the previously identified matching point in the color image frame produced immediately preceding the last-input color image frame;

(c) computing the average of the difference measures of the remaining matching point pairs and determining whether the computed average exceeds a prescribed minimum level;

(d) whenever the computed average exceeds a prescribed minimum level,

ordering the difference measures by magnitude and identifying a prescribed percentage of the difference measures starting with the largest and going down in magnitude,

de-designating as matching points the previously identified matching point in the color image frame produced immediately preceding the last-input color image frame and its corresponding matching point in the last-input color image frame of each matching point pair associated with one of the identified difference measures,

re-estimating the transformation matrix using the remaining matching points;

(e) determining if a prescribed maximum number of transformation matrix re-estimation iterations has been reached; and

(f) whenever the prescribed maximum number of transformation matrix re-estimation iterations has not been reached, repeating the execution of sub-modules (a) through (f).

6. The system of claim 2, further comprising a keyframe program module that is executed before executing the program module for identifying matching features between the last-input color image frame and a color image frame produced immediately preceding the last-input color image frame, said keyframe program module comprising sub-modules for:

determining if a new keyframe is to be established, and

whenever it is determined a new keyframe is to be established, recording the last-input color image frame and designating the recorded frame as a keyframe.

7. The system of claim 6, wherein the sub-module for determining if a new keyframe is to be established, comprises determining that a new keyframe is to be establish whenever the last-input color image frame is the first color image frame input, and whenever the last-input color frame depicts less than a prescribed portion of the scene depicted in the previously established keyframes.

8. The system of claim 6, wherein the keyframe program module further comprises sub-modules for:

whenever it is determined that a new keyframe is not to be established, determining whether a keyframe matching is to occur;

whenever it is determined that a keyframe matching is to occur, or a new keyframe has been established,

identifying keyframe matching points between the last-input color image frame and a recorded keyframe that depicts at least a prescribed portion of the scene depicted in the last-input color image frame,

estimating a second transformation matrix using the identified keyframe matching points and the corresponding three dimensional locations of the keyframe matching features in both the last-input color image frame and the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame, said second transformation matrix defining the translation and rotation of points from the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame to the last-input color image frame,

estimating a third transformation matrix using the second transformation matrix and a previously-computed keyframe transformation matrix that defines the translation and rotation of points from the first color image frame input to the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame, said third transformation matrix defining the translation and rotation of points from the first color image frame input to the last-input color image frame,

designating the third transformation matrix as a keyframe transformation matrix that defines the translation and rotation of points from the first color image frame input to the newly established keyframe whenever the last-input color image frame is the newly established keyframe,

estimating a fourth transformation matrix that defines the translation and rotation of points from the first color image frame input to the last-input color image frame by accumulating a final transformation matrix defining the translation and rotation of points from the first-input color image frame to the color image frame input immediately preceding the last-input frame, with the first transformation matrix defining the translation and rotation of points to the last-input color image frame from the frame immediately preceding it,

fusing the third and the fourth transformation matrices to produce a single combined transformation matrix that defines the translation and rotation of points from the first color image frame input to the last-input color image frame, and

designating the combined transformation matrix to be the final transformation matrix for the last-input color image frame.

9. The system of claim 8, wherein the sub-module for identifying keyframe matching points, comprises using a Speeded Up Robust Features (SURF) matching procedure.

10. The system of claim 8, wherein the sub-module for fusing the third and the fourth transformation matrices to produce a single combined transformation matrix, comprises employing a weighted interpolation procedure.

11. The system of claim 8, wherein the sub-module for estimating the third transformation matrix using the second transformation matrix and the previously-computed keyframe transformation matrix, comprises multiplying the second transformation matrix by the keyframe transformation matrix.

12. The system of claim 8, further comprising:

a depth correcting sub-module that is executed after executing the sub-module for identifying keyframe matching points and prior to executing the sub-module for estimating the second transformation matrix, said depth correcting program sub-module comprising sub-modules for,

defining a window around each identified keyframe matching point;

identifying the closest point to the camera within each window; and

designating the identified closest point within each window as a keyframe matching point in lieu of the previously identified keyframe matching point associated with the window,

a keyframe matching point outlier de-designation sub-module that is executed after executing the sub-module for depth correcting and prior to executing the sub-module for estimating the second transformation matrix, said keyframe matching point outlier de-designation program module comprising de-designating as keyframe matching points each pair of identified keyframe matching points having a matching quality level that falls below a prescribed minimum level, and

a second keyframe matching point outlier de-designation sub-module that is executed after executing the sub-module for estimating the second transformation matrix and before executing the sub-module for estimating the third transformation matrix, said second keyframe matching point outlier de-designation sub-module comprising sub-modules for,

(a) applying the second transformation matrix to the keyframe matching points identified in the last-input color image frame to ascertain keyframe matching points in the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame that correspond to keyframe matching points in the last-input color image frame;

(b) for each previously identified keyframe matching point in the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame, computing a difference measure between the coordinates of the previously identified keyframe matching point and the coordinates of that keyframe matching point ascertained by applying the second transformation matrix to the keyframe matching point in the last-input color image frame that corresponds to the previously identified keyframe matching point in the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame,

(c) computing the average of the difference measures of the remaining keyframe matching point pairs and determining whether the computed average exceeds a prescribed minimum level;

(d) whenever the computed average exceeds a prescribed minimum level,

ordering the difference measures by magnitude and identifying a prescribed percentage of the difference measures starting with the largest and going down in magnitude,

de-designating as keyframe matching points the previously identified keyframe matching point in the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame and its corresponding keyframe matching point in the last-input color image frame of each keyframe matching point pair associated with one of the identified difference measures,

re-estimating the second transformation matrix using the remaining keyframe matching points;

(e) determining if a prescribed maximum number of transformation matrix re-estimation iterations has been reached; and

(f) whenever the prescribed maximum number of transformation matrix re-estimation iterations has not been reached, repeating the execution of sub-modules (a) through (f).

13. The system of claim 8, wherein the sub-module determining whether a keyframe matching is to occur, comprises:

determining whether a prescribed number of frames have been input since the last time keyframe matching was performed,

whenever it is determined that a prescribed number of frames have not been input since the last time keyframe matching was performed, designating that it is not time for keyframe matching to occur, and

whenever it is determined that a prescribed number of frames have been input since the last time keyframe matching was performed, designating that it is time for keyframe matching to occur.

14. The system of claim 1, wherein the final transformation matrix and current head pose location and orientation in the scene are employed in an augmented reality application to position a virtual object in the scene.

15. A system for head pose tracking, comprising:

a group of head pose tracking sensors configured such that all the sensors in the group are disposed on a user'"'"'s head, said group of sensors comprising a depth sensor apparatus used to identify the three dimensional locations of points within a scene sensed by the group of sensors, a color video camera, and a suite of inertial sensors that measures angular velocity around three axes and linear acceleration along the three axes, said depth sensor apparatus and the color video camera being synchronized so as to periodically produce contemporaneous scene data in the form of a depth frame and a color image frame and calibrated so as to map each pixel in each color image frame to a corresponding three dimensional scene location in the contemporaneously-produced depth frame, and said inertial sensors providing frames of angular velocity and linear acceleration data at a rate equaling or exceeding the rate at which the depth and color image frames are provided;

a computing device; and

a computer program comprising program modules executed by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input each inertial sensors frame produced;

for each inertial sensors frame input after the first,

estimate a current inertial sensor-based transformation matrix using the last-input inertial sensors frame, said transformation matrix defining the translation and rotation of points from the immediately preceding inertial sensors frame input to the last-input inertial sensors frame,

estimate a final inertial sensor-based transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input inertial sensors frame by accumulating a previously-computed transformation matrix, which defines the translation and rotation of points from the first inertial sensors frame to the inertial sensors frame input immediately preceding the last-input inertial sensors frame, with the current inertial sensor-based transformation matrix that defines the translation and rotation of points from the immediately preceding inertial sensors frame input to the last-input inertial sensors frame,

determine if new color video and depth frames have been produced,

whenever new color video and depth frames have not been produced, apply the final inertial sensor-based transformation matrix to a previously determined head pose location and orientation associated with the first inertial sensors frame input to identify the current head pose location and orientation,

whenever new color video and depth frames have been produced,

 

input the new color image and depth frames,

 

identify matching features between the last-input color image frame and a color image frame produced immediately preceding the last-input color image frame,

 

estimate a current image-based transformation matrix using the identified matching features and the corresponding three dimensional locations of the matching features in both the last-input color image frame and the color image frame produced immediately preceding the last-input color image frame, said image-based transformation matrix defining the translation and rotation of points to the last-input color image frame from the color image frame produced immediately preceding the last-input color image frame,

 

estimate a final image-based transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input color image frame by accumulating a previously-computed transformation matrix, which defines the translation and rotation of points from the first inertial sensors frame to the color image frame input immediately preceding the last-input color image frame, with the current image-based transformation matrix defining the translation and rotation of points to the last-input color image frame from the immediately preceding color image frame,

 

fuse the final image-based transformation matrix and the final inertial sensor-based transformation matrix to produce a single combined transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input inertial sensors frame, and

 

apply the combined transformation matrix to a previously determined head pose location and orientation associated with the first inertial sensors frame input to identify the current head pose location and orientation.

16. The system of claim 15, further comprising a keyframe program module that is executed after executing the program module for inputting the new color image and depth frames and before executing the program module for identifying matching features between the last-input color image frame and a color image frame produced immediately preceding the last-input color image frame, said keyframe program module comprising sub-modules for:

determining if a new keyframe is to be established, and

whenever it is determined a new keyframe is to be established, recording the last-input color image frame and designating the recorded frame as a keyframe.

17. The system of claim 16, wherein the sub-module for determining if a new keyframe is to be established, comprises determining that a new keyframe is to be establish whenever the last-input color image frame is the first color image frame input, and whenever the last-input color frame depicts less than a prescribed portion of the scene depicted in the previously-established keyframes.

18. The system of claim 16, wherein the keyframe program module further comprises sub-modules for:

whenever it is determined that a new keyframe is not to be established, determining whether a keyframe matching is to occur;

whenever it is determined that a keyframe matching is to occur, or a new keyframe has been established,

identifying keyframe matching points between the last-input color image frame and a recorded keyframe that depicts at least a prescribed portion of the scene depicted in the last-input color image frame,

estimating a second transformation matrix using the identified keyframe matching points and the corresponding three dimensional locations of the keyframe matching features in both the last-input color image frame and the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame, said second transformation matrix defining the translation and rotation of points from the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame to the last-input color image frame,

estimating a third transformation matrix using the second transformation matrix and a previously-computed keyframe transformation matrix that defines the translation and rotation of points from the first color image frame input to the recorded keyframe that depicts at least the prescribed portion of the scene depicted in the last-input color image frame, said third transformation matrix defining the translation and rotation of points from the first color image frame input to the last-input color image frame,

designating the third transformation matrix as a keyframe transformation matrix that defines the translation and rotation of points from the first color image frame input to the newly established keyframe whenever the last-input color image frame is the newly established keyframe,

estimating a fourth transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input color image frame by accumulating said previously-computed transformation matrix, which defines the translation and rotation of points from the first inertial sensors frame to the color image frame input immediately preceding the last-input color image frame, with the current image-based transformation matrix defining the translation and rotation of points to the last-input color image frame from the immediately preceding color image frame,

fusing the third and the fourth transformation matrices to produce a single combined image-based transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input color image frame, and

designating the combined image-based transformation matrix to be said final image-based transformation matrix.

19. The system of claim 15, wherein the inertial sensor-based transformation matrix and current head pose location and orientation in the scene are employed in an augmented reality application to position a virtual object in the scene whenever new color video and depth frames have not been produced, and the combined transformation matrix and current head pose location and orientation in the scene are employed in the augmented reality application to position a virtual object in the scene whenever new color video and depth frames have been produced.