HEAD POSE TRACKING USING A DEPTH CAMERA
First Claim
1. A system for head pose tracking, comprising:
- a group of sensors configured so as to be disposed on a user'"'"'s head, said group of sensors comprising a depth sensor apparatus used to identify the three dimensional locations of features within a scene sensed by the group of sensors, and at least one other type of sensor;
a computing device; and
a computer program comprising program modules executed by the computing device, wherein the computing device is directed by the program modules of the computer program to,periodically input data output by each sensor in the group of sensors,each time data is input from one or more of the sensors, use the inputted data to compute a transformation matrix which when applied to a previously determined head pose location and orientation established when the first sensor data was input identifies a current head pose location and orientation, andapply the transformation matrix to the previously determined head pose location and orientation to identify the current head pose location and orientation.
2 Assignments
0 Petitions
Accused Products
Abstract
Head pose tracking technique embodiments are presented that use a group of sensors configured so as to be disposed on a user'"'"'s head. This group of sensors includes a depth sensor apparatus used to identify the three dimensional locations of features within a scene, and at least one other type of sensor. Data output by each sensor in the group of sensors is periodically input, and each time the data is input it is used to compute a transformation matrix that when applied to a previously determined head pose location and orientation established when the first sensor data was input identifies a current head pose location and orientation. This transformation matrix is then applied to the previously determined head pose location and orientation to identify a current head pose location and orientation.
56 Citations
20 Claims
-
1. A system for head pose tracking, comprising:
-
a group of sensors configured so as to be disposed on a user'"'"'s head, said group of sensors comprising a depth sensor apparatus used to identify the three dimensional locations of features within a scene sensed by the group of sensors, and at least one other type of sensor; a computing device; and a computer program comprising program modules executed by the computing device, wherein the computing device is directed by the program modules of the computer program to, periodically input data output by each sensor in the group of sensors, each time data is input from one or more of the sensors, use the inputted data to compute a transformation matrix which when applied to a previously determined head pose location and orientation established when the first sensor data was input identifies a current head pose location and orientation, and apply the transformation matrix to the previously determined head pose location and orientation to identify the current head pose location and orientation.
-
-
2. A system for head pose tracking, comprising:
-
a group of sensors configured so as to be disposed on a user'"'"'s head, said group of sensors comprising a depth sensor apparatus used to identify the three dimensional locations of points within a scene sensed by the group of sensors, and a color video camera, said depth sensor apparatus and the color video camera being synchronized so as to periodically produce contemporaneous scene data in the form of a depth frame and a color image frame, and calibrated so as to map each pixel in each color image frame to a corresponding three dimensional scene location in the contemporaneously-produced depth frame; a computing device; and a computer program comprising program modules executed by the computing device, wherein the computing device is directed by the program modules of the computer program to, input each contemporaneously-produced depth frame and color image frame, for each contemporaneously-produced depth frame and color image frame pair input after the first, identify matching features between the last-input color image frame and a color image frame produced immediately preceding the last-input color image frame, estimate a first transformation matrix using the identified matching features and the corresponding three dimensional locations of the matching features in both the last-input color image frame and the color image frame produced immediately preceding the last-input color image frame, said transformation matrix defining the translation and rotation of points to the last-input color image frame from the color image frame produced immediately preceding the last-input color image frame, estimate a final transformation matrix that defines the translation and rotation of points from the first color image frame input to the last-input color image frame by accumulating a previously-computed transformation matrix, which defines the translation and rotation of points from the first color image frame to the color image frame input immediately preceding the last-input color image frame, with the first transformation matrix defining the translation and rotation of points to the last-input color image frame from the immediately preceding color image frame, and apply the final transformation matrix to a previously determined head pose location and orientation within the scene depicted in the first color image frame input to identify the current head pose location and orientation in the scene depicted in the last-input color image frame. - View Dependent Claims (3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
-
-
16. A system for head pose tracking, comprising:
-
a group of sensors configured so as to be disposed on a user'"'"'s head, said group of sensors comprising a depth sensor apparatus used to identify the three dimensional locations of points within a scene sensed by the group of sensors, a color video camera, and a suite of inertial sensors that measures angular velocity around three axes and linear acceleration along the three axes, said depth sensor apparatus and the color video camera being synchronized so as to periodically produce contemporaneous scene data in the form of a depth frame and a color image frame and calibrated so as to map each pixel in each color image frame to a corresponding three dimensional scene location in the contemporaneously-produced depth frame, and said inertial sensors providing frames of angular velocity and linear acceleration data at a rate equaling or exceeding the rate at which the depth and color image frames are provided; a computing device; and a computer program comprising program modules executed by the computing device, wherein the computing device is directed by the program modules of the computer program to, input each inertial sensors frame produced; for each inertial sensors frame input after the first, estimate a current inertial sensor-based transformation matrix using the last-input inertial sensors frame, said transformation matrix defining the translation and rotation of points from the immediately preceding inertial sensors frame input to the last-input inertial sensors frame, estimate a final inertial sensor-based transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input inertial sensors frame by accumulating a previously-computed transformation matrix, which defines the translation and rotation of points from the first inertial sensors frame to the inertial sensors frame input immediately preceding the last-input inertial sensors frame, with the current inertial sensor-based transformation matrix that defines the translation and rotation of points from the immediately preceding inertial sensors frame input to the last-input inertial sensors frame, determine if new color video and depth frames have been produced, whenever new color video and depth frames have not been produced, apply the final inertial sensor-based transformation matrix to a previously determined head pose location and orientation associated with the first inertial sensors frame input to identify the current head pose location and orientation, whenever new color video and depth frames have been produced,
input the new color image and depth frames,
identify matching features between the last-input color image frame and a color image frame produced immediately preceding the last-input color image frame,
estimate a current image-based transformation matrix using the identified matching features and the corresponding three dimensional locations of the matching features in both the last-input color image frame and the color image frame produced immediately preceding the last-input color image frame, said image-based transformation matrix defining the translation and rotation of points to the last-input color image frame from the color image frame produced immediately preceding the last-input color image frame,
estimate a final image-based transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input color image frame by accumulating a previously-computed transformation matrix, which defines the translation and rotation of points from the first inertial sensors frame to the color image frame input immediately preceding the last-input color image frame, with the current image-based transformation matrix defining the translation and rotation of points to the last-input color image frame from the immediately preceding color image frame,
fuse the final image-based transformation matrix and the final inertial sensor-based transformation matrix to produce a single combined transformation matrix that defines the translation and rotation of points from the first inertial sensors frame input to the last-input inertial sensors frame, and
apply the combined transformation matrix to a previously determined head pose location and orientation associated with the first inertial sensors frame input to identify the current head pose location and orientation. - View Dependent Claims (17, 18, 19, 20)
-
Specification