Target orientation estimation using depth sensing

• US 20050058337A1
• Filed: 06/14/2004
• Published: 03/17/2005
• Est. Priority Date: 06/12/2003
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company), The Ohio State University Research Foundation
• Original Assignee: The Ohio State University Research Foundation
• Inventors: Fujimura, Kikuo, Zhu, Youding
• Granted Patent: US 7,620,202 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G06V 10/76 based on eigen-space representations, e.g. from pose or different illumination conditions; Shape manifolds
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/164 using holistic features
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

A system for estimating orientation of a target based on real-time video data uses depth data included in the video to determine the estimated orientation. The system includes a time-of-flight camera capable of depth sensing within a depth window. The camera outputs hybrid image data (color and depth). Segmentation is performed to determine the location of the target within the image. Tracking is used to follow the target location from frame to frame. During a training mode, a target-specific training image set is collected with a corresponding orientation associated with each frame. During an estimation mode, a classifier compares new images with the stored training set to determine an estimated orientation. A motion estimation approach uses an accumulated rotation/translation parameter calculation based on optical flow and depth constrains. The parameters are reset to a reference value each time the image corresponds to a dominant orientation.

Claims

1. A computer based method for estimating an orientation of a target using depth image data, the method comprising the steps of:

receiving a feed of depth images capturing a target, the depth images including pixel depth information;

determining a correlation between each image and a corresponding orientation measure for a set of images in the feed, the set of images representative of a set of target orientations, and the correlation providing a match between an image and a corresponding orientation measure representative of the orientation of the target captured in the image;

storing the set of depth images of the target and the correlation; and

comparing, based in appearance, a current depth image of the target with the depth images in the set of depth images to determine, based on the correlation, a corresponding orientation measure indicative of a current orientation of the target as captured in the current depth image.

2. The method of claim 1, wherein the correlation includes a mapping between a set of known orientations and the set of images.

3. The method of claim 1, wherein determining the correlation further includes:

determining a primary orientation by analyzing the set of images;

assigning a reference orientation measure to the images in the set of images that capture the target in the primary orientation; and

determining an orientation measure for images in the set of images that capture the target in orientations other than the primary orientation, the orientation measure based on optical flow of feature points in the images with respect to the reference orientation measure.

4. The method of claim 3, wherein the optical flow of feature points is based in part on the pixel depth information corresponding to the feature points in the image.

5. The method of claim 1, further comprising:

determining an appearance variation value of the target; and

in response to the appearance variation value exceeding a maximum variation, determining the correlation for a second set of images of the target;

storing the correlation; and

replacing the stored set of images with the second set of images.

6. The method of 1, wherein the target is a human head and the orientation is a head pose.

7. A computer readable medium for estimating an orientation of a target using depth image data, comprising a computer program that when executed by a computer processor implements the steps of:

receiving a feed of depth images capturing a target, the depth images including pixel depth information;

determining a correlation between each image and a corresponding orientation measure for a set of images in the feed, the set of images representative of a set of target orientations, and the correlation providing a match between an image and a corresponding orientation measure representative of the orientation of the target captured in the image;

storing the set of depth images of the target and the correlation; and

comparing, based in appearance, a current depth image of the target with the depth images in the set of depth images to determine, based on the correlation, a corresponding orientation measure indicative of a current orientation of the target as captured in the current depth image.

8. A system for estimating an orientation of a target using depth image data, the system comprising:

means for receiving a feed of depth images capturing a target, the depth images including pixel depth information;

means for determining a correlation between each image and a corresponding orientation measure for a set of images in the feed, the set of images representative of a set of target orientations, and the correlation providing a match between an image and a corresponding orientation measure representative of the orientation of the target captured in the image;

means for storing the set of depth images of the target and the correlation; and

means for comparing, based in appearance, a current depth image of the target with the depth images in the set of depth images to determine, based on the correlation, a corresponding orientation measure indicative of a current orientation of the target as captured in the current depth image.

9. A computer based method for estimating a real-time orientation measure for a target using depth video image data, the method comprising the steps of:

receiving as input a feed of depth video frames, the depth video frames capturing the target in real-time and including depth pixel data;

storing a target-specific training set of depth video frames during an initial training period;

identifying frames of the target-specific training set of depth video frames capturing the target in a dominant orientation by analyzing the target-specific training set of depth video frames;

comparing, based in appearance, a current depth video frame with the target-specific training set of depth video frames to determine whether the current depth video frame includes the target in the dominant orientation;

determining a current orientation measure based in part on depth data corresponding to the current depth video frame, the current orientation measure corresponding to a current orientation of the target captured in the current depth video frame; and

setting the current orientation measure to a reference orientation measure in response to determining that the current depth video frame includes the target in the dominant orientation.

10. The method of claim 9, wherein analyzing the target-specific training set of depth video frames includes performing a Parzen-window based PDF and determining the mode of the PDF.

11. The method of claim 9, wherein analyzing the target-specific training set of depth video frames includes segmenting each depth video frame to determine a segment of the frame that includes image data corresponding to the target.

12. The method of claim 11, wherein analyzing the target-specific training set of depth video frames further includes tracking the segment position from frame to frame based on one of an elliptic fitting method or a Mean shift algorithm.

13. The method of claim 9, wherein comparing, based in appearance, further comprises:

determining a projection matrix and a pose curve using PCA analysis of the target-specific training set of depth video frames;

projecting the current depth video frame onto a set of multidimensional eigenspaces based on the projection matrix; and

estimating an orientation based on the projected current depth video frame.

14. The method of claim 13, wherein estimating the orientation includes finding a nearest neighbor based on the projection of the current depth video frame and points of the pose curve.

15. The method of claim 13, wherein estimating the orientation includes performing a linear interpolation between points in the pose curve corresponding to the projected current depth video frame.

16. The method of claim 9, wherein determining the current orientation measure further comprises:

estimating an optical flow between feature points in a previous depth video frame and the current depth video frame to determine two-dimensional motion fields;

recovering three-dimensional rotation and translation parameters between the previous depth video frame and the current depth video frame using the two-dimensional motion fields and depth constrains based on the depth pixel data corresponding to the feature points; and

setting the current orientation measure to an accumulated orientation value based on an orientation measure for the previous depth video frame and the three-dimensional rotation and translation parameters.

17. The method of claim 9, wherein the target is a driver'"'"'s head and the orientation is a head pose.

18. A computer readable medium for estimating a real-time orientation measure for a target using depth video image data, comprising a computer program that when executed by a computer processor implements the steps of:

receiving as input a feed of depth video frames, the depth video frames capturing the target in real-time and including depth pixel data;

storing a target-specific training set of depth video frames during an initial training period;

identifying frames of the target-specific training set of depth video frames capturing the target in a dominant orientation by analyzing the target-specific training set of depth video frames;

comparing, based in appearance, a current depth video frame with the target-specific training set of depth video frames to determine whether the current depth video frame includes the target in the dominant orientation;

determining a current orientation measure based in part on depth data corresponding to the current depth video frame, the current orientation measure corresponding to a current orientation of the target captured in the current depth video frame; and

setting the current orientation measure to a reference orientation measure in response to determining that the current depth video frame includes the target in the dominant orientation.

19. A system for estimating a real-time orientation measure for a target using depth video image data, the system comprising:

means for receiving as input a feed of depth video frames, the depth video frames capturing the target in real-time and including depth pixel data;

means for storing a target-specific training set of depth video frames during an initial training period;

means for identifying frames of the target-specific training set of depth video frames capturing the target in a dominant orientation by analyzing the target-specific training set of depth video frames;

means for comparing, based in appearance, a current depth video frame with the target-specific training set of depth video frames to determine whether the current depth video frame includes the target in the dominant orientation;

means for determining a current orientation measure based in part on depth data corresponding to the current depth video frame, the current orientation measure corresponding to a current orientation of the target captured in the current depth video frame; and

means for setting the current orientation measure to a reference orientation measure in response to determining that the current depth video frame includes the target in the dominant orientation.

20. A system comprising:

a time-of-flight camera, the time-of-flight camera configured to capture a real-time video of a target and to operate within a depth window, the real-time video including a stream of frames, each frame including hybrid image data that includes color and depth information for each pixel; and

a computer processing system coupled to the time-of-flight camera for receiving the real-time video and configured to output an estimated orientation of the target, the estimated orientation of the target based on an appearance based classification using a target-specific training image set and based on motion estimation calculations using position changes of feature points of the target between frames of the video stream and constrains given by the corresponding depth information.

21. The system of claim 20, wherein the computer processing system comprises:

an input/output module for receiving the real-time video stream from the time-of-flight camera and for providing the estimated orientation of the target;

a segmentation and tracking module coupled to the input/output module for receiving hybrid image data for each frame of the real-time video stream and is configured to provide a segment of the image data consisting of a set of pixels determined to correspond to the target, the segment of the image data determined based on depth information included in the hybrid image data;

an adaptive learning module coupled to the segmentation and tracking module for receiving the hybrid image data and the corresponding segment, the adaptive learning module configured to compose the target-specific training image set from an initial set of video frames of the target captured by the time-of-flight camera during a training period, the target-specific training image set including at least one frame with a corresponding orientation measure; and

an appearance classifier module coupled to the segmentation and tracking module for receiving at least the segment of the image data of a current video frame comprising the target and coupled to the input/output module for providing the estimated orientation of the target in the current video frame, the appearance classifier module configured to compare the current video frame with the target-specific training image set and to estimate the current orientation based on the orientation corresponding to the frame most similar to the current frame.

22. The system of claim 21, wherein the computer processing system further comprises:

a motion estimation module coupled to the segmentation and tracking module for receiving at least the segment of the image data of the current video frame comprising the target and coupled to the input/output module for providing the estimated orientation of the target in the current video frame, and further coupled to the appearance classifier module for receiving an indication upon determining that the current video frame corresponds to a dominant orientation of the target, the motion estimation module configured to calculate a current orientation measure corresponding to the orientation of the target in the current frame based on the position changes of feature points of the target between frames of the video stream and the constrains given by the corresponding depth information, and further configured to reset the current orientation measure to a reference value upon receiving the indication that the current video frame corresponds to the dominant orientation.

23. The system of claim 21, wherein the computer processing system further comprises:

a memory module coupled to the adaptive learning module and the appearance classifier, the memory module configured to store the target-specific training image set.

24. The system of claim 20, wherein the time-of-flight camera is mounted in a vehicle cabin for capturing the real-time video of a target driver, and further wherein the computer processing system is part of a vehicle safety system that includes safety features configured to operate based on the estimated orientation of the driver.