Target orientation estimation using depth sensing

• US 7,620,202 B2
• Filed: 06/14/2004
• Issued: 11/17/2009
• Est. Priority Date: 06/12/2003
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company), The Ohio State University Research Foundation
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company), The Ohio State University Research Foundation
• Inventors: Fujimura, Kikuo, Zhu, Youding
• Primary Examiner(s): KIM, CHONG R
• Application Number:
  US10/868,707
• Publication Number:
  US 20050058337A1
• Time in Patent Office: 1,982 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G06V 10/76 based on eigen-space representations, e.g. from pose or different illumination conditions; Shape manifolds
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/164 using holistic features
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

A system for estimating orientation of a target based on real-time video data uses depth data included in the video to determine the estimated orientation. The system includes a time-of-flight camera capable of depth sensing within a depth window. The camera outputs hybrid image data (color and depth). Segmentation is performed to determine the location of the target within the image. Tracking is used to follow the target location from frame to frame. During a training mode, a target-specific training image set is collected with a corresponding orientation associated with each frame. During an estimation mode, a classifier compares new images with the stored training set to determine an estimated orientation. A motion estimation approach uses an accumulated rotation/translation parameter calculation based on optical flow and depth constrains. The parameters are reset to a reference value each time the image corresponds to a dominant orientation.

Claims

1. A computer based method for estimating an orientation of a target using depth image data, the method comprising the steps of:

receiving a feed of depth images capturing a target, the depth images including pixel depth information;

determining a correlation between each image and a corresponding orientation measure for a set of images in the feed, the set of images representative of a set of target orientations, and the correlation providing a match between an image and a corresponding orientation measure representative of the orientation of the target captured in the image;

wherein determining the correlation comprises;

determining a primary orientation by analyzing the set of images,

assigning a reference orientation measure to the images in the set of images that capture the target in the primary orientation, and

determining an orientation measure for images in the set of images that capture the target in orientations other than the primary orientation, the orientation measure based on optical flow of feature points in the images with respect to the reference orientation measure;

storing the set of depth images of the target and the correlation; and

comparing, based in appearance, a current depth image of the target with the depth images in the set of depth images to determine, based on the correlation, a corresponding orientation measure indicative of a current orientation of the target as captured in the current depth image.

2. The method of claim 1, wherein the correlation includes a mapping between a set of known orientations and the set of images.

3. The method of claim 1, wherein the optical flow of feature points is based in part on the pixel depth information corresponding to the feature points in the image.

4. The method of claim 1, further comprising:

determining an appearance variation value of the target; and

in response to the appearance variation value exceeding a maximum variation,

determining the correlation for a second set of images of the target;

storing the correlation; and

replacing the stored set of images with the second set of images.

5. The method of claim 1, wherein the target is a human head and the orientation is a head pose.

6. A computer readable medium for estimating an orientation of a target using depth image data, comprising a computer program that when executed by a computer processor implements the steps of:

receiving a feed of depth images capturing a target, the depth images including pixel depth information;

determining a correlation between each image and a corresponding orientation measure for a set of images in the feed, the set of images representative of a set of target orientations, and the correlation providing a match between an image and a corresponding orientation measure representative of the orientation of the target captured in the image, wherein determining the correlation comprises;

determining a primary orientation by analyzing the set of images,

assigning a reference orientation measure to the images in the set of images that capture the target in the primary orientation, and

determining an orientation measure for images in the set of images that capture the target in orientations other than the primary orientation, the orientation measure based on optical flow of feature points in the images with respect to the reference orientation measure;

storing the set of depth images of the target and the correlation; and

comparing, based in appearance, a current depth image of the target with the depth images in the set of depth images to determine, based on the correlation, a corresponding orientation measure indicative of a current orientation of the target as captured in the current depth image.

7. A system for estimating an orientation of a target using depth image data, the system comprising:

means for receiving a feed of depth images capturing a target, the depth images including pixel depth information;

means for determining a correlation between each image and a corresponding orientation measure for a set of images in the feed, the set of images representative of a set of target orientations, and the correlation providing a match between an image and a corresponding orientation measure representative of the orientation of the target captured in the image, wherein the means for determining the correlation comprises;

means for determining a primary orientation by analyzing the set of images,

means for assigning a reference orientation measure to the images in the set of images that capture the target in the primary orientation, and

means for determining an orientation measure for images in the set of images that capture the target in orientations other than the primary orientation, the orientation measure based on optical flow of feature points in the images with respect to the reference orientation measure;

means for storing the set of depth images of the target and the correlation; and

means for comparing, based in appearance, a current depth image of the target with the depth images in the set of depth images to determine, based on the correlation, a corresponding orientation measure indicative of a current orientation of the target as captured in the current depth image.