Body feature detection and human pose estimation using inner distance shape contexts

• US 9,904,845 B2
• Filed: 02/19/2010
• Issued: 02/27/2018
• Est. Priority Date: 02/25/2009
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Dariush, Behzad, Gopalan, Raghuraman
• Primary Examiner(s): Moyer, Andrew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US12/709,221
• Publication Number:
  US 20100215271A1
• Time in Patent Office: 2,930 Days
• Field of Search: 382173
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system, method, and computer program product for estimating human body pose are described. According to one aspect, a human figure silhouette is segmented from a depth image of a human actor. Contour points are sampled along the human figure silhouette. Inner Distance Shape Context (IDSC) descriptors of the sample contour points are determined and compared to IDSC descriptors of the feature points in an IDSC gallery for similarity. For each of the feature points, the sample contour point with the IDSC descriptor that is most similar to an IDSC of the feature point is identified as that feature point in the depth image. An estimated pose of a human model is estimated based on the detected feature points and kinematic constraints of the human model.

Claims

1. A computer based method for detecting a feature point of an object in an image of the object, the method comprising:

receiving a plurality of sequential images including the image and a previous image captured earlier in time than the image;

detecting a set of feature points from within the previous image;

estimating a pose of a human actor in a human model based on enforcing joint limitations and self-penetration avoidance based on the detected set of feature points from within the previous image;

segmenting an image region of the object from an image region of background in the image based on the estimated pose;

sampling a plurality of points along a contour of the segmented image region of the object;

determining Inner Distance Shape Context (IDSC) descriptors for the sampled plurality of points;

for each of the sampled plurality of points, comparing a threshold value with a difference between the IDSC descriptor of a point and a feature point IDSC descriptor of the feature point;

responsive to the threshold value exceeding differences associated with two or more of the sampled plurality of points, selecting one of the two or more of the sampled plurality of points as the feature point of the object in the image, wherein the object comprises a human actor;

augmenting a position of a missing feature point with the detected set of feature points based on the selected feature point; and

reconstructing a pose of the human actor based at least in part on the augmented missing feature point.

2. The method of claim 1, wherein the feature point IDSC descriptor is retrieved from an IDSC gallery comprising IDSC descriptors for each feature point of the object.

3. The method of claim 1, wherein the sampled plurality of points is sampled uniformly along the contour of the segmented image region of the object.

4. The method of claim 1, wherein the image comprises a depth image.

5. The method of claim 4, wherein segmenting the image region of the object comprises:

identifying image regions in the depth image with depth values exceeding a predetermined depth working volume as background; and

identifying image regions with vertical depth image normal vectors as background.

6. The method of claim 4, wherein the feature point comprises one of:

head top, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left waist, right waist, groin, left knee, right knee, left ankle, and right ankle.

7. The method of claim 4, wherein estimating the pose of the human actor in the human model further comprises:

tracking the estimated pose of the human model with an observed pose of the human actor.

8. The method of claim 4, further comprising:

generating a predicted feature point based on the augmented feature point and the joint limitations and self-penetration avoidance of the human model.

9. The method of claim 4, further comprises:

constructing a virtual surface surrounding an actual surface of a body segment of the human model;

monitoring a distance between the body segment and an unconnected structure;

detecting that the unconnected structure penetrates the virtual surface;

determining a redirected joint motion that prevents the unconnected structure from colliding with the body segment; and

redirecting the body segment based on the redirected joint motion to avoid colliding with the unconnected structure.

10. The method of claim 4, further comprising:

performing a skeleton analysis on the image region of the human actor to generate a skeleton image of the human actor;

performing distance transformation on the skeleton image to generate a distance transformed skeleton image of the human actor; and

detecting the feature point of the human actor in the distance transformed skeleton image.

11. The method of claim 10, wherein detecting the feature point of the human actor in the distance transformed skeleton image further comprises:

determining whether self occlusion is present in the depth image based on the distance transformed skeleton image; and

responsive to self occlusion being determined present in the depth image, conducting additional analysis of the depth image to detect the feature point of the human actor.

12. The method of claim 4, wherein the depth image is taken by a single time-of-flight camera.

13. The method of claim 1, further comprising:

labeling the detected feature point in the image.

14. The method of claim 1, wherein the set of feature points from within the previous image are detected based on a closed loop inverse kinematics computation of the reconstructed pose of the object in a prior image captured earlier in time than the previous image.

15. A non-transitory computer program product for detecting a feature point of an object in an image of the object, the computer program product comprising a computer-readable storage medium containing executable computer program code for performing a method comprising:

receiving a plurality of sequential images including the image and a previous image captured earlier in time than the image;

detecting a set of feature points from within the previous image;

estimating a pose of a human actor in a human model based on enforcing joint limitations and self-penetration avoidance based on the detected set of feature points from within the previous image;

segmenting an image region of the object from an image region of background in the image based on the estimated pose;

sampling a plurality of points along a contour of the segmented image region of the object;

determining Inner Distance Shape Context (IDSC) descriptors for the sampled plurality of points;

for each of the sampled plurality of points, comparing a threshold value with a difference between the IDSC descriptor of a point and a feature point IDSC descriptor of the feature point;

responsive to the threshold value exceeding differences associated with two or more of the sampled plurality of points, selecting one of the two or more of the sampled plurality of points as the feature point of the object in the image, wherein the object comprises a human actor;

augmenting a position of a missing feature point with the detected set of feature points based on the selected feature point; and

reconstructing a pose of the human actor based at least in part on the augmented missing feature point.

16. A system for detecting a feature point of an object in an image of the object, the system comprising:

a computer processor for executing executable computer program code;

a computer-readable storage medium containing the executable computer program code for performing a method comprising;

receiving a plurality of sequential images including the image and a previous image captured earlier in time than the image;

detecting a set of feature points from within the previous image;

estimating a pose of a human actor in a human model based on enforcing joint limitations and self-penetration avoidance based on the detected set of feature points from within the previous image;

segmenting an image region of the object from an image region of background in the image based on the estimated pose;

sampling a plurality of points along a contour of the segmented image region of the object;

determining Inner Distance Shape Context (IDSC) descriptors for the sampled plurality of points;

for each of the sampled plurality of points, comparing a threshold value with a difference between the IDSC descriptor of a point and a feature point IDSC descriptor of the feature point;

responsive to the threshold value exceeding differences associated with two or more of the sampled plurality of points, selecting one of the two or more of the sampled plurality of points as the feature point of the object in the image, wherein the object comprises a human actor;

augmenting a position of a missing feature point with the detected set of feature points based on the selected feature point; and

reconstructing a pose of the human actor based at least in part on the augmented missing feature point.