Pose estimation based on critical point analysis

• US 20060274947A1
• Filed: 03/17/2006
• Published: 12/07/2006
• Est. Priority Date: 03/17/2005
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company), The Ohio State University Research Foundation
• Original Assignee: The Ohio State University Research Foundation
• Inventors: Fujimura, Kikuo, Zhu, Youding
• Granted Patent: US 7,317,836 B2
• Time in Patent Office: Days
• US Class Current: 382/201
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Methods and systems for estimating a pose of a subject. The subject can be a human, an animal, a robot, or the like. A camera receives depth information associated with a subject, a pose estimation module to determine a pose or action of the subject from images, and an interaction module to output a response to the perceived pose or action. The pose estimation module separates portions of the image containing the subject into classified and unclassified portions. The portions can be segmented using k-means clustering. The classified portions can be known objects, such as a head and a torso, that are tracked across the images. The unclassified portions are swept across an x and y axis to identify local minimums and local maximums. The critical points are derived from the local minimums and local maximums. Potential joint sections are identified by connecting various critical points, and the joint sections having sufficient probability of corresponding to an object on the subject are selected.

Claims

1. A method for estimating a pose of a subject in an image, comprising:

receiving the image including depth information associated with the subject;

identifying critical points from local minimums and local maximums of the subject, the local minimums and maximums relative to horizontal and vertical edges of the subject as determined from the depth information;

generating a skeletal structure for the subject from joint positions, the joint positions formed by connecting critical points within an area of the subject; and

determining a pose of the subject based on the skeletal structure.

2. The method of claim 1, wherein the step of generating the skeletal structure comprises:

determining a set of possible joint positions by identifying pairs of critical points that, when connected, remain within the subject area.

3. The method of claim 1, wherein the step of generating the skeletal structure comprises:

determining a set of possible joint positions by identifying pairs of critical points that, when connected, span closest to a center of the subject area.

4. The method of claim 1, wherein the step of generating the skeletal structure comprises:

determining a set of possible joint positions by identifying pairs of critical points that, when connected, preserve a known human configuration.

5. The method of claim 1, wherein the step of generating the skeletal structure comprises:

receiving a subsequent image; and

determining a set of possible joint positions by identifying pairs of critical points that, when connected, preserve continuity between the images.

6. The method of claim 1, further comprising:

spatially partitioning the subject area with k-means clustering,

wherein the local minimums and maximums correspond to minimums and maximums within spatial partitions.

7. The method of claim 1, further comprising:

classifying parts of the subject area as known human body parts,

wherein the skeletal structure is generated from unclassified parts of the subject area.

8. The method of claim 1, wherein the step of generating the skeletal structure comprises:

based on the skeletal structure, classifying parts of the subject area as known human body parts.

9. The method of claim 1, wherein the step of generating the skeletal structure comprises:

receiving a subsequent image;

identifying subsequent critical points; and

generating a subsequent skeletal image by comparing the critical points against the subsequent critical points.

10. A computer-readable medium storing a computer program product configured to perform a method for estimating a pose of a subject in an image, the method comprising:

receiving the image including depth information associated with the subject;

identifying critical points from local minimums and local maximums of the subject, the local minimums and maximums relative to horizontal and vertical edges of the subject as determined from the depth information;

generating a skeletal structure for the subject from joint positions, the joint positions formed by connecting critical points within an area of the subject; and

determining a pose of the subject based on the skeletal structure.

11. The computer program product of claim 10, wherein the step of generating the skeletal structure comprises:

determining a set of possible joint positions by identifying pairs of critical points that, when connected, remain within the subject area.

12. The computer program product of claim 10, wherein the step of generating the skeletal structure comprises:

determining a set of possible joint positions by identifying pairs of critical points that, when connected, span closest to a center of the subject area.

13. The computer program product of claim 10, wherein the step of generating the skeletal structure comprises:

determining a set of possible joint positions by identifying pairs of critical points that, when connected, preserve a known human configuration.

14. The computer program product of claim 10, wherein the step of generating the skeletal structure comprises:

receiving a subsequent image; and

determining a set of possible joint positions by identifying pairs of critical points that, when connected, preserve continuity between the images.

15. The computer program product of claim 10, further comprising:

spatially partitioning the subject area with k-means clustering,

wherein the local minimums and maximums correspond to minimums and maximums within spatial partitions.

16. The computer program product of claim 10, further comprising:

classifying parts of the subject area as known human body parts,

wherein the skeletal structure is generated from unclassified parts of the subject area.

17. The computer program product of claim 10, wherein the step of generating the skeletal structure comprises:

based on the skeletal structure, classifying parts of the subject area as known human body parts.

18. The computer program product of claim 10, wherein the step of generating the skeletal structure comprises:

receiving a subsequent image;

identifying subsequent critical points; and

generating a subsequent skeletal image by comparing the critical points against the subsequent critical points.

19. A system for estimating a pose of a subject in an image, comprising:

an input to receive the image including depth information associated with the subject;

a critical points module, coupled in communication with the input, the critical points module configured to identify critical points from local minimums and local maximums of the subject, the local minimums and maximums relative to horizontal and vertical edges of the subject as determined from the depth information;

a skeletal generation module, coupled in communication with the critical points module, the skeletal generation module configured to form a skeletal structure for the subject from joint positions, the joint positions formed by connecting critical points within an area of the subject; and

an estimation module, coupled in communication with the skeletal generation module, the estimation module configured to determine a pose of the subject based on the skeletal structure.

20. The system of claim 19, wherein the skeletal generation module determines a set of possible joint positions by identifying pairs of critical points that, when connected, remain within the subject area.

21. The system of claim 19, wherein the skeletal generation module determines a set of possible joint positions by identifying pairs of critical points that, when connected, span closest to a center of the subject area.

22. The system of claim 19, wherein the skeletal generation module determines a set of possible joint positions by identifying pairs of critical points that, when connected, preserve a known human configuration.

23. The system of claim 19, wherein the input receives a subsequent image, and the skeletal generation module determines a set of possible joint positions by identifying pairs of critical points that, when connected, preserve continuity between the images.

24. The system of claim 19, wherein, wherein the critical points module spatially partitions the subject area with k-means clustering, wherein the local minimums and maximums correspond to minimums and maximums within spatial partitions.

25. The system of claim 19, wherein the critical points module classifies parts of the subject area as known human body parts, wherein the skeletal structure is generated from unclassified parts of the subject area.

26. The system of claim 19, wherein the skeletal generation module based on the skeletal structure, classifies parts of the subject area as known human body parts.

27. The system of claim 19, wherein the input receives a subsequent image, the critical points module identifies subsequent critical points, and the skeletal generation module generates a subsequent skeletal image by comparing the critical points against the subsequent critical points.