Intelligent Part Identification for Use with Scene Characterization or Motion Capture

• US 20110206273A1
• Filed: 02/25/2010
• Published: 08/25/2011
• Est. Priority Date: 02/25/2010
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Plagemann, Christian, Ganapathi, Hariraam Varun, Thrun, Sebastian
• Granted Patent: US 8,611,670 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A variety of methods, systems, devices and arrangements are implemented for use with motion capture. One such method is implemented for identifying salient points from three-dimensional image data. The method involves the execution of instructions on a computer system to generate a three-dimensional surface mesh from the three-dimensional image data. Lengths of possible paths from a plurality of points on the three-dimensional surface mesh to a common reference point are categorized. The categorized lengths of possible paths are used to identify a subset of the plurality of points as salient points.

Claims

1. A method for identifying salient points from three-dimensional image data, the method comprising:

executing instructions on a computer system to

generate a three-dimensional surface mesh from the three-dimensional image data;

categorize lengths of paths on the three-dimensional surface mesh, the paths connected to one or more points of a plurality of points on the three-dimensional surface mesh; and

identify a subset of the plurality of points as salient points based upon the categorized lengths of paths.

2. The method of claim 1, wherein the lengths of paths are categorized according to geodesic distances between the plurality of points on the three-dimensional surface mesh and a common reference point

3. The method of claim 1, wherein executing instructions on a computer system is further to identify the orientation of the salient points according to a direction determined from tracing the calculated longest surface paths backwards.

4. The method of claim 1, wherein executing instructions on a computer system is further to extract local descriptors at the salient points and classify the local descriptors according to body part type.

5. The method of claim 1, further including the step of using a depth camera to acquire the three-dimensional image data.

6. The method of claim 1, further including the step of using stereo reconstruction to produce the three-dimensional image data.

7. The method of claim 1 wherein the 3D data is acquired using pattern projection to produce the three-dimensional image data.

8. The method of claim 1, wherein the algorithm runs on a segmented 2D image (e.g., a color or texture segment) rather than on 3D data.

9. The method of claim 1, wherein executing instructions on a computer system is further to develop a set of part types by learning a set of local descriptors from training data.

10. The method of claim 1, wherein the step of determining a plurality of paths on the surface mesh includes determining respective path lengths between points on the surface mesh and selecting path lengths as a function of respective lengths.

11. A method for determining an orientation for a model of a structure from a sequence of depth-based image frames of the structure, the method implemented by a processing circuit and comprising:

for a given image frame of the sequence of image frames,

generating a set of surface meshes from the given image frame, each of the surface meshes including a plurality of points connected by vertices;

identifying interest points on each surface mesh by

selecting a central point on each surface mesh,

determining geodesic distances between the central point and the plurality of points as a function of the vertices, and

selecting one or more points on each surface mesh based upon the determined geodesic distances;

determining an orientation for identified interest point of a surface mesh as a function of a path between each identified interest point and corresponding vertices connecting the identified interest point to a corresponding central point; and

classifying the set of surface meshes according to a set of parts of the model of the structure by

selecting subsets of the image frame of the sequence of image frames that include a corresponding one of the identified interest points, and

assessing a correlation between candidate locations of the set of parts and the selected subsets of the image frame.

12. The method of claim 11, wherein the step of selecting one or more points on each surface mesh based upon the determined geodesic distances, includes selecting the one or more points having substantially maximized geodesic distances.

13. The method of claim 11, wherein the set of parts include one or more of a hand, a foot, a head or a shoulder of the structure.

14. The method of claim 11, wherein the structure is a human.

15. The method of claim 11, wherein the method is implemented in real-time for sets of images captured at about 20 frames per second.

16. The method of claim 11, wherein the step of selecting a central point on each surface mesh includes selection of a geodesic centroid of each surface mesh.

17. The method of claim 11, wherein the step of determining location probabilities includes a learning-based system that predicts for an entire depth image what the poses of object parts.

18. The method of claim 11, further including the step of determining, before the step of determining geodesic distances between the central point and the plurality of points, geodesic distances between points of the plurality of points and therefrom, selecting points of the plurality of points that are endpoints of the longest determined geodesic distances.

19. A system comprising:

a processing arrangement, including at least one processing circuit, configured and arranged to

generate a three-dimensional surface mesh from the three-dimensional image data;

categorize lengths of possible paths on the three-dimensional surface mesh from a plurality of points on the three-dimensional surface mesh to a common reference point; and

identify, as salient points, a subset of the plurality of points based upon the categorized lengths of possible paths

20. The system of claim 19, wherein the processing arrangement is further configured and arranged to learn a part classifier by generating modeled sensor data and learning parts from the modeled sensor data, and to determine location probabilities of a structure by identifying features of the structure using a geometry-based identification system to analyze connectivity structure of the three-dimensional image data and use the learned part classifier to identify the parts of the structure.

21. The system of claim 19, wherein the processing arrangement is further configured and arranged to use propagation of belief in a graphical model that represents an object as a kinematic chain.

22. The system of claim 19, wherein the at least one processing circuit cost includes a graphics processing unit configured and arranged to use parallel processing to generate proposed poses of a structure by initializing an algorithm according to the salient points.

23. A method for removing noise and measurement artifacts from depth images having a plurality of depth measurements the method implemented by a processing circuit and comprising:

identifying a constellation of unobserved structures from each depth measurement of the plurality of depth measurements; and

removing depth measurements that are inconsistent with the identified constellation of unobserved structures that surround the depth measurements.

24. The method of claim 23, wherein the step of removing the three-dimensional depth measurements includes removing depth measurements that exceed a threshold distance from depth measurements of identified constellation of unobserved structures that surround the depth measurements.