3D body modeling, from a single or multiple 3D cameras, in the presence of motion

• US 9,235,928 B2
• Filed: 01/24/2013
• Issued: 01/12/2016
• Est. Priority Date: 01/24/2012
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Medioni, Gerard Guy, Choi, Jongmoo
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Chin, Michelle
• Application Number:
  US13/749,318
• Publication Number:
  US 20130187919A1
• Time in Patent Office: 1,083 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

The present disclosure describes systems and techniques relating to generating three dimensional (3D) models from range sensor data. According to an aspect, 3D point clouds are captured using a 3D camera, where each of the 3D point clouds corresponds to a different relative position of the 3D camera with respect to a body. One of the 3D point clouds can be set as a reference point cloud, and transforms can be determined for coordinates of the other captured 3D point clouds to transform these to coordinates of the reference point cloud. The body represented in the reference point cloud can be segmented into body parts corresponding to elements of a 3D part-based volumetric model including cylindrical representations, and a segmented representation of the physical object of interest can be generated in accordance with the 3D part-based volumetric model, while taking localized articulated motion into account.

Claims

1. A method performed by a computer system comprising processor electronics and at least one memory device, the method comprising:

capturing three dimensional (3D) point clouds using a 3D camera, each of the 3D point clouds corresponding to a different relative position of the 3D camera with respect to a body;

setting one of the 3D point clouds as a reference point cloud;

determining transforms for coordinates of the captured 3D point clouds, other than the reference point cloud, to transform to coordinates of the reference point cloud;

segmenting the body represented in the reference point cloud into body parts corresponding to elements of a 3D part-based volumetric model comprising cylindrical representations; and

generating a segmented representation of the physical object of interest in accordance with the 3D part-based volumetric model, wherein the generating comprises, for each of the captured 3D point clouds other than the reference point cloud,

transforming the captured 3D point cloud using its transform,

segmenting the body represented in the transformed 3D point cloud using the body parts corresponding to the elements of the 3D part-based volumetric model, and

determining local motion, for each of the body parts corresponding to the elements of the 3D part-based volumetric model, between the transformed 3D point cloud and the reference point cloud;

wherein a junction between at least two of the cylindrical representations, which are processed as unwrapped cylindrical maps, is handled by setting a first of the cylindrical maps as a reference map, transforming points of a second of the cylindrical maps to the reference map, and blending the transformed points of the second of the cylindrical maps with points of the first of the cylindrical maps to smooth the junction.

2. The method of claim 1, wherein the 3D point clouds are captured using a single 3D camera.

3. The method of claim 1, wherein the segmented representation comprises a segmented mesh representing the physical object.

4. The method of claim 1, wherein each of the cylindrical representations is handled using an image R(x,θ

), where R represents a radius value of the cylindrical representation at position x along the axis, and θ

the angle;

wherein an axis of at least one of the cylindrical representations is curved; and

the method comprises smoothing the captured clouds of 3D points using image-based operators.

5. The method of claim 4, comprising performing interpolation, in an image domain, on the unwrapped cylindrical maps to fill holes.

6. A non-transitory computer-readable medium encoding a program that causes data processing apparatus to perform operations to perform local alignment of limbs identified in three dimensional (3D) point clouds captured using a 3D camera, each of the 3D point clouds corresponding to a different relative position of the 3D camera with respect to a body having the limbs, to model the body in the data processing apparatus;

wherein the operations comprise;

capturing the 3D point clouds using the 3D camera;

setting one of the 3D point clouds as a reference point cloud;

determining transforms for coordinates of the captured 3D point clouds, other than the reference point cloud, to transform to coordinates of the reference point cloud;

segmenting the body represented in the reference point cloud into body parts, including the limbs, corresponding to elements of a 3D part-based volumetric model comprising cylindrical representations; and

generating a segmented representation of the physical object of interest in accordance with the 3D part-based volumetric model, wherein the generating comprises, for each of the captured 3D point clouds other than the reference point cloud,

transforming the captured 3D point cloud using its transform,

segmenting the body represented in the transformed 3D point cloud using the body parts corresponding to the elements of the 3D part-based volumetric model, and

determining local motion, for each of the body parts corresponding to the elements of the 3D part-based volumetric model, between the transformed 3D point cloud and the reference point cloud; and

wherein a junction between at least two of the cylindrical representations, which are processed as unwrapped cylindrical maps, is handled by;

setting a first of the cylindrical maps as a reference map,

transforming points of a second of the cylindrical maps to the reference map, and

blending the transformed points of the second of the cylindrical maps with points of the first of the cylindrical maps to smooth the junction.

7. The non-transitory computer-readable medium of claim 6, wherein the 3D point clouds are captured using multiple 3D cameras, and each of the 3D point clouds corresponds to a different relative position of the multiple 3D cameras with respect to the body.

8. The non-transitory computer-readable medium of claim 6, wherein the segmented representation comprises a segmented mesh representing the physical object.

9. The non-transitory computer-readable medium of claim 6, wherein each of the cylindrical representations is handled using an image R(x,θ

), where R represents a radius value of the cylindrical representation at position x along the axis, and θ

the angle;

wherein an axis of at least one of the cylindrical representations is curved; and

the operations comprise smoothing the captured clouds of 3D points using image-based operators.

10. The non-transitory computer-readable medium of claim 9, wherein the operations comprise performing interpolation, in an image domain, on the unwrapped cylindrical maps to fill holes.

11. A system comprising:

processor electronics; and

computer-readable media configured and arranged to cause the processor electronics to perform operations comprising;

capturing three dimensional (3D) point clouds using a 3D camera, each of the 3D point clouds corresponding to a different relative position of the 3D camera with respect to a body;

setting one of the 3D point clouds as a reference point cloud;

determining transforms for coordinates of the captured 3D point clouds other than the reference point cloud to transform to coordinates of the reference point cloud;

segmenting the body represented in the reference point cloud into body parts corresponding to elements of a 3D part-based volumetric model comprising cylindrical representations; and

generating a segmented representation of the physical object of interest in accordance with the 3D part-based volumetric model, wherein the generating comprises, for each of the captured 3D point clouds other than the reference point cloud, transforming the captured 3D point cloud using its transform, segmenting the body represented in the transformed 3D point cloud using the body parts corresponding to the elements of the 3D part-based volumetric model, and determining local motion, for each of the body parts corresponding to the elements of the 3D part-based volumetric model, between the transformed 3D point cloud and the reference point cloud; and

wherein a junction between at least two of the cylindrical representations, which are processed as unwrapped cylindrical maps, is handled by;

setting a first of the cylindrical maps as a reference map,

transforming points of a second of the cylindrical maps to the reference map, and

blending the transformed points of the second of the cylindrical maps with points of the first of the cylindrical maps to smooth the junction.

12. The system of claim 11, wherein the 3D point clouds are captured using a single 3D camera.

13. The system of claim 11, wherein the segmented representation comprises a segmented mesh representing the physical object.

14. The system of claim 11, wherein each of the cylindrical representations is handled using an image R(x,θ

), where R represents a radius value of the cylindrical representation at position x along the axis, and θ

the angle;

wherein an axis of at least one of the cylindrical representations is curved; and

the computer-readable media is configured and arranged to cause the processor electronics to smooth the captured clouds of 3D points using image-based operators.

15. The system of claim 14, wherein the computer-readable media is configured and arranged to cause the processor electronics to perform interpolation, in an image domain, on the unwrapped cylindrical maps to fill holes.

16. The system of claim 11, comprising a server computer system and a user-interface computer, wherein the user-interface computer comprises the processor electronics, and the server computer system comprises the computer-readable media.