Generating computer models of 3D objects

• US 9,053,571 B2
• Filed: 06/06/2011
• Issued: 06/09/2015
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shotton, Jamie Daniel Joseph, Izadi, Shahram, Hilliges, Otmar, Kim, David, Molyneaux, David, Kohli, Pushmeet, Fitzgibbon, Andrew, Hodges, Stephen Edward
• Primary Examiner(s): Chow, Jeffrey
• Assistant Examiner(s): Cain, II, Leon T
• Application Number:
  US13/154,288
• Publication Number:
  US 20120306876A1
• Time in Patent Office: 1,464 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/251 involving models

Abstract

Generating computer models of 3D objects is described. In one example, depth images of an object captured by a substantially static depth camera are used to generate the model, which is stored in a memory device in a three-dimensional volume. Portions of the depth image determined to relate to the background are removed to leave a foreground depth image. The position and orientation of the object in the foreground depth image is tracked by comparison to a preceding depth image, and the foreground depth image is integrated into the volume by using the position and orientation to determine where to add data derived from the foreground depth image into the volume. In examples, the object is hand-rotated by a user before the depth camera. Hands that occlude the object are integrated out of the model as they do not move in sync with the object due to re-gripping.

Claims

1. A computer-implemented method of generating a 3D model of an object, comprising:

organizing, in a memory device, storage of data associated with a three-dimensional volume associated with the model, the volume comprising a plurality of voxels;

receiving a depth image from a substantially static depth camera, the depth image comprising a depth value at each image element relating to a distance from the depth camera to a scene comprising the object;

selecting and removing one or more image elements determined to relate to an unwanted portion of the scene to leave a wanted portion of the depth image comprising the object;

tracking the position and orientation of at least a portion of the object in the wanted portion of the depth image by computing registration parameters, the registration parameters being parameters of a transformation for aligning the wanted portion of the depth image and a preceding depth image; and

integrating the wanted portion of the depth image into the volume by, for each voxel;

using the position and orientation to determine a corresponding location in the depth image for the voxel, determining a factor relating to the distance between each voxel and the depth value at the corresponding location, and updating a stored value at the voxel using the factor,

wherein determining the factor comprises calculating a signed distance function between the associated voxel and a point in the scene at the corresponding location.

2. A method according to claim 1, wherein the step of tracking the position and orientation of at least a portion of the object in the wanted portion of the depth image comprises tracking the position and orientation of the largest rigid object present in the wanted portion of the depth image.

3. A method according to claim 1, wherein the step of selecting and removing one or more image elements determined to relate to the unwanted portion of the scene comprises selecting and removing image elements located outside a three-dimensional region within the frustum of the depth camera.

4. A method according to claim 3, wherein the dimensions of the three-dimensional region are defined by a user using at least one of an input device and a gesture read by the depth camera.

5. A method according to claim 1, wherein the step of selecting and removing one or more image elements determined to relate to the unwanted portion of the scene comprises comparing the depth image with the preceding depth image to detect movement within the scene, and removing one or more image elements relating to substantially stationary portions of the scene.

6. A method according to claim 1, wherein the image elements from the wanted portion of the depth image are displayed on a display device in substantially real-time.

7. A method according to claim 1, wherein the volume is created on the memory device as linear pitched memory.

8. A method as claimed in claim 1, wherein the preceding depth image is estimated from the model stored in the volume.

9. A method according to claim 1, further comprising receiving a red-green-blue image from an RGB camera substantially co-located with the depth camera, the red-green-blue image comprising a color value at each image element.

10. A method according to claim 9, further comprising determining a corresponding voxel in the volume for each image element of the red-green-blue image and integrating the color value into the volume at the corresponding voxel by updating a previously stored color value.

11. A method according to claim 9, wherein the step of selecting and removing one or more image elements determined to relate to the unwanted portion of the scene comprises analyzing the red-green-blue image to detect one or more image elements relating to hands of the user and removing corresponding image elements from the wanted portion of the depth image.

12. A method according to claim 1, wherein computing the registration parameters comprises using an iterative process to identify corresponding points in pairs of depth images without computing shapes depicted within the depth images, and by using a parallel computing unit to optimize an error metric applied to the identified corresponding points such that the error metric is applied to each of the identified corresponding points in parallel.

13. A method according to claim 12, wherein computing the registration parameters further comprises using an iterative process to identify corresponding points in a pair of red-green-blue images from an RGB camera substantially co-located with the depth camera, the red-green-blue image comprising a color value at each image element.

14. A method as claimed in claim 12, wherein using the parallel computing unit to optimize an error metric comprises, for each pair of corresponding points, forming a linear system for a numerical least squares optimization and reducing the linear systems to a single 6 by 6 matrix at the parallel computing unit.

15. A method according to claim 1, wherein the step of integrating the wanted portion of the depth image into the volume comprises:

assigning a separate execution thread to each voxel in a plane of the volume;

each execution thread performing the steps of using the position and orientation to determine a corresponding location in the depth image for its associated voxel, determining a factor relating to the distance between the associated voxel and the point in the environment at the corresponding location, and updating a stored value at the associated voxel using the factor; and

each execution thread iterating through an equivalent voxel in each remaining plane of the volume and repeating the steps of determining the factor and updating the stored value for each voxel.

16. A method according to claim 1, wherein determining the factor comprises calculating the signed distance function between the associated voxel and the point in the scene at the corresponding location, such that if the associated voxel is located at a first side of the point, then the factor has a positive value, and if the associated voxel is located at an opposite side of the point, then the factor has a negative value.

17. A 3D object scanning system, comprising:

an input interface arranged to receive a depth image from a substantially static depth camera, the depth image comprising a depth value at each image element relating to a distance from the depth camera to a scene comprising the object;

a memory device; and

at least one processor arranged to;

organize, in the memory device, storage of data associated with a three-dimensional volume associated with a scanned model of the object the volume comprising a plurality of voxels;

select and remove one or more image elements determined to relate to an unwanted portion of the scene to leave a wanted portion of the depth image comprising the object, identification of the unwanted portion being based at least in part on the unwanted portion not moving in tandem with the wanted portion;

track the position and orientation of at least a portion of the object in the wanted portion of the depth image by computing registration parameters, the registration parameters being parameters of a transformation for aligning the wanted portion of the depth image and a preceding depth image; and

integrate the wanted portion of the depth image into the volume by using the position and orientation to determine a corresponding location in the depth image for each voxel, determining a factor relating to the distance between each voxel and the depth value at its corresponding location, and updating a stored value at each voxel using the factor, wherein determining the factor comprises calculating a signed distance function between the associated voxel and a point in the scene at the corresponding location.

18. A system according to claim 17, wherein the processor is a graphics processing unit arranged to provide concurrent execution of a plurality of threads, and the processor is further arranged to assign a separate thread to each voxel in a plane of the volume.

19. A system according to claim 17, wherein the object is rotated in the frustum of the depth camera by a user'"'"'s hands or a turntable.

20. One or more device-readable media, the device-readable media being hardware with device-executable instructions that, when executed by a processor, direct the processor to perform steps comprising:

organizing, in a memory device, storage of data associated with a three-dimensional volume associated with a 3D model of all sides of a real-world object, the volume comprising a plurality of voxels;

receiving a depth image from a substantially static depth camera, the depth image comprising a depth value at each pixel relating to a distance from the depth camera to a point in a scene comprising the object;

selecting and removing pixels from the depth image that relate to points in the scene that are outside a predefined three-dimensional region within the depth camera frustum to leave a foreground depth image;

tracking the position and orientation of at least a portion of the object in the foreground depth image by computing registration parameters, the registration parameters being parameters of a transformation for aligning the foreground depth image and a preceding depth image;

integrating the foreground depth image into the volume by, for each voxel;

using the position and orientation to determine a corresponding location in the depth image for the voxel, determining a factor relating to the distance between each voxel and the depth value at the corresponding location, and updating a stored value at the voxel using the factor; and

rendering an image of the model in substantially real-time for display on a display device using ray-casting from the volume.