Selective surface mesh regeneration for 3-dimensional renderings

• US 10,366,534 B2
• Filed: 06/10/2015
• Issued: 07/30/2019
• Est. Priority Date: 06/10/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schoenberg, Michael John
• Primary Examiner(s): Zimmerman, Mark K
• Assistant Examiner(s): Doan, Phuc N
• Application Number:
  US14/735,789
• Publication Number:
  US 20160364907A1
• Time in Patent Office: 1,511 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 17/205 Re-meshing

Abstract

A 3D voxel grid of an environment is generated, with a signed value assigned to each voxel representative of that voxel'"'"'s distance from a nearest surface. For each subset of the 3D voxel grid comprising a surface, an initial surface mesh is generated. As new depth information is received, an updated surface is generated only for those subsets of the 3D voxel grid that exhibit greater than threshold change in signed values over time.

Claims

1. A method, comprising:

computer-generating a 3D voxel grid of an environment based on initial depth information received via a depth camera;

for each voxel in the 3D voxel grid, computer-assigning an initial signed value representative of that voxel'"'"'s distance from a nearest surface;

for each subset of voxels in the 3D voxel grid comprising a surface, computer-generating an initial surface mesh;

receiving, via the depth camera, updated depth information for the environment;

for each voxel in the 3D voxel grid, computer-assigning an updated signed value based on the updated depth information;

for each subset of voxels in the 3D voxel grid, determining an absolute difference between the updated signed value and the initial signed value for all voxels within the subset;

comparing the determined absolute difference to a non-zero threshold, the non-zero threshold based on a maximum allowed magnitude of discontinuity between adjacent subsets of voxels;

selectively computer-generating an updated surface mesh on only those subsets of voxels in the 3D voxel grid that exhibit greater than the non-zero threshold absolute difference between the updated signed value and the initial signed value; and

maintaining the initial surface mesh on those subsets of voxels in the 3D voxel grid where the absolute difference between the updated signed value and the initial signed value is less than the non-zero threshold, regardless of whether an updated surface mesh is computer-generated for adjacent subsets of voxels.

2. The method of claim 1, wherein the depth information comprises:

one or more depth images of the environment; and

information indicating a 6DOF pose of the depth camera relative to the environment at time of depth image capture.

3. The method of claim 2, wherein each signed value for a voxel is a signed distance field value proportionate to a distance between the voxel and the nearest surface.

4. The method of claim 3, wherein the nearest surface includes a surface point nearest to the voxel when compared to all other imaged surface points.

5. The method of claim 3, wherein the nearest surface includes a surface point located on a ray extending from the depth camera through the voxel.

6. The method of claim 3, wherein the signed distance field value is normalized and then truncated if an absolute value of the normalized signed distance field value is greater than a threshold.

7. The method of claim 1, wherein computer-generating the initial surface mesh comprises extracting an isosurface from the 3D voxel grid based on the initial signed values for each voxel within the 3D voxel grid.

8. The method of claim 7, wherein extracting an isosurface from the 3D voxel grid comprises executing a Marching Cubes algorithm on the 3D voxel grid.

9. The method of claim 1, wherein the determined absolute difference between each updated signed value and each previously assigned signed value is summed over time, and wherein an updated surface mesh is generated for subsets of voxels in the 3D voxel grid responsive to a sum of determined absolute differences exceeding the non-zero threshold.

10. The method of claim 1, where determining the absolute difference between the updated signed values and previously assigned signed values for all voxels within the subset comprises:

calculating a sum of absolute differences between the updated signed values and previously assigned signed values at each voxel within the subset.

11. A 3D modeling computer, comprising:

a depth camera interface configured to receive depth information from a depth camera;

a logic machine; and

a storage machine holding instructions executable by a logic machine to;

receive initial depth information for an environment;

generate a 3D voxel grid of the environment;

for each voxel in the 3D voxel grid, assign an initial signed value representative of that voxel'"'"'s distance from a nearest surface;

for each subset of the voxels in the 3D voxel grid comprising a surface, generate an initial surface mesh based on the signed values;

receive updated depth information for the environment;

for each voxel in the 3D voxel grid, assign an updated signed value based on the updated depth information;

for each subsets of voxels in the 3D voxel grid, determine an absolute difference between the updated signed value and the initial signed value for all voxels within the subset;

compare the determined absolute difference to a non-zero threshold, the non-zero threshold based on a maximum allowed magnitude of discontinuity between adjacent subsets of voxels;

selectively generate an updated surface mesh on only those subsets of voxels in the 3D voxel grid that exhibit greater than the non-zero threshold absolute difference between the updated signed value and the initial signed value; and

maintain the initial surface mesh on those subsets of voxels in the 3D voxel grid where the absolute difference between the updated signed value and the initial signed value is less than the non-zero threshold, regardless of whether an updated surface mesh is computer-generated for adjacent subsets of voxels.

12. The 3D modeling computer of claim 11, wherein receiving the depth information for the environment comprises:

receiving one or more depth images of the environment from the depth camera; and

receiving 6DOF pose information from a sensor subsystem indicating a location and orientation of the depth camera at time of depth image capture.

13. The 3D modeling computer of claim 12, wherein each signed value for a voxel is a signed distance field value proportionate to a distance between the voxel and a nearest corresponding point in the one or more depth images.

14. The 3D modeling computer of claim 13, wherein the signed distance field value is normalized and then truncated if an absolute value of the normalized signed distance field value is greater than a threshold.

15. The 3D modeling computer of claim 11, wherein the updated depth information includes two or more subsequent depth images, wherein the determined absolute difference between each updated signed value and each previously assigned signed value is summed over time, and wherein an updated surface mesh is generated for subsets of voxels in the 3D voxel grid responsive to a sum of determined absolute differences exceeding the non-zero threshold.

16. The 3D modeling computer of claim 11, wherein the storage machine further holds instructions executable by the logic machine to:

determine the absolute difference between the updated signed values and previously assigned signed values by calculating a sum of absolute differences between the updated signed values and previously assigned signed values at each voxel within the sub set.

17. A method, comprising:

receiving depth information for an environment via a depth camera;

computer-generating a 3D voxel grid of the environment based on initial depth information;

for each voxel in the 3D voxel grid, computer-assigning an initial signed value representative of that voxel'"'"'s distance from a nearest surface;

for each subset of voxels in the 3D voxel grid comprising a surface, computer-generating with a marching cubes algorithm an initial surface mesh;

receiving, via the depth camera, updated depth information for the environment;

for each voxel in the 3D voxel grid, computer-assigning an updated signed value based on the updated depth information;

for each subset of voxels in the 3D voxel grid, determining an absolute difference between the updated signed value and the initial signed value for all voxels within the subset;

comparing the determined absolute difference to a non-zero threshold, the non-zero threshold based on a maximum allowed magnitude of discontinuity between adjacent subsets of voxels;

selectively computer-generating with the marching cubes algorithm an updated surface mesh on only those subsets of voxels in the 3D voxel grid that exhibit greater than the non-zero threshold absolute difference between the updated signed value and the initial signed value; and

maintaining the initial surface mesh on those subsets of voxels in the 3D voxel grid where the absolute difference between the updated signed value and the initial signed value is less than the non-zero threshold, regardless of whether an updated surface mesh is computer-generated for adjacent subsets of voxels.

18. The method of claim 17, where greater than the non-zero threshold absolute difference between the updated signed value and the initial signed value includes a sum of determined changes exceeding the non-zero threshold.