Using free-form deformations in surface reconstruction

• US 9,483,879 B2
• Filed: 09/18/2014
• Issued: 11/01/2016
• Est. Priority Date: 09/18/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Gourlay, Michael Jason, Clark, Justin Avram
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US14/489,802
• Publication Number:
  US 20160086385A1
• Time in Patent Office: 775 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

Volumes of a 3D physical space are used in a surface reconstruction process, where adjacent volumes share vertices so that no gaps or overlaps between the volumes exist. As a result, a continuous surface is obtained in the surface reconstruction process. The vertices are anchored to nodes in a pose graph, such that locations of the vertices are adjusted as the pose graph is updated. As a result, a deformation of the volumes is permitted. Based on the deformation of a volume, a region of a depth map of the physical space is deformed correspondingly. Each vertex can be anchored to a closest node of the pose graph, or to a point which is based on a combination of nodes. In one approach, the point is defined based on the closest node and other nodes within a defined radius of the closest node.

Claims

1. A method for reconstructing a physical space, comprising:

obtaining a plurality of images of the physical space using a camera carried by a rig while the rig moves in the physical space through a plurality of poses;

determining an orientation and a coordinate system of each pose;

partitioning the physical space into a plurality of volumes, each volume of the plurality of volumes comprises vertices;

for each volume of the plurality of volumes, obtaining a depth map of the physical space using a depth sensor carried by the rig;

providing a pose graph in a first state, the pose graph in the first state comprising a plurality of nodes and links between nodes of the plurality of nodes, each node of the plurality of nodes is associated with;

one of the poses, one of the plurality of images associated with the one of the poses and one of the coordinate systems associated with the one of the poses, and the links define correspondences between the plurality of images and transforms between the coordinate systems;

for each volume of the plurality of volumes, anchoring the vertices of the volume to at least two different points in the pose graph in the first state, the vertices of the volume comprise shared vertices, the shared vertices are shared with an adjacent volume of the plurality of volumes when the pose graph is in the first state;

updating the pose graph to a second state, the updating comprising updating one or more of the nodes and one or more of the links; and

for each volume of the plurality of volumes, maintaining the shared vertices when the pose graph is in the second state, wherein the maintaining the shared vertices results in changing a shape of at least one volume of the plurality of volumes.

2. The method of claim 1, further comprising:

reconstructing a surface in the at least one volume of the plurality of volumes according to the second state of the pose graph; and

during the reconstructing, for the at least one volume of the plurality of volumes, distorting a depth map of the physical space based on the shape of the at least one volume of the plurality of volumes.

3. The method of claim 1, wherein:

each node of the plurality of nodes is associated with a location of the rig in the physical space; and

for one or more volumes of the plurality of volumes, the anchoring the vertices of the volume to at least two different points in the pose graph comprises anchoring one vertex of the volume to one of the nodes in the pose graph for which the location is closest to the one vertex.

4. The method of claim 1, wherein:

each node of the plurality of nodes is associated with a location of the rig in the physical space; and

for one or more volumes of the plurality of volumes, the anchoring the vertices of the volume to at least two different points in the pose graph comprises anchoring one vertex of the volume to a combination of nodes in the pose graph, the combination of nodes comprises one of the nodes in the pose graph for which the location is closest to the one vertex.

5. The method of claim 4, wherein:

the combination of nodes comprises one or more nodes within a specified distance of the one of the nodes in the pose graph for which the location is closest to the one vertex.

6. The method of claim 5, wherein:

in the combination of nodes, each node of the one or more nodes within the specified distance are weighted according to distance of the node to the one of the nodes in the pose graph for which the location is closest to the one vertex.

7. The method of claim 1, wherein:

for one or more volumes of the plurality of volumes, the anchoring the vertices of the volume to at least two different points in the pose graph comprises anchoring one vertex of the volume to one node in the pose graph and anchoring another vertex of the volume to another node in the pose graph.

8. The method of claim 1, further comprising:

for one or more volumes of the plurality of volumes, anchoring each vertex of the volume to a different point in the pose graph.

9. The method of claim 1, wherein:

the updating the pose graph comprises adding one or more nodes to the pose graph; and

for one or more volumes of the plurality of volumes, a number of points in the pose graph to which the vertices of the volume are anchored increases during the adding the one or more nodes to the pose graph.

10. The method of claim 1, wherein:

the plurality of volumes are non-overlapping with one another and gaps are not present between the volumes.

11. An apparatus for reconstructing a physical space, comprising:

an orientation sensor;

a depth sensor;

a camera; and

a processor in communication with the orientation sensor, the depth sensor and the camera, the processor is adapted to;

obtain a plurality of images of the physical space from the camera,

associate a pose and a coordinate system with each image based on the orientation sensor,

partition the physical space into a plurality of volumes, each volume of the plurality of volumes comprises vertices,

for each volume of the plurality of volumes, obtain a depth map of the physical space from the depth sensor,

provide a pose graph in a first state comprising a plurality of nodes and links between the nodes, each node of the plurality of nodes is associated with;

one of the poses, one of the plurality of images associated with the one of the poses and one of the coordinate systems associated with the one of the poses, and the links define correspondences between the plurality of images and transforms between the coordinate systems,

for each volume of the plurality of volumes, anchor the vertices of the volume to at least two different points in the pose graph in the first state, the vertices of the volume comprise shared vertices, the shared vertices are shared with an adjacent volume of the plurality of volumes when the pose graph is in the first state,

update the pose graph to a second state, using an update of one or more of the nodes and one or more of the links,

for each volume of the plurality of volumes, to maintain the shared vertices when the pose graph is in the second state, change a shape of at least one volume of the plurality of volumes, and

reconstruct a surface in one or more of the volumes according to the second state of the pose graph.

12. The apparatus of claim 11, wherein the processor is adapted to:

during the reconstructing of the surface, for the at least one volume of the plurality of volumes, distort a depth map of the physical space based on the shape of the at least one volume of the plurality of volumes.

13. The apparatus of claim 11, wherein:

each node of the plurality of nodes is associated with a location in the physical space; and

for one or more volumes of the plurality of volumes, the anchoring the vertices of the volume to at least two different points in the pose graph comprises anchoring one vertex of the volume to one of the nodes in the pose graph for which the location is closest to the one vertex.

14. The apparatus of claim 11, wherein:

each node of the plurality of nodes is associated with a location in the physical space; and

for one or more volumes of the plurality of volumes, the anchoring the vertices of the volume to at least two different points in the pose graph comprises anchoring one vertex of the volume to a combination of nodes in the pose graph, the combination of nodes comprises one of the nodes in the pose graph for which the location is closest to the one vertex.

15. The apparatus of claim 11, wherein:

the plurality of volumes are non-overlapping with one another and gaps are not present between the volumes.

16. An apparatus, comprising:

a storage device comprising code; and

a processor configured to execute the code to;

partition a physical space into a plurality of volumes, each volume of the plurality of volumes comprises vertices;

for each volume of the plurality of volumes, obtain a depth map of the physical space;

provide a pose graph in a first state, the pose graph in the first state comprising a plurality of nodes and links between nodes of the plurality of nodes;

for each volume of the plurality of volumes, anchor the vertices of the volume to at least two different points in the pose graph in the first state, the vertices of the volume comprise shared vertices, the shared vertices are shared with an adjacent volume of the plurality of volumes when the pose graph is in the first state;

update the pose graph to a second state, the updating comprising updating one or more of the nodes and one or more of the links; and

for each volume of the plurality of volumes, maintain the shared vertices when the pose graph is in the second state, wherein the maintaining the shared vertices results in changing a shape of at least one volume of the plurality of volumes.

17. The apparatus of claim 16, wherein:

the plurality of volumes are non-overlapping with one another and gaps are not present between the volumes.

18. The apparatus of claim 16, wherein the processor is configured to execute the code to:

reconstruct a surface in the at least one volume of the plurality of volumes according to the second state of the pose graph; and

during the reconstructing, for the at least one volume of the plurality of volumes, distort a depth map of the physical space based on the shape of the at least one volume of the plurality of volumes.

19. The apparatus of claim 16, wherein:

each node of the plurality of nodes is associated with a location in the physical space; and

for one or more volumes of the plurality of volumes, to anchor the vertices of the volume to at least two different points in the pose graph, the processor is configured to execute the code to anchor one vertex of the volume to one of the nodes in the pose graph for which the location is closest to the one vertex.

20. The apparatus of claim 16, wherein:

each node of the plurality of nodes is associated with a location in the physical space; and

for one or more volumes of the plurality of volumes, to anchor the vertices of the volume to at least two different points in the pose graph, the processor is configured to execute the code to anchor one vertex of the volume to a combination of nodes in the pose graph, the combination of nodes comprises one of the nodes in the pose graph for which the location is closest to the one vertex.