MIXED THREE DIMENSIONAL SCENE RECONSTRUCTION FROM PLURAL SURFACE MODELS

• US 20170004649A1
• Filed: 06/30/2015
• Published: 01/05/2017
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sullivan, Steve, Collet Romea, Alvaro, Kirk, Adam
• Granted Patent: US 9,646,410 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A three-dimensional (3D) scene is computationally reconstructed using a combination of plural modeling techniques. Point clouds representing an object in the 3D scene are generated by different modeled techniques and each point is encoded with a confidence value which reflects a degree of accuracy in describing the surface of the object in the 3D scene based on strengths and weaknesses of each modeling technique. The point clouds are merged in which a point for each location on the object is selected according to the modeling technique that provides the highest confidence.

Claims

1. A method of modeling a three-dimensional object from plural image data sources, the method comprising:

providing a first point cloud including a first plurality of points defined in space, the first plurality of points being derived from a first one or more images of the object, the first one or more images being of a first image type, each point in the first plurality representing a location on a surface of the three-dimensional object, and each point in the first plurality having a first confidence value on a first confidence scale;

providing a second point cloud including a second plurality of points defined in space, the second plurality of points being derived from a second one or more images of the object, the second one or more images being of a second image type, each point in the second plurality representing a location on the surface of the three-dimensional object, and each point in the second plurality having a second confidence value on a second confidence scale;

merging the first plurality and the second plurality of points into a third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including;

normalizing one of the first or second confidence scales with the respective second or first confidence scale; and

for each location of the object for which a corresponding point exists in both the first point cloud and the second point cloud, selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud having a greater first or second normalized confidence value.

2. The method according to claim 1, further comprising:

providing the first point cloud having therein the first plurality of points derived using a multiview stereo (MVS) process.

3. The method according to claim 2, further comprising:

providing the first point cloud having therein the first plurality of points, each point in the first plurality having the first confidence value being based upon an average texturedness value of the first one or more images.

4. The method according to claim 1, further comprising:

providing the second point cloud having therein the second plurality of points derived using a visual hull process.

5. The method according to claim 4, further comprising:

providing the second point cloud having therein the second plurality of points, each point in the second plurality having the second confidence value being based on a depth of the visual hull of the second one or more images.

6. The method according to claim 1, further comprising:

normalizing one of the first and second confidence scales with the respective second or first confidence scale including using a plurality of training objects of progressively varying dimensions and varying texture, each of the plurality of training objects having a preferred imaging technique associated therewith, to train a binary classifier to select a point from the first or second pluralities of points corresponding to the imaging technique used to derive the first or second plurality of points, based upon the texture or dimensions of the object for any location on the object in space.

7. The method according to claim 1, further comprising:

merging the first plurality and the second plurality of points into the third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud in response to the first or second normalized confidence value being greater than a predetermined threshold.

8. A computing device configured for modeling a three-dimensional object from plural image data sources, the computing device comprising:

one or more processors;

a network interface for supporting communications with the rendering device; and

one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to perform a method for controlling access to data from the remote client device comprising the steps of;

provide a first point cloud including a first plurality of points defined in space, the first plurality of points being derived from a first one or more images of the object, the first one or more images being of a first image type, each point in the first plurality representing a location on a surface of the three-dimensional object, and each point in the first plurality having a first confidence value on a first confidence scale;

provide a second point cloud including a second plurality of points defined in space, the second plurality of points being derived from a second one or more images of the object, the second one or more images being of a second image type, each point in the second plurality representing a location on the surface of the three-dimensional object, and each point in the second plurality having a second confidence value on a second confidence scale;

merge the first plurality and the second plurality of points into a third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including;

normalizing one of the first and second confidence scales with the respective second or first confidence scale; and

for each location of the object for which a corresponding point exists in both the first point cloud and the second point cloud, selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud having a greater first or second normalized confidence value.

9. The computing device according to claim 8, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the first point cloud having therein the first plurality of points derived using a multiview stereo (MVS) process.

10. The computing device according to claim 9, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the first point cloud having therein the first plurality of points, each point in the first plurality having the first confidence value being based upon an average texturedness value of the first one or more images.

11. The computing device according to claim 8, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the second point cloud having therein the first second plurality of points defined in space derived using a visual hull process.

12. The computing device according to claim 11, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the second point cloud having therein the second plurality of points, each point in the second plurality having the second confidence value being based on a depth of the visual hull of the second one or more images.

13. The computing device according to claim 8, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to normalize one of the first and second confidence scales with the respective second or first confidence scale including using a plurality of training objects of progressively varying dimensions and varying texture, each of the plurality of training objects having a preferred imaging technique associated therewith, to train a binary classifier to select a point from the first or second pluralities of points corresponding to the imaging technique used to derive the first or second plurality of points, based upon the texture or dimensions of the object for any location on the object in space.

14. The computing device according to claim 8, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to merge the first plurality and the second plurality of points into the third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud in response to the first or second normalized confidence value being greater than a predetermined threshold.

15. A system for modeling a three-dimensional object from plural image data sources, the system comprising:

first and second image capture devices, the first and second image capture devices each being operative to produce image data representing an image of the three-dimensional object, at least one of the first or second image capture devices further being operative to produce image data representing a silhouette image of the object; and

a computing device configured for modeling a three-dimensional object from plural image data sources, the computing device including

one or more processors;

a network interface for supporting communications with the rendering device; and

one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to perform a method for modeling a three-dimensional object comprising the steps of;

provide a first point cloud including a first plurality of points defined in space, the first plurality of points being derived from a first one or more images of the object, the first one or more images being of a first image type, each point in the first plurality representing a location on a surface of the three-dimensional object, and each point in the first plurality having a first confidence value on a first confidence scale;

provide a second point cloud including a second plurality of points defined in space, the second plurality of points being derived from a second one or more images of the object, the second one or more images being of a second image type, each point in the second plurality representing a location on the surface of the three-dimensional object, and each point in the second plurality having a second confidence value on a second confidence scale;

merge the first plurality and the second plurality of points into a third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including;

normalizing one of the first and second confidence scales with the respective second or first confidence scale; and

for each location of the object for which a corresponding point exists in both the first point cloud and the second point cloud, selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud having a greater first or second normalized confidence value.

16. The system according to claim 15, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the first point cloud having therein the first plurality of points derived using a multiview stereo (MVS) process.

17. The system according to claim 16, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the first point cloud having therein the first plurality of points, each point in the first plurality having the first confidence value being based upon an average texturedness value of the first one or more images.

18. The system according to claim 15, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the second point cloud having therein the first second plurality of points defined in space derived using a visual hull process.

19. The system according to claim 18, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to provide the second point cloud having therein the second plurality of points, each point in the second plurality having the second confidence value being based on a depth of the visual hull of the second one or more images.

20. The system according to claim 15, further comprising:

the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to normalize one of the first and second confidence scales with the respective second or first confidence scale including using a plurality of training objects of progressively varying dimensions and varying texture, each of the plurality of training objects having a preferred imaging technique associated therewith, to train a binary classifier to select a point from the first or second pluralities of points corresponding to the imaging technique used to derive the first or second plurality of points, based upon the texture or dimensions of the object for any location on the object in space.