Mobile capture visualization incorporating three-dimensional and two-dimensional imagery

• US 10,127,722 B2
• Filed: 06/30/2016
• Issued: 11/13/2018
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: Matterport, Inc
• Original Assignee: Matterport, Inc
• Inventors: Shakib, Babak Robert, Bjorke, Kevin Allen, Bell, Matthew Tschudy
• Primary Examiner(s): Craddock, Robert J
• Application Number:
  US15/199,853
• Publication Number:
  US 20180144547A1
• Time in Patent Office: 866 Days
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning

Abstract

This application generally relates to systems and methods for generating and rendering visualizations of an object or environment using 2D and 3D image data of the object or the environment captured by a mobile device. In one embodiment, a method includes providing, by the system, a representation of a 3D model of an environment from a first perspective of the virtual camera relative to the 3D model, receiving, by the system, input requesting movement of the virtual camera relative to the 3D model, and selecting, by the system, a first 2D image from a plurality of two dimensional images associated with different capture positions and orientations relative to the 3D model based on association of a capture position and orientation of the first 2D image with a second perspective of the virtual camera relative to the 3D model determined based on the movement.

Claims

1. A method comprising:

generating, by a system comprising a processor, a three-dimensional model of an environment using sets of three-dimensional data respectively comprising points that have been aligned relative to a three-dimensional coordinate space, wherein the sets of the three-dimensional data are respectively associated with two-dimensional images of the environment captured by a camera at different capture positions and capture orientations relative to the environment;

facilitating, by the system, navigating and viewing the three-dimensional model as displayed via an interface from different perspectives of a virtual camera relative to the three-dimensional model;

providing, by the system, a representation of the three-dimensional model from a first position and a first orientation of the virtual camera relative to the three-dimensional model corresponding to a first perspective of the three-dimensional model;

determining, by the system, a second position and a second orientation of the virtual camera relative to the three-dimensional model corresponding to a second perspective of the virtual camera relative to the three-dimensional model based on user input indicating movement of the virtual camera from the first perspective to the second perspective; and

identifying, by the system, a two-dimensional image of the two-dimensional images providing a view of the three-dimensional model from the second perspective based on the two-dimensional image having a capture position associated with the second position and a capture orientation associated with the second orientation.

2. The method of claim 1, further comprising, based on the identifying:

facilitating, by the system, transitioning from displaying the representation to displaying the two-dimensional image via the interface; and

facilitating, by the system, generating a smooth visual transition between the representation and the two-dimensional image in association with the transitioning.

3. The method of claim 1, further comprising:

ranking, by the system, the two-dimensional images based a first degree of correspondence between their respective capture positions and the second position, and a second degree of correspondence between their respective capture orientations and the second orientation; and

selecting, by the system, the two-dimensional image for rendering based on the ranking.

4. The method of claim 1, wherein the representation comprises a two-dimensional image and the user input comprises selection of a location on the two-dimensional image as displayed via the interface.

5. The method of claim 1, wherein the user input comprises a type of movement of the virtual camera selected from a defined set of movement types included in a group consisting of:

pan left, pan right, orbit left around an object, orbit right around an object, move left, and move right, and wherein the identifying the two-dimensional image is based on the type of movement.

6. The method of claim 5, wherein the facilitating comprises rendering interactive buttons on the interface that provide for control of movement of the virtual camera according to the defined set of movement types.

7. The method of claim 1, wherein the determining the second position and the second orientation comprises determining the second position and the second orientation based on a navigation mode and a selected anchor point relative to the three-dimensional model.

8. The method of claim 7, wherein the selected navigation mode comprises a navigation mode selected from a group consisting of:

walking mode, a feature view mode, a dollhouse mode, and a floor plan mode.

9. The method of claim 7, wherein the determining the second position and the second orientation further comprises panning the virtual camera from the first perspective to the second perspective relative to the selected anchor point and based on the navigation mode.

10. The method of claim 1, wherein the two-dimensional image comprises a first two-dimensional image, and wherein the method further comprises:

identifying, by the system, a second two-dimensional image of the two-dimensional images having a second capture position associated with the first position and a second capture orientation associated with the first orientation; and

combining, by the system, the first two-dimensional image and the second two-dimensional image to generate a third two-dimensional image of the environment from the second perspective and having a wider field of view of the environment relative to the first two-dimensional image or the second two-dimensional image.

11. The method of claim 1, further comprising, wherein the representation comprises a first representation and wherein the method further comprises generating, by the system, a second representation of the three-dimensional model from the second perspective, comprising:

generating, by the system, a three-dimensional representation of the three-dimensional model from the second perspective; and

projecting, by the system, image data of the two-dimensional image onto the three-dimensional representation.

12. The method of claim 11, wherein the generating the second representation further comprises:

identifying, by the system, a hole in the three-dimensional representation; and

filling, by the system, the hole based on the projecting.

13. The method of claim 1, further comprising:

generating, by the system, a three-dimensional representation of the three-dimensional model from the second perspective;

identifying, by the system, an occluded region in the three-dimensional representation; and

employing, by the system, shadow mapping and depth data associated with the occluded region to identify an area or volume of the three-dimensional model corresponding to the occluded region.

14. The method of claim 13, further comprising:

identifying, by the system, image data included in one or more sets of the sets of three-dimensional data and associated with area or volume; and

applying, by the system, the image data to the occluded region in the three-dimensional representation.

15. The method of claim 14, wherein the image data comprises color data from the one or more sets that was captured at different positions and orientations relative to the occluded region.

16. The method of claim 14, wherein the image data comprises texture data from the one or more sets that was captured at different positions and orientations relative to the occluded region.

17. The method of claim 1, wherein the representation comprises a first representation and wherein the method further comprises:

identifying, by the system, three-dimensional mesh data associated with the three-dimensional model and the second perspective; and

generating, by the system, a second representation of the three-dimensional model from the second perspective by combining the two-dimensional image and the three-dimensional mesh data, wherein the second representation has a wider field of view of the three-dimensional model relative to the two-dimensional image alone.

18. A system comprising

a processor;

and

a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising;

receiving navigational input in association with navigation of a three-dimensional model of an object or environment, wherein the navigational input indicates a request to move a virtual camera from a first perspective of the three-dimensional model to a second perspective of the three-dimensional model;

determining, based on the navigational input, a second position and a second orientation for the virtual camera relative to the three-dimensional model corresponding to the second perspective;

identifying two or more two-dimensional images captured of the object or environment having capture positions or capture orientations that have a defined degree of similarity to the second position or the second orientation, respectively; and

combining the two or more two-dimensional images to generate a representation of the three-dimensional model based on the second position and the second orientation.

19. The system of claim 18, wherein the representation comprises a second representation and the receiving the navigational input comprises receiving the navigational input relative to a first representation of the three-dimensional model, wherein the operations further comprise:

transitioning from rendering the first representation to rendering the second representation based on the receiving the navigational input.

20. The system of claim 18, wherein the generating the representation of the three-dimensional model comprises:

identifying a two-dimensional image captured of the object or environment having a capture position and a capture orientation that corresponds to the second position and the second orientation, respectively, wherein the representation comprises the two-dimensional image.

21. A non-transitory machine-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:

providing a first representation of a three-dimensional model of an object or environment from a first position and a first orientation of a virtual camera relative to the three-dimensional model corresponding to a first perspective of the three-dimensional model;

receiving navigational input in association with navigation of the three-dimensional model, wherein the navigational input indicates a request to move the virtual camera from the first perspective to a second perspective of the three-dimensional model;

determining, based on the navigational input, a second position and a second orientation for the virtual camera relative to the three-dimensional model corresponding to the second perspective; and

identifying, a two-dimensional image included in a set of two-dimensional images captured of the object or environment that provides a view of the three-dimensional model from the second perspective based on the two-dimensional image having a capture position associated with the second position and a capture orientation associated with the second orientation.

22. The non-transitory machine-readable storage medium claim 21, wherein the operations further comprise:

based on the identifying, facilitating transitioning from displaying the first representation to displaying the two-dimensional image via the interface.

23. The non-transitory machine-readable storage medium claim 21, wherein the two-dimensional image comprises a first two-dimensional image, and wherein the operations further comprise:

identifying a second two-dimensional image included in the set of two-dimensional images having a second capture position associated with the first position and a second capture orientation associated with the first orientation; and

combining the first two-dimensional image and the second two-dimensional image to generate a second representation of the three-dimensional model from the second perspective.