Image-based rendering of real spaces

• US 10,102,673 B2
• Filed: 12/05/2017
• Issued: 10/16/2018
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Appliance Computing III, Inc.
• Original Assignee: Appliance Computing III, Inc.
• Inventors: Eraker, David, Kalaiah, Aravind, McGarty, Robert
• Primary Examiner(s): Chen, Frank
• Application Number:
  US15/832,653
• Publication Number:
  US 20180096527A1
• Time in Patent Office: 315 Days
• Field of Search: None
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06Q 50/16 Real estate
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Under an embodiment of the invention, an image capturing and processing system creates 3D image-based rendering (IBR) for real estate. The system provides image-based rendering of real property, the computer system including a user interface for visually presenting an image-based rendering of a real property to a user; and a processor to obtain two or more photorealistic viewpoints from ground truth image data capture locations; combine and process two or more instances of ground truth image data to create a plurality of synthesized viewpoints; and visually present a viewpoint in a virtual model of the real property on the user interface, the virtual model including photorealistic viewpoints and synthesized viewpoints.

Claims

1. A method, comprising:

receiving image data of a plurality of spaces in a property, the image data including a plurality of images captured from a plurality of viewpoints;

creating a plurality of panoramas of the plurality of spaces using the image data;

rendering a virtual model of a selected space among the plurality of spaces using the plurality of panoramas;

causing a device to display the virtual model with a first label indicating a location of the selected space; and

defining spatial boundaries of the plurality of spaces in the property using the image data, the plurality of spaces including a plurality of rooms in the property,

wherein the image data includes metadata associated with the plurality of images, the metadata indicating capture locations of the images,

wherein rendering the virtual model of the selected space includes rendering a 3D scene of the selected space using the plurality of panoramas, and

wherein rendering the 3D scene includes;

determining camera geometry using the plurality of images;

receiving a selected viewpoint in the selected space;

generating a point cloud of the 3D model;

determining a geometric proxy for the selected viewpoint using the determined camera geometry and the point cloud; and

generating the 3D model using the geometric proxy.

2. The method of claim 1, further comprising:

calibrating the plurality of panoramas using the camera geometry,

wherein the camera geometry is determined by a first pass feature detection using the plurality of images.

3. The method of claim 1, wherein the plurality of panoramas are grouped according to a panorama bundle adjustment heuristic.

4. The method of claim 3, wherein the panorama bundle adjustment heuristic is initialized using the metadata, the metadata including one or both of GPS and WiFi positioning coordinates.

5. The method of claim 1, further comprising:

generating the plurality of spatial boundaries of the plurality of spaces using the plurality of images, each of the spatial boundaries defining a parcel outline of one of the plurality of spaces;

annotating the plurality of images with second labels indicating capture locations of the plurality of images using the plurality of spatial boundaries; and

identifying the selected space among the plurality of spaces where a selected viewpoint is located using the second labels,

wherein causing the device to display the virtual model with the first label indicating the location of the selected space includes visually presenting the virtual model on a first portion of a user interface, a map overlay on a second portion of the user interface, and a text overlay on a third portion of the user interface, the map overlay indicating a position of the selected viewpoint in the selected space, the text overlay including the second label identifying the selected space in the property where the selected viewpoint is located.

6. The method of claim 1, further comprising:

obtaining depth data of the plurality of spaces,

wherein creating the plurality of panoramas of the plurality of spaces includes generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data.

7. The method of claim 6 wherein generating the plurality of stitched panoramas includes:

generating a panorama neighborhood graph of the property;

generating a panorama spanning tree of the panorama neighborhood graph; and

performing panorama bundle adjustment on the panorama spanning tree.

8. The method of claim 6, rendering the virtual model includes registering image RGB data from the plurality of images with the depth data to generate a 3D point cloud.

9. The method of claim 1, further comprising:

receiving the plurality of images from a smart phone.

10. A system, comprising:

a processor;

a memory storing non-transitory program commands, which, when executed by the processor, cause the processor to;

receive image data of a plurality of spaces in a property, the image data including a plurality of images captured from a plurality of viewpoints;

create a plurality of panoramas of the plurality of spaces using the image data;

render a virtual model of a selected space among the plurality of spaces using the plurality of panoramas;

cause a device to display the virtual model with a first label indicating a location of the selected space,

wherein the image data includes metadata associated with the plurality of images, the metadata indicating capture locations of the images,

wherein the program commands cause the processor to render the virtual model of the selected space by rendering a 3D scene of the selected space using the plurality of panoramas, and

wherein the program commands cause the processor to render the 3D scene by;

determining camera geometry using the plurality of images;

receiving a selected viewpoint in the selected space;

generating a point cloud of the 3D model;

determining a geometric proxy for the selected viewpoint using the determined camera geometry and the point cloud; and

generating the 3D model using the geometric proxy.

11. The system of claim 10, wherein the program commands further cause the processor to:

obtain depth data of the plurality of spaces, and

wherein the program commands cause the processor to create the plurality of panoramas of the plurality of spaces by generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data.

12. The system of claim 10, wherein the program commands further cause the processor to:

generate a plurality of spatial boundaries of the plurality of spaces using the plurality of images, each of the spatial boundaries defining a parcel outline of one of the plurality of spaces;

annotate the plurality of images with second labels indicating capture locations of the plurality of images using the plurality of spatial boundaries; and

identify the selected space among the plurality of spaces where a selected viewpoint is located using the second labels,

wherein the program commands cause the processor to cause the device to display the virtual model with the first label indicating the location of the selected space by visually presenting the virtual model on a first portion of a user interface, a map overlay on a second portion of the user interface, and a text overlay on a third portion of the user interface, the map overlay indicating a position of the selected viewpoint in the selected space, the text overlay including the second label identifying the selected space in the property where the selected viewpoint is located.

13. A method, comprising:

receiving image data of a plurality of spaces in a property, the image data including a plurality of images captured from a plurality of viewpoints;

creating a plurality of panoramas of the plurality of spaces using the image data;

rendering a virtual model of a selected space among the plurality of spaces using the plurality of panoramas;

causing a device to display the virtual model with a first label indicating a location of the selected space; and

obtaining depth data of the plurality of spaces,

wherein creating the plurality of panoramas of the plurality of spaces includes generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data, and

wherein generating the plurality of stitched panoramas includes;

generating a panorama neighborhood graph of the property;

generating a panorama spanning tree of the panorama neighborhood graph; and

performing panorama bundle adjustment on the panorama spanning tree.

14. The method of claim 13, wherein the image data includes metadata associated with the plurality of images, the metadata indicating capture locations of the images, and

wherein rendering the virtual model of the selected space includes rendering a 3D scene of the selected space using the plurality of panoramas.

15. The method of claim 14, further comprising:

obtaining depth data of the plurality of spaces,

wherein creating the plurality of panoramas of the plurality of spaces includes generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data.

16. The method of claim 14, further comprising:

defining spatial boundaries of the plurality of spaces in the property using the image data, the plurality of spaces including a plurality of rooms in the property.

17. The method of claim 16, further comprising:

calibrating the plurality of panoramas using camera geometry,

wherein rendering the 3D scene includes;

determining the camera geometry using the plurality of images;

receiving a selected viewpoint in the selected space;

generating a point cloud of the 3D model;

determining a geometric proxy for the selected viewpoint using the determined camera geometry and the point cloud; and

generating the 3D model using the geometric proxy, and

wherein the camera geometry is determined by a first pass feature detection using the plurality of images.

18. The method of claim 13, wherein the plurality of panoramas are grouped according to a panorama bundle adjustment heuristic.

19. The method of claim 18, wherein the panorama bundle adjustment heuristic is initialized using the metadata, the metadata including one or both of GPS and WiFi positioning coordinates.