Image-based rendering of real spaces

• US 10,592,973 B1
• Filed: 11/12/2019
• Issued: 03/17/2020
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Appliance Computing III, Inc.
• Original Assignee: Appliance Computing III, Inc.
• Inventors: Eraker, David, Kalaiah, Aravind, McGarty, Robert
• Primary Examiner(s): Chen, Frank S
• Application Number:
  US16/681,287
• Time in Patent Office: 126 Days
• Field of Search: None
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06Q 50/16 Real estate
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Under an embodiment of the invention, an image capturing and processing system creates 3D image-based rendering (IBR) for real estate. The system provides image-based rendering of real property, the computer system including a user interface for visually presenting an image-based rendering of a real property to a user; and a processor to obtain two or more photorealistic viewpoints from ground truth image data capture locations; combine and process two or more instances of ground truth image data to create a plurality of synthesized viewpoints; and visually present a viewpoint in a virtual model of the real property on the user interface, the virtual model including photorealistic viewpoints and synthesized viewpoints.

Claims

1. A computer-readable memory medium containing program commands for controlling a computer processor, when executed, to provide image-based rendering of a real property, by performing a method comprising:

obtaining two or more views from capture locations in a plurality of spaces of the real property using ground truth image data, the ground truth image data including images of the real property captured from the capture locations;

defining a plurality of spatial boundaries of the plurality of spaces of the real property, each spatial boundary delineating a volume of one or more of the plurality of spaces, at least one of the plurality of spatial boundaries defining a parcel outline of the real property, and at least one of the plurality of spatial boundaries defining a space located within the parcel outline;

annotating the ground truth image data by annotating at least some of the captured images of the real property with information indicating a capture location of a corresponding image relative to a corresponding spatial boundary, each capture location identifying a location within the corresponding spatial boundary defined space or identifying the space delineated by the corresponding spatial boundary;

combining and processing two or more instances of the ground truth image data to create a plurality of synthesized views, wherein the synthesized views comprise one or more of composites, transitions, or projections derived from processing ground truth image data, and wherein the synthesized views are associated with corresponding spaces of the plurality of spaces of the real property;

generating and rendering a virtual model of a current space within the plurality of spaces of the real property from a perspective of a current view, the current view being one of the plurality of views obtained using ground truth image data or one of the plurality of synthesized views derived from processing ground truth data and associated with the current space;

identifying the current space in the real property where the current view is located using the capture locations indicated by the annotated ground truth image data;

visually presenting, on the user interface, at least a portion of the virtual model, a map user interface element, and a text user interface element, the map user interface element indicating a position of the current view in the current space, the text user interface element including a label identifying the current space in the real property within which the current view is located;

wherein the portion of the virtual model, the map user interface element, and the text user interface element are functionally linked based on the position of the current view in the real property.

2. The computer-readable memory medium of claim 1 wherein the label identifying the current space in the real property is a floor designation, and wherein the annotating at least some of the captured images of the real property with information indicating a capture location of a corresponding image relative to a corresponding spatial boundary further comprises determining one or more spatial boundaries associated with the floor designation identifying the current space.

3. The computer-readable memory medium of claim 1 wherein a plurality of the visually presented at least a portion of the virtual model, the map user interface element, and the text user interface element are presented to the user at the same time.

4. The computer-readable memory medium of claim 1 wherein the map user interface element is a 2D projection of the rendered virtual model.

5. The computer-readable memory medium of claim 1 wherein the method further comprises:

presenting a navigation interface that facilitates virtual movement to different positions in the real property using at least one of a visually presented portion of the virtual model, a map user interface element, or a text user interface element.

6. The computer-readable memory medium of claim 1 wherein the combining and processing two or more instances of the ground truth image data to create the plurality of synthesized views further comprises:

constructing one or more point clouds using the ground truth image data;

generating one or more geometric proxies from the constructed point clouds, each geometric proxy comprising a model of geometric data with registered captured images or texture maps at associated points in the model of geometric data; and

rendering the at least some of the plurality of synthesized views from the generated geometric proxies.

7. The computer-readable memory medium of claim 6 wherein the generating and rendering the virtual model of the current space using at least one of the plurality of synthesized views is rendered from the generated one or more geometric proxies.

8. The computer-readable memory medium of claim 6 wherein the method further comprises:

capturing geometric data collected from one or more sensors, and wherein the one or more point clouds are constructed from a combination of ground truth image data and geometric data, and the one or more generated geometric proxies utilize a combination of ground truth image data and geometric data.

9. The computer-readable memory medium of claim 1 wherein the method further comprises:

creating a synthesized view that is a spherical panorama of captured ground truth image data, and wherein the rendered virtual model of the current space renders the spherical panorama.

10. The computer-readable memory medium of claim 1 wherein the synthetic views enable rotation of a presented portion of the virtual model about a vertical axis and a horizontal axis.

11. The computer-readable memory medium of claim 1 wherein the image-based rendering is performed using image-based rendering techniques comprising:

techniques for rendering with no geometry, techniques for rendering with implicit geometry, or techniques for rendering with explicit geometry.

12. The computer-readable memory medium of claim 1 wherein the current space is a room in a building of the real property.

13. The computer-readable memory medium of claim 1 wherein the method further comprises:

generating the map user interface element and the text user interface element using the annotations associated with the ground truth image data; and

displaying an advertisement for an object in the at least the portion of the virtual model presented on the user interface, based upon a type of the object.

14. The computer-readable memory medium of claim 1,

wherein the text user interface element further includes a second label indicating a location of a different space of the plurality of spaces of the real property;

and

wherein the method further comprises changing the current view to a different view upon receiving a user input selecting the second label, the different view being positioned in the different space of the plurality of spaces of the real property.

15. The computer-readable memory medium of claim 1 wherein the method further comprises:

receiving a request to schedule an in-person tour of the real property.

16. The computer-readable memory medium of claim 1 wherein the method further comprises:

facilitating a transaction regarding the real property.

17. The computer-readable memory medium of claim 16 wherein the transaction further comprises one or more of:

scheduling a tour, initiating messaging, initiating audio communication, customer relationship management, transaction document signing or tracking, on-line negotiations, offer submissions, inspection support, financing support, real-estate transaction workflows, data tracking, or reporting functions.

18. A computer-readable memory medium containing program commands for controlling a computer processor, when executed, to provide image-based rendering of a real property, by performing a method comprising:

obtaining two or more views from capture locations in a plurality of spaces of the real property using ground truth image data, the ground truth image data including images of the real property captured from the capture locations;

defining a plurality of spatial boundaries of the plurality of spaces of the real property, each spatial boundary delineating a volume of one or more of the plurality of spaces, at least one of the plurality of spatial boundaries defining a parcel outline of the real property, and at least one of the plurality of spatial boundaries defining a space located within the parcel outline;

annotating the ground truth image data by annotating at least some of the captured images of the real property with information indicating a capture location of a corresponding image relative to a corresponding spatial boundary, each capture location identifying a location within the corresponding spatial boundary defined space or identifying the space delineated by the corresponding spatial boundary;

combining and processing two or more instances of the ground truth image data to create a plurality of synthesized views, wherein the synthesized views comprise one or more composites, transitions, or projections derived from processing ground truth image data, and wherein the synthesized views are associated with corresponding spaces of the plurality of spaces of the real property by;

constructing one or more point clouds using the ground truth image data; and

generating one or more geometric proxies from the constructed point clouds, each geometric proxy comprising a model of geometric data with registered captured images or texture maps at associated points in the model of geometric data; and

generating and rendering a virtual model of a current space within the plurality of spaces of the real property from a perspective of a current view, the current view associated with the current space and being one of the plurality of views obtained using ground truth image data or one of the plurality of synthesized views derived from processing ground truth data;

identifying the current space in the real property where the current view is located using the capture locations indicated by the annotated ground truth image data;

visually presenting a portion of the rendered virtual model of the current space in a user interface as a navigable first panorama; and

in response to receiving a user indication to navigate to a second location within the plurality of spaces of the real property not contiguous to the current space,

determining a second panorama located nearby the second location using a point in the geometric proxy that corresponds to the second location; and

visually presenting a transition to the determined second panorama resulting in visual presentation of the second panorama.

19. The computer-readable memory medium of claim 18 wherein the second panorama is not adjacent to the first panorama in the virtual model.

20. The computer-readable memory medium of claim 18 wherein the first and second panoramas are spherical panoramas.

21. The computer-readable memory medium of claim 18 wherein the transition is performed using one or more image based rendering techniques for rendering with no geometry, rendering with implicit geometry, or rendering with explicit geometry.

22. The computer-readable memory medium of claim 18 wherein the user indication to navigate to the second location is selection of a label corresponding to one of the defined spatial boundaries of the plurality of spaces of the real property.

23. The computer-readable memory medium of claim 22 wherein the label corresponds to an indoor area, an outdoor area, a floor, or a room of the property.