Image-based rendering of real spaces

• US 9,836,885 B1
• Filed: 10/27/2014
• Issued: 12/05/2017
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Appliance Computing III, Inc.
• Original Assignee: Appliance Computing III, Inc.
• Inventors: Eraker, David, Kalaiah, Aravind, McGarty, Robert
• Primary Examiner(s): Chen, Frank
• Application Number:
  US14/525,057
• Time in Patent Office: 1,135 Days
• Field of Search: None
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06Q 50/16 Real estate
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Under an embodiment of the invention, an image capturing and processing system creates 3D image-based rendering (IBR) for real estate. The system provides image-based rendering of real property, the computer system including a user interface for visually presenting an image-based rendering of a real property to a user; and a processor to obtain two or more photorealistic viewpoints from ground truth image data capture locations; combine and process two or more instances of ground truth image data to create a plurality of synthesized viewpoints; and visually present a viewpoint in a virtual model of the real property on the user interface, the virtual model including photorealistic viewpoints and synthesized viewpoints.

Claims

1. A computer system for providing image-based rendering of real property, the computer system comprising:

a processor implementing program commands and processing data;

a memory storing the program commands and the processing data, the memory being operatively coupled to the processor;

a user interface visually presenting an image-based rendering of a real property, the user interface being operatively coupled with the memory and processor;

wherein the program commands cause the processor to;

obtain two or more photorealistic viewpoints from capture locations in a plurality of spaces of the real property using ground truth image data, the ground truth image data including images of the real property captured from the capture locations;

combine and process two or more instances of ground truth image data to create a plurality of synthesized viewpoints;

generate a virtual model of a current space from the plurality of spaces in the real property from the perspective of a current viewpoint, the current viewpoint being one of the plurality of photorealistic viewpoints or one of the plurality of synthesized viewpoints associated with the current space;

identify the current space in the real property where the current viewpoint is located using metadata associated with the ground truth data;

visually present the virtual model on a first portion of the user interface, a map overlay on a second portion of the user interface, and a text overlay on a third portion of the user interface, the map overlay indicating a position of the current viewpoint in the current space, the text overlay including a first label identifying the current space in the real property where the current viewpoint is located;

generate a plurality of spatial boundaries of the plurality of spaces of the real property, each of the plurality of spatial boundaries defining a parcel outline of a corresponding space among the plurality of spaces; and

generate the ground truth image data by annotating the images of the real property with second labels indicating the capture locations using the plurality of spatial boundaries,

wherein the virtual model, the map overlay, and the text overlay are functionally linked based on the position of the current viewpoint in the real property, and

wherein identifying the current space in the real property where the current viewpoint is located using the metadata associated with the ground truth data includes identifying the first label of the current space using the second labels indicating the capture locations.

2. The computer system of claim 1, further including a navigation interface for enabling virtual movement to different positions in the real property.

3. The computer system of claim 1 wherein the synthetic viewpoints enable rotation about a vertical axis and a pitch.

4. The computer system of claim 1, wherein the virtual model, the map overlay and the text overlay function as operatively linked navigation elements.

5. The computer system of claim 1, wherein the image-based rendering is performed by a plurality of image-based rendering techniques chosen from the group of view-dependent texture mapping, synthetic rendering from geometric proxies, and light field rendering.

6. The computer system of claim 1, wherein the current space is a room in a building of the real property.

7. The computer system of claim 1, wherein the program commands further cause the processor to:

generate the map overlay and the text overlay using metadata associated with the ground truth data; and

display an advertisement for the type of object in the virtual model on the user interface,

wherein the metadata further includes object metadata that identifies the type of the object in the virtual model, and

wherein the metadata includes any of global positioning system (GPS) coordinates and infrared (IR) point clouds.

8. The computer system of claim 1,

wherein the text overlay further includes a second label indicating a location of a different space in the real property, and

wherein the program commands further cause the processor to change the current viewpoint to a different viewpoint when a user input selecting the second label is received, the different viewpoint being positioned in the different space.

9. The computer system of claim 1, wherein the program commands further cause the processor to:

receive an offer to schedule an in-person tour of the real property.

10. A computer-implemented method, comprising:

obtaining two or more photorealistic viewpoints of a plurality of spaces in a real property from capture locations using ground truth image data, the ground truth image data including images of the real property captured from the capture locations;

generating a plurality of synthesized viewpoints by combining and processing two or more instances of the obtained ground truth image data;

generating a virtual model of a current space from the plurality of spaces in the real property from the perspective of a current viewpoint, the current viewpoint being one of the plurality of photorealistic viewpoints or one of the plurality of synthesized viewpoints associated with the current space;

identifying the current space in the real property where the current viewpoint is located using metadata associated with the ground truth data;

presenting, via a user interface, the virtual model on a first portion of the user interface, a map overlay on a second portion of the user interface, and a text overlay on a third portion of the user interface, the map overlay indicating a position of the current viewpoint in the current space, the text overlay including a first label identifying the current space in the real property where the current viewpoint is located;

generating a plurality of spatial boundaries of the plurality of spaces of the real property, each of the plurality of spatial boundaries defining a parcel outline of a corresponding space among the plurality of spaces; and

generating the ground truth image data by annotating the images of the real property with second labels indicating the capture locations using the plurality of spatial boundaries,

wherein identifying the current space in the real property where the current viewpoint is located using the metadata associated with the ground truth data includes identifying the first label of the current space using the second labels indicating the capture locations.

11. The method of claim 10, wherein generating the plurality of synthesized viewpoints includes performing view-dependent texture mapping of the obtained ground truth image data.

12. The method of claim 10, wherein generating the plurality of synthesized viewpoints includes performing synthetic rendering from a plurality of geometric proxies of the obtained ground truth image data.

13. The method of claim 10, wherein generating the plurality of synthesized viewpoints includes performing light field rendering on the obtained ground truth image data.

14. The method of claim 10, wherein the virtual model, the map overlay and the text overlay function as operatively linked navigation elements.

15. The method of claim 10, wherein the current space is a room in a building of the real property.

16. The method of claim 10, further comprising:

generating the map overlay and the text overlay using metadata associated with the ground truth data;

displaying an advertisement for the type of object in the virtual model on the user interface,

wherein the metadata further includes object metadata that identifies the type of the object in the virtual model, and

wherein the metadata includes any of global positioning system (GPS) coordinates and infrared (IR) point clouds.

17. The method of claim 10,

wherein the text overlay further includes a second label identifying a different space in the real property, and

wherein the method further comprises changing the current viewpoint to a different viewpoint when a user input selecting the second label is received, the different viewpoint being positioned in the different space.

18. The method of claim 10, further comprising:

receiving an offer to schedule an in-person tour of the real property.