Image-based rendering of real spaces

• US 10,510,111 B2
• Filed: 09/13/2018
• Issued: 12/17/2019
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Appliance Computing III, Inc.
• Original Assignee: Appliance Computing III, Inc.
• Inventors: Eraker, David, Kalaiah, Aravind, McGarty, Robert
• Primary Examiner(s): Chen, Frank S
• Application Number:
  US16/130,555
• Publication Number:
  US 20190012833A1
• Time in Patent Office: 460 Days
• Field of Search: None
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06Q 50/16 Real estate
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Under an embodiment of the invention, an image capturing and processing system creates 3D image-based rendering (IBR) for real estate. The system provides image-based rendering of real property, the computer system including a user interface for visually presenting an image-based rendering of a real property to a user; and a processor to obtain two or more photorealistic viewpoints from ground truth image data capture locations; combine and process two or more instances of ground truth image data to create a plurality of synthesized viewpoints; and visually present a viewpoint in a virtual model of the real property on the user interface, the virtual model including photorealistic viewpoints and synthesized viewpoints.

Claims

1. A method, comprising:

receiving image data of a plurality of spaces in a real estate property, the image data including a plurality of images captured from a plurality of viewpoints, wherein each of at least some of the images correspond to a respective one of a plurality of rooms of the real estate property;

creating a plurality of panoramas of the plurality of spaces by processing and/or compositing the image data;

defining a plurality of spatial boundaries of the plurality of spaces in the real estate property using the image data, some of the spatial boundaries delineating a volume of one or more of the plurality of spaces of one or more rooms within the real estate property, and wherein a defined first spatial boundary is a further delineation within a second spatial boundary, the second spatial boundary corresponding to a parcel outline of the real estate property;

in response to receiving a user indication of a location within the real estate property, rendering a virtual model of a selected space of the one or more spaces delineated by a spatial boundary of the plurality of spatial boundaries that corresponds to the indicated location, the virtual model rendered using one or more of the plurality of panoramas created from the captured images that correspond to the one or more spaces delineated by the spatial boundary that corresponds to the indicated location; and

causing a device to display the virtual model with a first label indicating a location of the selected space in the spatial boundary that corresponds to the indicated location.

2. The method of claim 1, further comprising:

annotating the plurality of captured images by associating the plurality of captured images with second labels indicating capture locations of the plurality of images within the plurality of defined spatial boundaries; and

using the second labels, identifying the selected space where a selected viewpoint is located within the spatial boundary that corresponds to the indicated location.

3. The method of claim 2 wherein the annotations are manually entered during capture, or asynchronously generated during processing of the captured image data.

4. The method of claim 1, wherein the first label indicates a floor level of the property.

5. The method of claim 1, wherein the first label indicates at least one of a indoor area and/or a outdoor area.

6. The method of claim 1, wherein the first label indicates a kitchen, a bathroom, or a bedroom.

7. The method of claim 1, wherein the second label indicates at least one of a room in the property, an indoor area of the property, an outdoor area of the property, or a floor of the property.

8. The method of claim 1, further comprising:

calibrating the plurality of panoramas using camera geometry,

wherein the camera geometry is determined by a first pass feature detection using the plurality of images.

9. The method of claim 1, wherein the plurality of panoramas are grouped according to a panorama bundle adjustment heuristic to spatially orient the plurality of panoramas.

10. The method of claim 1 wherein the received image data for one or more of the plurality of spaces in the real estate property includes annotations indicating one or more of:

sequencing information, depth data, sensor information, and/or labels that describe location information or information usable in processing, rendering, or real estate related transactions.

11. The method of claim 10 wherein the sensor information is automatically generated metadata in a form of one or more of:

GPS coordinates, wifi-location triangulation data, compass readings, accelerometer data, barometric data, and/or altimeter data.

12. The method of claim 1 wherein the received image data for one or more of the plurality of spaces in the real estate property includes geometric data, and further comprising:

using the geometric data and the captured images to construct one or more point clouds;

using 3D reconstruction algorithms, generating one or more geometric proxies from the constructed point clouds, each geometric proxy comprising a 3D model with registered images and/or texture maps at associated points in the 3D model; and

in response to receiving a user indication of a second location in the real estate property, determining whether a panorama created from captured images that corresponds to the second location exists;

when the panorama from captured images that corresponds to the second location exists, rendering a view of a selected space by rendering the panorama that corresponds to the second location; and

when the panorama from captured images that corresponds to the second location does not exist, synthesizing and rendering a 3D view using at least one of the geometric proxies and registered captured images and/or texture maps that are associated with a point that corresponds to the second location.

13. The method of claim 12, further comprising:

when the panorama from captured images that corresponds to the second location does not exist, using at least one of the geometric proxies to render a transition between the panorama that corresponds to the indicated location and a panorama that corresponds to a third location.

14. The method of claim 12 wherein the 3D model is a view dependent texture mapped mesh.

15. The method of claim 12 wherein the geometric data is ground truth geometric data from a laser device and/or the geometric data is depth map data from an infrared structured light device.

16. The method of claim 12 wherein the user indication of the location and the second location are the same.

17. The method of claim 1, further comprising:

defining a spatial boundary delineating a room or floor of the real estate property and annotating the spatial boundary with a label identifying the room or floor;

receiving image data including a plurality of images captured from a plurality of viewpoints within the room or floor;

using the defined spatial boundary delineating the room or floor, generating a point cloud associated with the received image data;

generating a geometric proxy from the point cloud, the geometric proxy including the spatial boundary with the annotated label identifying the room or floor;

determining using the geometric proxy whether the user indicated location corresponds to a point within the spatial boundary delineating the room or floor; and

when it is determined that the user indicated location corresponds to the point within the spatial boundary delineating the room or floor, causing the device to display the virtual model with the label identifying the room or floor.

18. A method, comprising:

receiving image data of a plurality of spaces in a plurality of rooms in a real estate property, the image data including a plurality of images captured from a plurality of viewpoints, each image corresponding to a respective one of the plurality of rooms;

creating a plurality of panoramas of the plurality of spaces by processing and/or compositing the image data;

defining a plurality of spatial boundaries of the plurality of spaces in the plurality of rooms in the real estate property using the image data, each of the spatial boundaries delineating a volume of one or more of the plurality of spaces, wherein the volume delineated by a first spatial boundary corresponds to a first floor of the real estate property and the volume delineated by a second spatial boundary corresponds to a second floor of the real estate property, and wherein the defined first spatial boundary and second spatial boundary are further delineations within a third spatial boundary corresponding to a parcel outline of the real estate property;

in response to receiving a user indication of a floor of the real estate property, using the defined spatial boundary corresponding to the indicated floor, rendering a virtual model of a selected space of the one or more spaces delineated by the spatial boundary corresponding to the indicated floor using one or more of the created plurality of panoramas that correspond to one or more rooms located within the spatial boundary corresponding to the indicated floor; and

causing a device to display the virtual model with a first label indicating a location of the selected space in one of the generated plurality of spatial boundaries.

19. The method of claim 18, further comprising:

annotating each of the plurality of captured images by associating the plurality of captured images with second labels indicating a capture location of the image within the plurality of defined spatial boundaries; and

using the second labels, identifying the selected space where a selected viewpoint is located within the spatial boundary that corresponds to the indicated floor.

20. The method of claim 19 wherein the second label specifies a room label and wherein the first label specifies a floor label of the real estate property.

21. The method of claim 18, further comprising:

using the defined plurality of spatial boundaries to determine connectivity between the plurality of rooms and/or floors in the real estate property, the connectivity defining spatial navigation options in a user interface; and

causing the device to display the virtual model of the selected space consistent with the spatial navigation options.

22. The method of claim 18 wherein the receiving a user indication of a floor of the real estate property and the rendering a virtual model of a selected space of the one or more spaces delineated by the generated spatial boundary corresponding to the indicated floor, further comprises:

generating and displaying a floor plan corresponding to a portion of the real estate property; and

receiving a user indication of navigation to a floor of the real estate property for performing a virtual tour of the real estate property by receiving an indication to a corresponding location on the floor plan; and

rendering a virtual model of a space that corresponds to the indicated corresponding location on the floor plan.

23. The method of claim 18 wherein the receiving the user indication of the floor of the real estate property further comprises:

displaying one or more floor labels that indicate one or more floors of a portion of the real estate property; and

receiving a user indication of navigation to a floor of the one or more floors of the real estate property for performing a virtual tour of the real estate property by receiving an indication to a corresponding one of the displayed one or more floor labels, thereby enabling navigation to a non-contiguous space within the real estate property.