CONNECTING AND USING BUILDING INTERIOR DATA ACQUIRED FROM MOBILE DEVICES

• US 20190020817A1
• Filed: 07/13/2017
• Published: 01/17/2019
• Est. Priority Date: 07/13/2017
• Status: Active Grant

• Current Assignee: MFTB Holdco Incorporated
• Original Assignee: Zillow Group, Inc.
• Inventors: Shan, Qi, Colburn, Alex, Guan, Li, Boyadzhiev, Ivaylo
• Granted Patent: US 10,530,997 B2
• Time in Patent Office: Days
• CPC Class Codes: G01P 1/127 for acceleration values
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

Techniques are described for automated operations involving acquiring and analyzing information from an interior of a house, building or other structure, for use in generating and providing a representation of that interior. Such techniques may include using a user'"'"'s mobile device to capture video data from multiple viewing locations (e.g., 360° video at each viewing location) within multiple rooms, and capturing data linking the multiple viewing locations (e.g., by recording video, acceleration and/or other data from the mobile device as the user moves between the two viewing locations), creating a panorama image for each viewing location, analyzing the linking information to model the user'"'"'s travel path and determine relative positions/directions between at least some viewing locations, creating inter-panorama links in the panoramas to each of one or more other panoramas based on such determined positions/directions, and providing information to display multiple linked panorama images to represent the interior.

Claims

1. A computer-implemented method comprising:

obtaining, by at least one device, linking information for a sequence of viewing locations within an interior of a building, including video information and acceleration data generated by a mobile device as a user carries the mobile device between each successive pair of viewing locations in the sequence;

determining, by the at least one device, and for each successive pair of viewing locations in the sequence, relative positional information that includes at least a direction from a starting viewing location of the successive pair to an ending viewing location of the successive pair, including analyzing the acceleration data of the linking information to model a travel path of the user from the starting viewing location to the ending viewing location of the successive pair and using the modeled travel path as part of determining the direction;

generating, by the at least one device, and using the determined direction for each successive pair of viewing locations in the sequence, information to link panorama images for the viewing locations, wherein each of the panorama images is associated with one of the viewing locations and has views from the one viewing location in each of multiple directions, and wherein the generating includes, for each of the viewing locations other than a last viewing location in the sequence, generating an inter-panorama link for the panorama image associated with the viewing location that points toward a next viewing location in the sequence; and

providing, by the at least one device and for display on a client device, information about the interior of the building that includes the panorama images and that includes the generated inter-panorama links for the panorama images.

2. The computer-implemented method of claim 1 wherein the determining of the relative positional information for one of the successive pairs of viewing locations in the sequence further includes determining a departure direction from the starting viewing location of the one successive pair by analyzing a first portion of the video information that corresponds to that starting viewing location, and determining an arrival direction at the ending viewing location of the one successive pair by analyzing a second portion of the video information that corresponds to that ending viewing location, and wherein modeling of the travel path for the one successive pair is further based in part on the determined departure direction and on the determined arrival direction.

3. The computer-implemented method of claim 2 wherein the determining of the relative positional information for the one successive pair further includes determining a second direction from the ending viewing location of the one successive pair to the starting viewing location of the one successive pair, and wherein the generating of the relative positional information further includes using the determined second direction to generate an inter-panorama link that points toward the panorama image associated with the starting viewing location of the one successive pair from the panorama image for the ending viewing location of the one successive pair.

4. The computer-implemented method of claim 2 wherein the analyzing of the acceleration data of the linking information for the one successive pair further includes performing a double integration operation on each of multiple data points of the acceleration data to determine velocity and location information for the data point, and combining the determined velocity and location information for the multiple data points to construct at least some of the travel path of the user for the one successive pair.

5. The computer-implemented method of claim 4 wherein the analyzing of the acceleration data of the linking information for the one successive pair further includes modeling bias from one or more sensors of the mobile device that provide the acceleration data, and using the modeled bias as part of performing the double integration operation.

6. The computer-implemented method of claim 4 wherein the determining of the relative positional information for the one successive pair further includes generating a confidence value for the direction from the starting viewing location of the one successive pair to the ending viewing location of the one successive pair, and determining to use the direction from the starting viewing location of the one successive pair to the ending viewing location of the one successive pair based at least in part on the generated confidence value exceeding a defined threshold.

7. The computer-implemented method of claim 6 wherein the generating of the confidence value is based at least in part on a length of the travel path for the one successive pair, on a straightness of the travel path for the one successive pair, on matching of the first portion of the video information to the panorama image for the starting viewing location of the one successive pair, and on matching of the second portion of the video information to the panorama image for the ending viewing location of the one successive pair.

8. The computer-implemented method of claim 1 wherein the determining of the relative positional information for one of the successive pairs of viewing locations in the sequence further includes identifying first images taken from the panorama image for the starting viewing location of the one successive pair that have features matching second images taken from the panorama image for the ending viewing location of the one successive pair, and using information from the first images and the second images as part of determining the direction from the starting viewing location of the one successive pair to the ending viewing location of the one successive pair.

9. The computer-implemented method of claim 8 wherein the using of the information from the first images and the second images as part of determining the direction further includes generating an essential matrix for each of multiple pairs of images from the first and second images, determining a value for the direction for each of the multiple pairs based on the essential matrix for the pair, and generating an aggregate value for the direction from the values determined for the multiple pairs.

10. The computer-implemented method of claim 8 wherein the using of the information from the first images and the second images as part of determining the direction further includes generating a homography matrix for each of multiple pairs of images from the first and second images, determining a value for the direction for each of the multiple pairs based on the homography matrix for the pair, and generating an aggregate value for the direction from the values determined for the multiple pairs.

11. The computer-implemented method of claim 8 wherein the using of the information from the first images and the second images as part of determining the direction further includes determining, for each of multiple pairs of images from the first and second images, whether to use an essential matrix or a homography matrix to determine a value for the direction for the pair.

12. The computer-implemented method of claim 8 wherein the using of the information from the first images and the second images as part of determining the direction further includes determining, for each of multiple pairs of images from the first and second images, and by using reprojection, that a degree of match between images of the pair exceeds a defined threshold before further using the pair to determine a value for the direction.

13. The computer-implemented method of claim 8 wherein the using of the information from the first images and the second images as part of determining the direction further includes generating a confidence value based on the information from the first images and the second images, and determining to use the direction from the starting viewing location of the one successive pair to the ending viewing location of the one successive pair based at least in part on the generated confidence value exceeding a defined threshold.

14. The computer-implemented method of claim 13 wherein the generating of the confidence value is based at least in part on a number of peaks in an aggregate consensus distribution generated using multiple pairs of images from the first and second images, and on circular coverage of samples in at least one of the panorama image for the starting viewing location of the one successive pair or the panorama image for the ending viewing location of the one successive pair.

15. The computer-implemented method of claim 1 wherein the generating of the information to link the panorama images for the viewing locations further includes modifying one or more of the directions from the starting viewing locations of each successive pair to the ending viewing locations of each successive pair by performing a global optimization on the directions, and wherein the generating of at least one inter-panorama link is based on at least one of the modified directions.

16. The computer-implemented method of claim 1 further comprising:

recording, by the mobile device, videos of the interior from each of the viewing locations as the user turns at the viewing locations; and

creating, by the at least one device and from the recorded videos, the panorama images for the viewing locations.

17. The computer-implemented method of claim 1 wherein the providing of the information includes receiving an indication from the client device of a user selection of one of the included generated inter-panorama links during a display of one of the panorama images, and causing, based at least in part on the user selection of the one included generated inter-panorama link, the client device to display another of the panorama images that is associated with the one included generated inter-panorama link.

18. The computer-implemented method of claim 1 wherein the at least one device includes the mobile device, such that the determining of the relative positional information, the generating of the information to link the panorama images, and the providing of the information are performed at least in part by the mobile device.

19. The computer-implemented method of claim 1 wherein the at least one device includes one or more server computing devices that are located remotely from the building and that receive via one or more intervening computer networks data recorded by the mobile device including the linking information and additional representing the sequence of viewing locations, such that one or more of the determining of the relative positional information, the generating of the information to link the panorama images, and the providing of the information are performed at least in part by the one or more server computing devices.

20. A non-transitory computer-readable medium having stored contents that cause one or more devices to perform automated operations, the automated operations including:

obtaining linking information for a sequence of viewing locations within an interior of a structure, including acceleration data generated by a device as the device moves between each successive pair of viewing locations in the sequence;

determining, by the one or more devices, and for one or more pairs of successive viewing locations in the sequence, relative positional information that includes at least a direction from a starting viewing location of the pair to an ending viewing location of the pair, including analyzing the acceleration data of the linking information to model a travel path of the device from the starting viewing location to the ending viewing location and using the modeled travel path as part of determining the direction;

generating, by the one or more devices, and using the determined direction for each of the one or more pairs of successive viewing locations, information to link visual information associated with the starting viewing location for the pair to visual information associated with the ending viewing location for the pair, including generating a link for the visual information associated with the starting viewing location for the pair that points toward the ending viewing location for the pair; and

providing, by the one or more devices, information for display about the interior of the structure that includes the visual information associated with the starting viewing location of each pair, the visual information associated with the ending viewing location of each pair, and the generated inter-panorama link for each pair.

21. The non-transitory computer-readable medium of claim 20 wherein the device is a mobile computing device associated with a user that participates in recording data for the sequence of viewing locations, wherein the visual information associated with the starting viewing location of each pair is a panorama image from that starting viewing location, wherein the visual information associated with the ending viewing location of each pair is a panorama image from that ending viewing location, and wherein the stored contents include software instructions that, when executed by the one or more devices, program the one or more devices to perform the automated operations.

22. A system, comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing devices to provide data representing a building, including;

obtaining data for a sequence of recording locations associated with the building that include at least one recording location within an interior of the building, including data that is recorded at each of the recording locations, and linking information that includes acceleration data associated with movement between successive recording locations in the sequence;

determining, based at least in part on analyzing the acceleration data, and for one or more pairs of successive recording locations in the sequence, a direction from a starting recording location of the pair to an ending recording location of the pair;

generating, using the determined direction for each of the one or more pairs of successive recording locations, information to link the data recorded at the starting recording location for the pair to the data recorded for the ending recording location for the pair, including generating a link that points from the starting recording location for the pair toward the ending recording location for the pair; and

providing information for presentation about the interior of the building that includes the data recorded at one or more of the recording locations and that includes at least one of the generated links.

23. The system of claim 22 wherein, for two successive recording locations in the sequence of recording locations, the data recorded for each of the successive recording locations includes recorded video, the obtaining of the linking information between the two successive recording locations further includes capturing visual data associated with the movement between the two successive recording locations, the stored instructions include software instructions that further cause the one or more computing devices to create a panorama image for each of the successive recording locations, and the providing of the information includes:

presenting, on a client device of a end user, the panorama image for a first recording location of the two successive recording locations, including presenting a visual representation for a generated link to a second recording location of the two successive recording locations; and

presenting, on the client device and after selection of the visual representation for the generated link, additional visual information that includes the panorama image for the second recording location.

24. The system of claim 22 wherein the obtaining of the data for the sequence of recording locations further includes recording data for one or more recording locations external to the building to include data for at least some of an exterior of the building.

25. The system of claim 22 wherein the obtaining of the data for the sequence of recording locations includes, for each of the at least some recording locations, capturing visual data at the recording location that includes multiple images captured in multiple directions from the recording location.

26. The system of claim 22 wherein the obtaining of the data for the sequence of recording locations includes, for each of the at least some recording locations, capturing audio data at the recording location.

27. A computer-implemented method comprising:

generating, by a mobile device carried by a user, visual data representing multiple locations within an interior of a house, including recording first visual information from a first viewing location within the house for a first panorama image having a 360-degree view around a vertical axis at the first viewing location, capturing, as the user carries the mobile device along a travel path from the first viewing location to a second viewing location within the house, linking information between the first and second viewing locations that includes acceleration data associated with movement of the mobile device along the travel path and that includes multiple images along the travel path, and recording second visual information from the second viewing location for a second panorama image having a 360-degree view around a vertical axis at the second viewing location;

automatically determining, based at least in part on analyzing the acceleration data and the multiple images included in the captured linking information to model the travel path of the user, relative positional information that includes direction information between the first and second viewing locations and that includes a distance between the first and second viewing locations;

automatically generating, using at least the determined direction information, information to link the first and second panorama images, including generating a first inter-panorama link for the first panorama image to be displayed in a direction of the second viewing location and a second inter-panorama link for the second panorama image to be displayed in a direction of the first viewing location; and

presenting, using the first and second panorama images and the generated first and second inter-panorama links, a display of the interior of the house on a client device that includes the first and second panorama images, including displaying the generated first inter-panorama link in the first panorama image in a direction of the second panorama image that is selectable by the user to display the second panorama image, and displaying the generated second inter-panorama link in the second panorama image in a direction of the first panorama image that is selectable by the user to display the first panorama image.

28. The computer-implemented method of claim 27 wherein the capturing of the linking information between the first and second viewing locations includes recording a third video as the user carries the mobile device from the first viewing location to the second viewing location, and wherein the automatic determining of the relative positional information is based in part on the recorded third video.

29. The computer-implemented method of claim 28 wherein the generating of the visual data representing the multiple locations within the interior of the house further includes, for each of multiple additional viewing locations in the house that are part of a sequence from the first viewing location to a final viewing location, and beginning with the second viewing location as a current viewing location in the sequence:

capturing, as the user carries the mobile device from the current viewing location in the sequence to a next viewing location in the sequence, additional linking information between the current and next viewing locations that includes acceleration data associated with movement of the mobile device;

recording an additional video of the interior of the house from the next viewing location that is captured as the user performs at least a full rotation around a vertical axis at the next viewing location, such that the next viewing location becomes a new current viewing location in the sequence; and

creating additional panorama images for the multiple additional viewing locations,

and wherein the automatic determining of the relative positional information includes determining relative positional information between each pair of viewing locations in the sequence, and

wherein the presenting includes presenting the additional panorama images including additional displayed user-selectable links between at least some of the additional panorama images.

30. The computer-implemented method of claim 29 wherein the recording of the additional video for each of the multiple additional viewing locations includes recording audio data, wherein one or more of the additional viewing locations are outside of the house, and wherein the additional video recorded at each of the one or more additional viewing locations includes a portion of an exterior of the house.