Automated spatial indexing of images based on floorplan features

• US 10,467,804 B2
• Filed: 06/28/2018
• Issued: 11/05/2019
• Est. Priority Date: 06/29/2017
• Status: Active Grant

• Current Assignee: Open Space Labs, Inc.
• Original Assignee: Open Space Labs, Inc.
• Inventors: Fleischman, Michael Ben, DeCamp, Philip, Kalanithi, Jeevan, Allen, Thomas Friel
• Primary Examiner(s): Demeter, Hilina K
• Application Number:
  US16/022,477
• Publication Number:
  US 20190005719A1
• Time in Patent Office: 495 Days
• Field of Search: None
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A spatial indexing system receives a sequence of images depicting an environment, such as a floor of a construction site, and performs a spatial indexing process to automatically identify the spatial locations at which each of the images were captured. The spatial indexing system also generates an immersive model of the environment and provides a visualization interface that allows a user to view each of the images at its corresponding location within the model.

Claims

1. A method comprising:

receiving a sequence of images from an image capture system, the sequence of images captured by a camera of the image capture system as the image capture system is moved along a camera path through an environment;

generating a first estimate of the camera path, the first estimate of the camera path specifying, for images in the sequence of images, a position of the image relative to a reference point;

obtaining a floorplan of the environment, the floorplan specifying positions of a plurality of physical features in the environment;

generating a combined estimate of the camera path based on the first estimate of the camera path and the positions of the plurality of physical features specified in the floorplan at least in part by generating a grid map based on the floorplan, the grid map comprising a plurality of nodes and edges, each of the edges connecting a first node and a second node of the plurality of nodes and specifying a transition score representing a traversabilitv of a space between the first node and the second node, the transition score generated based on physical features in the floorplan between the first node and the second node; and

automatically generating an immersive model of the environment based on the combined estimate of the camera path and received sequence of images, the immersive model specifying, for each image of a plurality of the images, a location of the image within the floorplan and at least one route vector defining a spatial distance between the image and at least one of the other images of the plurality of images.

2. The method of claim 1, wherein the camera is a 360-degree camera and the images are 360-degree images.

3. The method of claim 1, wherein the first estimate of the camera path is generated by performing a simultaneous localization and mapping process on the sequence of images.

4. The method of claim 1, wherein global satellite navigation system (GNSS) signals are substantially attenuated in the environment.

5. The method of claim 1, wherein an indoor positioning system (IPS) is not available in the environment.

6. The method of claim 1, further comprising:

receiving motion data from the image capture system, the motion data collected by an inertial measurement unit (IMU) of the image capture system as the image capture system is moved along the camera path;

generating a second estimate of the camera path based on the motion data, wherein generating the combined estimate of the camera path is further based on the second estimate of the camera path.

7. The method of claim 1, wherein the physical features in the floorplan include a doorway, and wherein the presence of a doorway in the floorplan between the first node and the second node to a higher transition score for the edge between the first node and the second node.

8. The method of claim 1, wherein the physical features in the floorplan include a wall, and wherein the presence of a wall in the floorplan between the first node and the second node leads to a lower transition score for the edge between the first node and the second node.

9. The method of claim 1, wherein generating the combined estimate of the camera path comprises:

performing a map matching process on the grid map and the first estimate of the camera path to generate the combined estimate of the camera path.

10. The method of claim 9, wherein performing the map matching process comprises:

identifying a node along the camera path;

generating a score for each of a plurality of other nodes in the grid map, the score for each of the other nodes generated based on the transition scores for the one or more edges between the identified node and the other node;

identifying one of the other nodes having a highest score; and

selecting the identified other node as a next node along the camera path.

11. The method of claim 1, further comprising:

determining, for each image in the sequence of images, an orientation of the camera when the image was captured.

12. The method of claim 1, wherein generating the first estimate of the camera path comprises generating a plurality of camera path segments.

13. The method of claim 12, wherein generating the combined estimate of the camera path comprises aligning each camera path segment of the plurality of camera path segments with the floorplan, the aligning of each camera path segment comprising:

generating a plurality of candidate camera paths, each of the candidate camera paths generated by performing one or more of a scaling operation and a rotation operation on the camera path segment,

generating a consistency score for each of the candidate camera paths, the consistency score representing a degree of consistency between the candidate camera path and the physical features in the floorplan; and

selecting the candidate camera path having the highest consistency score.

14. The method of claim 1, wherein automatically generating the immersive model is performed automatically and without manual user input.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform steps comprising:

receiving a sequence of images from an image capture system, the sequence of images captured by a camera of the image capture system as the image capture system is moved along a camera path through an environment;

generating a first estimate of the camera path, the first estimate of the camera path specifying, for images in the sequence of images, a position of the image relative to a reference point;

obtaining a floorplan of the environment, the floorplan specifying positions of a plurality of physical features in the environment;

generating a combined estimate of the camera path based on the first estimate of the camera path and the positions of the plurality of physical features specified in the floorplan at least in part by generating a grid map based on the floorplan, the grid map comprising a plurality of nodes and edges, each of the edges connecting a first node and a second node of the plurality of nodes and specifying a transition score representing a traversabilitv of a space between the first node and the second node, the transition score generated based on physical features in the floorplan between the first node and the second node; and

automatically generating an immersive model of the environment based on the combined estimate of the camera path and received sequence of images, the immersive model specifying, for each image of a plurality of the images, a location of the image within the floorplan and at least one route vector defining a spatial distance between the image and at least one of the other images of the plurality of images.

16. The non-transitory computer-readable storage medium of claim 15, wherein generating the first estimate of the camera path comprises generating a plurality of camera path segments.

17. The non-transitory computer-readable storage medium of claim 16, wherein generating the combined estimate of the camera path comprises aligning each camera path segment of the plurality of camera path segments with the floorplan, the aligning of each camera path segment comprising:

generating a plurality of candidate camera paths, each of the candidate camera paths generated by performing one or more of a scaling operation and a rotation operation on the camera path segment,

generating a consistency score for each of the candidate camera paths, the consistency score representing a degree of consistency between the candidate camera path and the physical features in the floorplan; and

selecting the candidate camera path having the highest consistency score.

18. The non-transitory computer-readable storage medium of claim 15, wherein automatically generating the immersive model is performed automatically and without manual user input.

19. A computing system comprising:

a processor;

a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform steps comprising;

receiving a sequence of images from an image capture system, the sequence of images captured by a camera of the image capture system as the image capture system is moved along a camera path through an environment;

generating a first estimate of the camera path, the first estimate of the camera path specifying, for images in the sequence of images, a position of the image relative to a reference point;

obtaining a floorplan of the environment, the floorplan specifying positions of a plurality of physical features in the environment;

generating a combined estimate of the camera path based on the first estimate of the camera path and the positions of the plurality of physical features specified in the floorplan at least in part by generating a grid map based on the floorplan, the grid map comprising a plurality of nodes and edges, each of the edges connecting a first node and a second node of the plurality of nodes and specifying a transition score representing a traversabilitv of a space between the first node and the second node, the transition score generated based on physical features in the floorplan between the first node and the second node; and

automatically generating an immersive model of the environment based on the combined estimate of the camera path and received sequence of images, the immersive model specifying, for each image of a plurality of the images, a location of the image within the floorplan and at least one route vector defining a spatial distance between the image and at least one of the other images of the plurality of images.

20. A method comprising:

receiving a sequence of images from an image capture system, the sequence of images captured by a camera of the image capture system as the image capture system is moved along a camera path through an environment;

generating a first estimate of the camera path, the first estimate of the camera path specifying, for images in the sequence of images, a position of the image relative to a reference point;

obtaining a floorplan of the environment, the floorplan specifying positions of a plurality of physical features in the environment;

generating a combined estimate of the camera path based on the first estimate of the camera path and the positions of the plurality of physical features specified in the floorplan at least in part by generating a grid map based on the floorplan, the grid map comprising a plurality of nodes and edges, each of the edges connecting a first node and a second node of the plurality of nodes and specifying a transition score representing a traversabilitv of a space between the first node and the second node, the transition score generated based on physical features in the floorplan between the first node and the second node; and

automatically generating an immersive model of the environment based on the combined estimate of the camera path and received sequence of images without using an existing immersive model, the immersive model specifying, for each image of a plurality of the images, a location of the image within the floorplan and at least one route vector defining a spatial distance between the image and at least one of the other images of the plurality of images.