VISUAL LOCALISATION

• US 20150243080A1
• Filed: 09/23/2013
• Published: 08/27/2015
• Est. Priority Date: 09/21/2012
• Status: Active Grant

• Current Assignee: NavVis GmbH
• Original Assignee: NavVis GmbH
• Inventors: Steinbach, Eckehard, Huitl, Robert, Schroth, Georg, Hilsenbeck, Sebastian
• Granted Patent: US 10,319,146 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G06F 16/3328 using graphical result space presentation or visualisation
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

In an embodiment of the invention there is provided a method of visual localisation, comprising: generating a plurality of virtual views, wherein each of the virtual views is associated with a location; obtaining a query image; determining the location where the query image was obtained on the basis of a comparison of the query image with said virtual views.

Claims

1. A method of visual localisation, comprising:

generating a plurality of virtual views, wherein each of the virtual views is associated with a location;

obtaining at least one query image;

determining the location where the query image was obtained on the basis of a comparison of the query image with said virtual views.

2. The method of claim 1, further comprising:

generating said virtual views from a model of an environment, for example the inside of a building, in which said visual localisation is to be performed;

or

generating said virtual views from one or more reference images of an environment, for example the inside of a building, in which said visual localisation is to be performed.

3. The method of claim 2, further comprising:

extracting one or more features from one of said reference images; and

transforming said one or more features to obtain a visual representation of said features at one or more locations different from the location at which said one reference image was generated.

4. The method of claim 2, comprising:

transforming one or more of said reference images to the location of one or more said virtual views.

5. The method of claim 1, wherein said query image is generated by a mobile device, the method further comprising determining the pose of the mobile device on the basis of said comparison of the query image with said virtual views.

6. The method of claim 1, comprising:

determining one or more planes within said virtual views; and

allocating points of said query image to one or more of said planes.

7. The method of claim 1, comprising:

determining a plurality of points in an environment;

identify one or more planes in said environment from said plurality of points; and

mapping some or all of said points to said planes.

8. The method of claim 7, comprising:

for each point that has been mapped to a plane, determining one or more reference images that include said point.

9. The method of claim 7, comprising:

determining the trajectory of rays between one or more of said points and a given virtual viewpoint, thereby to detect planes within the virtual view from the given virtual viewpoint, and preferably repeating this step for each pixel associated with the virtual view from the given viewpoint, and preferably sorting the planes by the number of pixels that belong to each plane.

10. The method of claim 9, comprising:

processing each of the detected planes to determine reference images containing points that have been mapped to the detected planes, and assigning detected planes to reference images, in particular assigning that reference image that best matches a given plane; and

optionally repeating said assigning for some or all of the detected planes, in particular until the number of pixels associated with non-assigned planes reaches a predetermined threshold or there are no more unmapped reference images.

11. The method of claim 10, further comprising:

warping a reference image, in particular the reference image that best matches said given plane, to a given virtual viewpoint, thereby to generate a virtual view from said viewpoint.

12. The method of claim 10, wherein said assigning is performed only if the angle between the normal of the reference image and the normal of a given detected plane is below a predetermined angle, and/or wherein, if there is more than one candidate reference image for assigning, the one of the candidate reference images closest to the virtual viewpoint is selected.

13. The method of claim 1, comprising:

determining a plurality of points in an environment;

estimating a normal vector of each of said plurality of points; and

determining a planar surface element or a polygon for each of said plurality of points based on the respective normal vector.

14. The method of claim 13, comprising:

determining a polygon mesh comprising said polygons.

15. The method of claim 13, comprising:

for some or all of said points or said surface elements or said polygons determining one or more reference images that include said points or surface elements or polygons, respectively.

16. The method of claim 15, comprising:

for each reference image, creating a first depth map on the basis of said points that have been mapped to planar surface elements.

17. The method of claim 16, comprising:

generating said virtual views from one or more of reference images;

for each virtual view, generating a second depth map.

18. The method of claim 17, comprising:

warping a selected reference image to a given virtual viewpoint based on said first and/or second depth map, thereby to generate a virtual view from said viewpoint.

19. The method of claim 17, comprising:

comparing a predicted depth with a corresponding depth stored in the first and/or second depth map, thereby to detect occlusions or obstructions.

20. The method of claim 1, comprising extracting features from said virtual views, and preferably combining extracted features into a single bag-of-features vector that is indexed by a CBIR system for retrieval during localization.

21. The method of claim 20, comprising extracting features from said query image and retrieving the most similar virtual views from the CBIR system, thereby to determine the location and the orientation of a camera used to generate the query image, wherein the retrieving step is preferably performed by using inverted files.

22. An article storing non-transitory instructions that, when executed by a data processing system, cause the method of claim 1 to be performed.