RELOCALIZATION SYSTEMS AND METHODS

• US 20170161919A1
• Filed: 12/05/2016
• Published: 06/08/2017
• Est. Priority Date: 12/04/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Schroeder, Brigit, Malisiewicz, Tomasz J., Rabinovich, Andrew
• Granted Patent: US 10,909,711 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

A method of determining a pose of an image capture device includes capturing an image using an image capture device. The method also includes generating a data structure corresponding to the captured image. The method further includes comparing the data structure with a plurality of known data structures to identify a most similar known data structure. Moreover, the method includes reading metadata corresponding to the most similar known data structure to determine a pose of the image capture device.

Claims

1. A method of determining a pose of an image capture device, comprising capturing an image using an image capture device;

generating a data structure corresponding to the captured image;

comparing the data structure with a plurality of known data structures to identify a most similar known data structure;

reading metadata corresponding to the most similar known data structure to determine a pose of the image capture device.

2. The method of claim 1, wherein the data structure is a compact representation of the captured image.

3. The method of claim 2, wherein the data structure is an N dimensional vector.

4. The method of claim 3, wherein generating the data structure corresponding to the captured image comprises using a neural network to map the captured image to the N dimensional vector.

5. The method of claim 4, wherein the neural network is a convolutional neural network.

6. The method of claim 3, wherein the data structure is a 128 dimensional vector.

7. The method of claim 1, wherein each of the plurality of known data structures is a respective known N dimensional vector in an N dimensional space.

8. The method of claim 7, wherein the data structure is an N dimensional vector, and

wherein comparing the data structure with the plurality of known data structures to identify the most similar known data structure comprises

determining respective Euclidean distances between the N dimensional vector and each respective known N dimensional vector, and

identifying a known N dimensional vector having a smallest distance to the N dimensional vector as the most similar known data structure.

9. The method of claim 7, wherein each of the plurality of known data structures is a respective known 128 dimensional vector in a 128 dimensional space.

10. The method of claim 1, further comprising training a neural network by mapping a plurality of known images to the plurality of known data structures.

11. The method of claim 10, wherein the neural network is a convolutional neural network.

12. The method of claim 10, wherein training the neural network comprises modifying the neural network based on comparing a pair of known images of the plurality.

13. The method of claim 10, wherein training the neural network comprises modifying the neural network based on comparing a triplet of known images of the plurality.

14. The method of claim 10, wherein each known image of the plurality has respective metadata, including pose data, and

wherein training the neural network comprises accessing a database of the known images annotated with the respective metadata.

15. The method of claim 14, wherein the pose data encodes a translation and a rotation of a camera corresponding to a known image.

16. The method of claim 14, wherein each known data structure of the plurality is a respective known N dimensional vector in an N dimensional space,

wherein a first known image of the triplet is a matching image for a second known image of the triplet, and

wherein a third known image of the triplet is a non-matching image for the first known image of the triplet.

17. The method of claim 16, wherein a first Euclidean distance between respective first and second pose data corresponding to the matching first and second known images is less than a predefined threshold, and

wherein a second Euclidean distance between respective first and third pose data corresponding to the non-matching first and third known images is more than the predefined threshold.

18. The method of claim 16, wherein training the neural network comprises:

decreasing a first Euclidean distance between first and second known N dimensional vectors respectively corresponding to the matching first and second known images in an N dimensional space; and

increasing a second Euclidean distance between first and third known N dimensional vectors respectively corresponding to the non-matching first and third known images in the N dimensional space.

19. The method of claim 1, further comprising comparing the data structure with the plurality of known data structures to identify the most similar known data structure in real time.

20. The method of claim 1, wherein the metadata corresponding to the most similar known data structure includes pose data corresponding to the most similar known data structure.

21. The method of claim 20, further comprising determining a pose of the image capture device from the pose data in the metadata of the most similar known data structure.