WIDE AREA LOCALIZATION FROM SLAM MAPS

• US 20140323148A1
• Filed: 12/23/2013
• Published: 10/30/2014
• Est. Priority Date: 04/30/2013
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Schmalstieg, Dieter, Arth, Clemens, Pirchheim, Christian, Reitmayr, Gerhard, Ventura, Johnathan
• Application Number:
  US14/139,856
• Publication Number:
  US 20140323148A1
• Time in Patent Office: Days
• US Class Current: 455/456.1
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01V 3/38 Processing data, e.g. for analysis, for interpretation, for correction
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

Exemplary methods, apparatuses, and systems for performing wide area localization from simultaneous localization and mapping (SLAM) maps are disclosed. A mobile device can select a first keyframe based SLAM map of the local environment with one or more received images. A respective localization of the mobile device within the local environment can be determined, and the respective localization may be based on the keyframe based SLAM map. The mobile device can send the first keyframe to a server and receive a first global localization response representing a correction to a local map on the mobile device. The first global localization response can include rotation, translation, and scale information. A server can receive keyframes from a mobile device, and localize the keyframes within a server map by matching keyframe features received from the mobile device to server map features.

Claims

1. A method of performing wide area localization at a mobile device, comprising:

receiving, one or more images of a local environment of the mobile device;

initializing, a keyframe based simultaneous localization and mapping (SLAM) map of the local environment with the one or more images, wherein the initializing comprises selecting a first keyframe from one of the images;

determining, a respective localization of the mobile device within the local environment, wherein the respective localization is based on the keyframe based SLAM map;

sending, the first keyframe to a server; and

receiving, a first global localization response from the server.

2. The method of claim 1, further comprising:

referencing the keyframe based SLAM map to provide relative six degrees of freedom mobile device motion detection.

3. The method of claim 1, wherein the first global localization response is determined based on matching feature points and associated descriptors of the first keyframe to feature points and associated descriptors of a server map, and wherein the first global localization response provides a correction to a local map on the mobile device and includes one or more of:

rotation, translation, and scale information.

4. The method of claim 1, wherein the first keyframe sent to the server contains one or more new objects or scenes to extend a server map.

5. The method of claim 1, further comprising:

generating, a second keyframe as a result of the SLAM of the local environment;

sending, the second keyframe to the server as an incremental update; and

receiving, in response to the server receiving the incremental update, a second global localization response from the server.

6. The method of claim 1, further comprising:

displaying, at the mobile device, an augmented reality representation of the local environment upon initializing the keyframe based SLAM map; and

updating the augmented reality representation of the environment while tracking movement of the mobile device.

7. The method of claim 1, wherein the first keyframe comprises a camera image, camera position, and camera orientation when the camera image was captured.

8. A non-transitory storage medium having stored thereon instructions that, in response to being executed by a processor in a mobile device device, perform a method comprising:

receiving, one or more images of a local environment of the mobile device;

initializing, a keyframe based simultaneous localization and mapping (SLAM) map of the local environment with the one or more images, wherein the initializing comprises selecting a first keyframe from one of the images;

determining, a respective localization of the mobile device within the local environment, wherein the respective localization is based on the keyframe based SLAM map;

sending, the first keyframe to a server; and

receiving, a first global localization response from the server.

9. The medium of claim 8, further comprising:

referencing the keyframe based SLAM map to provide relative six degrees of freedom mobile device motion detection.

10. The medium of claim 8, wherein the first global localization response is determined based on matching feature points and associated descriptors of the first keyframe to feature points and associated descriptors of a server map, and wherein the first global localization response provides a correction to a local map on the mobile device which includes one or more of:

rotation, translation, and scale information.

11. The medium of claim 8, wherein the first keyframe sent to the server contains one or more new objects or scenes to extend a server map.

12. The medium of claim 8, further comprising:

selecting, a second keyframe from the one or more images of the local environment;

sending, the second keyframe to the server as an incremental update; and

receiving, in response to the server receiving the incremental update, a second global localization response from the server.

13. The medium of claim 8, further comprising:

displaying, at the mobile device, an augmented reality representation of the local environment upon initializing the keyframe based SLAM map; and

updating the augmented reality representation of the environment while tracking movement of the mobile device.

14. The medium of claim 8, wherein the first keyframe comprises a camera image, camera position, and camera orientation when the camera image was captured.

15. A mobile device for performing wide area localization comprising:

means for receiving, one or more images of a local environment of the mobile device;

means for initializing, a keyframe based simultaneous localization and mapping (SLAM) map of the local environment with the one or more images, wherein the initializing comprises selecting a first keyframe from one of the images;

means for determining, a respective localization of the mobile device within the local environment, wherein the respective localization is based on the keyframe based SLAM map;

means for sending, the first keyframe to a server; and

means for receiving, a first global localization response from the server.

16. The mobile device of claim 15, further comprising:

means for referencing the keyframe based SLAM map to provide relative six degrees of freedom mobile device motion detection.

17. The mobile device of claim 15, wherein the first global localization response is determined based on means for matching feature points and associated descriptors of the first keyframe to feature points and associated descriptors of a server map, and wherein the first global localization response provides a correction to a local map on the mobile device which includes one or more of:

rotation, translation, and scale information.

18. The mobile device of claim 15, wherein the first keyframe sent to the server contains one or more new objects or scenes to extend a server map.

19. The mobile device of claim 15, further comprising:

means for selecting, a second keyframe from the one or more images of the local environment;

means for sending, the second keyframe to the server as an incremental update; and

means for receiving, in response to the server receiving the incremental update, a second global localization response from the server.

20. The mobile device of claim 15, further comprising:

means for displaying, at the mobile device, an augmented reality representation of the local environment upon initializing the keyframe based SLAM map; and

means for updating the augmented reality representation of the environment while tracking movement of the mobile device.

21. The mobile device of claim 15, wherein the first keyframe comprises a camera image, camera position, and camera orientation when the camera image was captured.

22. A mobile device comprising:

a processor;

a storage device coupled to the processor and configurable for storing instructions, which, when executed by the processor cause the processor to;

receive, at an image capture device coupled to the mobile device, one or more images of a local environment of the mobile device;

initialize, a keyframe based simultaneous localization and mapping (SLAM) map of the local environment with the one or more images, wherein the initializing comprises selecting a first keyframe from one of the images;

determine, a respective localization of the mobile device within the local environment, wherein the respective localization is based on the keyframe based SLAM map;

send, the first keyframe to a server; and

receive, a first global localization response from the server.

23. The mobile device of claim 22, further comprising instructions to:

reference the keyframe based SLAM map to provide relative six degrees of freedom mobile device motion detection.

24. The mobile device of claim 22, wherein the first global localization response is determined based on matching feature points and associated descriptors of the first keyframe to feature points and associated descriptors of a server map, and wherein the first global localization response provides a correction to a local map on the mobile device which includes one or more of:

rotation, translation, and scale information.

25. The mobile device of claim 22, wherein the first keyframe sent to the server contains one or more new objects or scenes to extend a server map.

26. The mobile device of claim 22, further comprising instructions to cause the processor to:

select, a second keyframe from the one or more images of the local environment;

send, the second keyframe to the server as an incremental update; and

receive, in response to the server receiving the incremental update, a second global localization response from the server.

27. The mobile device of claim 22, further comprising instructions to cause the processor to:

display, at the mobile device, an augmented reality representation of the local environment upon initializing the keyframe based SLAM map; and

update the augmented reality representation of the environment while tracking movement of the mobile device.

28. The mobile device of claim 22, wherein the first keyframe comprises a camera image, camera position, and camera orientation when the camera image was captured.