Coarse relocalization using signal fingerprints

• US 10,600,252 B2
• Filed: 06/03/2019
• Issued: 03/24/2020
• Est. Priority Date: 03/30/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bhatnagar, Tushar Cyril, Lazarow, Gavin Dean, Rosenberg, Jonathan Paul
• Primary Examiner(s): West, Lewis G
• Application Number:
  US16/429,512
• Publication Number:
  US 20190287311A1
• Time in Patent Office: 295 Days
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 2205/01 specially adapted for specific applications
• CPC Class Codes: G01S 5/02521 using a radio-map
• CPC Class Codes: G01S 5/02523 Details of interaction of receiver with radio-map
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04B 17/318 Received signal strength
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions

Abstract

A first display device and method are provided for accelerating the coarse relocalization process of the first display device by generating a session-specific identifier and sending it to a second display device, which transmits the identifier to a cloud service, which returns at least a portion of a map data set corresponding to the identifier. The returned map data set is then stitched into the local map data of the first display device to create an integrated map, which is used to render one or a plurality of holograms. The first display device may comprise a processor, a memory operatively coupled to the processor, and a fingerprint program stored in the memory and executed by the processor.

Claims

1. A first display device, comprising:

a processor;

a display;

a memory operatively coupled to the processor and the display; and

a client fingerprint program stored in the memory and executed by the processor of the first display device, wherein

the client fingerprint program is configured to;

receive a session-specific identifier from a second device, the session-specific identifier corresponding to a map data set;

transmit the session-specific identifier to a remote system;

receive from the remote system at least a portion of the map data set corresponding to the session-specific identifier; and

stitch the map data set into a local map data of the first display device to create an integrated map.

2. The first display device of claim 1, wherein

the client fingerprint program is further configured to cause the display of the first display device to display one or more holograms based on the map data corresponding to the session-specific identifier.

3. The first display device of claim 2, wherein

the map data set comprises keyframes;

and

the keyframes and pose-graphs record rotational and translational motion of the first display device through a real world three-dimensional environment.

4. The first display device of claim 3, wherein

at least two of the keyframes are linked to a common fingerprint.

5. The first display device of claim 3, wherein

the map data set corresponding to the session-specific identifier comprises neighboring map data of a neighborhood around one or a plurality of keyframes linked to the session-specific identifier.

6. The first display device of claim 1, wherein

when the client fingerprint program is unsuccessful in receiving the session-specific identifier from the second device, the client fingerprint program is configured to:

receive one or a plurality of signals from one or a plurality of signal sources by at least one sensor of the first display device;

generate a current signal fingerprint based on the received signals;

determine map data to load by comparing the current signal fingerprint against signal fingerprint data to generate a ranked list of matching fingerprints, including performing a statistical analysis to evaluate for proximity or degree of disambiguation of the current signal fingerprint to the signal fingerprint data; and

based on the ranked list, retrieve corresponding map data corresponding to the matching fingerprints in the ranked list.

7. The first display device of claim 1, wherein

the client fingerprint program generates the session-specific identifier based on signals from signal sources, the signal sources comprising at least one of wireless access points, radiofrequency sources, electromagnetic sources, light sources, sound sources, and heat sources.

8. The first display device of claim 7, wherein

the signals comprise beacon frames.

9. The first display device of claim 1, wherein

the session-specific identifier is received from the second device via low energy beacons.

10. The first display device of claim 1, wherein

the client fingerprint program transmits an identifier request;

and

the client fingerprint program receives the session-specific identifier from the second device that received the transmitted identifier request.

11. A method comprising:

receiving a session-specific identifier from a second device at a first display device, the session-specific identifier corresponding to a map data set;

transmitting the session-specific identifier from the first display device to a remote system;

receiving from the remote system at least a portion of the map data set corresponding to the session-specific identifier at the first display device; and

stitching the map data set into a local map data to create an integrated map at the first display device.

12. The method of claim 11, further comprising:

displaying one or more holograms based on the map data set corresponding to the session-specific identifier at the first display device.

13. The method of claim 12, wherein

the map data set comprises keyframes;

and

the keyframes and pose-graphs record rotational and translational motion of the first display device through a real world three-dimensional environment.

14. The method of claim 13, wherein

at least two of the keyframes are linked to a common fingerprint.

15. The method of claim 13, wherein

the map data set corresponding to the session-specific identifier comprises neighboring map data of a neighborhood around one or a plurality of keyframes linked to the session-specific identifier.

16. The method of claim 11, wherein

the session-specific identifier is generated at the second device based on beacon frames of signals from signal sources.

17. The method of claim 11, wherein

the session-specific identifier is generated at the second device based on signals from signal sources, the signal sources comprising at least one of wireless access points, radiofrequency sources, electromagnetic sources, light sources, sound sources, and heat sources.

18. The method of claim 11, wherein

the session-specific identifier is received at the first display device from the second device via low energy beacons.

19. The method of claim 11, further comprising:

transmitting an identifier request at the first display device; and

at the first display device, receiving the session-specific identifier from the second device that received the transmitted identifier request.

20. A server computing device, comprising:

a processor;

a memory operatively coupled to the processor;

signal fingerprint data stored in the memory; and

a server fingerprint program stored in the memory and executed by the processor of the computing device, wherein

the server fingerprint program is configured to;

receive from a first mobile device a session-specific identifier generated by a second mobile device based on one or a plurality of signals from one or a plurality of signal sources received by at least one sensor of the second mobile device; and

send to the first mobile device a corresponding map data set corresponding to the session-specific identifier, the corresponding map data set comprising neighboring map data of a neighborhood around one or a plurality of keyframes linked to the session-specific identifier, the neighboring map data captured and recorded by the second mobile device.