Optically augmenting electromagnetic tracking in mixed reality

• US 10,254,546 B2
• Filed: 06/06/2016
• Issued: 04/09/2019
• Est. Priority Date: 06/06/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Poulos, Adam G., Tomlin, Arthur, Balan, Alexandru Octavian, Dulu, Constantin, Edmonds, Christopher Douglas
• Primary Examiner(s): Bhat, Aditya
• Application Number:
  US15/174,920
• Publication Number:
  US 20170351094A1
• Time in Patent Office: 1,037 Days
• Field of Search: None
• CPC Class Codes: G01B 11/022 by means of tv-camera scanning
• CPC Class Codes: G01B 7/003 for measuring position, not involving coordinate determination coordinate measuring G01B7/004
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 2027/0192 Supplementary details
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A mixed reality system may comprise a head-mounted display (HMD) device with a location sensor and a base station, mounted a predetermined offset from the location sensor, that emits an electromagnetic field (EMF). An EMF sensor affixed to an object may sense the EMF, forming a magnetic tracking system. The HMD device may determine a relative location of the EMF sensor therefrom and determine a location of the EMF sensor in space based on the relative location, the predetermined offset, and the location of the location sensor. An optical tracking system comprising a marker an optical sensor configured to capture optical data may be included to augment the magnetic tracking system based on the optical data and a location of the optical sensor or marker. The HMD device may display augmented reality images and overlay a hologram corresponding to the location of the EMF sensor over time.

Claims

1. A mixed reality system comprising:

a head-mounted display (HMD) device comprising;

a location sensor from which the HMD device determines a location of the location sensor in space; and

a base station mounted at a fixed position relative to the HMD device a non-zero predetermined offset from the location sensor and configured to emit an electromagnetic field; and

an electromagnetic field sensor affixed to an object and configured to sense a strength of the electromagnetic field, the base station and electromagnetic field sensor together forming a magnetic tracking system;

wherein the HMD device includes a processor configured to;

determine a location of the electromagnetic field sensor relative to the base station based on the sensed strength;

determine a location of the electromagnetic field sensor in space based on the relative location, the predetermined offset, and the location of the location sensor in space;

determine that a confidence level of the location of the electromagnetic field sensor in space determined using the magnetic tracking system is less than a predetermined threshold; and

determine a secondary location of the electromagnetic field sensor in space using the optical tracking system;

wherein the mixed reality system comprises an optical tracking system comprising at least one marker and at least one optical sensor configured to capture optical data, and the processor is further configured to augment the magnetic tracking system based on the optical data and a location of the optical sensor or marker.

2. The mixed reality system of claim 1, wherein the optical tracking system is configured with the at least one optical sensor on the HMD device and the at least one marker on the object.

3. The mixed reality system of claim 1, wherein the optical tracking system is configured with the at least one optical sensor on the object and the at least one marker on the HMD device.

4. The mixed reality system of claim 1, wherein the marker comprises a light source.

5. The mixed reality system of claim 1, wherein the marker is reflective.

6. The mixed reality system of claim 1, wherein the processor is configured to:

determine a plurality of possible locations of the electromagnetic field sensor in space using the magnetic tracking system; and

disambiguate between the possible locations using the optical tracking system.

7. The mixed reality system of claim 1, wherein the object is a handheld input device configured to provide user input to the HMD device.

8. The mixed reality system of claim 7, wherein the handheld input device comprises a housing including a grip area and the at least one marker or the at least one optical sensor is located on at least one protuberance that extends outside of the grip area.

9. The mixed reality system of claim 1, wherein the confidence level is based at least on a change in the location of the electromagnetic field sensor in space over time.

10. A method of locating an object in a mixed reality system, the method comprising:

determining a location of a location sensor of a head-mounted display (HMD) device in space;

emitting an electromagnetic field from a base station mounted at a fixed position relative to the HMD device a non-zero predetermined offset from the location sensor;

sensing a strength of the electromagnetic field with an electromagnetic field sensor affixed to the object, the base station and electromagnetic field sensor together forming a magnetic tracking system;

determining, with a processor of the HMD device, a location of the electromagnetic field sensor relative to the base station based on the sensed strength;

determining, with the processor, a location of the electromagnetic field sensor in space based on the relative location, the predetermined offset, and the location of the location sensor in space;

determining that a confidence level of the location of the electromagnetic field sensor in space determined using the magnetic tracking system is less than a predetermined threshold;

determining a secondary location of the electromagnetic field sensor in space using the optical tracking system; and

using an optical tracking system comprising at least one marker and at least one optical sensor configured to capture optical data, augmenting the magnetic tracking system based on the optical data and a location of the optical sensor or marker.

11. The method of claim 10, further comprising configuring the at least one optical sensor on the HMD device and the at least one marker on the object.

12. The method of claim 10, further comprising configuring the at least one optical sensor on the object and the at least one marker on the HMD device.

13. The method of claim 10, wherein the marker comprises a light source.

14. The method of claim 10, wherein the marker is reflective.

15. The method of claim 10, further comprising:

determining a plurality of possible locations of the electromagnetic field sensor in space using the magnetic tracking system; and

disambiguating between the possible locations using the optical tracking system.

16. The method of claim 10, wherein the object is a handheld input device configured to provide user input to the HMD device.

17. The method of claim 16, wherein the handheld input device comprises a housing including a grip area and the at least one marker or the at least one optical sensor is located on at least one protuberance that extends outside of the grip area.

18. A mixed reality system comprising:

a head-mounted display (HMD) device comprising;

a location sensor from which the HMD device determines a location of the location sensor in space;

a base station mounted at a fixed position relative to the HMD device a non-zero predetermined offset from the location sensor and configured to emit an electromagnetic field;

at least one optical sensor configured to capture optical data; and

an at least partially see-through display configured to display augmented reality images;

an electromagnetic field sensor affixed to an object and configured to sense a strength of the electromagnetic field, the base station and electromagnetic field sensor together forming a magnetic tracking system; and

at least one marker on the object, the optical sensor and the marker together forming an optical tracking system;

wherein

the object includes a processor configured to determine a location of the electromagnetic field sensor relative to the base station based on the sensed strength;

the HDM device includes a processor configured to;

determine a location of the electromagnetic field sensor in space based on the relative location, the predetermined offset, and the location of the location sensor in space;

determine that a confidence level of the location of the electromagnetic field sensor in space determined using the magnetic tracking system is less than a predetermined threshold; and

determine a secondary location of the electromagnetic field sensor in space using the optical tracking system;

augment the magnetic tracking system based on the optical data and a location of the optical sensor or marker; and

the at least partially see-through display is configured to overlay a hologram that corresponds to the location of the electromagnetic field sensor in space over time.