Localizing devices in augmented reality environment

• US 9,865,091 B2
• Filed: 09/02/2015
• Issued: 01/09/2018
• Est. Priority Date: 09/02/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Giraldi, Riccardo, Gavriliuc, Anatolie, Chua, Michelle, Muehlhausen, Andrew Frederick, Held, Robert Thomas, Heuvel, Joseph van den, Omotani, Todd Alan, Wifall, Richard J., Sadak, Christian, Alt, Gregory
• Primary Examiner(s): Mazumder, Tapas
• Application Number:
  US14/843,596
• Publication Number:
  US 20170061692A1
• Time in Patent Office: 860 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04R 29/005 Microphone arrays

Abstract

Examples are disclosed herein that relate to identifying and localizing devices in an environment via an augmented reality display device. One example provides, on a portable augmented reality computing device, a method including establishing a coordinate frame for an environment, and discovering, via a location-sensitive input device, a location of a physical manifestation of the device in the environment, assigning a device location for the device in the coordinate frame based upon the location of the physical manifestation, and modifying an output of the portable augmented reality computing device based upon a change in relative position between the portable augmented reality computing device and the physical manifestation in environment.

Claims

1. On a portable augmented reality computing device, a method comprising:

establishing a coordinate frame for an environment;

discovering, via a location-sensitive input device, a location of a physical manifestation of a device in the environment;

assigning a device location for the device in the coordinate frame based upon the location of the physical manifestation;

modifying an augmented reality image displayed by the portable augmented reality computing device based upon a change in relative position between the portable augmented reality computing device and the device in the environment by modifying a portion of the augmented reality image that corresponds to the device location to reduce occlusion of a view of the device in the environment by the augmented reality image.

2. The method of claim 1, wherein the location-sensitive input device comprises one or more of an image sensor and a microphone array.

3. The method of claim 1, wherein discovering the location of the physical manifestation comprises detecting a light controlled by the device in the environment.

4. The method of claim 1, wherein discovering the location of the physical manifestation comprises detecting motion of an object controlled by the device.

5. The method of claim 1, wherein discovering the location of the physical manifestation comprises detecting a location of a display screen.

6. The method of claim 1, further comprising discovering the device on a wireless network.

7. The method of claim 1, wherein the output is an augmented reality image, and wherein modifying the output of the portable augmented reality computing device comprises modifying the augmented reality image to avoid occluding the physical manifestation with the augmented reality image.

8. The method of claim 7, further comprising tracking motion of the portable augmented reality computing device and updating the augmented reality image based upon the tracking to avoid occluding the physical manifestation.

9. An augmented reality display device, comprising:

a see-through display device;

a location-sensitive input device;

a logic device; and

a storage device comprising instructions executable by the logic device to

establish a coordinate frame for an environment based upon data from the location-sensitive input device,

conduct communication with a device to direct the device to alter a physical manifestation of the device in the environment, the physical manifestation being detectable by the location-sensitive input device,

detect, via the location-sensitive input device, a location of the physical manifestation based on an alteration of the physical manifestation of the device in the environment,

assign a device location for the device in the coordinate frame based upon the location of the physical manifestation, and

modify an output of the augmented reality display device based upon a change in relative position between the augmented reality display device and the physical manifestation in the environment by modifying a portion of an augmented reality image that corresponds to the device location to reduce occlusion of a view of the device in the environment by the augmented reality image.

10. The augmented reality display device of claim 9, wherein the location-sensitive input device comprises one or more of an image sensor and a microphone array.

11. The augmented reality display device of claim 9, wherein the physical manifestation comprises a light controlled by the device in the environment.

12. The augmented reality display device of claim 9, wherein the physical manifestation comprises a motion of an object controlled by the device in the environment.

13. The augmented reality display device of claim 9, wherein the physical manifestation comprises a display screen.

14. The augmented reality display device of claim 9, wherein the instructions are further executable to discover the device on a wireless network.

15. The augmented reality display device of claim 9, wherein the output is an augmented reality image, and wherein the instructions are executable to modify the augmented reality image to avoid occluding the physical manifestation with the augmented reality image.

16. The augmented reality display device of claim 9, further comprising a motion sensor configured to track motion of the augmented reality display device.

17. The augmented reality display device of claim 9, wherein the augmented reality display device comprises a head-mounted display device.

18. A computing device, comprising:

a logic device; and

a storage device comprising instructions executable by the logic device to

establish a connection with a portable augmented reality display device remote from the computing device,

receive instruction from the portable augmented reality display device requesting the computing device to control each of a plurality of devices in the environment to alter a physical manifestation of the device in the environment,

send to the portable augmented reality display device information regarding a device address of each device in the environment,

send instruction to a first device of the plurality of devices in the environment to alter a physical manifestation output by the first device that is detectable by a location-sensitive input device of the portable augmented reality display device,

receive confirmation from the portable augmented reality display device confirming discovery of the first device,

send instruction to a second device of the plurality of devices in the environment to alter a physical manifestation output by the second device that is detectable by the location-sensitive input device of the portable augmented reality display device, and

receive confirmation from the portable augmented reality display device confirming discovery of the second device.

19. The computing device of claim 18, wherein the instructions are further executable to control the plurality of devices in the environment to alter the physical manifestations in a specified order.

20. The computing device of claim 18, wherein the instructions are executable to receive a request from the portable augmented reality display device for the device address of each device in the environment in a specified order.