Control device with holographic element

• US 10,126,553 B2
• Filed: 06/16/2016
• Issued: 11/13/2018
• Est. Priority Date: 06/16/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Poulos, Adam, Jentz, Lorenz Henric, Brown, Cameron, Ambrus, Anthony, Tomlin, Arthur, Dack, James, Kohler, Jeffrey, Rehmeyer, Eric Scott, Parker, Edward Daniel, Denhez, Nicolas, Boesel, Benjamin
• Primary Examiner(s): Haley, Joseph
• Application Number:
  US15/184,681
• Publication Number:
  US 20170363867A1
• Time in Patent Office: 880 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0103 comprising holographic elements
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A head-mounted display device may display a holographic element with a portable control device. Image data of a physical environment including the control device may be received and used to generate a three dimensional model of at least a portion of the environment. Using position information of the control device, a holographic element is displayed with the control device. Using the position information, it is determined that the control device is within a predetermined proximity of either a holographic object or a physical object. Based on determining that the control device is within the predetermined proximity, the displayed holographic element is modified.

Claims

1. A method for displaying via a head-mounted display device a holographic element with a portable control device, the method comprising:

receiving image data of a physical environment including the control device;

using the image data, generating a three dimensional model of at least a portion of the physical environment;

receiving position information of the control device within the physical environment;

based at least in part on the position information, displaying the holographic element in a world-locked manner with the control device;

based at least in part on the position information, determining that the control device is within a predetermined proximity of either a holographic object displayed via the head-mounted display device or a physical object; and

based at least in part on determining that the control device is within the predetermined proximity, modifying the holographic element displayed via the head-mounted display device.

2. The method of claim 1, wherein modifying the holographic element comprises providing visual feedback indicating a new capability of the control device.

3. The method of claim 2, wherein the new capability of the control device is related to the holographic object or the physical object.

4. The method of claim 1, wherein modifying the holographic element comprises providing visual feedback indicating an amount of pressure applied to the holographic object or the physical object.

5. The method of claim 1, wherein the control device is a first control device and the physical object is a second control device, the method further comprising, based at least in part on determining that the first control device is within the predetermined proximity of the second control device, displaying another holographic element via the head-mounted display device with the second control device.

6. The method of claim 1, wherein the control device comprises a light source, the method further comprising, based at least in part on determining that the control device is within the predetermined proximity, modifying light output by the light source.

7. The method of claim 6, wherein the predetermined proximity is from the control device to the holographic object displayed via the head-mounted display device.

8. The method of claim 1, wherein the control device is a hand-held controller.

9. The method of claim 1, wherein the control device is a wearable device.

10. The method of claim 1, wherein receiving the position information comprises receiving a signal generated by the control device.

11. A head-mounted display device, comprising:

an at least partially see-through display;

a processor; and

a memory holding instructions executable by the processor to;

receive image data of a physical environment including a control device;

using the image data, generating a three dimensional model of at least a portion of the physical environment;

receive position information of the control device within the physical environment;

based at least in part on the position information, display a holographic element in a world-locked manner with the control device;

based at least in part on the position information, determine that the control device is within a predetermined proximity of either a holographic object displayed via the head-mounted display device or a physical object; and

based at least in part on determining that the control device is within the predetermined proximity, modify the holographic element.

12. The head-mounted display device of claim 11, wherein modifying the holographic element comprises providing visual feedback indicating a new capability of the control device.

13. The head-mounted display device of claim 12, wherein the new capability of the control device is related to the holographic object or the physical object.

14. The head-mounted display device of claim 11, wherein the control device is a first control device and the physical object is a second control device, and the instructions are executable by the processor to, based at least in part on determining that the first control device is within the predetermined proximity of the second control device, display another holographic element via the head-mounted display device with the second control device.

15. The head-mounted display device of claim 11, wherein the control device comprises a light source and the instructions are executable by the processor to, based at least in part on determining that the control device is within the predetermined proximity, modify light output by the light source.

16. The head-mounted display device of claim 15, wherein the predetermined proximity is from the control device to the physical object.

17. The head-mounted display device of claim 11, wherein the control device is a hand-held controller.

18. The head-mounted display device of claim 11, wherein the control device is a wearable device.

19. The head-mounted display device of claim 11, wherein receiving the position information comprises receiving a signal generated by the control device.

20. A method for displaying via a head-mounted display device a holographic element with a hand-held control device, the method comprising:

receiving image data of a physical environment including the hand-held control device;

using the image data, generating a three dimensional model of at least a portion of the physical environment;

receiving position information of the control device within the physical environment;

based at least in part on the position information, displaying the holographic element in a world-locked manner with the hand-held control device;

based at least in part on the position information, determining that the hand-held control device is within a predetermined proximity of either a holographic object displayed via the head-mounted display device or a physical object; and

based at least in part on determining that the hand-held control device is within the predetermined proximity, modifying the holographic element displayed via the head-mounted display device to provide visual feedback indicating a new capability of the holographic element.