Augmented reality overlay for control devices

• US 9,329,678 B2
• Filed: 08/14/2012
• Issued: 05/03/2016
• Est. Priority Date: 08/14/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Krauss, Aaron, Latta, Stephen, Scavezze, Mike, McCulloch, Daniel, Mount, Brian, Geisner, Kevin
• Primary Examiner(s): Thompson, James A
• Application Number:
  US13/585,520
• Publication Number:
  US 20140049558A1
• Time in Patent Office: 1,358 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: H04N 21/42208 Display device provided on the remote control
• CPC Class Codes: H04N 21/42226 Reprogrammable remote control devices

Abstract

Embodiments for providing instructional information for control devices are disclosed. In one example, a method on a see-through display device comprising a see-through display and an outward-facing image sensor includes acquiring an image of a scene viewable through the see-through display and detecting a control device in the scene. The method also includes retrieving information pertaining to a function of an interactive element of the control device and displaying an image on the see-through display augmenting an appearance of the interactive element of the control device with image data related to the function of the interactive element.

Claims

1. On a see-through display device comprising a see-through display and an outward-facing image sensor, a method for providing instructional information for control devices, the method comprising:

acquiring an image of a physical scene viewable through the see-through display;

detecting a control device in the physical scene, the control device comprising a physical user-actuatable input device configured to control an associated computing device separate from the see-through display device;

retrieving information pertaining to a function of the physical user-actuatable input device of the control device, the function of the physical user-actuatable input device comprising a control instruction for the associated computing device;

displaying an image on the see-through display augmenting an appearance of the physical user-actuatable input device of the control device with image data related to the function of the physical user-actuatable input device;

detecting a change in function of the physical user-actuatable input device; and

displaying an updated image augmenting the appearance of the physical user-actuatable input device, the updated image providing updated information related to the change in function of the physical user-actuatable input device.

2. The method of claim 1, wherein the image comprises a graphical element related to the function of the physical user-actuatable input device, the graphical element being displayed on the see-through display over the physical user-actuatable input device.

3. The method of claim 1, wherein the image comprises a text box having text information describing the physical user-actuatable input device.

4. The method of claim 3, further comprising receiving a selection of the text box, and in response displaying additional information on the see-through display device.

5. The method of claim 1, wherein the image comprises an animation.

6. The method of claim 1, further comprising detecting a gaze of a user of the see-through display device at a selected physical user-actuatable input device of the control device, and wherein the image augments an appearance of the selected physical user-actuatable input device and not another input device.

7. The method of claim 1, wherein retrieving the information comprises retrieving the information from the control device.

8. The method of claim 1, wherein retrieving the information comprises retrieving the information from a network-based service.

9. A portable see-through display device, comprising:

an image sensor;

a see-through display;

a logic subsystem; and

a data-holding subsystem comprising instructions executable by the logic subsystem to;

acquire an image of a physical scene viewable through the see-through display;

detect a control device in the physical scene, the control device comprising a physical user-actuatable input device configured to control an associated computing device separate from the portable see-through device;

retrieve information pertaining to a function of the physical user-actuatable input device of the control device, the function of the physical user-actuatable input device comprising a control instruction for the associated computing device;

display on the see-through display an image augmenting an appearance of the physical user-actuatable input device of the control device, the image being related to the-function of the physical user-actuatable input device;

detect a change in function of the physical user-actuatable input device; and

display an updated image augmenting the appearance of the physical user-actuatable input device, the updated image providing updated information related to the change in function of the physical user-actuatable input device.

10. The device of claim 9, wherein the instructions are executable to detect a change in function of the physical user-actuatable input device by detecting a change in a current state of an application being controlled by the control device.

11. The device of claim 9, wherein the instructions are executable to detect a keyboard in the physical scene and display on the see-through display an image augmenting an appearance of keys of the keyboard in accordance with an application being controlled by the keyboard.

12. The device of claim 9, wherein the instructions are executable to display the image in the form of a graphical element related to the function of the physical user-actuatable input device that is displayed on the see-through display over the physical user-actuatable input device.

13. The device of claim 9, wherein the instructions are executable to display the image in the form of a text box having text information describing the physical user-actuatable input device.

14. The device of claim 13, wherein the instructions are further executable to receive a selection of the text box, and in response display additional information on the see-through display device.

15. The device of claim 9, wherein the instructions are executable to display the image in the form of an animation.

16. The device of claim 9, further comprising an inward-facing sensor configured to detect a gaze of a user of the see-through display device at a selected physical user-actuatable input device of the control device, and wherein the instructions are further executable to display the image augmenting an appearance of the selected physical user-actuatable input device and not another input device.

17. On a first computing device, a method for providing information regarding operation of a control device, the method comprising:

receiving a request from a second computing device for information regarding one or more functions of one or more physical user-actuatable input devices of the control device, the one or more functions including an instruction that is provided to an associated computing device being controlled by the control device and being separate from the second computing device;

in response to the request, sending the information to the second computing device;

receiving information regarding a change in one or more functions of one or more physical user-actuatable input devices of the control device; and

sending updated information to the second computing device, the updated information related to the change in function of the physical user-actuatable input device.

18. The method of claim 17, wherein the first computing device is configured to receive input from the control device, and wherein the method further comprises establishing a direct connection with the second computing device and receiving the request via the direct connection.

19. The method of claim 17, further comprising establishing a connection with the first device over a network and receiving the request via the network.

20. The method of claim 17, further comprising receiving application information with the request and sending a plurality of mappings for the one or more physical user-actuatable input devices of the control device, the plurality of mappings each corresponding to a different state of an application that is executed on the associated computing device being controlled by the control device and associated with the application information.