RECOGNITION OF IMAGE ON EXTERNAL DISPLAY

• US 20130187835A1
• Filed: 01/25/2012
• Published: 07/25/2013
• Est. Priority Date: 01/25/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Vaught, Ben, Sugden, Ben, Latta, Stephen
• Granted Patent: US 9,024,844 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

Embodiments are disclosed that relate to the recognition via a see-through display system of an object displayed on an external display device at which a user of the see-through display system is gazing. For example, one embodiment provides a method of operating a see-through display system comprising acquiring an image of an external display screen located in the background scene via an outward facing image sensor, determining via a gaze detection subsystem a location on the external display screen at which the user is gazing, obtaining an identity of an object displayed on the external display screen at the location determined, and performing an action based upon the identity of the object.

Claims

1. A method of operating a see-through display system, the see-through display system comprising a see-through display screen, a gaze detection subsystem configured to determine a direction of gaze of each eye of the user, and an outward facing image sensor configured to acquire images of a background scene relative to a user of the see-through display system, the method comprising:

acquiring an image of an external display screen located in the background scene via the outward facing image sensor;

determining via the gaze detection subsystem a location on the external display screen at which the user is gazing;

obtaining an identity of an object displayed on the external display screen at the location determined; and

performing an action based upon the identity of the object.

2. The method of claim 1, wherein obtaining the identity of the object comprises sending image information regarding the object to a remote computing device, and receiving the identity from the remote computing device.

3. The method of claim 2, wherein the remote computing device is not in control of the external display screen.

4. The method of claim 3, wherein performing an action comprises displaying contextual information related to the object on the see-through display.

5. The method of claim 4, further comprising sending a request for the contextual information to a network-based service after obtaining the identity, and receiving the contextual information from the network-based service.

6. The method of claim 3, wherein the remote computing device is in control of the external display screen.

7. The method of claim 6, wherein the object comprises a control on a user interface displayed on the external display screen, and wherein performing an action based upon the identity of the object comprises sending a user interface input command to the remote computing device.

8. The method of claim 6, wherein obtaining the identity comprises receiving from the remote computing device identities and locations of multiple objects displayed on the external display screen.

9. The method of claim 2, further comprising sending location information to the remote computing device.

10. The method of claim 1, further comprising tracking a change in location on the external display screen at which the user is gazing, and displaying a response to the change in location on the see-through display system.

11. The method of claim 10, wherein tracking changes comprises tracking changes via image data.

12. The method of claim 10, wherein tracking changes comprises tracking changes via motion data.

13. A head-mounted display system, comprising:

a see-through display screen;

one or more inward facing image sensors configured to acquire images of each eye of a user;

one or more outward facing image sensors configured to acquire images of a background scene relative to the user;

a logic subsystem configured to execute instructions; and

a data holding subsystem comprising instructions stored thereon that are executable by the logic subsystem to;

acquire via the one or more outward facing image sensors an image of an external display screen located in the background scene;

determine via image data from the one or more inward facing image sensors and the image of the external display screen a location on the external display screen at which the user is gazing;

send to a remote computing device image information for identifying an object displayed on the external display screen at the location at which the user is gazing;

receive information from the remote computing device comprising an identity of the object; and

display contextual information related to the object on the see-through display screen.

14. The head-mounted display system of claim 13, wherein the instructions executable to send the image information for identifying the object to the remote computing device are further executable to send the image information to a computing device that is not in control of the external display screen.

15. The head-mounted display system of claim 14, wherein the instructions are further executable to send a request for the contextual information to a network-based service after obtaining the identity, and to receive the contextual information from the network-based service.

16. The head-mounted display system of claim 13, wherein the remote computing device is in control of the external display screen.

17. The head-mounted display system of claim 16, wherein the object comprises a control on a user interface displayed on the external display screen, and wherein the instructions are further executable to send to the remote computing device a user interface input command based upon the identity of the object.

18. The head-mounted display system of claim 16, wherein the information from the remote computing device comprises identities and locations of a plurality of objects displayed on the external display screen.

19. A head-mounted display system, comprising:

a see-through display screen;

one or more inward facing image sensors configured to acquire images of each eye of a user;

one or more outward facing image sensors configured to acquire images of a background scene relative to the user;

a logic subsystem configured to execute instructions; and

a data holding subsystem comprising instructions stored thereon that are executable by the logic subsystem to;

acquire via the one or more outward facing image sensors an image of an external display screen located in the background scene;

communicate with a remote computing device that is controlling the display of images on the external display screen;

determine via image data from the one or more inward facing image sensors a location on the external display screen at which the user is gazing;

identify a user input control displayed on the external display screen at a location at which the user is gazing;

send to the remote computing device a user input command based upon the user input control.

20. The head-mounted display device of claim 19, wherein the instructions are executable to identify the user input control by receiving from the remote computing device information regarding locations of objects displayed on the external display screen controlled by the remote computing device, and to compare the location at which the user is gazing to in the information regarding locations of objects.