Interaction and management of devices using gaze detection

• US 10,067,563 B2
• Filed: 10/27/2017
• Issued: 09/04/2018
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Parker, Crystal Lee, O'Hanlon, Mark Louis Wilson, Lovitt, Andrew, Farmer, Jason Ryan
• Primary Examiner(s): Lee, Jr., Kenneth B
• Application Number:
  US15/796,295
• Publication Number:
  US 20180059781A1
• Time in Patent Office: 312 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context

Abstract

User gaze information, which may include a user line of sight, user point of focus, or an area that a user is not looking at, is determined from user body, head, eye and iris positioning. The user gaze information is used to select a context and interaction set for the user. The interaction sets may include grammars for a speech recognition system, movements for a gesture recognition system, physiological states for a user health parameter detection system, or other possible inputs. When a user focuses on a selected object or area, an interaction set associated with that object or area is activated and used to interpret user inputs. Interaction sets may also be selected based upon areas that a user is not viewing. Multiple devices can share gaze information so that a device does not require its own gaze detector.

Claims

1. A device, comprising:

a gaze detector adapted to generate gaze information; and

a user-interface controller adapted to;

in response to the gaze information indicating that the user is gazing at the device, recognize an audio command uttered by the user, and provide a response to the audio command via the device; and

in response to the gaze information indicating that the user is not gazing at the device, recognize an audio command uttered by the user, and route response information to a second device for presentation to the user.

2. The device of claim 1, wherein the second device is selected from one of multiple alternative devices.

3. The device of claim 1, wherein the user-interface controller is further adapted to:

activate a set of audio inputs recognizable by the device in response to the gaze information indicating that the user is gazing at the device.

4. The device of claim 1, wherein the gaze information identifies a user line of sight, or a user point of focus, or an area that the user is not viewing.

5. The device of claim 1, wherein the gaze information identifies whether the user is looking at a graphical user interface on the device.

6. The device of claim 5, wherein the response is a presented on the graphical user interface.

7. The device of claim 1, wherein the response to the audio command is an audio response.

8. The device of claim 1, wherein the response information routed to the second device is provided in response to an audio command received by the device from the user, and wherein the second device is a mobile communication device or personal computing device operated by the user.

9. A method, comprising:

receiving, by a device, user gaze information;

in response to the user gaze information indicating that a user is gazing at a first portion of a graphical interface;

activating a first set of speech inputs recognizable by the device; and

associating a first set of inputs with the first portion of the graphical interface; and

in response to the gaze information indicating that the user is not gazing at any portion of the device;

activating a second set of speech inputs recognizable by the device; and

routing response information to a selected one of alternative devices.

10. The method of claim 9, wherein the user gaze information is a direction that a user is looking, a user'"'"'s point of focus, a user line of sight, or an area that the user is not viewing.

11. The method of claim 10, wherein the gaze information is provided by an external device or external cameras.

12. The method of claim 9, wherein the first set of inputs includes one or more verbal commands, and wherein the second set of inputs includes one or more different verbal commands.

13. The method of claim 9, wherein the first portion of the graphical interface is configured to render a first application, and wherein the second portion of the graphical interface is configured to render a second application distinct from the first application.

14. The method of claim 13, wherein the first application is a media application, and wherein the second application is a communications or navigation application.

15. The method of claim 13, wherein the non-verbal inputs control certain aspects of the first application if the user gaze information indicates the user is gazing at the first portion of the graphical interface and wherein the non-verbal inputs control certain aspects of the second application if the user gaze information indicates the user is gazing at the second portion of the graphical interface.

16. The method of claim 9, wherein the user gaze information comprises a gaze duration.

17. The method of claim 16, further comprising:

providing an alert if the gaze duration exceeds a predefined threshold.

18. The method of claim 9, further comprising:

in response to the user gaze information indicating that the user is gazing at a second portion of the graphical interface, wherein the first portion and the second portion are displayed simultaneously on the graphical interface;

activating a third set of speech inputs recognizable by the device; and

associating the third set of inputs with the second portion of the graphical interface.

19. A method, comprising:

receiving, by a device, user gaze information;

in response to the user gaze information indicating that a user is gazing at a graphical interface on the device, providing responses to user inputs via the graphical interface; and

in response to the gaze information indicating that the user is not gazing at the device, routing response information to the user via a selected one of alternative devices.

20. The method of claim 19, wherein the selected alternative device comprises a mobile communication device or a personal computing device operated by the user;

and

in response to the user gaze information indicating that the user is not gazing at the device, providing responses to user inputs via the graphical interface and routing response information to the user via the selected one of alternative devices.