Interaction and management of devices using gaze detection

• US 9,823,742 B2
• Filed: 05/18/2012
• Issued: 11/21/2017
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Parker, Crystal Lee, O'Hanlon, Mark Louis Wilson, Lovitt, Andrew, Farmer, Jason Ryan
• Primary Examiner(s): Lee, Jr., Kenneth B
• Application Number:
  US13/474,723
• Publication Number:
  US 20130307771A1
• Time in Patent Office: 2,013 Days
• Field of Search: 345156-172
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context

Abstract

User gaze information, which may include a user line of sight, user point of focus, or an area that a user is not looking at, is determined from user body, head, eye and iris positioning. The user gaze information is used to select a context and interaction set for the user. The interaction sets may include grammars for a speech recognition system, movements for a gesture recognition system, physiological states for a user health parameter detection system, or other possible inputs. When a user focuses on a selected object or area, an interaction set associated with that object or area is activated and used to interpret user inputs. Interaction sets may also be selected based upon areas that a user is not viewing. Multiple devices can share gaze information so that a device does not require its own gaze detector.

Claims

1. A method, comprising:

receiving, by a device, user gaze information;

in response to the user gaze information indicating that a user is gazing at a first portion of a graphical interface;

activating a first set of speech inputs recognizable by the device; and

associating a first set of inputs with the first portion of the graphical interface; and

in response to the gaze information indicating that the user is not gazing at any portion of the device, providing information to the user via another device.

2. The method of claim 1, wherein the user gaze information is a direction that a user is looking or a user'"'"'s point of focus or both.

3. The method of claim 1, wherein the user gaze information identifies an area that is not being covered by the user'"'"'s gaze.

4. The method of claim 2, wherein the user point of focus is provided by an external device or external cameras.

5. The method of claim 1, further comprising:

the other device including a mobile communication device or a personal computing device operated by the user; and

in response to the user gaze information indicating that the user is gazing at a second portion of the graphical interface, wherein the first portion and the second portion are displayed simultaneously on the graphical interface;

activating a second set of speech inputs recognizable by the device; and

associating the second set of inputs with the second portion of the graphical interface.

6. The method of claim 5, wherein the first portion of the graphical interface is configured to render a first application, and wherein the second portion of the graphical interface is configured to render a second application distinct from the first application.

7. The method of claim 6, wherein the first application is a media reproduction application and wherein the second application is a communications or navigation application.

8. The method of claim 1, wherein the user gaze information comprises a gaze duration.

16. The method of claim 6, wherein the non-verbal inputs control certain aspects of the first application if the user gaze information indicates the user is gazing at the first portion of the graphical interface and wherein the non-verbal inputs control certain aspects of the second application if the user gaze information indicates the user is gazing at the second portion of the graphical interface.

17. The method of claim 8, further comprising:

providing an alert if the gaze duration exceeds a predefined threshold.

18. The method of claim 5, wherein the first set of inputs includes one or more verbal commands and wherein the second set of inputs includes one or more different verbal commands.

9. A method, comprising:

receiving, by a device, user gaze information;

in response to the user gaze information indicating that a user is gazing at a first portion of a graphical interface, activating a first set of gesture inputs recognizable by the device; and

in response to the gaze information indicating that the user is not gazing at any portion of the device, providing information to the user via another device.

10. The method of claim 9, further comprising:

the other device including a mobile communication device or a personal computing device operated by the user; and

in response to the user gaze information indicating that the user is gazing at a second portion of the graphical interface, wherein the first portion and the second portion are displayed simultaneously on the graphical interface, activating a second set of gesture inputs recognizable by the device.

11. The method of claim 10, wherein the first portion of the graphical interface is configured to render a first application, and wherein the second portion of the graphical interface is configured to render a second application distinct from the first application.

12. The method of claim 9, wherein the gaze information comprises one or more of a direction of a user line of sight and a user point of focus.

13. The method of claim 10, further comprising:

allowing the user to interact with the device using the first or second sets of recognizable gesture inputs and no other inputs for a predetermined period.

19. The method of claim 10, further comprising:

identifying a user'"'"'s gesture; and

comparing the user'"'"'s gesture to the first or second set of recognizable gesture inputs to determine if the user'"'"'s gesture will be processed by the device.

14. A device, comprising:

a gaze detector adapted to generate gaze information identifying a user line of sight or a user point of focus or an area that the user is not viewing; and

a user-interface controller coupled to or integrated into the device, the user-interface controller adapted to;

in response to the gaze information indicating that the user is gazing at a portion of the device, activate a set of audio inputs recognizable by the device, recognize an audio command uttered by the user, and provide an audio response to the audio command via the device; and

in response to the gaze information indicating that the user is not gazing at any portion of the device, provide information to the user via another device.

15. The device of claim 14, wherein the information is provided in response to the audio command received by the device from the user, and wherein the other device is a mobile communication device or personal computing device operated by the user.