Device control using gaze information

• US 10,860,096 B2
• Filed: 08/28/2019
• Issued: 12/08/2020
• Est. Priority Date: 09/28/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kelly, Sean B., Bacim De Araujo E Silva, Felipe, Bark, Karlin Y.
• Primary Examiner(s): Yang, Nan-Ying
• Application Number:
  US16/553,622
• Publication Number:
  US 20200103963A1
• Time in Patent Office: 468 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 1/72415 for remote control of appliances

Abstract

The present disclosure generally relates to controlling electronic devices. In some examples, the electronic device uses gaze information to activate a digital assistant. In some examples, the electronic device uses gaze information to identify an external device on which to act. In some examples, the electronic device provides an indication that distinguishes between different speakers.

Claims

1. An electronic device, comprising:

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

while a digital assistant of the electronic device is not activated;

receiving a first portion of an audio user input request to perform a command;

obtaining, using one or more camera sensors, first gaze information;

independent of audio input;

in accordance with a determination that the first gaze information satisfies a set of one or more activation criteria;

activating the digital assistant of the electronic device; and

providing an indication that the set of one or more activation criteria has been satisfied;

while the digital assistant of the electronic device is activated, receiving a second portion of the audio user input request to perform the command; and

in response to receiving the audio user input request to perform the command;

in accordance with a determination that a set of one or more performance criteria is met, performing the command based on a content of the first portion and the second portion of the audio user input request.

2. The electronic device of claim 1, the one or more programs further including instructions for:

independent of audio input;

in accordance with a determination that the first gaze information does not satisfy the set of one or more activation criteria;

forgo activating the digital assistant.

3. The electronic device of claim 1, the one or more programs further including instructions for:

producing an audio output, wherein the audio output is being produced at a first volume prior to receiving the audio user input request; and

in response to beginning to receive the audio user input request, reducing a volume of the audio output to a second volume that is lower than the first volume, wherein the second volume is based on a distance between a user and the electronic device.

4. The electronic device of claim 1, the one or more programs further including instructions for:

in response to receiving the audio user input request to perform the command;

in accordance with a determination that the set of one or more performance criteria is not met, forgo performing the command based on the content of the audio user input request, wherein the set of one or more performance criteria includes a continuous gaze criterion that is met when the electronic device determines that second gaze information indicates that the user does not break a gaze directed at the electronic device for more than a threshold duration between activation of the digital assistant and termination of the audio user input request.

5. The electronic device of claim 1, the one or more programs further including instructions for:

in response to receiving the audio user input request to perform the command;

in accordance with a determination that the set of one or more performance criteria is not met, providing a headshake indication, wherein the headshake indication includes;

displaying a first pattern of lights on a first side of a display of the electronic device,

subsequent to displaying the first pattern of lights, displaying a second pattern of lights on a second side of the display of the electronic device, and

subsequent to displaying the second pattern of lights, displaying the first pattern of lights on the first side of the display of the electronic device.

6. The electronic device of claim 1, wherein the indication is a visual indication that indicates that the digital assistant is activated.

7. The electronic device of claim 1, wherein the indication is an audio indication that indicates that the digital assistant is activated.

8. The electronic device of claim 1, wherein the set of one or more activation criteria includes a field of view criterion that is met when a determined offset angle is less than a threshold offset angle, wherein the determined offset angle is an angle formed between:

a line-of-sight of a gaze of a user, and

a calculated line-of-sight from the user to the electronic device.

9. The electronic device of claim 8, wherein the line-of-sight of the gaze of the user is based on a head position of the user.

10. The electronic device of claim 1, wherein the set of one or more activation criteria includes a dwell time criterion that is satisfied when a dwell time of the gaze determined from the first gaze information is more than a threshold period of time.

11. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for:

while a digital assistant of the electronic device is not activated;

receiving a first portion of an audio user input request to perform a command;

obtaining, using one or more camera sensors, first gaze information;

independent of audio input;

in accordance with a determination that the first gaze information satisfies a set of one or more activation criteria;

activating the digital assistant of the electronic device; and

providing an indication that the set of one or more activation criteria has been satisfied;

while the digital assistant of the electronic device is activated, receiving a second portion of the audio user input request to perform the command; and

in response to receiving the audio user input request to perform the command;

in accordance with a determination that a set of one or more performance criteria is met, performing the command based on a content of the first portion and the second portion of the audio user input request.

13. The non-transitory computer-readable storage medium of claim 11, the one or more programs further including instructions for:

independent of audio input;

in accordance with a determination that the first gaze information does not satisfy the set of one or more activation criteria;

forgo activating the digital assistant.

14. The non-transitory computer-readable storage medium of claim 11 the one or more programs further including instructions for:

producing an audio output, wherein the audio output is being produced at a first volume prior to receiving the audio user input request; and

in response to beginning to receive the audio user input request, reducing a volume of the audio output to a second volume that is lower than the first volume, wherein the second volume is based on a distance between a user and the electronic device.

15. The non-transitory computer-readable storage medium of claim 11, the one or more programs further including instructions for:

in response to receiving the audio user input request to perform the command;

in accordance with a determination that the set of one or more performance criteria is not met, forgo performing the command based on the content of the audio user input request, wherein the set of one or more performance criteria includes a continuous gaze criterion that is met when the electronic device determines that second gaze information indicates that the user does not break a gaze directed at the electronic device for more than a threshold duration between activation of the digital assistant and termination of the audio user input request.

16. The non-transitory computer-readable storage medium of claim 11, the one or more programs further including instructions for:

in response to receiving the audio user input request to perform the command;

in accordance with a determination that the set of one or more performance criteria is not met, providing a headshake indication, wherein the headshake indication includes;

displaying a first pattern of lights on a first side of a display of the electronic device,

subsequent to displaying the first pattern of lights, displaying a second pattern of lights on a second side of the display of the electronic device, and

subsequent to displaying the second pattern of lights, displaying the first pattern of lights on the first side of the display of the electronic device.

17. The non-transitory computer-readable storage medium of claim 11, wherein the indication is a visual indication that indicates that the digital assistant is activated.

18. The non-transitory computer-readable storage medium of claim 11, wherein the indication is an audio indication that indicates that the digital assistant is activated.

19. The non-transitory computer-readable storage medium of claim 11, wherein the set of one or more activation criteria includes a field of view criterion that is met when a determined offset angle is less than a threshold offset angle, wherein the determined offset angle is an angle formed between:

a line-of-sight of a gaze of a user, and

a calculated line-of-sight from the user to the electronic device.

20. The non-transitory computer-readable storage medium of claim 19, wherein the line-of-sight of the gaze of the user is based on a head position of the user.

21. The non-transitory computer-readable storage medium of claim 11, wherein the set of one or more activation criteria includes a dwell time criterion that is satisfied when a dwell time of the gaze determined from the first gaze information is more than a threshold period of time.

12. A method, comprising:

at an electronic device;

while a digital assistant of the electronic device is not activated;

receiving a first portion of an audio user input request to perform a command;

obtaining, using one or more camera sensors, first gaze information;

independent of audio input;

in accordance with a determination that the first gaze information satisfies a set of one or more activation criteria;

activating the digital assistant of the electronic device; and

providing an indication that the set of one or more activation criteria has been satisfied;

while the digital assistant of the electronic device is activated, receiving a second portion of the audio user input request to perform the command; and

in response to receiving the audio user input request to perform the command;

in accordance with a determination that a set of one or more performance criteria is met, performing the command based on a content of the first portion and the second portion of the audio user input request.

22. The method of claim 12, further comprising:

independent of audio input;

in accordance with a determination that the first gaze information does not satisfy the set of one or more activation criteria;

forgo activating the digital assistant.

23. The method of claim 12, further comprising:

producing an audio output, wherein the audio output is being produced at a first volume prior to receiving the audio user input request; and

in response to beginning to receive the audio user input request, reducing a volume of the audio output to a second volume that is lower than the first volume, wherein the second volume is based on a distance between a user and the electronic device.

24. The method of claim 12, further comprising:

in response to receiving the audio user input request to perform the command;

in accordance with a determination that the set of one or more performance criteria is not met, forgo performing the command based on the content of the audio user input request, wherein the set of one or more performance criteria includes a continuous gaze criterion that is met when the electronic device determines that second gaze information indicates that the user does not break a gaze directed at the electronic device for more than a threshold duration between activation of the digital assistant and termination of the audio user input request.

25. The method of claim 12, further comprising:

in response to receiving the audio user input request to perform the command;

in accordance with a determination that the set of one or more performance criteria is not met, providing a headshake indication, wherein the headshake indication includes;

displaying a first pattern of lights on a first side of a display of the electronic device,

subsequent to displaying the first pattern of lights, displaying a second pattern of lights on a second side of the display of the electronic device, and

subsequent to displaying the second pattern of lights, displaying the first pattern of lights on the first side of the display of the electronic device.

26. The method of claim 12, wherein the indication is a visual indication that indicates that the digital assistant is activated.

27. The method of claim 12, wherein the indication is an audio indication that indicates that the digital assistant is activated.

28. The method of claim 12, wherein the set of one or more activation criteria includes a field of view criterion that is met when a determined offset angle is less than a threshold offset angle, wherein the determined offset angle is an angle formed between:

a line-of-sight of a gaze of a user, and

a calculated line-of-sight from the user to the electronic device.

29. The method of claim 28, wherein the line-of-sight of the gaze of the user is based on a head position of the user.

30. The method of claim 12, wherein the set of one or more activation criteria includes a dwell time criterion that is satisfied when a dwell time of the gaze determined from the first gaze information is more than a threshold period of time.