INITIATING PERSONAL ASSISTANT APPLICATION BASED ON EYE TRACKING AND GESTURES

• US 20150154001A1
• Filed: 12/03/2013
• Published: 06/04/2015
• Est. Priority Date: 12/03/2013
• Status: Active Grant

• Current Assignee: Lenovo PC International Limited (Lenovo Group Ltd.)
• Original Assignee: Lenovo Singapore Pte Limited (Lenovo Group Ltd.)
• Inventors: Knox, Jonathan Gaither, Waltermann, Rod D., Chen, Liang, Cohen, Mark Evan
• Granted Patent: US 9,110,635 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A device includes a processor and a memory accessible to the processor. The memory bears instructions executable by the processor to receive at least one signal from at least one camera in communication with the device, determine that a user of the device is looking at least substantially at a personal assistant feature presented on a display of the device at least partially based on the signal, and initiate a personal assistant which processes audible input from the user received at the device in response to a determination that the user is looking at least substantially at the personal assistant feature. The personal assistant is associated with the personal assistant feature.

Claims

1. A device, comprising:

a processor;

a memory accessible to the processor and bearing instructions executable by the processor to;

receive at least one signal from at least one camera in communication with the device;

at least partially based on the signal, determine that a user is looking at least substantially at a display;

in response to a determination that the user looks at least substantially at the display, determine whether the user looks at the display for a period satisfying a threshold;

in response to a determination that the user looks at the display for a period satisfying the threshold, execute at least a first computer process; and

in response to a determination that the user looks at the display for a period not satisfying the threshold, not execute the first computer process.

2. The device of claim 1, wherein the instructions are further executable by the processor to receive audible input from the user.

3. The device of claim 2, wherein the instructions are further executable by the processor to execute a function on the device which is executed in response to receipt of the audible input and in response to processing a command derived from the audible input, the command being derived from the audible input at least in part using an assistant that is at least part of the first computer process.

5. The device of claim 1, wherein the determination that the user is looking at least substantially at the display includes a determination that the user is looking at a feature presented on the display.

6. The device of claim 1, wherein the determination that the user is looking at least substantially at the display includes a determination that the user is looking at a feature presented on the display, and wherein the first computer process is initiated in response to at least both of the determination that the user is looking at the feature and a determination that the user'"'"'s current facial expression is indicative of being at least about to provide audible input.

10. The device of claim 1, wherein the first computer process is executed responsive to a determination that the user gestures a predefined gesture defined by a person prior to the looking at the display.

21. The device of claim 1, wherein the first computer process is executed responsive to a determination that the user gestures a predefined gesture which includes pointing at the device.

4. (canceled)

7-9. -9. (canceled)

11. A method, comprising:

receiving at least a first signal from at least one camera in communication with a device;

at least partially based on the first signal, determining whether a user is looking at least substantially at the device for at least a threshold amount of time;

in response to determining that the user is looking at least substantially at the device for at least the threshold amount of time, initiating a personal assistant application, wherein the personal assistant application executes a command in response to processing audible input from the user; and

in response to determining that the user is looking at least substantially at the device but not for the threshold amount of time, not initiating the personal assistant application.

13. The method of claim 11, wherein determining that the user is looking at least substantially at the device for at least the threshold amount of time includes determining that the user is looking at a personal assistant icon presented on a display of the device for at least the threshold amount of time, wherein the personal assistant icon is selectable to initiate the personal assistant.

14. The method of claim 11, further comprising:

controlling the personal assistant application to begin executing a command in response to audible input received at the device;

determining that the user has stopped speaking at least partially based on a second signal from the camera;

responsive to determining that the user has stopped speaking, ceasing to execute the command;

determining that the user has resumed speaking based at least partially based on a third signal from the camera; and

responsive to determining that the user has resumed speaking, continuing to execute the command.

15. The method of claim 11, further comprising:

controlling the personal assistant application to begin executing a command in response to audible input received at the device;

determining that the user is no longer looking at least substantially at the device;

receiving additional audible input at the device; and

in response to determining that the user is no longer looking at least substantially at the device and in response to receiving the audible input, declining to process the additional audible input as part of the command.

16. The method of claim 11, further comprising:

at least partially based on the first signal, determining that the user is speaking, wherein the determining that the user is speaking is executed by the device responsive to executing facial expression software on a facial expression of the user from an image gathered by the camera.

22. The method of claim 11, wherein the determining that a user is looking at least substantially at the device for at least the threshold amount of time comprises determining that the user is looking at least substantially at a personal assistant icon presented on the device that is associated with the personal assistant application, and wherein the determining that the user is looking at least substantially at the personal assistant icon is at least in part based on identifying at the device a vector established by the user'"'"'s actual line of sight toward the device as being within a threshold number of degrees of looking directly at at least a portion of the personal assistant icon.

12. (canceled)

17. (canceled)

18. An apparatus, comprising:

a first processor;

a network adapter;

storage bearing instructions executable by a second processor for;

processing an audible input command at a device associated with the second processor, the processing of the audible input command being responsive to a determination based on at least one signal from at least one camera in communication with the device that a user is gesturing a first predefined gesture; and

presenting a user interface (UI) on the device associated with the second processor, the UI indicating plural predefined gestures;

receiving a selection of at least one of the predefined gestures; and

establishing the first predefined gesture at least in part based on the selection;

wherein the first processor transfers the instructions over a network via the network adapter.

23. The apparatus of claim 18, wherein the instructions are executable by the processor to initiate processing the audible input command further responsive to determining that the user is gesturing a predefined gesture and responsive to determining that the predefined gesture is gestured by the user from within a threshold distance of the device, wherein the threshold distance is less than a maximum distance from the device at which gestures are identifiable by the device, wherein the threshold distance is defined by a user, and wherein the determining that the predefined gesture is gestured by the user from within the threshold distance of the device is based at least in part on input from a sensor which senses body heat.

24. The apparatus of claim 18, wherein the instructions are executable by the processor to initiate processing the audible input command further responsive to determining that the user is gesturing a predefined gesture comprising a head nod, the head nod comprising movement of the user'"'"'s head.

25. The apparatus of claim 18, wherein the instructions are executable by the processor to initiate processing the audible input command further responsive to determining that the user is gesturing a predefined gesture, wherein the predefined gesture is established at least in part based on first input from a person, the first input detected by the device prior to the processing of the audible input command, the first input comprising the person performing a gesture.

26. The apparatus of claim 18, wherein the instructions are executable for receiving the selection of the at least one of the predefined gestures based at least in part on receipt of input from the at least one camera used to determine that the user is looking at least substantially at one of the indications, and based at least in part on receipt of input from a keyboard corresponding to selection of at least one key from the keyboard at at least one time during which the user is looking at least substantially at the one of the indications.

27. The apparatus of claim 18, wherein the UI includes a selector element selectable to initiate a computer process in which the device at least attempts to detect performance of a particular gesture which the device is to establish as being the first predefined gesture.

28. The apparatus of claim 27, wherein upon initiation of the computer process, the device detects user input of the particular gesture and establishes the particular gesture as being the first predefined gesture.

19-20. -20. (canceled)