Natural human-computer interaction for virtual personal assistant systems

• US 10,198,069 B2
• Filed: 03/07/2017
• Issued: 02/05/2019
• Est. Priority Date: 05/20/2013
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Deleeuw, William C.
• Primary Examiner(s): Sarma, Abhishek
• Application Number:
  US15/452,131
• Publication Number:
  US 20170177080A1
• Time in Patent Office: 700 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 21/003 Changing voice quality, e.g. pitch or formants

Abstract

Technologies for natural language interactions with virtual personal assistant systems include a computing device configured to capture audio input, distort the audio input to produce a number of distorted audio variations, and perform speech recognition on the audio input and the distorted audio variants. The computing device selects a result from a large number of potential speech recognition results based on contextual information. The computing device may measure a user'"'"'s engagement level by using an eye tracking sensor to determine whether the user is visually focused on an avatar rendered by the virtual personal assistant. The avatar may be rendered in a disengaged state, a ready state, or an engaged state based on the user engagement level. The avatar may be rendered as semitransparent in the disengaged state, and the transparency may be reduced in the ready state or the engaged state. Other embodiments are described and claimed.

Claims

1. A computing device comprising:

a display;

a virtual personal assistant to display an avatar of the virtual personal assistant on the display in a disengaged state, a ready state, and an engaged state; and

an engagement module to;

determine whether a user of the computing device has requested activation of the virtual personal assistant while the avatar is displayed in the disengaged state;

determine an engagement level of the user while the avatar is displayed in the ready state; and

monitor the engagement level of the user while the avatar is displayed in the engaged state;

wherein the virtual personal assistant is to;

display the avatar in the ready state in response to a determination that the user has requested activation of the virtual personal assistant;

display the avatar in the engaged state in response to a determination that the user has an engagement level greater than a threshold level; and

display the avatar in the disengaged state in response to a determination that the user has an engagement level less than the threshold level.

2. The computing device of claim 1, further comprising an eye tracking sensor, wherein to determine whether the user has requested activation of the virtual personal assistant comprises to:

receive eye tracking data from the eye tracking sensor; and

determine whether the user has focused on the avatar based on the eye tracking data.

3. The computing device of claim 1, further comprising an eye tracking sensor, wherein to determine the engagement level of the user comprises to:

receive eye tracking data from the eye tracking sensor;

determine, based on the eye tracking data, whether the user has visually focused on the avatar for a period of time greater than a threshold time;

determine that the user has an engagement level greater than the threshold level in response to a determination that the user has visually focused on the avatar for a period of time greater than the threshold time; and

determine that the user has an engagement level less than the threshold level in response to a determination that the user has not visually focused on the avatar for a period of time greater than the threshold time.

4. The computing device of claim 1, wherein to display the avatar in the ready state and to display the avatar in the engaged state comprise to display an anthropomorphic representation of eye contact of the avatar with the user.

5. The computing device of claim 4, further comprising an eye tracking sensor, wherein to display the anthropomorphic representation of eye contact comprises to:

receive at least one of eye tracking data or head position data from the eye tracking sensor; and

display an anthropomorphic representation of eyes of the avatar following the user.

6. The computing device of claim 1, wherein:

to display the avatar of the virtual personal assistant in the disengaged state comprises to display the avatar as semitransparent, to allow a user interface element of the computing device to shine through the avatar;

to display the avatar in the ready state comprises to decrease a transparency of the avatar; and

to display the avatar in the engaged state comprises one of to;

decrease the transparency of the avatar or eliminate the transparency of the avatar.

7. The computing device of claim 1, further comprising an eye tracking sensor, wherein:

to display the avatar in the ready state and to display the avatar in the engaged state comprise to move the avatar on the display to a position closer to a position where the user is focused, based on eye tracking data received from the eye tracking sensor.

8. A method comprising:

displaying, by a computing device, an avatar of a virtual personal assistant on a display of the computing device in a disengaged state, a ready state, and an engaged state;

determining, by the computing device, whether a user of the computing device has requested activation of the virtual personal assistant while the avatar is displayed in the disengaged state;

determining, by the computing device, an engagement level of the user while the avatar is displayed in the ready state;

monitoring, by the computing device, the engagement level of the user while the avatar is displayed in the engaged state;

displaying, by the computing device, the avatar in the ready state in response to a determination that the user has requested activation of the virtual personal assistant;

displaying, by the computing device, the avatar in the engaged state in response to a determination that the user has an engagement level greater than a threshold level; and

displaying, by the computing device, the avatar in the disengaged state in response to a determination that the user has an engagement level less than the threshold level.

9. The method of claim 8, wherein determining whether the user has requested activation of the virtual personal assistant comprises:

receiving eye tracking data from an eye tracking sensor of the computing device; and

determining whether the user has focused on the avatar based on the eye tracking data.

10. The method of claim 8, wherein determining the engagement level of the user comprises:

receiving eye tracking data from an eye tracking sensor of the computing device;

determining, based on the eye tracking data, whether the user has visually focused on the avatar for a period of time greater than a threshold time;

determining that the user has an engagement level greater than the threshold level in response to a determination that the user has visually focused on the avatar for a period of time greater than the threshold time; and

determining that the user has an engagement level less than the threshold level in response to a determination that the user has not visually focused on the avatar for a period of time greater than the threshold time.

11. The method of claim 8, wherein displaying the avatar in the ready state and displaying the avatar in the engaged state comprise displaying an anthropomorphic representation of eye contact of the avatar with the user.

12. The method of claim 11, wherein displaying the anthropomorphic representation of eye contact comprises:

receiving at least one of eye tracking data or head position data from an eye tracking sensor of the computing device; and

displaying an anthropomorphic representation of eyes of the avatar following the user.

13. The method of claim 8, wherein:

displaying the avatar of the virtual personal assistant in the disengaged state comprises displaying the avatar as semitransparent, to allow a user interface element of the computing device to shine through the avatar;

displaying the avatar in the ready state comprises decreasing a transparency of the avatar; and

displaying the avatar in the engaged state comprises one of decreasing the transparency of the avatar or eliminating the transparency of the avatar.

14. The method of claim 8, wherein:

displaying the avatar in the ready state and displaying the avatar in the engaged state comprise moving the avatar on the display to a position closer to a position where the user is focused, based on eye tracking data received from an eye tracking sensor of the computing device.

15. One or more non-transitory computer readable media comprising a plurality of instructions stored thereon that, when executed by a computing device, cause the computing device to:

display an avatar of a virtual personal assistant on a display of the computing device in a disengaged state, a ready state, and an engaged state;

determine whether a user of the computing device has requested activation of the virtual personal assistant while the avatar is displayed in the disengaged state;

determine an engagement level of the user while the avatar is displayed in the ready state;

monitor the engagement level of the user while the avatar is displayed in the engaged state;

display the avatar in the ready state in response to a determination that the user has requested activation of the virtual personal assistant;

display the avatar in the engaged state in response to a determination that the user has an engagement level greater than a threshold level; and

display the avatar in the disengaged state in response to a determination that the user has an engagement level less than the threshold level.

16. The one or more non-transitory computer readable media of claim 15, wherein to determine whether the user has requested activation of the virtual personal assistant comprises to:

receive eye tracking data from an eye tracking sensor of the computing device; and

determine whether the user has focused on the avatar based on the eye tracking data.

17. The one or more non-transitory computer readable media of claim 15, wherein to determine the engagement level of the user comprises to:

receive eye tracking data from an eye tracking sensor of the computing device;

determine, based on the eye tracking data, whether the user has visually focused on the avatar for a period of time greater than a threshold time;

determine that the user has an engagement level greater than the threshold level in response to a determination that the user has visually focused on the avatar for a period of time greater than the threshold time; and

determine that the user has an engagement level less than the threshold level in response to a determination that the user has not visually focused on the avatar for a period of time greater than the threshold time.

18. The one or more non-transitory computer readable media of claim 15, wherein to display the avatar in the ready state and to display the avatar in the engaged state comprise to display an anthropomorphic representation of eye contact of the avatar with the user.

19. The one or more non-transitory computer readable media of claim 18, wherein to display the anthropomorphic representation of eye contact comprises to:

receive at least one of eye tracking data or head position data from an eye tracking sensor of the computing device; and

display an anthropomorphic representation of eyes of the avatar following the user.

20. The one or more non-transitory computer readable media of claim 15, wherein:

to display the avatar of the virtual personal assistant in the disengaged state comprises to display the avatar as semitransparent, to allow a user interface element of the computing device to shine through the avatar;

to display the avatar in the ready state comprises to decrease a transparency of the avatar; and

to display the avatar in the engaged state comprises one of to decrease the transparency of the avatar or to eliminate the transparency of the avatar.

21. The one or more non-transitory computer readable media of claim 15, wherein:

to display the avatar in the ready state and to display the avatar in the engaged state comprise to move the avatar on the display to a position closer to a position where the user is focused, based on eye tracking data received from an eye tracking sensor of the computing device.