Digital assistant user interface amalgamation

• US 10,831,442 B2
• Filed: 10/19/2018
• Issued: 11/10/2020
• Est. Priority Date: 10/19/2018
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Fox, Jeremy R., Boss, Gregory J., Anders, Kelley, Rakshit, Sarbajit K.
• Primary Examiner(s): Yen, Eric
• Application Number:
  US16/165,777
• Publication Number:
  US 20200125321A1
• Time in Patent Office: 753 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: H04R 1/04 Structural association of microphone with electric circuitry therefor in deaf-aid sets H04R25/00

Abstract

An approach is provided that receives, from a user, an amalgamation at a digital assistant. The amalgamation includes one or more words spoken by the user that are captured by a digital microphone and a set of digital images corresponding to one or more gestures that are performed by the user with the digital images captured by a digital camera. The system then determines an action that is responsive to the amalgamation and then performs the determined action.

Claims

1. A method implemented by an information handling system that includes a processor and a memory accessible by the processor, the method comprising:

receiving, from a user, an amalgamation at a digital assistant, wherein the amalgamation includes a first set of words spoken by the user and captured by a microphone and a set of digital images corresponding to one or more gestures performed by the user captured by a digital camera;

determining an action responsive to the amalgamation;

performing the action by the digital assistant;

determining that the action is incorrect based on a facial expression received from the user and responsively;

receiving, from the user, a set of further amalgamations at the digital assistant;

indicating a set of responsive actions corresponding to the set of further amalgamations;

receiving a set of user feedback to the set of responsive actions; and

selecting one amalgamation from the set of further amalgamations based on the set of user feedback; and

storing the selected amalgamation and a corresponding responsive action from the set of responsive actions in a data store.

2. The method of claim 1 further comprising:

training a machine learning system, wherein the training includes the determined action and a corresponding amalgamation.

3. The method of claim 1 wherein the set of user feedback comprises a subsequent facial expression of the user captured by the digital camera.

4. The method of claim 1 further comprising:

receiving, from the user, a second amalgamation at the digital assistant, wherein the second amalgamation includes a second set of words spoken by the user and captured by the microphone and a second set of digital images corresponding to a second set of one or more gestures performed by the user captured by a digital camera;

identifying that the second amalgamation matches the stored selected amalgamation and responsively receiving the stored corresponding responsive action from the data store; and

performing, by the digital assistant, the stored corresponding responsive action.

5. The method of claim 1 wherein the selected amalgamation and the stored corresponding responsive action are stored in a question-answering (QA) system.

6. The method of claim 1 further comprising:

determining a meaning of at least one of the one or more gestures;

determining a set of ingested words related to the first set of words, wherein the set of ingested words correspond to the determined meaning of the at least one of the one or more gestures; and

identify the determined action based on the set of ingested words and the meaning of the at least one of the one or more gestures.

7. An information handling system comprising:

one or more processors;

a memory coupled to at least one of the one or more processors;

a digital microphone accessible by at least one of the one or more processors;

a digital camera accessible by at least one of the one or more processors; and

a set of computer program instructions stored in the memory and executed by at least one of the one or more processors in order to perform actions comprising;

receiving, from a user, an amalgamation at a digital assistant, wherein the amalgamation includes a first set of words spoken by the user and captured by the digital microphone and a set of digital images corresponding to one or more gestures performed by the user captured by the digital camera;

determining an action responsive to the amalgamation;

performing the action by the digital assistant;

determining that the action is incorrect based on a facial expression received from the user and responsively;

receiving, from the user, a set of further amalgamations at the digital assistant;

indicating a set of responsive actions corresponding to the set of further amalgamations;

receiving a set of user feedback to the set of responsive actions; and

selecting one amalgamation from the set of further amalgamations based on the set of user feedback; and

storing the selected amalgamation and a corresponding responsive action from the set of responsive actions in a data store.

8. The information handling system of claim 7 wherein the actions further comprise:

training a machine learning system, wherein the training includes the determined action and a corresponding amalgamation.

9. The information handling system of claim 7 wherein the set of user feedback comprises a subsequent facial expression of the user captured by the digital camera.

10. The information handling system of claim 7 wherein the actions further comprise:

receiving, from the user, a second amalgamation at the digital assistant, wherein the second amalgamation includes a second set of words spoken by the user and captured by the microphone and a second set of digital images corresponding to a second set of one or more gestures performed by the user captured by a digital camera;

identifying that the second amalgamation matches the stored selected amalgamation and responsively receiving the stored corresponding responsive action from the data store; and

performing, by the digital assistant, the stored corresponding responsive action.

11. The information handling system of claim 7 wherein the selected amalgamation and the stored corresponding responsive action are stored in a question-answering (QA) system.

12. The information handling system of claim 7 wherein the actions further comprise:

determining a meaning of at least one of the one or more gestures;

determining a set of ingested words related to the first set of words, wherein the set of ingested words correspond to the determined meaning of the at least one of the one or more gestures; and

identify the determined action based on the set of ingested words and the meaning of the at least one of the one or more gestures.

13. A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, performs actions comprising:

receiving, from a user, an amalgamation at a digital assistant, wherein the amalgamation includes a first set of words spoken by the user and captured by a microphone and a set of digital images corresponding to one or more gestures performed by the user captured by a digital camera;

determining an action responsive to the amalgamation;

performing the action by the digital assistant;

determining that the action is incorrect based on a facial expression received from the user and responsively;

receiving, from the user, a set of further amalgamations at the digital assistant;

indicating a set of responsive actions corresponding to the set of further amalgamations;

receiving a set of user feedback to the set of responsive actions; and

selecting one amalgamation from the set of further amalgamations based on the set of user feedback; and

storing the selected amalgamation and a corresponding responsive action from the set of responsive actions in a data store.

14. The computer program product of claim 13 wherein the actions further comprise:

training a machine learning system, wherein the training includes the determined action and a corresponding amalgamation.

15. The computer program product of claim 13 wherein the set of user feedback comprises a subsequent facial expression of the user captured by the digital camera.

16. The computer program product of claim 13 wherein the actions further comprise:

receiving, from the user, a second amalgamation at the digital assistant, wherein the second amalgamation includes a second set of words spoken by the user and captured by the microphone and a second set of digital images corresponding to a second set of one or more gestures performed by the user captured by a digital camera;

identifying that the second amalgamation matches the stored selected amalgamation and responsively receiving the stored corresponding responsive action from the data store; and

performing, by the digital assistant, the stored corresponding responsive action.

17. The computer program product of claim 13 wherein the selected amalgamation and the stored corresponding responsive action are stored in a question-answering (QA) system.