LEARNING INTENDED USER ACTIONS

• US 20160239258A1
• Filed: 02/16/2015
• Published: 08/18/2016
• Est. Priority Date: 02/16/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Lenchner, Jonathan, Venkataraman, Vinay
• Granted Patent: US 10,048,934 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

A method and system are provided. The method includes receiving, by a microphone and camera, user utterances indicative of user commands and associated user gestures for the user utterances. The method further includes parsing, by a hardware-based recognizer, sample utterances and the user utterances into verb parts and noun parts. The method also includes recognizing, by a hardware-based recognizer, the user utterances and the associated user gestures based on the sample utterances and descriptions of associated supporting gestures for the sample utterances. The recognizing step includes comparing the verb parts and the noun parts from the user utterances individually and as pairs to the verb parts and the noun parts of the sample utterances. The method additionally includes selectively performing a given one of the user commands responsive to a recognition result.

Claims

1-14. -14. (canceled)

15. A computer program product for recognizing intended user actions, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

receiving, by a microphone and camera, user utterances indicative of user commands and associated user gestures for the user utterances;

parsing, by a hardware-based recognizer, sample utterances and the user utterances into verb parts and noun parts;

recognizing, by a hardware-based recognizer, the user utterances and the associated user gestures based on the sample utterances and descriptions of associated supporting gestures for the sample utterances, wherein said recognizing step comprises comparing the verb parts and the noun parts from the user utterances individually and as pairs to the verb parts and the noun parts of the sample utterances; and

selectively performing a given one of the user commands responsive to a recognition result.

16. A system, comprising

a microphone and camera for receiving user utterances indicative of user commands and associated user gestures for the user utterances;

a hardware-based recognizer for parsing sample utterances and the user utterances into verb parts and noun parts, and recognizing the user utterances and the associated user gestures based on the sample utterances and descriptions of associated supporting gestures for the sample utterance by comparing the verb parts and the noun parts from the user utterances individually and as pairs to the verb parts and the noun parts of the sample utterances; and

a user command selective execution device for selectively performing a given one of the user commands responsive to a recognition result.

17. The system of claim 16, wherein the user speech and associated gesture recognizer forms triples of a verb, a noun, and a gesture, and compares the verb and the noun to the gesture as a pair and individually.

18. The method of claim 16, further comprising a user command learning device for learning from multiple recognition sessions by acquiring user accepted examples and user rejected examples of the user utterances and the associated user gestures, and wherein the user command selective execution device selectively performs a given one of the user commands responsive to the user accepted examples and the user rejected examples.

19. The system of claim 18, further comprising a confidence indication generator for generating respective confidence values for at least one of the noun, the verb, the gesture, and a combination thereof including at least the gesture, responsive to at least one of a number of positive examples and a number of negative examples involving the gesture and at least one of the noun and the verb for a particular one of the user commands.

20. The system of claim 18, further comprising an error indication generator for generating respective error values for at least one of the noun, the verb, the gesture, and a combination thereof including at least the gesture, responsive to at least one of a number of user accepted examples and a number of user rejected examples involving the gesture and at least one of the noun and the verb for a particular one of the user commands.