Multimodal Input System

• US 20120105257A1
• Filed: 11/01/2010
• Published: 05/03/2012
• Est. Priority Date: 11/01/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Murillo, Oscar E., Cluts, Jonathan C., Creasey, Jack, Bhattacharyay, Subha, Galore, Janet E., Estrada, Colleen G., Koenig, Michael
• Granted Patent: US 9,348,417 B2
• Time in Patent Office: Days
• US Class Current: 341/20
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

The subject disclosure relates to user input into a computer system, and a technology by which one or more users interact with a computer system via a combination of input modalities. When the input data of two or more input modalities are related, they are combined to interpret an intended meaning of the input. For example, speech when combined with one input gesture has one intended meaning, e.g., convert the speech to verbatim text for consumption by a program, while the exact speech when combined with a different input gesture has a different meaning, e.g., convert the speech to a command that controls the operation of that same program.

Claims

1. In a computing environment, a method performed at least in part on at least one processor, comprising, receiving sets of input data corresponding to a plurality of input modalities, determining that a first set of input data is related to a second set of input data, and using the second set of input data to interpret an intent of the first set of input data.

2. The method of claim 1 wherein receiving the sets of input data comprises receiving the first set from a user via one input device and the second set from the user via another input device.

3. The method of claim 1 wherein receiving the sets of input data comprises receiving the first set from one user and the second set from another user.

4. The method of claim 1 wherein using the second set of input data to interpret intent of the first set of input data comprises accessing a combined input dictionary.

5. The method claim 1 wherein receiving the sets of input data comprises receiving at least one set as input that is synthesized from output of another device.

6. The method of claim 1 wherein determining that a first set of input data is related to a second set of input data comprises accessing preference data.

7. The method of claim 1 wherein determining that a first set of input data is related to a second set of input data comprises accessing data provided by at least one sensor.

8. In a computing environment, a system, comprising:

a plurality of input devices that each input raw data corresponding to an input modality;

an input processor configured to process the raw data into input events; and

a recommendation engine configured to evaluate the input events and to determine a user-intended meaning of a combination of at least two of the input events.

9. The system of claim 8 wherein one of the input devices inputs raw data from one user and another of the input devices inputs raw data from another user.

10. The system of claim 8 wherein at least one of the input devices is coupled to the system via a remote connection.

11. The system of claim 8 wherein the input devices include a camera, a microphone, a digitizer, a pen, or a pointing device, or any combination of a camera, a microphone, a digitizer, a pen, or a pointing device.

12. The system of claim 8 wherein the input processor includes an audio processing component, a visual processing component, a touch processing component, a gesture processing component, a pointer processing component or a text processing component, or any combination of an audio processing component, a visual processing component, a touch processing component, a gesture processing component, a pointer processing component or a text processing component.

13. The system of claim 8 further comprising a personalization engine configured to communicate with the recommendation engine to provide personalization information that the recommendation engine uses in determining the user-intended meaning.

14. The system of claim 13 wherein the personalization engine accesses one or more dictionaries to provide the personalization information.

15. The system of claim 8 further comprising a combined dictionary corresponding to two or more input modalities.

16. The system of claim 8 further comprising at least one sensor that provides sensed data, the recommendation engine configured to use the sensed data in determining the user-intended meaning.

17. The system of claim 8 wherein the recommendation engine is configured to use preference data in determining the user-intended meaning.

20. The one or more computer-readable media of claim 17 having further computer-executable instructions comprising, changing the input data of one device based upon the combined meaning into modified data, and wherein providing the output data comprises providing the modified data.

18. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:

receiving input data from a plurality of input devices;

accessing at least one dictionary to determine a combined meaning of the input data; and

providing output data for input by a program, the output data corresponding to the combined meaning of the input data.

19. The one or more computer-readable media of claim 18 wherein providing the output data comprises providing a recommendation.