MULTIMODAL AGGREGATING UNIT

• US 20120044183A1
• Filed: 09/23/2011
• Published: 02/23/2012
• Est. Priority Date: 03/07/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Faisman, Alexander, Kanevsky, Dimitri, Nahamoo, David, Sicconi, Roberto, Viswanathan, Mahesh
• Granted Patent: US 8,370,163 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

In a voice processing system, a multimodal request is received from a plurality of modality input devices, and the requested application is run to provide a user with the feedback of the multimodal request. In the voice processing system, a multimodal aggregating unit is provided which receives a multimodal input from a plurality of modality input devices, and provides an aggregated result to an application control based on the interpretation of the interaction ergonomics of the multimodal input within the temporal constraints of the multimodal input. Thus, the multimodal input from the user is recognized within a temporal window. Interpretation of the interaction ergonomics of the multimodal input include interpretation of interaction biometrics and interaction mechani-metrics, wherein the interaction input of at least one modality may be used to bring meaning to at least one other input of another modality.

Claims

1-20. -20. (canceled)

21. A method of processing user input, the method comprising:

receiving a first input associated with a first application, wherein the first input has a first modality;

determining, using at least one processor, whether the first application accepts only input of the first modality;

sending, in response to determining that the first application accepts only input of the first modality, the first input to the first application; and

storing the first input in response to determining that the first application does not accept only input of the first modality.

22. The method of claim 21, further comprising:

in response to receiving the first input, performing, by the at least one first application, at least one action based at least in part on the first input.

23. The method of claim 21, further comprising:

receiving a second input associated with the first application, wherein the second input has a second modality that is different from the first modality; and

aggregating the stored first input and the second input to obtain an aggregated input.

24. The method of claim 23, further comprising:

decoding the aggregated input to provide a decoded result.

25. The method of claim 24, further comprising:

sending the decoded result to the first application; and

in response to receiving the decoded result, performing, by the first application, at least one action based at least in part on the decoded result.

26. The method of claim 23, further comprising:

sending the aggregated input to the first application; and

in response to receiving the aggregated input, performing, by the first application, at least one application based at least in part on the aggregated input.

27. The method of claim 23, wherein aggregating the stored first input and the second input comprises:

determining a first life cycle for the first input;

determining a second life cycle for the second input;

determining whether the first life cycle and the second life cycle overlap in time; and

aggregating, in response to determining that the first life cycle and the second life cycle overlap in time, the first input and the second input.

28. The method of claim 21, wherein the first modality is selected from the group consisting of touch, speech, gestures, eye movements, and face direction.

29. The method of claim 21, wherein the first modality is touch;

and

wherein the first input is a touch gesture selected from the group consisting of at least one stable mark left on a touch-sensitive display by a user and a touch movement across the touch-sensitive display.

30. The method of claim 23, wherein the first modality is a first member selected from the group consisting of touch, speech, gestures, eye movements, and face direction and the second modality is a second member selected from the group consisting of touch, speech, gestures, eye movements, and face direction.

31. The method of claim 21, wherein the first input is received by a system that includes the at least one processor.

32. The method of claim 31, wherein the first input is a produced by the system to provide an alert to a user.

33. The method of claim 32, wherein the first input is selected from the group comprising a tone, one or more visual/audio stable behavioral patterns on a user display, a set of visual/audio patterns changing/appearing/disappearing on the user display in a predetermined fashion, a set of geometric figures appearing on the user display, a system mechanical response that reflects the user'"'"'s interaction with at least one sensor, and a system mechanical response that provides an information response to a user.

34. At least one non-transitory computer readable medium encoded with instructions that, when executed by at least one processor, perform a method comprising:

receiving a first input associated with a first application, wherein the first input has a first modality;

determining whether the first application accepts only input of the first modality;

sending, in response to determining that the first application accepts only input of the first modality, the first input to the first application; and

storing the first input in response to determining that the first application does not accept only input of the first modality.

35. An apparatus comprising:

at least one processor programmed to;

receive a first input associated with a first application, wherein the first input has a first modality;

determine whether the first application accepts only input of the first modality;

send, in response to determining that the first application accepts only input of the first modality, the first input to the first application; and

store the first input in response to determining that the first application does not accept only input of the first modality.