Aggregating multimodal inputs based on overlapping temporal life cycles

• US 8,370,162 B2
• Filed: 09/23/2011
• Issued: 02/05/2013
• Est. Priority Date: 03/07/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Faisman, Alexander, Kanevsky, Dimitri, Nahamoo, David, Sicconi, Roberto, Viswanathan, Mahesh
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US13/242,538
• Publication Number:
  US 20120046945A1
• Time in Patent Office: 501 Days
• Field of Search: None
• US Class Current: 704/275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

In a voice processing system, a multimodal request is received from a plurality of modality input devices, and the requested application is run to provide a user with the feedback of the multimodal request. In the voice processing system, a multimodal aggregating unit is provided which receives a multimodal input from a plurality of modality input devices, and provides an aggregated result to an application control based on the interpretation of the interaction ergonomics of the multimodal input within the temporal constraints of the multimodal input. Thus, the multimodal input from the user is recognized within a temporal window. Interpretation of the interaction ergonomics of the multimodal input include interpretation of interaction biometrics and interaction mechani-metrics, wherein the interaction input of at least one modality may be used to bring meaning to at least one other input of another modality.

Claims

1. A method of processing user input, the method comprising:

receiving a first input having a first modality;

determining, with at least one processor, a first temporal window for the first input based, at least in part, on first data stored on at least one storage device, wherein the first data indicates a duration of the first temporal window associated with inputs received via the first modality;

receiving a second input from the user having a second modality that is different than the first modality;

determining whether the second input is received during the first temporal window;

aggregating based, at least in part on whether the second input is received during the first temporal window, the first input and the second input to obtain an aggregated input;

determining whether any portion of the second input is received after the first temporal window has expired; and

wherein aggregating the first input and the second input comprises aggregating the first input and only a portion of the second input received during the first temporal window.

2. The method of claim 1, wherein aggregating the first input and only a portion of the second input comprises aggregating the first input and the entire second input, in response to determining that no portion of the second input is received after the first temporal window has expired.

3. The method of claim 1, further comprising:

determining whether the first input makes sense; and

aggregating the first input and the portion of the second input received during the first temporal window only when it is determined that the first input makes sense.

4. The method of claim 3, wherein determining whether the first input makes sense comprises determining whether an application associated with the first input accepts input according to the first modality.

5. The method of claim 3, wherein the first modality is speech;

and

wherein determining whether the first input makes sense comprises determining whether the speech input matches an active grammar.

6. The method of claim 3, wherein the first modality is touch;

and

wherein determining whether the first input makes sense comprises determining whether a location of the touch input corresponds to at least one field or button on a touch-sensitive display.

7. The method of claim 3, further comprising:

determining whether the second input makes sense; and

aggregating the first input and the portion of the second input received during the first temporal window only when the second input makes sense.

8. The method of claim 7, wherein determining whether the second input makes sense comprises determining whether the second input provides meaning to the first input.

9. The method of claim 7, wherein determining whether the second input makes sense comprises determining whether an application associated with the second input accepts input according to the second modality.

10. The method of claim 1,

wherein aggregating the first input and the portion of the second input comprises aggregating, in response to determining that at least some of the second input was received after the first temporal window expired, only the portion of the second input received during the first temporal window, wherein the portion of the second input received during the first temporal window is less than the entire second input.

11. The method of claim 1, further comprising:

determining a context based, at least in part, on the received first input; and

interpreting the received second input based, at least in part, on the context.

12. The method of claim 11, wherein the first modality is touch;

and

wherein the context relates to a particular field or button corresponding to a location of the touch input on a touch-sensitive display.

13. The method of claim 12, further comprising:

determining whether the second modality is an acceptable modality for the particular field or button; and

aggregating the first input and the portion of the second input received during the first temporal window only when the second modality is an acceptable modality for the particular field or button.

14. The method of claim 12, wherein the second modality is speech;

and

wherein determining whether the second modality is an acceptable modality comprises determining whether the particular field or button selected by the touch input is speech-enabled.

15. The method of claim 11, further comprising:

decoding the aggregated input to provide a decoded result; and

sending the decoded result to an application control.

16. At least one non-transitory computer readable medium encoded with instructions that, when executed by at least one processor, perform a method comprising:

receiving a first input having a first modality;

determining a first temporal window for the first input based, at least in part, on first data stored on at least one storage device, wherein the first data indicates a duration of the first temporal window associated with inputs received via the first modality;

receiving a second input from the user having a second modality that is different than the first modality;

determining whether the second input is received during the first temporal window;

aggregating based, at least in part, on whether the second input is received during the first temporal window, the first input and the second input to obtain an aggregated input;

determining whether any portion of the second input is received after the first temporal window has expired; and

wherein aggregating the first input and the second input comprises aggregating the first input and only a portion of the second input received during the first temporal window.

17. An apparatus comprising:

at least one processor programmed to;

receive a first input having a first modality;

determine a first temporal window for the first input based, at least in part, on first data stored on at least one storage device, wherein the first data indicates a duration of the first temporal window associated with inputs received via the first modality;

receive a second input from the user having a second modality that is different than the first modality;

determine whether the second input is received during the first temporal window;

aggregate based, at least in part, on whether the second input is received during the first temporal window, the first input and the second input to obtain an aggregated input;

determine whether any portion of the second input is received after the first temporal window has expired; and

wherein aggregating the first input and the second input comprises aggregating the first input and only a portion of the second input received during the first temporal window.