Multimodal aggregating unit

• US 20050197843A1
• Filed: 03/07/2004
• Published: 09/08/2005
• Est. Priority Date: 03/07/2004
• Status: Abandoned Application

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Viswanathan, Mahesh, Faisman, Alexander, Kanevsky, Dimitri, Sicconi, Roberto, Nahamoo, David
• Application Number:
  US10/795,495
• Publication Number:
  US 20050197843A1
• Time in Patent Office: Days
• US Class Current: 704/276
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

In a voice processing system, a multimodal request is received from a plurality of modality input devices, and the requested application is run to provide a user with the feedback of the multimodal request. In the voice processing system, a multimodal aggregating unit is provided which receives a multimodal input from a plurality of modality input devices, and provides an aggregated result to an application control based on the interpretation of the interaction ergonomics of the multimodal input within the temporal constraints of the multimodal input. Thus, the multimodal input from the user is recognized within a temporal window. Interpretation of the interaction ergonomics of the multimodal input include interpretation of interaction biometrics and interaction mechani-metrics, wherein the interaction input of at least one modality may be used to bring meaning to at least one other input of another modality.

Claims

1. A voice processing system, comprising:

a plurality of modality input devices; and

a multimodal aggregating unit comprising a life cycle determining device for receiving input from the plurality of modality input devices, whereby the multimodal aggregating unit provides an aggregated output based on the input from the plurality of modality input devices and temporal constraints determined by the life cycle determining device.

2. The system of claim 1, wherein the life cycle determining device determines a temporal window for aggregating the input of the plurality of modality input devices based on life cycles of each input.

3. The system of claim 2, wherein the plurality of modality input devices include a first modality input device and a second modality input device, wherein input from the second modality input device is received within the temporal window to bring meaning to input from the first modality input device.

4. The system of claim 3, wherein the temporal window is determined as the overlap between the life cycles of the input from the first modality input device and the input from the second modality input device.

5. The system of claim 1, wherein the multimodal aggregating unit interprets interaction ergonomics of the input from the plurality of modality input devices and an input of at least one modality may be used to bring meaning to at least one other input of another modality.

6. The system of claim 4, wherein interpreting the interaction ergonomics comprises interpretation of interaction biometrics and interaction mechani-metrics.

7. The system of claim 1, wherein the input from the plurality of modality input devices comprises at least one of touch, speech, gestures, eye movements, face direction, and the like.

8. A multimodal aggregating unit, comprising:

an input unit that receives an input from a plurality of modality input devices; and

a decoder comprising a life cycle determining device for providing an aggregated output based on the input from the plurality of modality input devices and temporal constraints determined by the life cycle determining device.

9. The multimodal aggregating unit of claim 8, wherein the life cycle determining device determines a temporal window for aggregating the input of the plurality of modality input devices based on life cycles of each input.

10. The multimodal aggregating unit of claim 9, wherein the input includes input from a first modality input device and input from a second modality input device, wherein the input from the second modality input device is received within the temporal window to bring meaning to the input from the first modality input device.

11. The multimodal aggregating unit of claim 10, wherein the temporal window is determined as the overlap between the life cycles of the input from the first modality input device and the input from the second modality input device.

12. The multimodal aggregating unit of claim 8, wherein the multimodal aggregating unit interprets interaction ergonomics of the input from the plurality of modality input devices and an input of at least one modality may be used to bring meaning to at least one other input of another modality.

13. The multimodal aggregating unit of claim 12, wherein interpreting the interaction ergonomics comprises interpretation of interaction biometrics and interaction mechani-metrics.

14. The multimodal aggregating unit of claim 8, wherein the input from the plurality of modality input devices comprises at least one of touch, speech, gestures, eye movements, face direction, and the like.

15. A voice processing method, comprising:

receiving an input from a plurality of modality input devices; and

providing an aggregated output based on the input from the plurality of modality input devices and temporal constraints determined for aggregating the input of the plurality of modality input devices.

16. The method of claim 15, further comprising determining a temporal window based on life cycles of each input of the modality input devices to determine the temporal constraints for aggregating the input of the plurality of modality input devices.

17. The method of claim 16, wherein the input from the plurality of modality input devices includes a first modality input and a second modality input, wherein the second modality input is received within the temporal window to bring meaning to the first modality input.

18. The method of claim 17, further comprising determining the temporal window as the overlap between the life cycles of the first modality input and the second modality input.

19. The method of claim 15, further comprising interpreting interaction ergonomics of the input from the plurality of modality input devices, wherein an input of at least one modality may be used to bring meaning to at least one other input of another modality.

20. The method of claim 19, wherein interpreting the interaction ergonomics comprises interpretation of interaction biometrics and interaction mechani-metrics.