System and Method for Multi-Modal Input Synchronization and Disambiguation

• US 20100241431A1
• Filed: 03/18/2009
• Published: 09/23/2010
• Est. Priority Date: 03/18/2009
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Zou, Lincan, Shen, Zhongnan, Ren, Liu, Weng, Fuliang, Feng, Zhe, Yan, Baoshi
• Granted Patent: US 9,123,341 B2
• Time in Patent Office: Days
• US Class Current: 704/257
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

Embodiments of a dialog system that utilizes a multi-modal input interface for recognizing user input in human-machine interaction (HMI) systems are described. Embodiments include a component that receives user input from a plurality of different user input mechanisms (multi-modal input) and performs certain synchronization and disambiguation processes. The multi-modal input components synchronizes and integrates the information obtained from different modalities, disambiguates the input, and recovers from any errors that might be produced with respect to any of the user inputs. Such a system effectively addresses any ambiguity associated with the user input and corrects for errors in the human-machine interaction.

Claims

1. A computer-implemented method in a dialog system, comprising:

receiving input from a plurality of input modalities;

synchronizing the inputs based on the temporal order of events associated with the input;

generating an initial hypothesis for an input for an event;

receiving constraints for the input based on one or more other inputs of the plurality of inputs;

combining the constraints with the initial hypothesis; and

deriving a new hypothesis to interpret the input.

2. The method of claim 1 wherein the hypothesis comprises a hypothesis value (H) and a weighting value (W), wherein W represents a probability of proper interpretation of the input.

3. The method of claim 1 wherein the input is provided at least in part by a human user, and wherein the plurality of input modalities are selected from the group consisting of:

spoken input, gesture input, keyboard input, mouse input, and pen or stylus input.

4. The method of claim 3 wherein the input is provided at least in part by an application program or a device, and wherein the plurality of input modalities further include programmatic input and device input.

5. The method of claim 1 wherein the input contains one or more errors, and wherein the new hypothesis is used to disambiguate the input based on multi-modal information provided by the constraints.

6. The method of claim 5 wherein the dialog system comprises a speech recognition stage including a spoken language understanding unit.

7. The method of claim 6 wherein the dialog system further comprises a gesture recognition unit to interpret physical user input provided through a touch screen interface or sensed by one or more gesture sensors.

10. The system of claim 1 wherein the input is provided at least in part by a human user, and wherein the plurality of input modalities provided to the input stage are selected from the group consisting of:

spoken input, gesture input, keyboard input, mouse input, and pen or stylus input.

11. The system of claim 10 wherein the input is provided at least in part by an application program or a device, and wherein the plurality of input modalities further include programmatic input and device input.

8. A dialog system, comprising:

an input stage receiving input from a plurality of input modalities;

a synchronization module synchronizing the inputs based on the temporal order of events associated with the input;

a dialog manager generating an initial hypothesis for an input for an event;

a disambiguation module receiving constraints for the input based on one or more other inputs of the plurality of inputs, wherein the dialog manager combines the constraints with the initial hypothesis and derives a new hypothesis to interpret the input.

9. The system of claim 8 wherein the hypothesis comprises a hypothesis value (H) and a weighting value (W), wherein W represents a probability of proper interpretation of the input.

12. The system of claim 8 wherein the input contains one or more errors, and wherein the new hypothesis is used to disambiguate the input based on multi-modal information provided by the constraints.

13. The system of claim 12 wherein the dialog system comprises a speech recognition stage including a spoken language understanding unit.

14. The system of claim 13 wherein the dialog system further comprises a gesture recognition unit to interpret physical user input provided through a touch screen interface or sensed by one or more gesture sensors.

15. The system of claim 14 wherein the disambiguation module comprises a disambiguation component provided in each of the speech recognition stage and the gesture recognition unit.

16. The system of claim 15 further comprising:

a response generator generating a system output based on the user input;

a text-to-speech unit converting at least a portion of the system output to spoken language output;

an audio output device coupled to a text-to-speech unit playing the spoken language output for the user; and

a graphical output displaying graphical output from the response generator through a graphical user interface process.

17. A computer readable medium containing a plurality of program instructions, which when executed by a processor, cause the processor to perform the steps of:

receiving input from a plurality of input modalities;

synchronizing the inputs based on the temporal order of events associated with the input;

generating an initial hypothesis for an input for an event;

receiving constraints for the input based on one or more other inputs of the plurality of inputs;

combining the constraints with the initial hypothesis; and

deriving a new hypothesis to interpret the input.

18. The medium of claim 17 wherein the hypothesis comprises a hypothesis value (H) and a weighting value (W), wherein W represents a probability of proper interpretation of the input.

19. The medium of claim 18 wherein the input is provided at least in part by

a human user, and wherein the plurality of input modalities are selected from the group consisting of:

spoken input, gesture input, keyboard input, mouse input, and pen or stylus input; and

an application program or a device, and wherein the plurality of input modalities further include programmatic input and device input.

20. The medium of claim 19 wherein the input contains one or more errors, and wherein the new hypothesis is used to disambiguate the input based on multi-modal information provided by the constraints.