Method and system for assisting users in interacting with multi-modal dialog systems

• US 20070033526A1
• Filed: 08/03/2005
• Published: 02/08/2007
• Est. Priority Date: 08/03/2005
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Gupta, Anurag, Thompson, William, Lee, Hang
• Granted Patent: US 7,548,859 B2
• Time in Patent Office: Days
• US Class Current: 715/708
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A method and system for assisting a user in interacting with a multi-modal dialog system (104) is provided. The method includes interpreting a “What Can I Do? (WCID)” question from a user in a turn of the dialog. A multi-modal grammar (212) is generated, based on the current context of the dialog. One or more user multi-modal utterances are generated, based on the WCID question and the multi-modal grammar. One or more user multi-modal utterances are conveyed to the user.

Claims

1. A method for assisting a user in interacting with a multi-modal dialog system, the method comprising:

interpreting a question from the user;

generating one or more user multi-modal utterances based on the question and a multi-modal grammar, wherein the multi-modal grammar is based on a current context of a multi-modal dialog; and

conveying the one or more user multi-modal utterances to the user.

2. The method according to claim 1 further comprising de-referencing the one or more user multi-modal utterances using a visual context of the multi-modal dialog.

3. The method according to claim 2 further comprising storing the visual context of the multi-modal dialog.

4. The method according to claim 1, wherein generating the one or more user multi-modal utterances comprises generating the multi-modal grammar based on the current context of the multi-modal dialog.

5. The method according to claim 1 further comprising ranking the one or more user multi-modal utterances based on one or more of a group consisting of the current context of the multi-modal dialog, a user model and modality availability.

6. A multi-modal dialog system comprising:

a multi-modal input fusion (MMIF) component, the multi-modal input fusion component accepting a question from the user;

a dialog manager, the dialog manager generating a multi-modal grammar based on a current context of a multi-modal dialog;

a multi-modal utterance generator, the multi-modal utterance generator generating one or more user multi-modal utterances based on the question and the multi-modal grammar.

7. The multi-modal dialog system of claim 6 further comprising a multi-modal utterance ranker, the multi-modal utterance ranker ranking the one or more user multi-modal utterances based on one or more of a group consisting of a current context of the multi-modal dialog, a user model and modality availability.

8. The multi-modal dialog system of claim 6 further comprising a visual context manager, the visual context manager storing a visual context of the multi-modal dialog.

9. The multi-modal dialog system of claim 8, wherein the multi-modal utterance generator generates the one or more user multi-modal utterances based on the visual context of the multi-modal dialog.

10. The multi-modal dialog system of claim 6 further comprising an output generator, the output generator conveying the one or more user multi-modal utterances to the user.

11. The multi-modal dialog system of claim 6, wherein the MMIF component maintains and updates the modality availability.

12. The multi-modal dialog system of claim 6, wherein the dialog manager interprets the question from the user.

13. The multi-modal dialog system of claim 6, wherein the dialog manager maintains and updates the current context of the dialog.

14. An electronic equipment comprising:

means for interpreting a question from the user;

means for generating one or more user multi-modal utterances based on the question and a multi-modal grammar, wherein the multi-modal grammar is based on a current context of a multi-modal dialog; and

means for conveying the one or more user multi-modal utterances to the user.

15. The electronic equipment of claim 14 further comprising means for storing a visual context of the multi-modal dialog.