Method and system for assisting users in interacting with multi-modal dialog systems

• US 7,548,859 B2
• Filed: 08/03/2005
• Issued: 06/16/2009
• Est. Priority Date: 08/03/2005
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Gupta, Anurag K., Lee, Hang S., Thompson, William K.
• Primary Examiner(s): Chawan; Vijay B
• Application Number:
  US11/196,112
• Publication Number:
  US 20070033526A1
• Time in Patent Office: 1,413 Days
• Field of Search: 704/9, 704/8, 704/7, 704/4, 704/3, 704/270, 704/257, 704/276, 704/10, 704/251, 704/252, 704/255, 704/275
• US Class Current: 704/270
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A method and system for assisting a user in interacting with a multi-modal dialog system (104) is provided. The method includes interpreting a “What Can I Do? (WCID)” question from a user in a turn of the dialog. A multi-modal grammar (212) is generated, based on the current context of the dialog. One or more user multi-modal utterances are generated, based on the WCID question and the multi-modal grammar. One or more user multi-modal utterances are conveyed to the user.

Claims

1. A method for assisting a user in interacting with a multi-modal dialog apparatus, the method comprising:

interpreting by a processor of the multi-modal dialog apparatus a What Can I Do (WCID) type of question from the user that is received at a user interface device coupled to the multi-modal dialog apparatus;

generating by the processor one or more user multi-modal utterances as sequences of words that a user may provide in the multi-modal inputs in a next turn of the dialog based on the WCID question and a multi-modal grammar, wherein the multi-modal grammar is based on a current context of a multi-modal dialog; and

conveying the one or more user multi-modal utterances to the user at a user interface device coupled to the multi-modal dialog apparatus.

2. The method according to claim 1 further comprising de-referencing the one or more user multi-modal utterances using a visual context of the multi-modal dialog.

3. The method according to claim 2 further comprising storing the visual context of the multi-modal dialog.

4. The method according to claim 1, wherein generating the one or more user multi-modal utterances comprises generating the multi-modal grammar based on the current context of the multi-modal dialog.

5. The method according to claim 1 further comprising ranking the one or more user multi-modal utterances based on one or more of a group consisting of the current context of the multi-modal dialog, a user model and modality availability.

6. A multi-modal dialog apparatus comprising:

a user interface device;

a processor; and

a memory, wherein the memory stores programming instructions organized into functional groups that control the processor, the functional groups comprising

a multi-modal input fusion (MMIF) component, the multi-modal input fusion component accepting a What Can I Do (WCID) type of question from the user through the user interface device;

a dialog manager, the dialog manager generating a multi-modal grammar based on a current context of a multi-modal dialog;

a multi-modal utterance generator, the multi-modal utterance generator generating one or more user multi-modal utterances through the user interface device, as sequences of words that a user may provide in the multi-modal inputs in a next turn of the dialog based on the question and the multi-modal grammar.

7. The multi-modal dialog apparatus of claim 6 wherein the functional groups further comprise a multi-modal utterance ranker, the multi-modal utterance ranker ranking the one or more user multi-modal utterances based on one or more of a group consisting of a current context of the multi-modal dialog, a user model and modality availability.

8. The multi-modal dialog apparatus of claim 6 wherein the functional groups further comprise a visual context manager, the visual context manager storing a visual context of the multi-modal dialog.

9. The multi-modal dialog apparatus of claim 8, wherein the multi-modal utterance generator generates the one or more user multi-modal utterances based on the visual context of the multi-modal dialog.

10. The multi-modal dialog apparatus of claim 6 wherein the functional groups further comprise an output generator, the output generator conveying the one or more user multi-modal utterances to the user.

11. The multi-modal dialog apparatus of claim 6, wherein the MMIF component maintains and updates the modality availability.

12. The multi-modal dialog apparatus of claim 6, wherein the dialog manager interprets the question from the user.

13. The multi-modal dialog apparatus of claim 6, wherein the dialog manager maintains and updates the current context of the dialog.