System and method of handling problematic input during context-sensitive help for multi-modal dialog systems

• US 20040006480A1
• Filed: 12/19/2002
• Published: 01/08/2004
• Est. Priority Date: 07/05/2002
• Status: Active Grant

• Current Assignee: Interactions, LLC
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Johnston, Michael, Ehlen, Patrick, Hastie, Helen
• Granted Patent: US 7,177,816 B2
• Time in Patent Office: Days
• US Class Current: 704/276
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A method of presenting a multi-modal help dialog move to a user in a multi-modal dialog system is disclosed. The method comprises presenting an audio portion of the multi-modal help dialog move that explains available ways of user inquiry and presenting a corresponding graphical action performed on a user interface associated with the audio portion. The multi-modal help dialog move is context-sensitive and uses current display information and dialog contextual information to present a multi-modal help move that is currently related to the user. A user request or a problematic dialog detection module may trigger the multi-modal help move.

Claims

1. In a multi-modal dialog system, a method of presenting a multi-modal help dialog move to a user in a problematic dialog situation, the method comprising:

receiving a plurality of utterances that do not meet a successful recognition threshold;

presenting an audio portion of a multi-modal help dialog move that explains available ways of user inquiry; and

presenting a corresponding graphical action performed on a user interface associated with the audio portion.

2. The method of claim 1, wherein the presented audio portion and presented graphical action relate to what is currently being displayed to the user.

3. The method of claim 1, wherein the presented graphical action further comprises at least one graphical button corresponding to choices available to the user.

4. The method of claim 3, wherein the at least one graphical button relates to different kinds of help available to the user.

5. The method of claim 1, wherein the number of the plurality of user utterances that does not meet the successful recognition threshold is adjustable based on user experience.

6. The method of claim 1, wherein the number of the plurality of user utterances that does not meet the successful recognition threshold is adjustable based on automatic speech recognition performance.

12. The method of claim 1 l, wherein the problematic condition relates to the system receiving a plurality of user utterances that do not meet a recognition threshold.

7. In a multi-modal dialog system, a method of presenting a multi-modal help dialog move to a user in a problematic dialog situation, the method comprising:

receiving at least one utterance that does not meet an utterance recognition threshold and at least one graphical user input that does not meet a graphical user input recognition threshold;

presenting an audio portion of a multi-modal help dialog move that explains available ways of user inquiry; and

presenting a corresponding graphical action performed on a user interface associated with the audio portion.

8. The method of claim 7, wherein the presented audio portion and presented graphical action relate to what is currently being displayed to the user.

9. The method of claim 7, wherein the presented graphical action further comprises at least one graphical button corresponding to choices available to the user.

10. The method of claim 9, wherein the at least one graphical button relates to different kinds of help available to the user.

11. In a multi-modal dialog system, a method of presenting a multi modal help dialog move to a user in a problematic situation, comprising:

analyzing a current dialog state with the user, including a visual context, to determine whether a problematic condition exists;

if the multi-modal dialog system determines that a problematic condition exists;

presenting the multi-modal help dialog move having an audio portion and a graphical portion that utilize the visual context of the current dialog state with the user.

13. The method of claim 11, wherein the problematic condition relates to the system receiving a null signal.

14. The method of claim 11, wherein the problematic condition relates to the system receiving at least one user utterance that does not meet an utterance recognition threshold and at least one graphical user input that does not meet a graphical recognition threshold.

15. In a multi-modal dialog system, a method of presenting a multi-modal help dialog move to a user in a problematic dialog situation, the method comprising:

receiving a plurality of utterances that do not meet a successful recognition threshold, the number of plurality of utterances being set according to user experience and automatic speech recognition parameters;

presenting an audio portion of a multi-modal help dialog move that explains available ways of user inquiry; and

presenting a corresponding graphical action performed on a user interface associated with the audio portion.

16. In a multi-modal dialog system, a method of presenting a multi-modal help dialog move to a user, comprising:

upon detecting a problematic dialog situation wherein a multi-modal dialog system receives a plurality of user inputs that do not meet recognition thresholds, presenting the multi modal help dialog move having an audio portion and a graphical portion that utilize a visual context of the current dialog state with the user, wherein the multi-modal help dialog move provides information to the user regarding how to interact with the system in a multi-modal fashion.

17. The method of claim 16, wherein the presented graphical portion further comprises at least one graphical button corresponding to choices available to the user.

18. The method of claim 17, where the at least one graphical button relates to different kinds of help available to the user.

19. In a multi-modal dialog system using a map display having an entity of interest thereon, a method of presenting a multi-modal help dialog move to a user in a problematic situation, comprising:

determining whether the user is in a problematic situation wherein at least one type of multi-modal input is not recognized by the system, and if a problematic situation exists and when an entity of interest is currently being displayed to the user;

presenting a multi-modal help dialog move having an audio portion and a graphical portion, the audio portion stating a plurality of types of information available to the user associated with the entity of interest, and the graphical portion corresponding to the audio portion.

20. The method of claim 19, where the audio portion further comprises instructions to the user regarding how to multi-modally obtain information regarding the entity of interest.

21. The method of claim 20, wherein while the audio portion is being presented to the user explaining how to multi-modally obtain information regarding the entity of interest, the graphical portion provides an example graphical input that the user can make to obtain the information about the entity of interest.

22. A method of assisting a user to interact with a system having a multi-modal interface, the method comprising:

determining whether a problematic situation exists wherein a system with a multi-modal interface has received a plurality of multi-modal user inputs that do not meet a recognition threshold; and

upon determining that help is needed, presenting to the user a multi-modal dialog help move having an audio portion and a graphical portion, the multi-modal dialog help move audibly explaining to the user available multi-modal actions the user can take to obtain information while providing corresponding graphics demonstrating the kind of graphic input the user can make to obtain the information.

23. The method of claim 22, wherein the system comprises a problematic dialog situation detector to detect problematic situations.

24. The method of claim 22, wherein the presented multi-modal dialog help move utilizes current graphical information being presented to the user in demonstrating the kind of graphic input the user can make to obtain the information.

25. In a multi-modal dialog system using a map display having a plurality of entities of interest thereon, a method of presenting a multi modal help dialog move to a user in a problematic situation, comprising:

determining whether a problematic situation exists wherein the multi-modal dialog system has received a plurality of user inputs that do not meet recognition thresholds, and if a problematic situation exists;

selecting an entity of interest from the plurality of entities of interest currently displayed to the user; and

presenting a multi-modal help dialog move having an audio portion and a graphical portion, the audio portion stating a plurality of types of information available to the user associated with the selected entity of interest, and the graphical portion being associated with the audio portion.