Call steering data tagging interface with automatic semantic clustering

• US 8,903,712 B1
• Filed: 09/27/2011
• Issued: 12/02/2014
• Est. Priority Date: 09/27/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Tremblay, Real, Tremblay, Jerome, Ulug, Amy E., Marcus, Jeffrey N., Mauro, David Andrew, Fortier, Jean-Francois, Berleur, Francois
• Primary Examiner(s): Shah, Paras D
• Assistant Examiner(s): THOMAS-HOMESCU, ANNE L
• Application Number:
  US13/246,801
• Time in Patent Office: 1,162 Days
• Field of Search: 704/4, 704/9, 704/3, 704/10, 704/246, 704/272, 379/93.01, 707/737, 707/769, 707/706, 707/708, 707/705, 707/740, 707/739, 715/255, 715/840, 715/222, 715/738, 715/830, 715/753, 715/853, 715/777, 715/719, 715/256
• US Class Current: 704/9
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 2015/0638 Interactive procedures
• CPC Class Codes: H04M 11/00 Telephonic communication systems specially adapted for combination with other electrical systems
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals

Abstract

A system and method for providing an easy-to-use interface for verifying semantic tags in a steering application in order to generate a natural language grammar. The method includes obtaining user responses to open-ended steering questions, automatically grouping the user responses into groups based on their semantic meaning, and automatically assigning preliminary semantic tags to each of the groups. The user interface enables the user to validate the content of the groups to ensure that all responses within a group have the same semantic meaning and to add or edit semantic tags associated with the groups. The system and method may be applied to interactive voice response (IVR) systems, as well as customer service systems that can communicate with a user via a text or written interface.

Claims

1. A method for enabling a developer of a steering application to associate semantic tags with user responses, the method comprising:

obtaining user responses to an open-ended steering question posed by an interactive response system;

automatically grouping the user responses into groups, wherein each group is a set of sentences that are semantically related;

automatically assigning preliminary semantic tags to each of the groups; and

providing a computer user interface that enables a user to validate the content of the groups to ensure that all sentences within a group have the same semantic meaning and to view and edit the preliminary semantic tags associated with the groups, wherein the computer user interface includes;

a groups view that displays a list of the groups and corresponding semantic tags for each group, wherein the groups view enables a user to edit the preliminary semantic tags associated with each of the groups,

a sentence view that displays, for a selected group in the groups view, a list of unique sentences associated with the selected group, wherein in the sentence view a user is able to verify whether or not a sentence belongs to the group selected in the groups view, and

a related-groups view that displays, for a selected sentence in the sentence view, a plurality of groups most closely-related to the selected sentence,

wherein the computer user interface enables the user to apply semantic clustering to unverified sentences, and wherein applying semantic clustering to unverified sentences re-distributes the unverified sentences into groups based at least in part on the group memberships of verified sentences.

2. The method of claim 1, wherein the user responses are spoken utterances.

3. The method of claim 2, wherein, prior to grouping the user responses into groups, transcriptions are obtained for the spoken utterances.

4. The method of claim 1, wherein the user responses are in the form of written or textual inputs.

5. The method of claim 4, wherein the written or textual inputs include one or more of text messaging, emails, online chat system communications, and internet or web-based communications.

6. The method of claim 1, wherein the semantic tags comprise a value for an action and a value for an object.

7. The method of claim 1, wherein, in the groups view, the computer user interface enables a user to set a quality rating for each of the groups.

8. The method of claim 1, wherein the groups view displays the number of unique sentences in the group.

9. The method of claim 1, wherein, for each unique sentence in the sentence view, the computer user interface displays the number of user responses corresponding to the sentence.

10. The method of claim 1, wherein, from the sentence view, a user is able to move a displayed sentence to a different group.

11. The method of claim 1, wherein, from the sentence view, a user is able to move a displayed sentence to a group-later area.

12. The method of claim 1, wherein, from the sentence view, a user is able to move a displayed sentence to a rejected-sentence area.

13. The method of claim 1, wherein, in the related groups view, groups are displayed in a star diagram with the selected sentence in the middle of the star diagram and the plurality of most closely-related groups surrounding the selected sentence.

14. The method of claim 1, wherein, in the related groups view, the plurality of most closely-related groups are displayed in a list in order of semantic distance from the selected sentence.

15. The method of claim 1, wherein a sentence in the sentence view can be moved to a different group by dragging the sentence to one of the plurality of most closely-related groups in the related-groups view.

16. The method of claim 1, wherein a sentence in the sentence view can be moved to a different group by dragging the sentence to one of the plurality of groups in the groups view.

17. The method of claim 1, wherein the computer user interface enables the user to merge groups.

18. The method of claim 1, wherein the computer user interface enables the user to create a new group.

19. The method of claim 1, wherein the computer user interface enables the user to apply semantic clustering on new sentences that have been added to the computer user interface, wherein applying semantic clustering distributes the new sentences into groups based at least in part on the verified sentences.

20. The method of claim 1, wherein the computer user interface enables the user to apply an inconsistency checker to detect inconsistencies in the tagged data.

21. A method for enabling a developer of a steering application to associate semantic tags with user responses, the method comprising:

obtaining user responses to an open-ended steering question posed by an interactive response system;

selecting a subset of user responses;

automatically grouping the user responses within the subset into groups, wherein each group is a set of sentences that are semantically related;

automatically assigning preliminary semantic tags to each of the groups;

providing a computer user interface that enables a user to validate the content of the groups to ensure that all sentences within a group have the same semantic meaning and to view and edit the preliminary semantic tags associated with the groups, wherein the computer user interface includes;

a groups view that displays a list of the groups and corresponding semantic tags for each group, wherein the groups view enables a user to edit the preliminary semantic tags associated with each of the groups,

a sentence view that displays, for a selected group in the groups view, a list of unique sentences associated with the selected group, wherein in the sentence view a user is able to verify whether or not a sentence belongs to the group selected in the groups view, and

a related-groups view that displays, for a selected sentence in the sentence view, a plurality of groups most closely-related to the selected sentence,

wherein the computer user interface enables the user to apply semantic clustering to unverified sentences, and wherein applying semantic clustering to unverified sentences re-distributes the unverified sentences into groups based at least in part on the group memberships of verified sentences; and

iteratively repeating the selecting, grouping, assigning, and providing steps with different subsets of user responses until all user responses have been processed, wherein each iteration uses data from previously validated and tagged groups to increase the accuracy of the clustering and assigning steps.

22. The method of claim 21, wherein the subsets are selected and processed such that the most common user responses are processed first.

23. The method of claim 21, wherein the subsets are selected and processed such that user responses with key words or phrases are processed first.

24. The method of claim 21, wherein the user responses are spoken utterances and, prior to the selecting step, transcriptions are obtained for the utterances.

25. The method of claim 21, wherein the user responses are written responses.

26. The method of claim 21, wherein the selecting, grouping, assigning, and providing steps are repeated in response to the user pressing a button to initiate semantic clustering for a next subset of responses.

27. A computer system for enabling a developer of a steering application to associate semantic tags with user responses, the system comprising:

a database;

a processor coupled to the database; and

a memory coupled to the processor, wherein the memory stores instructions that when executed by the processor causes the system to perform the operations of;

obtaining user responses to an open-ended steering question;

automatically grouping the user responses into groups, wherein each group is a set of sentences that are semantically related;

automatically assigning preliminary semantic tags to each of the groups; and

providing a computer user interface that enables a user to validate the content of the groups to ensure that all sentences within a group have the same semantic meaning and to view and edit the preliminary semantic tags associated with the groups, wherein the computer user interface includes;

a groups view that displays a list of the groups and corresponding semantic tags for each group, wherein the groups view enables a user to edit the preliminary semantic tags associated with each of the groups,

a sentence view that displays, for a selected group in the groups view, a list of unique sentences associated with the selected group, wherein in the sentence view a user is able to verify whether or not a sentence belongs to the group selected in the groups view, and

a related-groups view that displays, for a selected sentence in the sentence view, a plurality of groups most closely-related to the selected sentence,

wherein the computer user interface enables the user to apply semantic clustering to unverified sentences, and wherein applying semantic clustering to unverified sentences re-distributes the unverified sentences into groups based at least in part on the group memberships of verified sentences.

28. A computer program embodied on a non-transitory computer-readable medium and comprising code, that, when executed by a computer system, enables the computer system to perform the following method:

obtaining user responses to an open-ended steering question;

automatically grouping the user responses into groups, wherein each group is a set of sentences that are semantically related;

automatically assigning preliminary semantic tags to each of the groups; and

providing a computer user interface that enables a user to validate the content of the groups to ensure that all sentences within a group have the same semantic meaning and to view and edit the preliminary semantic tags associated with the groups, wherein the computer user interface includes;

a groups view that displays a list of the groups and corresponding semantic tags for each group, wherein the groups view enables a user to edit the preliminary semantic tags associated with each of the groups,

a sentence view that displays, for a selected group in the groups view, a list of unique sentences associated with the selected group, wherein in the sentence view a user is able to verify whether or not a sentence belongs to the group selected in the groups view, and

a related-groups view that displays, for a selected sentence in the sentence view, a plurality of groups most closely-related to the selected sentence,

wherein the computer user interface enables the user to apply semantic clustering to unverified sentences, and wherein applying semantic clustering to unverified sentences re-distributes the unverified sentences into groups based at least in part on the group memberships of verified sentences.