Diagnosing recognition problems from untranscribed data

• US 20070005354A1
• Filed: 06/30/2005
• Published: 01/04/2007
• Est. Priority Date: 06/30/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Potter, Stephen, Bowker, Anthony
• Granted Patent: US 7,873,523 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

Analytical processes are applied to contextual information in log data of dialog turns (user-system exchange) in a speech enabled application to diagnose speech recognition problems. Diagnosis can include inferring the semantics of a given user response. From semantic inferences an estimate of semantic recognition accuracy can also be obtained.

Claims

1. A computer-implemented method of analyzing recognition results between a user and an interactive application having dialog turns, wherein a turn comprises a prompt from the system and a response received from the user, the method comprising:

receiving information indicative of dialog turns between the system and at least one user; and

associating a turn providing a response with an inferred value based on analyzing the received information without performing transcription of the user responses.

2. The computer-implemented method of claim 1 wherein analyzing the received information comprises analyzing turns where a user confirmation is logged.

3. The computer-implemented method of claim 2 wherein analyzing confirmation turns where the user confirmation is logged comprises identifying application storage locations associated with each confirmation turn.

4. The computer-implemented method of claim 3 wherein identifying application storage locations associated with each confirmation turn comprises identifying application storage locations where the value was at least one of confirmed, denied and corrected.

5. The computer-implemented method of claim 4 wherein analyzing confirmation turns comprises for each identified application storage location, identifying a closest prior turn in which a value of the confirmation turn was obtained.

6. The computer-implemented method of claim 5 wherein analyzing confirmation turns comprises identifying a semantic portion of a recognition result that is mapped to an application storage location having the value of the confirmation turn.

7. The computer-implemented method of claim 5 wherein the inferred value is the value of the confirmation turn and wherein associating the turn with the inferred value includes associating the closest prior turn with the inferred value.

8. The computer-implemented method of claim 4 wherein the turns are related to one or more tasks of the application, and wherein analyzing confirmation turns comprises for each identified application storage location, identifying a closest prior turn in which a value of the confirmation turn was obtained and within a selected task.

9. The computer-implemented method of claim 2 and providing a measure of recognition errors based on analysis of a plurality of turns each having the associated inferred value.

10. The computer-implemented method of claim 1 wherein the application is defined in terms of tasks, and wherein a task involves one or more turns, and wherein analyzing the received information comprises analyzing information related to a task being successfully completed.

11. The computer-implemented method of claim 10 wherein analyzing information related to the task being successfully completed comprises identifying application storage locations associated with the corresponding task having values with confirmed status on task completion.

12. The computer-implemented method of claim 11 wherein analyzing information related to the task being successfully completed comprises for each identified application storage location, identifying a closest prior turn in which a value corresponds to a value received when the task was completed successfully.

13. The computer-implemented method of claim 12 wherein analyzing information related to the task being successfully completed comprises identifying a semantic portion of a recognition result of the closest prior turn that is mapped to an application storage location having the when the task was completed successfully and adding the semantic portion to an array.

14. The computer-implemented method of claim 13 wherein the inferred value is the array and wherein associating the turn with the inferred value includes associating all turns in the task with the inferred value.

15. The computer-implemented method of claim 10 and providing a measure of recognition errors based on analysis of a plurality of turns each having the associated inferred value.

16. The computer-implemented method of claim 1 wherein the application is defined in terms of tasks, and wherein a task involves one or more turns, and wherein analyzing the received information comprises analyzing where a user accepts or denies a value in a confirmation turn in a task that has completed successfully.

17. The computer-implemented method of claim 16 wherein analyzing information comprises identifying an application storage location associated with the confirmation turn having a value with confirmed status on task completion.

18. The computer-implemented method of claim 16 wherein analyzing information comprises comparing the value of the associated with confirmation turn with the value of the storage location when the task was completed successfully.

19. The computer-implemented method of claim 16 wherein the inferred value comprises acceptance or denial of the value in the confirmation turn, and wherein associating the inferred value comprises associating the correctness of the acceptance or the denial.

20. A computer-readable medium having instructions executable on a computer for implementing the method of claim 1.