User feedback for speech interactions

• US 9,922,639 B1
• Filed: 01/11/2013
• Issued: 03/20/2018
• Est. Priority Date: 01/11/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Belin, Gilles Jean Roger, Rogers, III, Charles S., Owen, Robert David, Adams, Jeffrey Penrod, Ramachandran, Rajiv, Hart, Gregory Michael
• Primary Examiner(s): Tzeng, Forrest F
• Application Number:
  US13/739,826
• Time in Patent Office: 1,894 Days
• Field of Search: 704251, 704257, 704270, 704275, 704E15001
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/221 Announcement of recognition results

Abstract

An interactive system may be implemented in part by an audio device located within a user environment, which may accept speech commands from a user and may also interact with the user by means of generated speech. In order to improve performance of the interactive system, a user may use a separate device, such as a personal computer or mobile device, to access a graphical user interface that lists details of historical speech interactions. The graphical user interface may be configured to allow the user to provide feedback and/or corrections regarding the details of specific interactions.

Claims

1. A system, comprising:

a speech interface platform configured to (a) receive an audio signal from a speech interface device located in an environment of a user, (b) recognize user speech in the audio signal, (c) understand a user intent from the recognized user speech, and (d) perform an action based on the understood user intent;

a graphical interface that is provided by, and accessible by the user through, a user device that is physically separate from the speech interface device, the graphical interface being configured to display a first interface including a list of a plurality of historical interaction records, a historical interaction record of the plurality of historical interaction records associated with information regarding a historical speech interaction between the speech interface platform and the user, wherein the information includes (a) the recognized user speech, (b) the understood user intent, and (c) the action performed by the speech interface platform;

wherein the first interface provided by, and accessible by the user through, the user device that is physically separate from the speech interface device is further configured to accept, within the first interface provided by the user device, a selection of the historical interaction record;

wherein the graphical interface is configured to display, based at least in part on the selection, a second interface, the second interface configured to display at least a portion of the information regarding the historical speech interaction and to accept, within the second interface provided by the user device, one or more edits or evaluations from the user of the historical speech interaction regarding one or more of (a) the understood user intent and (b) the action performed by the speech interface platform; and

wherein the system is configured to receive, from the user device that is physically separate from the speech interface device, the one or more edits or evaluations for improving future speech recognition and intent understanding on a subsequent audio signal received from the speech interface device located in the environment of the user.

2. The system of claim 1, wherein the second interface allows the user to play the audio signal.

3. The system of claim 1, wherein the at least the portion of the information regarding the historical speech interaction further includes text of speech generated by the speech interface platform in response to the user speech.

4. The system of claim 1, wherein:

the at least the portion of the information regarding the historical speech interaction further includes text of speech generated by the speech interface platform in response to the user speech; and

the graphical interface allows the user to play audio of the generated speech and to edit or evaluate the generated speech.

26. The system of claim 1, wherein:

the user device includes a first processor and first memory and executes a first application that communicates with the speech interface platform via one or more networks; and

the speech interface device includes a second processor and second memory and executes a second application that communicates with the speech interface platform via the one or more networks.

27. The system of claim 26, wherein:

the user device further includes a first network interface; and

the first processor, the first memory and the first network interface are not involved in the receipt, by the speech interface platform, of the audio signal from the speech interface device.

28. The system of claim 1, wherein the speech interface device comprises a microphone and speaker and does not include a display for displaying the graphic interface for receiving the one or more edits or evaluations from the user of the historical speech interaction regarding the one or more of (a) the understood user intent and (b) the action performed by the speech interface platform.

29. The system of claim 1, wherein the list of the plurality of historical interaction records is presented in reverse chronological order.

30. The system of claim 1, wherein the one or more edits or evaluations is an edit to a first intent corresponding to the understood user intent, and wherein the graphical interface is further configured to accept, via the second interface, a user selection of a second intent to provide one or more corrections to the understood user intent, the second intent selected by the speech interface platform as a possible user intent.

31. The system of claim 1, wherein the one or more edits or evaluations is an evaluation of an accuracy of the understood user intent, wherein the graphical interface is further configured to accept, via the second interface, an input indicating that the understood user intent is incorrect.

5. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

providing, to a user device that is physically separate from a speech interface device, a first graphical interface displaying a plurality of historical interaction records, a historical interaction record of the plurality of historical interaction records associated with information regarding a historical speech interaction conducted at least in part through the speech interface device between a speech interface platform and a user, wherein the information indicates recognized speech from a user utterance and at least one of (a) an intent of the user that was understood by the speech interface platform from the recognized speech, (b) text corresponding to speech generated by the speech interface platform in response to the user utterance, or (c) an action performed in response to the user utterance;

accepting, from the user device that is physically separate from the speech interface device via the first graphical interface displayed at the user device, a selection of the historical interaction record; and

based at least in part on the selection, providing to the user device a second graphical interface displaying at least a portion of the information regarding the historical speech interaction and accepting, within the second graphical interface displayed by the user device, one or more edits or evaluations from the user regarding at least some of the at least the portion of the information regarding the historical speech interaction for improving future speech recognition and intent understanding on a subsequent audio signal received from the speech interface device.

6. The one or more non-transitory computer-readable media of claim 5, the acts further comprising updating at least one of a model for recognizing speech or a model for determining user intent.

7. The one or more non-transitory computer-readable media of claim 5, wherein the historical speech interaction relates to activating or deactivating the speech interface platform.

8. The one or more non-transitory computer-readable media of claim 5, wherein the one or more edits or evaluations comprise text entry by the user received via the second graphical interface.

9. The one or more non-transitory computer-readable media of claim 5, wherein the one or more edits or evaluations comprise a correction to text of the recognized speech.

10. The one or more non-transitory computer-readable media of claim 5, the acts further comprising identifying, by the speech interface platform, an identity of the user based at least in part on voice identification of the user utterance, and wherein the one or more edits or evaluations comprise a correction to the identity of the user.

11. The one or more non-transitory computer-readable media of claim 5, wherein the one or more edits or evaluations comprise a correction to the understood intent.

12. The one or more non-transitory computer-readable media of claim 5, wherein the one or more edits or evaluations comprise a rating of the generated speech, the rating received via the second graphical interface.

13. The one or more non-transitory computer-readable media of claim 5, wherein the one or more edits or evaluations comprise specifying, within the second graphical interface, an action that was intended by the user.

14. The one or more non-transitory computer-readable media of claim 5, wherein the selection is a first user selection, the acts further comprising at least one of:

playing audio of the user utterance in response to a second user selection;

or

playing audio of the generated speech in response to a third user selection.

15. The one or more non-transitory computer-readable media of claim 5, wherein the at least the portion of the information is displayed by an application that executes on the user device which is a mobile device of the user.

16. A method comprising:

receiving, by a user device that is physically separate from a speech interface device, information regarding a historical speech interaction conducted at least in part through the speech interface device between a user and a speech interface platform, wherein the information indicates recognized speech from a user utterance and at least one of (a) a user intent that was understood from the recognized speech, (b) text corresponding to speech generated by the speech interface platform in response to the user utterance, or (c) an action performed by the speech interface platform in response to the user utterance;

accepting, by the user device that is physically separate from the speech interface device via a first graphical interface displayed at the user device, a selection of a historical interaction record from a plurality of historical interaction records, the historical interaction record associated with the information regarding the historical speech interaction;

accepting, by the user device via a second graphical interface, one or more edits or evaluations from the user regarding the historical speech interaction, the second graphical interface presented at least in part based on the selection; and

providing, by the user device that is physically separate from the speech interface device and to the speech interface platform, the edits or evaluations regarding the historical speech interaction for use in conjunction with future speech interactions and for improving future speech recognition and intent understanding on a subsequent audio signal received from the speech interface device.

17. The method of claim 16, further comprising updating at least one of a model for recognizing speech or a model for determining user intent.

18. The method of claim 16, wherein the one or more edits or evaluations comprise a correction to the recognized speech.

19. The method of claim 16, wherein the one or more edits or evaluations comprise a correction to an identity of the user.

20. The method of claim 16, wherein the one or more edits or evaluations comprise a correction to the understood intent.

21. The method of claim 16, wherein the one or more edits or evaluations comprise a rating of the speech generated by the speech interface platform.

22. The method of claim 16, wherein the one or more edits or evaluations comprise specifying an action that was intended by the user.

23. The method of claim 16, the acts further comprising playing audio of the user utterance in response to a user selection.

24. The method of claim 16, the acts further comprising playing audio corresponding to the speech generated by the speech interface platform in response to a user selection.

25. The method of claim 16, wherein the information is displayed by an application that executes on the user device which is a mobile device of the user.