Outcome-oriented dialogs on a speech recognition platform

• US 9,721,570 B1
• Filed: 12/17/2013
• Issued: 08/01/2017
• Est. Priority Date: 12/17/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kumar, Nishant, Beal, Jeff Bradley, Gopalakrishnan, Ajay, Charter, Kevin Robert, Kshirsagar, Sumedha Arvind
• Primary Examiner(s): Hudspeth, David
• Assistant Examiner(s): Patel, Shreyans
• Application Number:
  US14/109,738
• Time in Patent Office: 1,323 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A speech recognition platform configured to receive an audio signal that includes speech from a user and perform automatic speech recognition (ASR) on the audio signal to identify ASR results. The platform may identify: (i) a domain of a voice command within the speech based on the ASR results and based on context information associated with the speech or the user, and (ii) an intent of the voice command. In response to identifying the intent, the platform may perform multiple actions corresponding to this intent. The platform may select a target action to perform, and may engage in a back-and-forth dialog to obtain information for completing the target action. The action may include streaming audio to the device, setting a reminder for the user, purchasing an item on behalf of the user, making a reservation for the user or launching an application for the user.

Claims

1. A system comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

receiving an audio signal that includes speech of a user;

analyzing the speech to generate speech-recognition results;

selecting an intent associated with the speech based at least in part on the speech-recognition results, the intent representing multiple possible tasks that the user may have requested in the speech;

identifying the multiple possible tasks represented by the intent;

identifying, for at least one of the multiple possible tasks;

(1) one or more fields that, when filled with respective values, results in the respective possible task being actionable, and (2) which of the one or more fields have values determined from the speech-recognition results;

selecting a target task to perform from the multiple possible tasks;

if a threshold number of the one or more fields have values, performing the target task;

if the threshold number of the one or more fields do not have values, performing an action from a plurality of actions to obtain one or more respective values for one or more fields that do not have values the action selected with reference to (1) a cost associated with the action, the cost being greater if the action is the first action that involves interacting with the user than if the action is the second action that is free from interacting with the user, and (2) a probability that performing the action will result in obtaining the one or more respective values.

2. A system as recited in claim 1, wherein the performing the action to obtain the one or more respective values for the one or more fields that do not have values is based at least in part on the speech-recognition results and contextual information associated with the user.

3. A system as recited in claim 1, wherein the audio signal comprises a first audio signal, the speech-recognition results comprise first speech-recognition results, and the performing of the action comprises:

identifying a question associated with the intent, the question for determining a corresponding value for at least one field of the one or more fields;

converting the question into a second audio signal;

sending the second audio signal for output to the user;

receiving a third audio signal that includes a reply of the user to the question;

analyzing the reply to generate second speech-recognition results;

identifying a value of the at least one field based at least in part on the second speech-recognition results; and

associating the value with the at least one field.

4. A system as recited in claim 1, wherein the probability comprises a first probability, and the acts further comprising:

determining, for the at least one of the multiple possible tasks, a second probability that the user requested the respective possible task in the speech;

and wherein the selecting the target task is based at least in part on the second probability.

5. A system as recited in claim 1, wherein the cost comprises a first cost, and the acts further comprising:

determining, for the at least one of the multiple possible tasks, a second cost associated with making the respective possible task actionable, the second cost increasing with a number of questions needed to ask the user to fill in the one or more fields associated with the respective possible task;

and wherein the selecting the target task is based at least in part on the second cost.

6. One or more computing devices comprising:

one or more processors;

memory; and

a dialog component, stored in the memory and executable on the one or more processors to perform acts comprising;

receiving an indication of a request made by a user, the request identified from an audio signal including speech of the user;

identifying a possible task to perform in response to receiving the request;

identifying, for the possible task;

(1) one or more fields that, when filled with respective values, results in the possible task being actionable, and (2) which of the one or more fields have values determined from the speech of the user;

identifying multiple possible actions, including a first action that involves to perform to obtain one or more values for fields that do not have values;

selecting an action to perform from the multiple possible actions, the selecting based at least in part on a probability that performing the action will result in obtaining the one or more values for the fields that do not have values; and

performing the action based at least in part on the probability being above a threshold value.

7. One or more computing devices as recited in claim 6, wherein the dialog component is further executable to perform acts comprising:

performing the possible task based at least in part on the possible task being actionable.

8. One or more computing devices as recited in claim 6, wherein the dialog component is further executable to perform acts comprising:

identifying multiple possible tasks to perform in response to receiving the request;

and wherein the identifying of the possible task to perform comprises identifying the possible task from the multiple possible tasks.

9. One or more computing devices as recited in claim 6, wherein performing the action comprises is based at least in part on the speech and contextual information associated with the user, the contextual information including information from previous speech from the user or previously stored data relating to the user.

10. One or more computing devices as recited in claim 9, wherein the contextual information includes at least one of information from previous speech from the user or previously stored data relating to the user.

11. One or more computing devices as recited in claim 6, wherein performing the action comprises:

generating a rendering directive for output on a device that is proximate to the user and that generated the audio signal that included the speech of the user.

12. One or more computing devices as recited in claim 6, wherein the audio signal comprises a first audio signal and performing the action comprises:

identifying a question associated with the possible task, the question for determining a value for at least one field of the one or more fields associated with the possible task;

converting the question into a second audio signal;

sending the second audio signal for output to the user;

receiving a third audio signal that includes a reply of the user to the question;

analyzing the reply to generate speech-recognition results;

identifying a value of the at least one field based at least in part on the speech-recognition results; and

associating the value with the at least one field.

13. One or more computing devices as recited in claim 6, wherein the one or more computing devices form a portion of a network-accessible computing platform that is remote from the user.

14. One or more computing devices as recited in claim 6, wherein the request comprises one or more of a request to set a reminder, a request to play a media file, a request to add an item to a list, a request to provide a recommendation to the user, a request to purchase an item, or a request to place a telephone call.

15. One or more computing devices as recited in claim 6, the probability comprising a first probability, and wherein the dialog component is further executable to perform acts comprising:

identifying multiple possible tasks to perform in response to receiving the request;

determining, for the multiple possible tasks, a second probability that the user requested the respective possible task in the speech;

and wherein the identifying of the possible task to perform comprises selecting the possible task to perform based at least in part on the second probability.

16. One or more computing devices as recited in claim 6, wherein selecting the action is also based at least in part on a first cost associated with the action, the first cost being greater if the action is the first action that involves interacting with the user than if the action is the second action that is free from interacting with the user, wherein a possible task is actionable when each field associated with the respective possible task is filled with a respective value, and wherein the dialog component is further executable to perform acts comprising:

identifying multiple possible tasks to perform in response to receiving the request;

determining, for the multiple possible tasks, a second cost associated with making the respective possible task actionable, the second cost increasing with a number of questions needed to ask the user to fill in the one or more fields associated with the respective possible task;

and wherein the identifying of the possible task to perform comprises selecting the possible task to perform based at least in part on the second cost.

17. One or more computing devices as recited in claim 6, wherein selecting the action is also based at least in part on the probability being at least a threshold amount higher than a cost associated with the action.

18. One or more computing devices as recited in claim 6, wherein the possible task corresponds to a domain and the multiple possible actions correspond to activities associated with the domain.

19. One or more computing devices as recited in claim 18, wherein the action is performed to determine at least one of the domain or an activity of the activities.

20. A method comprising:

receiving an audio signal that includes speech of a user;

identifying, from the audio signal, a request from the user;

identifying a task to perform in response to the request, the task being associated with one or more fields that, when filled with respective values, results in the task being actionable;

determining which of the one or more fields is associated with a respective value;

determining, for multiple possible actions to perform, which of the one or more fields each respective possible action is intended to fill;

selecting an action to perform from the multiple possible actions, the selecting based at least in part on a probability that performing the action will result in obtaining values for fields that do not have values; and

performing the action based at least in part on the probability.

21. A method as recited in claim 20, wherein selecting is based at least in part on at least one of the speech or contextual information associated with the user, the contextual information including information from previous speech from the user or previously stored data relating to the user.

22. A method as recited in claim 20, further comprising:

determining that at least one of the one or more fields does not contain a respective value; and

generating the respective value from at least one of the second action.

23. A method as recited in claim 20, wherein the identifying of the task comprises identifying multiple possible tasks to perform based at least in part on the request, the probability comprising a first probability, and further comprising:

determining, for each of the multiple possible tasks, a second probability that the user requested the respective possible task in the speech; and

selecting the task to perform based at least in part on the second probability.