Training a dialog system using user feedback for answers to questions

• US 10,504,521 B1
• Filed: 05/26/2017
• Issued: 12/10/2019
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Taubman, Gabriel, Hogue, Andrew W., Lee, John J.
• Primary Examiner(s): Washburn, Daniel C
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US15/606,373
• Time in Patent Office: 928 Days
• Field of Search: 7042701
• CPC Class Codes: G06F 16/3337 Translation of the query language, e.g. Chinese to English
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a dialog system using user feedback. According to one implementation, a method includes receiving, by a dialog engine and from a computing device, a voice input; determining, by the dialog engine, that the received voice input is classified as feedback to an answer of a question; identifying, by the dialog engine, a predetermined feedback score associated with the feedback; and adjusting a confidence score associated with the question and the answer based on the predetermined feedback score.

Claims

1. A computer-implemented method comprising:

receiving, at a dialog engine on a mobile device, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user;

in response to receiving the voice query;

identifying, by the dialog engine, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query;

providing, by the dialog engine, a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query;

receiving, at the dialog engine, feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query;

determining, by the dialog engine, whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and

when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, by the dialog engine, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query.

2. The computer-implemented method of claim 1, wherein receiving feedback from the user comprises receiving a voice utterance detected by the mobile device that is uttered by the user.

3. The computer-implemented method of claim 1, further comprising, when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, updating, by the dialog engine, the confidence score associated with the candidate answer corresponding to the first answer.

4. The computer-implemented method of claim 1, further comprising storing, by the dialog engine, multiple predefined queries that each correspond to a particular candidate answer.

5. The computer-implemented method of claim 4, further comprising:

determining, by the dialog engine, a level of similarity between the received voice query and one or more of the multiple predefined queries; and

in response to determining that the level of similarity of the received voice query to a stored query exceeds a similarity threshold, obtaining, by the dialog engine, the particular predetermined answer that corresponds to the stored query,

wherein the first answer is provided based on the particular predetermined answer that corresponds to the stored query.

6. The computer-implemented method of claim 1, wherein determining whether the feedback was received within the predetermined amount of time comprises determining that the user has performed one or more activities within the predetermined amount of time of providing the first answer for output using the speaker of the mobile device.

7. The computer-implemented method of claim 6, wherein the one or more activities include at least one of (i) calling a telephone number associated with the answer, (ii) sending an electronic message associated with the answer, or (iii) uttering one or more terms.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising;

receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user;

in response to receiving the voice query;

identifying, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query;

providing a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query;

receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query

determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and

when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query.

9. The system of claim 8, wherein receiving feedback from the user comprises receiving a voice utterance detected by the mobile device that is uttered by the user.

10. The system of claim 8, wherein the operations further comprise, when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, updating the confidence score associated with the candidate answer corresponding to the first answer.

11. The system of claim 8, wherein the operations further comprise:

storing multiple predefined queries that each correspond to a particular predetermined answer;

determining a level of similarity between the received voice query and one or more of the multiple predefined queries; and

in response to determining that the level of similarity of the received voice query to a stored query exceeds a similarity threshold, obtaining the particular predetermined answer that corresponds to the stored query,

wherein the audible output is based on the particular predetermined answer that corresponds to the stored query.

12. The system of claim 8, wherein determining whether the feedback was received within the predetermined amount of time comprises determining that the user has performed one or more activities within the predetermined amount of time of providing the first answer for output using the speaker of the mobile device.

13. The system of claim 12, wherein the one or more activities include at least one of (i) calling a telephone number associated with the answer, (ii) sending an electronic message associated with the answer, or (iii) uttering one or more terms.

14. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising:

receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user;

in response to receiving the voice query;

identifying a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query;

providing a first answer to the voice query for output using a speaker of the system, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query;

receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query

determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and

when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query.

15. The computer-readable medium of claim 14, wherein receiving feedback from the user comprises receiving a voice utterance detected by the mobile device that is uttered by the user.

16. The computer-readable medium of claim 14, wherein the operations further comprise, when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, updating the confidence score associated with the candidate answer corresponding to the first answer.

17. The computer-readable medium of claim 14, wherein the operations further comprise storing multiple predefined queries that each correspond to a particular predetermined answer.

18. The computer-readable medium of claim 17, wherein the operations comprise:

determining a level of similarity between the received voice query and one or more of the multiple predefined queries; and

in response to determining that the level of similarity of the received voice query to a stored query exceeds a similarity threshold, obtaining the particular predetermined answer that corresponds to the stored query,

wherein the first answer is provided based on the particular predetermined answer that corresponds to the stored query.

19. The computer-readable medium of claim 14, wherein determining whether the feedback was received within the predetermined amount of time comprises determining that the user has performed one or more activities within the predetermined amount of time of providing the first answer for output using the speaker of the mobile device.

20. The computer-readable medium of claim 19, wherein the one or more activities include at least one of (i) calling a telephone number associated with the answer, (ii) sending an electronic message associated with the answer, or (iii) uttering one or more terms.