Training a dialog system using user feedback

• US 9,679,568 B1
• Filed: 11/05/2012
• Issued: 06/13/2017
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hogue, Andrew W., Lee, John J., Taubman, Gabriel
• Primary Examiner(s): Hudspeth, David
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US13/668,603
• Time in Patent Office: 1,681 Days
• Field of Search: 7042701, 704270, 704277
• CPC Class Codes: G06F 16/3337 Translation of the query language, e.g. Chinese to English
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a dialog system using user feedback. According to one implementation, a method includes receiving, by a dialog engine, a first input that specifies a question; providing, by the dialog engine, an answer to the question; receiving, by the dialog engine, a second input; and determining, by the dialog engine, that the second input is classified as feedback to the answer, then determining a feedback score associated with the second input.

Claims

1. A computer-implemented method comprising:

providing, by a dialog engine of a computing device, an answer to a question;

after providing the answer to the question, receiving, by the dialog engine, a voice input;

determining, by the dialog engine, that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer to the question;

identifying, by the dialog engine, a predetermined feedback score that reflects an extent to which, in response to other questions, other users express satisfaction or dissatisfaction when speaking a particular term that is included in the received voice input;

adjusting, based at least on the predetermined feedback score that is associated with a transcription of the received voice input, a confidence score that, based on prior selections of the provided answer to the question by other users, indicates an estimation of relevance of the provided answer to the question; and

in response to a subsequent receipt of the question, selecting, by the answer generation engine, the answer or a different answer based at least on the adjusted confidence score.

2. The method of claim 1, wherein the predetermined feedback score is determined after determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer to the question.

3. The method of claim 1, wherein the predetermined feedback score is determined before determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer to the question.

4. The method of claim 1, comprising:

before providing the answer to the question;

receiving an initial voice input that specifies the question;

determining a confidence score associated with a question-answer pair that comprises (i) the question, and (ii) the provided answer to the question; and

determining to provide the provided answer to the question based on the confidence score associated with the question-answer pair.

5. The method of claim 1, wherein determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer comprises:

determining an amount of time that elapses between providing the answer to the question and receiving the voice input; and

determining that the amount of time that elapses between providing the answer to the question and receiving the voice input is less than a predetermined amount of time.

6. The method of claim 1, wherein determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer comprises:

determining (i) a semantic measure associated with the question, and (ii) a semantic measure associated with the provided answer to the question; and

determining that (i) the semantic measure associated with the question, and (ii) the semantic measure associated with the provided answer to the question, satisfy a predetermined semantic measure similarity threshold.

7. The method of claim 1, wherein determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer comprises:

identifying an action performed by a user of the computing device after the voice input was received.

8. The method of claim 7, wherein the action performed by the user of the computing device includes dialing a telephone number that is associated with the answer, or composing an email to an email address that is associated with the answer.

9. The method of claim 1, wherein identifying a predetermined feedback score that is associated with the transcription of the received voice input further comprises:

normalizing the transcription of the received voice input; and

identifying the predetermined feedback score that is associated with the normalized transcription.

10. The method of claim 1, further:

after adjusting the confidence score that is associated with the provided answer to the question;

identifying a different question-answer pair that (I) comprises (i) the question, and (ii) a different answer to the question, and (II) is associated with a different confidence score that is higher than the confidence score that is associated with the question-answer pair that comprises (i) the question, and (ii) the provided answer to the question; and

providing, to the computing device, the different answer to the question.

11. The method of claim 1, wherein the subsequent receipt of the question is based on a question that originated with a user that is different than the user that originated the received voice input.

12. A system comprising:

a non-transitory computer readable medium having instructions stored thereon; and

data processing apparatus programmed to execute the instructions to perform operations comprising;

providing, by a dialog engine, an answer to a question;

after providing the answer to the question, receiving, by the dialog engine, a voice input;

determining, by the dialog engine, that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer to the question;

identifying, by the dialog engine, a predetermined feedback score that reflects an extent to which, in response to other questions, other users express satisfaction or dissatisfaction when speaking a particular term that is included in the received voice input;

adjusting, based at least on the predetermined feedback score that is associated with a transcription of the received voice input, a confidence score that, based on prior selections of the provided answer to the question by other users, indicates an estimation of relevance of the provided answer to the question; and

in response to a subsequent receipt of the question, selecting, by the answer generation engine, the answer or a different answer based at least on the adjusted confidence score.

13. The system of claim 12, wherein the predetermined feedback score is determined after determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer to the question.

14. The system of claim 12, wherein the predetermined feedback score is determined before determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer to the question.

15. The system of claim 12, wherein the operations comprise:

before providing the answer to the question;

receiving an initial voice input that specifies the question;

determining a confidence score associated with a question-answer pair that comprises (i) the question, and (ii) the provided answer to the question; and

determining to provide the provided answer to the question based on the confidence score associated with the question-answer pair.

16. The system of claim 12, wherein determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer comprises:

determining an amount of time that elapses between providing the answer to the question and receiving the voice input; and

determining that the amount of time that elapses between providing the answer to the question and receiving the voice input is less than a predetermined amount of time.

17. The system of claim 12, wherein determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer comprises:

determining (i) a semantic measure associated with the question, and (ii) a semantic measure associated with the provided answer to the question; and

determining that (i) the semantic measure associated with the question, and (ii) the semantic measure associated with the provided answer to the question, satisfy a predetermined semantic measure similarity threshold.

18. The system of claim 12, wherein determining that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer comprises:

identifying, after the voice input was received, an action performed by a user of a computing device that includes the dialog engine.

19. The system of claim 12, wherein identifying a predetermined feedback score that is associated with the transcription of the received voice input further comprises:

normalizing the transcription of the received voice input; and

identifying the predetermined feedback score that is associated with the normalized transcription.

20. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:

providing, by a dialog engine, an answer to a question;

after providing the answer to the question, receiving, by the dialog engine, a voice input;

determining, by the dialog engine, that the received voice input is classified as feedback that indicates satisfaction or dissatisfaction with the provided answer to the question;

identifying, by the dialog engine, a predetermined feedback score that reflects an extent to which, in response to other questions, other users express satisfaction or dissatisfaction when speaking a particular term that is included in the received voice input;

adjusting, based at least on the predetermined feedback score that is associated with a transcription of the received voice input, a confidence score that, based on prior selections of the provided answer to the question by other users, indicates an estimation of relevance of the provided answer to the question; and

in response to a subsequent receipt of the question, selecting, by the answer generation engine, the answer or a different answer based at least on the adjusted confidence score.