Training a dialog system using user feedback for answers to questions
First Claim
Patent Images
1. A computer-implemented method comprising:
- receiving, at a dialog engine on a mobile device, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user;
in response to receiving the voice query;
identifying, by the dialog engine, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query;
providing, by the dialog engine, a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query;
receiving, at the dialog engine, feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query;
determining, by the dialog engine, whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and
when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, by the dialog engine, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query.
2 Assignments
0 Petitions
Accused Products
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a dialog system using user feedback. According to one implementation, a method includes receiving, by a dialog engine and from a computing device, a voice input; determining, by the dialog engine, that the received voice input is classified as feedback to an answer of a question; identifying, by the dialog engine, a predetermined feedback score associated with the feedback; and adjusting a confidence score associated with the question and the answer based on the predetermined feedback score.
-
Citations
20 Claims
-
1. A computer-implemented method comprising:
-
receiving, at a dialog engine on a mobile device, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user; in response to receiving the voice query; identifying, by the dialog engine, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query; providing, by the dialog engine, a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query; receiving, at the dialog engine, feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query; determining, by the dialog engine, whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, by the dialog engine, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising; receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user; in response to receiving the voice query; identifying, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query; providing a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query; receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query. - View Dependent Claims (9, 10, 11, 12, 13)
-
14. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising:
-
receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user; in response to receiving the voice query; identifying a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query; providing a first answer to the voice query for output using a speaker of the system, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query; receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query. - View Dependent Claims (15, 16, 17, 18, 19, 20)
-
Specification