Automated learning from a question and answering network of humans

• US 7,809,664 B2
• Filed: 12/21/2007
• Issued: 10/05/2010
• Est. Priority Date: 12/21/2007
• Status: Expired due to Fees

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Heck, Larry P.
• Primary Examiner(s): Trujillo; James
• Assistant Examiner(s): Casanova; Jorge A
• Application Number:
  US11/962,766
• Publication Number:
  US 20090162824A1
• Time in Patent Office: 1,019 Days
• Field of Search: 704/7, 704/10, 706/45, 706/46, 706/48, 707 3- 6, 707/706, 707/721, 707/751, 707/999.003, 707/999.004, 707/999.005, 707/999.006, 707/739, 707/748, 709/204
• US Class Current: 706/46
• CPC Class Codes: G06N 3/004 Artificial life, i.e. computing arrangements simulating life
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

A QA robot learns how to answer questions by observing human interaction over online social networks. The QA robot observes the way people ask questions and how other users respond to those questions. In addition, the QA robot observes which questions are most helpful and analyzes those questions to identify the characteristics of those questions that are most helpful. The QA robot then uses those observations to enhance the way it answers questions in the future.

Claims

1. A method comprising:

extracting, by a question answering (QA) system, a set of features from a current question;

performing, by the QA system, a comparison between (a) said set of features from said current question and (b) a set of features from one or more previously-submitted questions;

selecting, based on the comparison, an answer to said current question from a set of answers to said one or more previously-submitted questions;

displaying said answer in conjunction with a set of user-submitted answers to said current question;

receiving user input that rates at least one user-submitted answer of said user-submitted answers higher than said answer;

identifying one or more characteristics of the at least one user-submitted answer that are different from said answer;

based on the one or more characteristics, modifying one or more elements of the QA system; and

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein extracting said set of features includes:

matching the current question to an element in a grammar; and

based on the features defined in the element in the grammar, extracting features from the current question.

3. The method of claim 2, wherein the element in the grammar is a question template.

4. The method of claim 1, wherein performing the comparison includes comparing the set of features from the current question to a set of index values associated with a knowledgebase of said QA system.

5. The method of claim 1, wherein selecting an answer to said current question from said set of answers includes:

computing a confidence value for the answer; and

determining if said confidence value exceeds a specified threshold.

6. The method of claim 5, wherein the answer is displayed in conjunction with said set of user-submitted answers only if the confidence value exceeds a specified threshold.

7. The method of claim 1, wherein receiving user input that rates at least one user-submitted answer of said user-submitted answers higher than said answer includes receiving user input that indicates a score for the at least one user-submitted answer and said answer.

8. The method of claim 1, further comprising adding said at least one user submitted answer to a knowledge store.

9. The method of claim 1, wherein:

the one or more elements of the QA system include a QA router that, for each question of a plurality of questions, routes said each question, based on features extracted from said each question, to one of a plurality of QA experts; and

modifying the one or more elements of the QA system includes modifying routing information of the QA router.

10. The method of claim 1, wherein:

the one or more elements of the QA system include a grammar that comprises a set of template questions; and

modifying one or more elements of the QA system includes modifying at least one template question from the set of template questions.

11. The method of claim 1, wherein:

the one or more elements of the QA system include a plurality of QA experts, each of which is configured to select an answer to a given question; and

modifying one or more elements of the QA system includes modifying information used by one or more of the plurality of QA experts to answer subsequent questions.

12. One or more machine-readable storage medium storing instructions which, when executed by one or more processors, cause the performance of:

extracting, by a question answering (QA) system, a set of features from a current question;

performing, by the QA system, a comparison between (a) said set of features from said current question and (b) a set of features from one or more previously-submitted questions;

selecting, based on the comparison, an answer to said current question from a set of answers to said one or more previously-submitted questions;

displaying said answer in conjunction with a set of user-submitted answers to said current question;

receiving user input that rates at least one user-submitted answer of said user-submitted answers higher than said answer;

identifying one or more characteristics of the at least one user-submitted answer that are different from said answer; and

based on the one or more characteristics, modifying one or more elements of the QA system.

13. The one or more machine-readable storage media of claim 12, wherein extracting said set of features includes:

matching the current question to an element in a grammar; and

based on the features defined in the element in the grammar, extracting features from the current question.

14. The one or more machine-readable storage media of claim 13, wherein the element in the grammar is a question template.

15. The one or more machine-readable storage media of claim 12, wherein performing the comparison includes comparing the set of features from the current question to a set of index values associated with a knowledgebase of said question answering component.

16. The one or more machine-readable storage media of claim 12, wherein selecting an answer to said current question from said set of answers includes:

computing a confidence value for the answer; and

determining if said confidence value exceeds a specified threshold.

17. The one or more machine-readable storage media of claim 16, wherein the answer is displayed in conjunction with said set of user-submitted answers only if the confidence value exceeds a specified threshold.

18. The one or more machine-readable storage media of claim 12, wherein receiving user input that rates at least one user-submitted answer of said user-submitted answers higher than said answer includes receiving user input that indicates a score for the at least one user-submitted answer and said answer.

19. The one or more machine-readable storage media of claim 12, wherein the instructions, when executed by the one or more processors, further cause adding said at least one user-submitted answer to a knowledge store.

20. The one or more machine-readable storage media of claim 12, wherein:

the one or more elements of the QA system include a QA router that, for each question of a plurality of questions, routes said each question, based on features extracted from said each question, to one of a plurality of QA experts; and

modifying the one or more elements of the QA system includes modifying routing information of the QA router.

21. The one or more machine-readable storage media of claim 12, wherein:

the one or more elements of the QA system include a grammar that comprises a set of template questions; and

modifying one or more elements of the QA system includes modifying at least one template question from the set of template questions.

22. The one or more machine-readable storage media of claim 12, wherein:

the one or more elements of the QA system include a plurality of QA experts, each of which is configured to select an answer to a given question; and

modifying one or more elements of the QA system includes modifying information used by one or more of the plurality of QA experts to answer subsequent questions.