Scalable curation system

• US 9,582,757 B1
• Filed: 07/25/2014
• Issued: 02/28/2017
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Holmes, Petra Elisabeth, Hopkinson, Andrew, Copland, Neil, Fenn, Andrew Graham, Palfrey, David Spike, Tunstall-Pedoe, William
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US14/341,595
• Time in Patent Office: 949 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A system and method is disclosed for improving automated question-answering using real-world knowledge from a knowledge base. The system can be used to answer questions from various users. The system can generate answers to these questions using data stored in a knowledge base. In some embodiments, the system is provided with a list of questions separate from any user questions. The system can be trained using these questions in advance of a user question. The system can process the presented questions to determine that it can generate high-quality correct answers. The system can take various steps to determine a high-quality answer to a question. The system can utilize human intelligence providers to improve this process, such as through a human interactive task system. Human intelligence can be used to determine that a question is understood, that a question is answered, and that the answer is of high quality.

Claims

1. A system comprising:

a computer-readable memory storing executable instructions; and

one or more processors in communication with the computer-readable memory, wherein the one or more processors are programmed by the executable instructions to at least;

obtain a question for processing;

determine that the system is not able to understand the question;

take a first action to enable the system to understand the question;

determine that the system is not able to determine an initial answer to the question utilizing information stored in a knowledge base;

take a second action to enable the system to determine the initial answer;

determine the initial answer;

determine that a quality level associated with the initial answer is less than a threshold; and

take a third action to enable the system to determine a new answer having a higher quality level than the quality level of the initial answer.

2. The system of claim 1, wherein the determination that the system is not able to understand the question comprises one or more of:

determining that the question does not match a template, determining that a part of the question does not match a template, determining that the system does not understand an entity in the question, determining that the system does not understand a relationship in the question, or determining that the system is unable to choose between two or more possible understandings.

3. The system of claim 1, wherein the first action comprises one or more of:

identifying alternative strings related to an entity in the question, substituting a known entity in the question to determine that the system can answer the question using the known entity, determining a similar question to determine that the system can answer the similar question, or sending the question to a human interactive task system.

4. The system of claim 1, wherein the second action comprises one or more of:

breaking the question into one or more related questions and answering the one or more related questions utilizing information stored in the knowledge base, performing natural language processing on a text source to obtain additional information, or sending the question to a human interactive task system.

5. The system of claim 1, wherein the quality level is determined by one or more of:

receiving information regarding the quality level of the initial answer from a human interactive task system, receiving information regarding the quality level of the initial answer from a user, determining that the initial answer is factually incorrect, determining that the initial answer contains unnecessary information, determining that the initial answer does not include necessary information, determining that the initial answer is not grammatically correct, determining that the initial answer contains duplicative information, or determining that the initial answer is not in an expected format.

6. The system of claim 1, wherein the third action comprises one or more of:

utilizing information stored in an external database, performing natural language processing on a text source to obtain additional information, removing unnecessary information from the initial answer, adding necessary information to the initial answer, or sending information to a human interactive task system.

7. The system of claim 1, wherein the second action comprises receiving a request from a user to determine the initial answer using a human interactive task system.

8. The system of claim 7, wherein the second action further comprises sending a notification to the user after the system receives the new answer from the human interactive task system.

9. A computer-implemented method comprising:

under control of one or more computing devices configured with specific computer-executable instructions,

obtaining a question for processing;

determining that the one or more computing devices are not able to understand the question for processing;

taking a first action to enable the one or more computing devices to understand the question;

determining that the one or more computing devices are not able to answer the question;

taking a second action to determine the answer;

determining the answer;

determining that a quality level associated with the answer is less than a threshold; and

taking a third action to determine a new answer having a higher quality level than the quality level of the answer.

10. The one or more non-transitory computer readable media of claim 9, wherein determining that the one or more computing devices are not able to answer the question for processing comprises one or more of:

determining that the question does not match a template, determining that a part of the question does not match a template, determining that the system does not understand an entity in the question, determining that the system does not understand a relationship in the question, or determining that the system is unable to choose between two or more possible understandings of the question.

11. The one or more non-transitory computer readable media of claim 9, wherein the first action comprises one or more of:

identifying alternative strings related to an entity in the question, substituting a known entity in the question to determine that the system can answer the question using the known entity, determining a similar question to determine whether the system can answer the similar question, or sending the question to a human interactive task system.

12. The one or more non-transitory computer readable media of claim 9, wherein the second action comprises one or more of:

breaking the question into one or more related questions and answering the one or more related questions utilizing information stored in the knowledge base, performing natural language processing on a text source to obtain additional information, or sending the question to a human interactive task system.

13. The one or more non-transitory computer readable media of claim 9, wherein the quality level is determined by one or more of:

receiving information regarding the quality level of the answer from a human interactive task system, receiving information regarding the quality level of the answer from a user, determining that the answer is factually incorrect, determining that the answer contains unnecessary information, determining that the answer does not include necessary information, determining that the answer is not grammatically correct, determining that the answer contains duplicative information, or determining that the answer is not in an expected format.

14. The one or more non-transitory computer readable media of claim 9, wherein the third action comprises one or more of:

utilizing information stored in an external database, performing natural language processing on a text source to obtain additional information, removing unnecessary information from the answer, adding necessary information to the answer, or sending information to a human interactive task system.

15. The one or more non-transitory computer readable media of claim 9, wherein the process further comprises providing the improved answer to a user.

17. The one or more non-transitory computer readable media of claim 14, wherein determining whether the one or more computing devices are able to process a question comprises one or more of:

determining that the question does not match a template, determining that a part of the question does not match a template, determining that the system does not recognize an entity in the question, determining that the system cannot process a relationship in the question, or determining that the one or more computing devices are unable to choose between two or more possible interpretations of the question.

16. One or more non-transitory computer readable media comprising executable code that, when executed, cause one or more computing devices to perform a process comprising:

determining that the one or more computing devices are not able to understand a question;

taking a first action to enable the one or more computing devices to understand the question;

determining that the one or more computing devices are not able to answer the question;

taking a second action to enable the one or more computing devices to answer the question;

determining the answer;

determining a quality level associated with the answer; and

taking a third action to enable the one or more computing devices to determine a new answer having a higher quality level than the quality level of the answer.

18. The one or more non-transitory computer readable media of claim 16, wherein the first action comprises one or more of:

identifying alternative strings related to an entity in the question, substituting a known entity in the question to determine that the system can answer the question using the known entity, determining a similar question to determining that the system can answer the similar question, or sending the question to a human interactive task system.

19. The one or more non-transitory computer readable media of claim 16, wherein the second action comprises one or more of:

breaking the question into one or more related questions and answering the one or more related questions utilizing information stored in the knowledge base, performing natural language processing on a text source to obtain additional information, or sending the question to a human interactive task system.

20. The one or more non-transitory computer readable media of claim 16, wherein increasing the quality level of the answer comprises one or more of:

utilizing information stored in an external database, performing natural language processing on a text source to obtain additional information, removing unnecessary information from the answer, adding necessary information to the answer, or sending information to a human interactive task system.

21. The one or more non-transitory computer readable media of claim 16, wherein the question is obtained from a user.

22. The one or more non-transitory computer readable media of claim 21, wherein the process further comprises providing the new answer to a user.

23. The one or more non-transitory computer readable media of claim 16, wherein the question is obtained from a set of questions stored in the memory of the one or more computing devices.