Determining question and answer alternatives

• US 9,336,269 B1
• Filed: 03/14/2013
• Issued: 05/10/2016
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mihaila, George Andrei, Smith, David, Sahin, Engin Cinar
• Primary Examiner(s): Singh, Amresh
• Application Number:
  US13/803,628
• Time in Patent Office: 1,153 Days
• Field of Search: 707/723
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A computer-implemented method can include identifying one or more candidate topics from a query. The method can generate, for each candidate topic, a candidate topic-answer pair that includes both the candidate topic and an answer to the query for the candidate topic. The method can obtain search results based on the query, wherein one or more of the search results references an annotated resource. For each candidate topic-answer pair, the method can determine a score for the candidate topic-answer pair for use in determining a response to the query, based on (i) an occurrence of the candidate topic in the annotations of the resources referenced by one or more of the search results, and (ii) an occurrence of the answer in annotations of the resources referenced by the one or more search results, or in the resources referenced by the one or more search results.

Claims

1. A computer-implemented method comprising:

identifying one or more candidate topics from a query;

generating, for each candidate topic, a candidate topic-answer pair that includes (i) the candidate topic, and (ii) an answer to the query for the candidate topic;

obtaining search results based on the query, wherein one or more of the search results references an annotated resource, wherein an annotated resource is a resource that, based on an automated evaluation of the content of the resource, is associated with an annotation that identifies one or more likely topics associated with the resource;

for each candidate topic-answer pair, determining a score for the candidate topic-answer pair based on (i) an occurrence of the candidate topic in the annotations of the resources referenced by one or more of the search results, and (ii) an occurrence of the answer in annotations of the resources referenced by the one or more search results, or in the resources referenced by the one or more search results; and

determining whether to respond to the query with one or more answers from the candidate topic-answer pairs, based on the scores.

2. The computer-implemented method of claim 1, wherein

determining whether to respond to the query comprises comparing one or more of the scores to a predetermined threshold;

and

the method further comprises selecting, based on the comparison of the one or more scores to the predetermined threshold, one or more answers to respond to the query from among the candidate topic-answer pairs.

3. The computer-implemented method of claim 1, wherein

determining whether to respond to the query comprises comparing one or more of the scores to a predetermined threshold;

and

the method further comprises determining, based on the comparison of the one or more scores to the predetermined threshold, to not respond to the query with an answer from among the candidate topic-answer pairs.

4. The computer-implemented method of claim 1, further comprising selecting, based on the scores, one or more answers to respond to the query from among the candidate topic-answer pairs.

5. The computer-implemented method of claim 4, wherein the one or more selected answers includes an answer from the candidate topic-answer pair that has a highest score among the scores of each of the candidate topic-answer pairs.

6. The computer-implemented method of claim 4, further comprising providing the selected one or more answers to a user at a client computing device.

7. The computer-implemented method of claim 1, wherein each candidate topic is represented by a node in a graph of interconnected nodes that each represents a known topic.

8. The computer-implemented method of claim 7, wherein generating, for each candidate topic, a candidate topic-answer pair that includes (i) the candidate topic, and (ii) an answer to the query for the candidate topic, comprises identifying an attribute value from the node that represents each candidate topic, respectively.

9. The computer-implemented method of claim 7, wherein the answer in one or more of the candidate topic-answer pairs is represented by a node in the graph of interconnected nodes.

10. The computer-implemented method of claim 1, wherein, for one or more of the candidate topic-answer pairs, the score is further based on a respective query relevance score of the search results that include annotations in which the candidate topic occurs.

11. The computer-implemented method of claim 1, wherein, for one or more of the candidate topic-answer pairs, the score is further based on a confidence measure associated with (i) each of one or more annotations in which the candidate topic in a respective candidate topic-answer pair occurs, or (ii) each of one or more annotations in which the answer in a respective candidate topic-answer pair occurs.

12. The computer-implemented method of claim 1, wherein:

the obtained search results are ranked according to a determined relevance of each of the search results to the query; and

for each candidate topic-answer pair, the score for the candidate topic-answer pair is determined further based on the determined relevance to the query of at least one search result from the obtained search results that references an annotated resource for which the candidate topic or the answer from the candidate topic-answer pair occurs in the annotation for the annotated resource.

13. The computer-implemented method of claim 1, wherein, for each candidate topic-answer pair, the score for the candidate topic-answer pair is determined further based on a number of the search results that reference an annotated resource for which the candidate topic or the answer from the candidate topic-answer pair occurs in the annotation for the annotated resource.

14. The computer-implemented method of claim 1, wherein the search results are identified separately from the candidate topics and the answers in the candidate topic-answer pairs.

15. A computer system comprising:

one or more computing devices;

an interface at the one or more computing devices that is programmed to receive a query;

a knowledge repository that is accessible to the one or more computing devices and that includes a plurality of topics, each topic including one or more attributes and associated values for the attributes;

a mapping module that is installed on the one or more computing devices and that identifies one or more candidate topics from the topics in the knowledge repository, wherein the identified candidate topics are determined to relate to a possible subject of the query;

an answer generator that is installed on the one or more computing devices and that generates, for each candidate topic, a candidate topic-answer pair that includes (i) the candidate topic, and (ii) an answer to the query for the candidate topic, wherein the answer for each candidate topic is identified from information in the knowledge repository;

a search engine that is installed on the one or more computing devices and that returns search results based on the query, wherein one or more of the search results references an annotated resource, wherein an annotated resource is a resource that, based on an automated evaluation of the content of the resource, is associated with an annotation that identifies one or more likely topics associated with the resource;

a scoring module that is installed on the one or more computing devices and that determines a score for each candidate topic-answer pair based on (i) an occurrence of the candidate topic in the annotations of the resources referenced by one or more of the search results, and (ii) an occurrence of the answer in annotations of the resources referenced by the one or more search results, or in the resources referenced by the one or more search results; and

a front end system at the one or more computing devices that determines whether to respond to the query with one or more answers from the candidate topic-answer pairs, based on the scores.

16. The computer system of claim 15, wherein the front end system determines whether to respond to the query based on a comparison of one or more of the scores to a predetermined threshold.

17. The computer system of claim 15, wherein each of the plurality of topics that is included in the knowledge repository is represented by a node in a graph of interconnected nodes.

18. The computer system of claim 15, wherein one or more of the returned search results is associated with a respective query relevance score and the score determined by the scoring module for each candidate topic-answer pair is further based on the query relevance scores of one or more of the search results that reference an annotated resource in which the candidate topic occurs.

19. The computer system of claim 15, wherein, for one or more of the candidate topic-answer pairs, the score is further based on a confidence measure associated with (i) each of one or more annotations in which the candidate topic in a respective candidate topic-answer pair occurs, or (ii) each of one or more annotations in which the answer in a respective candidate topic-answer pair occurs.

20. A tangible computer-readable storage device having instructions stored thereon that, when executed by one or more computer processors, cause the processors to perform operations comprising:

identifying one or more candidate topics from a query;

generating, for each candidate topic, a candidate topic-answer pair that includes (i) the candidate topic, and (ii) an answer to the query for the candidate topic;

obtaining search results based on the query, wherein one or more of the search results references an annotated resource, wherein an annotated resource is a resource that, based on an automated evaluation of the content of the resource, is associated with an annotation that identifies one or more likely topics associated with the resource;

for each candidate topic-answer pair, determining a score for the candidate topic-answer pair based on (i) an occurrence of the candidate topic in the annotations of the resources referenced by one or more of the search results, and (ii) an occurrence of the answer in annotations of the resources referenced by the one or more search results, or in the resources referenced by the one or more search results; and

determining whether to respond to the query with one or more answers from the candidate topic-answer pairs, based on the scores.

21. The tangible computer-readable storage device of claim 20, wherein determining whether to respond to the query includes comparing one or more of the scores to a predetermined threshold.

22. The tangible computer-readable storage device of claim 20, wherein the operations further comprise determining a query relevance score for one or more of the returned search results, and wherein the score for one or more of the candidate topic-answer pairs is further based on the query relevance scores of one or more of the search results that reference an annotated resource in which the candidate topic occurs.

23. The tangible computer-readable storage device of claim 20, wherein for one or more of the candidate topic-answer pairs, the score is further based on a confidence measure associated with (i) each of one or more annotations in which the candidate topic in a respective candidate topic-answer pair occurs, or (ii) each of one or more annotations in which the answer in a respective candidate topic-answer pair occurs.