Question answering using entity references in unstructured data

• US 10,339,190 B2
• Filed: 08/30/2016
• Issued: 07/02/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Keysar, Dvir, Shmiel, Tomer
• Primary Examiner(s): Beausoliel, Jr., Robert W
• Assistant Examiner(s): Santos, Pedro J
• Application Number:
  US15/251,989
• Publication Number:
  US 20160371385A1
• Time in Patent Office: 1,036 Days
• Field of Search: 707723, 707728, 707748, 707731, 707727, 707707, 707730
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Methods, systems, and computer-readable media are provided for collective reconciliation. In some implementations, a query is received, wherein the query is associated at least in part with a type of entity. One or more search results are generated based at least in part on the query. Previously generated data is retrieved associated with at least one search result of the one or more of search results, the data comprising one or more entity references in the at least one search result corresponding to the type of entity. The one or more entity references are ranked, and an entity result is selected from the one or more entity references based at least in part on the ranking. An answer to the query is provided based at least in part on the entity result.

Claims

1. A method performed by one or more processors comprising:

receiving a query, the query being a natural language query that is received based on user input at a computing device;

obtaining search results for the query and a ranking of the search results, the search results generated based at least in part on the query and the ranking of the search results being based on relevance of the search results to the query;

identifying that the query is associated with a particular type of entity, wherein the particular type of entity defines a particular categorization that encompasses multiple entities, and wherein identifying that the query is associated with the particular type of entity comprises identifying the particular type of entity, from a plurality of candidate types of entities, based on at least one term of the query;

retrieving data comprising one or more entity references that occur in content of one or more of the search results, wherein retrieving the data comprises;

retrieving the entity references, that occur in the content of one or more of the search results, based on the entity references being identified as the particular type of entity, and based on the particular type of entity having been identified based on the at least one term of the query, and

retrieving the entity references based on their occurrence in the content of the one or more of the search results and based on the one or more of the search results being obtained for the query;

determining a ranking signal for each of the entity references;

ranking the entity references based on the determined ranking signals;

selecting an entity result from the entity references based at least in part on the ranking;

generating a natural language answer to the query based on the selected entity result and based on the query, wherein generating the natural language answer to the query based on the selected entity result and based on the query comprises including at least one query term, from the query, in the natural language answer and including at least one entity term, for the entity result, in the natural language answer; and

providing the natural language answer to the query for presentation at the computing device, wherein providing the natural language answer to the query comprises providing the answer for presentation in place of the search results.

2. The method of claim 1, further comprising:

determining, based on the ranking of the search results, a subset of the search results that are above a first ranking threshold;

wherein retrieving the entity references further comprises retrieving the entity references based on the entity references corresponding to the subset of the search results that are above the first ranking threshold.

3. The method of claim 1, wherein determining the ranking signal for each of the entity references comprises determining a weighted sum of the frequencies of occurrence of each respective entity reference.

4. The method of claim 1, wherein the particular type of entity is one of a person type, a location type, or a date type.

5. The method of claim 4, wherein the particular type of entity is the location type.

6. The method of claim 1, further comprising:

in response to determining that the ranking of the one or more entity references satisfies one or more criteria, reranking the one or more entity references;

wherein selecting the entity result is based at least in part on the reranking of the one or more entity references.

7. The method of claim 6, wherein determining that the ranking of the one or more entity references satisfies one or more criteria comprises:

determining that the ranking of the one or more entity references results in a tie.

8. The method of claim 1, wherein determining the ranking signal for each of the entity references comprises determining a topicality score of each respective entity reference, wherein the topicality score for a given entity reference and a document that contains the given entity reference is based on a relationship between the given entity reference and the document.

9. A system comprising:

one or more computers configured to perform operations comprising;

receiving a query, the query being a natural language query that is received based on user input at a computing device;

obtaining search results for the query and a ranking of the search results, the search results generated based at least in part on the query and the ranking of the search results being based on relevance of the search results to the query;

identifying that the query is associated with a particular type of entity, wherein the particular type of entity defines a particular categorization that encompasses multiple entities, and wherein identifying that the query is associated with the particular type of entity comprises identifying the particular type of entity, from a plurality of candidate types of entities, based on at least one term of the query;

retrieving data comprising one or more entity references that occur in content of one or more of the search results, wherein retrieving the data comprises;

retrieving the entity references based on the entity references being identified as the particular type of entity, and based on the particular type of entity having been identified based on the at least one term of the query, and

retrieving the entity references based on their occurrence in the content of the one or more of the search results and based on the one or more of the search results being obtained for the query;

determining a ranking signal for each of the entity references;

ranking the entity references based on the determined ranking signals;

selecting an entity result from the entity references based at least in part on the ranking; and

generating a natural language answer to the query based on the selected entity result and based on the query, wherein generating the natural language answer to the query based on the selected entity result and based on the query comprises including at least one query term, from the query, in the natural language answer and including at least one entity term, for the entity result, in the natural language answer;

providing the natural language answer to the query for presentation at the computing device, wherein providing the natural language answer to the query comprises providing the answer for display along with the search results, and positioned above the search results.

10. The system of claim 9, wherein the operations performed by the one or more computers further comprise:

determining, based on the ranking of the search results, a subset of the search results that are above a first ranking threshold;

wherein retrieving the entity references further comprises retrieving the entity references based on the entity references corresponding to the subset of the search results that are above the first ranking threshold.

11. The system of claim 9, wherein determining the ranking signal for each of the entity references comprises determining a weighted sum of the frequencies of occurrence of each respective entity reference.

12. The system of claim 9, wherein the particular type of entity is one of a person type, a location type, or a date type.

13. The system of claim 12, wherein the particular type of entity is the location type.

14. The system of claim 9, wherein the operations performed by the one or more computers further comprise:

in response to determining that the ranking of the one or more entity references satisfies one or more criteria, reranking the one or more entity references;

wherein selecting the entity result is based at least in part on the reranking of the one or more entity references.

15. The system of claim 14, wherein determining that the ranking of the one or more entity references satisfies one or more criteria comprises:

determining that the ranking of the one or more entity references results in a tie.

16. The system of claim 9, wherein determining the ranking signal for each of the entity references comprises determining a topicality score of each respective entity reference, wherein the topicality score for a given entity reference and a document that contains the given entity reference is based on a relationship between the given entity reference and the document.

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving a query, the query being a natural language query that is received based on user input at a computing device;

obtaining search results for the query and a ranking of the search results, the search results generated based at least in part on the query and the ranking of the search results being based on relevance of the search results to the query;

identifying that the query is associated with a particular type of entity, wherein the particular type of entity defines a particular categorization that encompasses multiple entities, and wherein identifying that the query is associated with the particular type of entity comprises identifying the particular type of entity, from a plurality of candidate types of entities, based on at least one term of the query;

retrieving data comprising one or more entity references that occur in content of one or more of the search results, wherein retrieving the data comprises;

retrieving the entity references based on the entity references being identified as the particular type of entity, and based on the particular type of entity having been identified based on the at least one term of the query, and

retrieving the entity references based on their occurrence in the content of the one or more of the search results and based on the one or more of the search results being obtained for the query;

determining a ranking signal for each of the entity references;

ranking the entity references based on the determined ranking signals;

selecting an entity result from the entity references based at least in part on the ranking;

generating a natural language answer to the query based on the selected entity result and based on the query; and

providing the answer to the query for presentation at the computing device, wherein providing the answer to the query comprises providing the answer for presentation in place of the search results.

18. The non-transitory computer-readable medium of claim 17, wherein the operations performed by the one or more processors further comprise:

determining, based on the ranking of the search results, a subset of the search results that are above a first ranking threshold;

wherein retrieving the entity references further comprises retrieving the entity references based on the entity references corresponding to the subset of the search results that are above the first ranking threshold.

19. The non-transitory computer-readable medium of claim 17, wherein determining the ranking signal for each of the entity references comprises determining a weighted sum of the frequencies of occurrence of each respective entity reference.