Automatic annotation for training and evaluation of semantic analysis engines

• US 9,224,103 B1
• Filed: 03/13/2013
• Issued: 12/29/2015
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Subramanya, Amarnag, Pereira, Fernando
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Sobkowski, Isidore
• Application Number:
  US13/801,197
• Time in Patent Office: 1,021 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Implementations include systems and methods generate data for training or evaluating semantic analysis engines. For example, a method may include receiving documents from a corpus that includes an authoritative set of documents from an authoritative source. Each document in the authoritative set may be associated with an entity. A second set of documents from the corpus that do not overlap with the first set may include at least one link to a document in the authoritative set, the at least one link being associated with anchor text. For each document in the second set, the method may include identifying entity mentions in the document based on the anchor text. The method may include associating the entity mention with the entity in a graph-structured knowledge base or associating entity types with the entity mention. The method may also include training a semantic analysis engine using the identified entity mentions and associations.

Claims

1. A computer system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, causes the computer system to perform operations comprising;

receiving documents from a corpus, the corpus comprising;

an authoritative set of documents from an authoritative source, each document in the authoritative set being associated with an entity, and

a second set of documents, the second set being documents that are not in the authoritative set and that are not copies of documents in the authoritative set but that each include at least one link to a document in the authoritative set, the at least one link being associated with anchor text,

identifying, for each document in the second set, entity mentions in the document based on the anchor text, each entity mention including the anchor text and an identifier of the linked-to authoritative document,

associating the identified entity mentions with respective entity types based on content in the linked-to authoritative document, and

training an entity tagging engine using the identified entity mentions and the entity types associated with the entity mentions.

2. The system of claim 1, wherein entity mentions are identified by matching a token in the anchor text with a token in a title of the linked-to authoritative document.

3. The system of claim 2, wherein entity mentions are further identified by matching a token in the anchor text with an alias for the linked-to authoritative document, the alias being defined by the authoritative source.

4. The system of claim 3, wherein alias is determined by a set of anchor texts within documents from the authoritative source linking to the linked-to authoritative document.

5. The system of claim 1, wherein the instructions, when executed by the at least one processor, further cause the computer system to perform the operations of:

computing a fingerprint for sentences in each document of the authoritative set;

computing a fingerprint for sentences in a particular document from the second set;

comparing the fingerprints of the particular document with the fingerprints for the authoritative set; and

removing particular document from the second set when a majority of fingerprints match.

6. The system of claim 1, wherein the entity type associated with a particular entity mention in a document in the second set is selected from categories associated with the linked-to authoritative document.

7. The system of claim 1, wherein the instructions, when executed by the at least one processor, further cause the computer system to perform the operation of extracting free text from the documents in the second set, so that entity mentions are identified from the free text.

8. The system of claim 1, wherein the identified entity mentions and associated entity types are further associated with respective offsets within the documents.

9. A computer system comprising:

at least one processor;

memory storing a graph-structured knowledge base; and

memory storing instructions that, when executed by at least one processor, causes the computer system to perform operations comprising;

accessing an authoritative set of documents from a corpus of documents, the authoritative set of documents being from an authoritative source, each document in the authoritative set being associated with a respective entity in the graph-structured knowledge base,

identifying a second set of documents from the corpus of documents, the second set being documents that are not in the authoritative set and that are not copies of documents in the authoritative set but that each include at least one hyperlink to a document in the authoritative set, the at least one hyperlink being associated with anchor text,

for each document in the second set;

identifying an entity mention in the document based on the anchor text, the entity mention including the anchor text and an identifier of the linked-to authoritative document, and

associating the entity mention with the entity in the graph-structured knowledge base associated with the linked-to authoritative document, and

training an entity matching engine using the identified entity mentions and associated entities.

10. The system of claim 9, wherein the entity mention is stored with an offset within the document for respective entity mentions.

11. The system of claim 10, wherein the offset is used to determine a context of the entity mention, the context being used in training the entity matching engine.

12. The system of claim 11, wherein entity mentions are identified by matching a token in the anchor text with a token in a title of the linked-to authoritative document.

13. The system of claim 12, wherein entity mentions are further identified by matching a token in the anchor text with a token in the title of the linked-to authoritative document.

14. A computer-implemented method comprising:

obtaining, using at least one processor, a first document in a corpus of documents that has a link to an authoritative document, the link being associated with anchor text, the authoritative document being from an authoritative source and being associated with an entity, and the first document being from a source other than the authoritative source;

determining, using the at least one processor, whether a majority of content of the first document matches content from one of the documents in the authoritative source;

identifying at least one entity mention in the first document when it is determined that the majority of the content does not match content from one of the documents in the authoritative source, the entity mention including the anchor text, an identifier of the linked-to authoritative document, and a position of the mention within the content of the first document;

storing the entity mentions in memory;

repeating the obtaining, determining, identifying, and storing for other documents in the corpus; and

evaluating a semantic analysis engine using the stored entity mentions and information associated with the documents in the authoritative source.

15. The method of claim 14, wherein entity mentions are identified by matching a token in the anchor text with a token in a title of the linked-to authoritative document.

16. The method of claim 15, wherein entity mentions are further identified by matching a token in the anchor text with an alias for the linked-to authoritative document, the alias being defined by the authoritative source.

17. The method of claim 16, wherein alias is determined by a set of anchor texts within documents from the authoritative source linking to the linked-to authoritative document.

18. The method of claim 14, wherein identifying at least one entity mention includes extracting free text from the first document, so that entity mentions are identified from the free text.

19. The method of claim 14, wherein the semantic analysis engine is an entity matching engine, wherein the anchor text is considered an entity.

20. The method of claim 14, wherein the semantic analysis engine is an entity tagging engine and the method further comprises:

associating the stored entity mentions with respective entity types based on categories associated with the linked-to authoritative document; and

evaluating the entity tagging engine using the stored entity mentions and the associated entity types.

21. The method of claim 14, wherein at least one million documents in the corpus includes links to documents in the authoritative source.