NLP-based entity recognition and disambiguation

• US 8,594,996 B2
• Filed: 10/15/2008
• Issued: 11/26/2013
• Est. Priority Date: 10/17/2007
• Status: Expired due to Fees

• Current Assignee: Fiver LLC
• Original Assignee: EVRI Incorporated
• Inventors: Tusk, Carsten, Dhillon, Navdeep S., Koperski, Krzysztof, Liang, Jisheng, Bhatti, Satish
• Primary Examiner(s): Godbold, Douglas
• Assistant Examiner(s): Villena, Mark
• Application Number:
  US12/288,158
• Publication Number:
  US 20090144609A1
• Time in Patent Office: 1,868 Days
• Field of Search: 704/9, 704/10, 707/737
• US Class Current: 704/9
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

Methods and systems for entity recognition and disambiguation using natural language processing techniques are provided. Example embodiments provide an entity recognition and disambiguation system (ERDS) and process that, based upon input of a text segment, automatically determines which entities are being referred to by the text using both natural language processing techniques and analysis of information gleaned from contextual data in the surrounding text. In at least some embodiments, supplemental or related information that can be used to assist in the recognition and/or disambiguation process can be retrieved from knowledge repositories such as an ontology knowledge base. In one embodiment, the ERDS comprises a linguistic analysis engine, a knowledge analysis engine, and a disambiguation engine that cooperate to identify candidate entities from a knowledge repository and determine which of the candidates best matches the one or more detected entities in a text segment using context information.

Claims

1. A computer-implemented method for disambiguating one or more entities in an indicated text segment, comprising:

processing the indicated text segment to determine a plurality of terms and their associated parts-of-speech tags and grammatical roles;

performing linguistic analysis of the processed text segment to determine one or more potential entities which are referred to in the text segment by potential entity names;

generating and storing, for each potential entity, an entity profile data structure storing one or more associated properties that characterize the entity based upon surrounding context and linguistic information, the surrounding context and linguistic information retrieved from the performed linguistic analysis of the processed text segment, by;

retrieving, from the linguistic analysis of the processed text segment, actions and/or modifiers associated with the determined potential entity name, the actions and/or modifiers appearing in the context surrounding the potential entity names in the indicated text segment;

determining one or more roles that are attributable to the potential entity based upon the retrieved actions and/or modifiers associated with the determined potential entity names; and

storing, in the entity profile data structure as part of the properties associated with the potential entity, the determined one or more roles; and

disambiguating which entities are being referred to in the indicated text segment by determining one or more most likely entities that are referred to in the text segment by comparing, using both linguistic and contextual information, the entity profiles generated for each potential entity with attributes of one or more candidate entities, including a comparison of the determined one or more roles stored in the profile generated for each potential entity with the one or more roles of each of the candidate entities.

2. The method of claim 1 wherein the candidate entities are entity entries retrieved from a knowledge repository.

3. The method of claim 2 wherein the knowledge repository is an ontology knowledge base.

4. The method of claim 1 wherein the disambiguating which entities are being referred to in the indicated text segment by determining of the one or more mostly likely entities which are referred to in the text segment by comparing, using both linguistic and contextual information, the entity profiles generated for each potential entity name with attributes of one or more candidate entities further comprises:

searching a knowledge repository for a set of candidate entities that have similar characteristics to the properties of one or more of the generated entity profiles;

ranking the candidate entities in the set of candidate entities to determine a set of mostly likely entities which are referred to in the text segment; and

providing the determined set of mostly likely entities.

5. The method of claim 4 wherein the ranking the candidate entities weights the candidate entities according to contextual information surrounding portions of the text segment that refer to the potential entity names.

6. The method of claim 4 wherein the ranking the candidate entities weights the candidate entities according to preference information.

7. The method of claim 4 wherein the ranking the candidate entities further comprises using a classifier, of a classification model constructed from training data, to classify the candidate entities and to order them based upon closest matches.

8. The method of claim 1 wherein the disambiguating which entities are being referred to in the indicated text segment by determining of the one or more mostly likely entities which are referred to in the text segment by comparing, using both linguistic and contextual information, the entity profiles generated for each potential entity name with attributes of one or more candidate entities further comprises:

resolving the one or more most likely entities to a single identified entity by performing iterative comparisons reusing entity disambiguation information gained from a prior comparison until no new entity disambiguation information is gained.

9. The method of claim 8 wherein the terms are weighted based upon at least one of a binary weight or a frequency weight.

10. The method of claim 1 wherein each entity profile comprises a feature vector of terms collected from modifiers and/or actions associated with the potential entity name based upon the linguistic analysis of the processed text segment.

11. The method of claim 1, further comprising providing a ranked list of the determined one or more most likely entities which are referred to in the text segment.

12. The method of claim 1, further comprising using the determined one or more likely entities to inform a relationship search.

13. The method of claim 1, further comprising:

performing the method in response to selection of a user interface component.

14. The method of claim 1 wherein the method is embedded in code that supports a widget presented on a web page.

15. The method of claim 1, further comprising:

invoking the method to annotate information on a web page.

16. A non-transitory computer-readable medium containing contents that, when executed causes a computing system to perform a method comprising:

processing the indicated text segment to determine a plurality of terms and their associated parts-of-speech tags and grammatical roles;

performing linguistic analysis of the processed text segment to determine one or more potential entities which are referred to in the text segment by potential entity names;

generating and storing, for each potential entity, an entity profile data structure storing one or more associated properties that characterize the entity based upon surrounding context and linguistic information, the surrounding context and linguistic information retrieved from the performed linguistic analysis of the processed text segment, by;

retrieving, from the linguistic analysis of the processed text segment, actions and/or modifiers associated with the determined potential entity name, the actions and/or modifiers appearing in the context surrounding the potential entity names in the indicated text segment;

determining one or more roles that are attributable to the potential entity based upon the retrieved actions and/or modifiers associated with the determined potential entity names; and

storing, in the entity profile data structure as part of the properties associated with the potential entity, the determined one or more roles; and

automatically disambiguating which entities are being referred to in the indicated text segment by determining one or more most likely entities that are referred to in the text segment by comparing, using both linguistic and contextual information, the entity profiles generated for each potential entity with attributes of one or more candidate entities, including a comparison of the determined one or more roles stored in the profile generated for each potential entity with the one or more roles of each of the candidate entities.

17. The non-transitory computer-readable medium of claim 16, wherein the medium is a memory in a computing system containing the contents.

18. The non-transitory computer-readable medium of claim 16 wherein the contents are instructions that, when executed, cause a computer processor in the computing device to perform the method.

19. The non-transitory computer-readable medium of claim 16 embedded in a computing system configured to perform indexing and storing of a corpus of documents for searching using natural language processing.

20. An entity recognition and disambiguation computing system, comprising;

a memory; and

a recognition and disambiguation module stored in the memory that is configured, when executed, to

receive a text segment for processing;

recognize one or more candidate named entities which are referred to by a potential entity detected in a received text segment based, at least in part, upon a natural language analysis of the text segment; and

disambiguate the candidate named entities to determine a single named entity to which the detected entity in the received text segment is deemed to refer based upon a combination of linguistic analysis, contextual information gleaned from text surrounding the detected entity mention in the received text segment, and information retrieved from one or more knowledge repositories,

wherein the contextual information and linguistic-analysis is used to determine and store;

as part of an entity profile data structure generated for the detected entity, at least one or more roles of the detected entity based upon determining actions and/or modifiers from the linguistic analysis of the surrounding text, and wherein the one or more roles of the entity profile data structure generated for the detected entity are compared to one or more roles of the candidate entities to help disambiguate the candidate named entities.

21. The system of claim 20, wherein the module is further configured, when executed, to disambiguate the candidate named entities using information based upon a relationship search.

22. The system of claim 20 wherein the module is further configured, when executed, to disambiguate the candidate named entities using a classifier of a classification modeling approach constructed from training data.

23. A computer-implemented method for presenting information regarding named entities comprising:

receiving an indication of a segment of text;

invoking the recognition and disambiguation module of claim 20 to process the indicated text segment to automatically determine one or more named entities referred to in the text segment; and

for each determined one or more named entities, presenting a link to information associated with the named entity.

24. The method of claim 23 wherein the information is based in part upon an ontology entry.