Detecting relationships in unstructured text

• US 8,001,144 B2
• Filed: 03/26/2008
• Issued: 08/16/2011
• Est. Priority Date: 09/20/2005
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Novak, Jasmine
• Primary Examiner(s): TIMBLIN, ROBERT M
• Application Number:
  US12/056,048
• Publication Number:
  US 20080177740A1
• Time in Patent Office: 1,238 Days
• Field of Search: 707/706, 707/776
• US Class Current: 707/776
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

Disclosed are embodiments of a system and a method for detecting relationships described in unstructured text-based electronic documents. The system and method incorporate the use of an input file that contains one or more text patterns that represent particular relationships. The text patterns each include regular text expressions that describe the particular relationship and slots for the location of each entity in that relationship. Document(s) are selected by a user and scanned by a proper noun tagger that identifies and tags every occurrence of proper names within the document(s). Then, a pattern matcher scans the document(s) to match text patterns. If a text pattern is matched within a document a relationship detector extracts all pairs of proper names found in the slots for each matched text pattern. The output from the relationship detector includes the names for each entity in the relationship, the type of relationship, and the identity of the document and the location of the sentence describing the relationship in the document.

Claims

1. A computer-implemented method of detecting that different entities identified by different proper names are in a relationship, said method comprising:

storing, in a storage device, an input file comprising text patterns, each text pattern comprising;

a specific text expression that represents a specific type of relationship between a first entity and a second entity;

a first slot location identifier indicating a location of a first slot for a first proper name of said first entity in said specific type of relationship relative to said specific text expression; and

a second slot location identifier indicating a location of a second slot for a second proper name of said second entity in said specific type of relationship relative to said specific text expression;

locating and tagging, by a processor, occurrences of all proper names within multiple text-based documents in order to generate a list of proper names and locations of said proper names within said multiple text-based documents;

analyzing, by said processor, said multiple text-based documents so as to locate a document that contains said specific text expression of a specific text pattern; and

accessing, by said processor, said list to determine if any of said proper names on said list are located in said document within said first slot relative to said specific text expression, as determined based on said first slot location identifier, and within said second slot relative to said specific text expression, as determined based on said second slot location identifier so as to detect said specific type of relationship and so as to identify, said first entity in said specific type of relationship by said first proper name and said second entity in said specific type of relationship by said second proper name, respectively.

2. The method of claim 1, said specific text expression comprising a plurality of words that describe said specific type of relationship with one of said words being a keyword and said method further comprising, before said analyzing of said multiple text-based documents, scanning said multiple text-based documents to determine if said keyword is located in said document and only performing said analyzing if said keyword is located in said document.

3. The method of claim 1, said specific text expression comprising a plurality of words that describe said specific type of said relationship, said first slot location identifier indicating whether said first slot for said first entity is located before, within, or after said plurality of words, and said second slot location identifier indicating whether said second slot for said second entity is located before, within, or after said plurality of words.

4. The method of claim 1, said locating and said tagging of said comprising:

scanning said multiple text-based documents to identify any proper names occurring within any of said multiple text-based documents based on a set of matching rules;

re-scanning said multiple text-based documents to tag locations for each of said proper names identified; and

recording said locations in said list.

5. The method of claim 4, said set of matching rules being based on at least one of word capitalization, sentence structure, sentence boundaries, and excluded words.

6. The method of claim 1, each of said text patterns further comprising a relationship order identifier indicating relative positions of said two different entities in said specific type of relationship as a function of said locations of said proper names within said first slot and said second slot.

7. The method of claim 6, further comprising storing a record comprising said first proper name of said first entity, said second proper name of said second entity, said specific type of relationship between said first entity and said second entity, a first position of said first entity and a second position of said second entity in said specific type of relationship, and an identifier for said document and a location in said document where said specific type of relationship is detected.

8. A system for detecting that different entities identified by different proper names are in a relationship, said system comprising:

a storage device storing an input file comprising text patterns, each text pattern comprising;

a specific text expression that represents a specific type of relationship between a first entity and a second entity;

a first slot location identifier indicating a location of a first slot for a first proper name of said first entity in said specific type of relationship relative to said specific text expression; and

a second slot location identifier indicating a location of a second slot for a second proper name of said second entity in said specific type of relationship relative to said specific text expression; and

a processor comprising;

a proper noun tagger locating and tagging occurrences of all proper names within multiple text-based documents in order to generate a list of proper names and locations of said proper names within said multiple text-based documents;

a pattern matcher accessing said input file in said storage device and analyzing said multiple text-based documents so as to locate a document that contains said specific text expression of a specific text pattern within said document; and

a relationship detector in communication with said pattern matcher and said proper noun tagger, said relationship detector accessing said list to determine if any of said proper names on said list are located in said document within said first slot relative to said specific text expression, as determined based on said first slot location identifier, and within said second slot relative to said specific text expression, as determined based on said second slot location identifier, so as to detect said specific type of relationship and so as to identify, by name, said first entity in said specific type of relationship by said first proper name and said second entity in said specific type of relationship by said second proper name, respectively.

9. The system of claim 8, said specific text expression comprising a plurality of words that describe said specific type of relationship with one of said words being a keyword and said system further comprising a keyword identifier accessing said input file in said storage device, scanning said multiple text-based documents for said keyword to determine if said keyword is located in said document, and forwarding said document to said pattern matcher only if said keyword is located in said document.

10. The system of claim 8, said specific text expression comprising a plurality of words that describe said specific type of relationship, said first slot location identifier indicating whether said first slot for said first entity is located before, within, or after said plurality of words, and said second slot location identifier indicating whether said second slot for said second entity is located before, within, or after said plurality of words.

11. The system of claim 8, each of said text patterns further comprising a relationship order identifier indicating relative positions of said two different entities in said specific type of relationship as a function of said locations of said proper names within said first slot and said second slot.

12. The system of claim 8, said proper noun tagger locating and tagging by performing the following:

scanning said multiple text-based documents to identify any proper names occurring within said multiple text-based documents based on a set of matching rules;

re-scanning said multiple text-based documents to tag locations for each of said proper names identified; and

recording said locations in said list.

13. The system of claim 12, said set of matching rules being based on at least one of word capitalization, sentence structure, sentence boundaries, and excluded words.

14. The system of claim 11, said storage device further storing a record comprising said first proper name of said first entity, said second proper name of said second entity, said specific type of relationship between said first entity and said second entity, a first position of said first entity and a second position of said second entity in said specific type of relationship, and an identifier for said document and a location in said document where said specific type of relationship is detected.

15. A program storage medium readable by a computer and tangibly embodying a program of instructions executable by said computer to perform a method of detecting a that different entities identified by different proper names are in a relationship, said method comprising:

storing an input file comprising text patterns, each text pattern comprising;

a specific text expression that represents a specific type of relationship between a first entity and a second entity;

a first slot location identifier indicating a location of a first slot for a first proper name of said first entity in said specific type of relationship relative to said specific text expression; and

a second slot location identifier indicating a location of a second slot for a second proper name of said second entity in said specific type of relationship relative to said specific text expression;

locating and tagging occurrences of all proper names within multiple text-based documents in order to generate a list of proper names and locations of said proper names within said multiple text-based documents;

analyzing said multiple text-based documents so as to locate a document that contains said specific text expression of a specific text pattern; and

accessing said list to determine if any of said proper names on said list are located in said document within said first slot relative to said specific text expression, as determined based on said first slot location identifier, and within said second slot relative to said specific text expression, as determined based on said second slot location identifier so as to detect said specific type of relation ship and so as to identify said first entity in said specific type of relationship by said first proper name and said second entity in said specific type of relationship by said second proper name, respectively.

16. The program storage medium of claim 15, said specific text expression comprising a plurality of words that describe said specific type of relationship with one of said words being a keyword and said method further comprising before said analyzing of said multiple text-based documents, scanning said multiple text-based documents to determine if said keyword is located in said document and only performing said analyzing if said keyword is located in said document.

17. The program storage medium of claim 15, said specific text expression comprising a plurality of words that describe said specific type of said relationship, said first slot location indicating whether said first slot for said first entity is located before, within, or after said plurality of words, and said second slot location indicating whether said second slot for said second entity is located before, within, or after said plurality of words.

18. The program storage medium of claim 15, said locating and said tagging comprising:

scanning said multiple text-based documents to identify any proper names occurring within said multiple text-based documents based on a set of matching rules;

re-scanning said multiple text-based documents to tag locations for each of said proper names identifier; and

recording said locations in said list.

19. The program storage medium of claim 18, said set of matching rules being based on at least one of word capitalization, sentence structure, sentence boundaries, and excluded words.

20. The program storage medium of claim 15, each of said text patterns further comprising a relationship order identifier indicating relative positions of said two different entities in said specific type of relationship as a function of said locations of said proper names within said first slot and said second slot.

21. A computer-implemented method of detecting that different entities identified by different proper names are in a relationship, said method comprising:

storing, in a storage device, an input file comprising text patterns, each text pattern comprising;

a specific text expression that represents a specific type of relationship between a first entity and a second entity;

a first slot location identifier indicating a location of a first slot for a first proper name of said first entity in said specific type of relationship relative to said specific text expression; and

a second slot location identifier indicating a location of a second slot for a second proper name of said second entity in said specific type of relationship relative to said specific text expression;

locating and tagging, by a processor, occurrences of all proper names within multiple text-based documents in order to generate a list of proper names and locations of said proper names within said multiple text-based documents;

analyzing, by said processor, said multiple text-based documents so as to locate a document that contains said specific text expression of a specific text pattern; and

accessing, by said processor, said list to determine if any of said proper names on said list are located in said document within said first slot relative to said specific text expression, as determined based on said first slot location identifier, and within said second slot relative to said specific text expression, as determined based on said second slot location identifier so as to detect said specific type of relationship and so as to identify said first entity in said specific type of relationship by said first proper name and said second entity in said specific type of relationship by said second proper name, respectively,

said specific text expression comprising a plurality of words that describe said specific type of relationship with one of said words being a keyword and said method further comprising before said analyzing of said multiple text-based documents, scanning said multiple text-based documents to determine if said keyword is located in said document and only performing said analyzing if said keyword is located in said document,

said specific text expression comprising a plurality of words that describe said specific type of relationship, said first slot location identifier indicating whether said first slot for said first entity is located before, within, or after said plurality of words, and said second slot location identifier indicating whether said second slot for said second entity is located before, within, or after said plurality of words,

said locating and said tagging comprising;

scanning said multiple text-based documents to identify any proper names occurring within said multiple text-based documents based on a set of matching rules;

re-scanning said document to tag locations for each of said proper names identified; and

recording said locations in said list,

said set of matching rules being based on at least one of word capitalization, sentence structure, sentence boundaries, and excluded words,

each of said test patterns further comprising a relationship order identifier indicating relative positions of said two different entities in said specific type of relationship as a function of said locations of said proper names within said first slot and said second slot, and

said method further comprising storing a record comprising said first proper name of said first entity, said second proper name of said second entity, said specific type of relationship between said first entity and said second entity, a first position of said first entity and a second position of said second entity in said specific type of relationship, and an identifier for said document and a location in said document where said specific type of relationship is detected.