EXTRACTING INFORMATION ABOUT REFERENCES TO ENTITIES ROM A PLURALITY OF ELECTRONIC DOCUMENTS

• US 20070016580A1
• Filed: 07/15/2005
• Published: 01/18/2007
• Est. Priority Date: 07/15/2005
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Zhang, Zengyan, Mann, John, Nguyen, Tram, Niblack, Carlton
• Application Number:
  US11/160,943
• Publication Number:
  US 20070016580A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

The present invention provides a method and system of extracting information about references to entities from a plurality of electronic documents. In an exemplary embodiment, the method and system include (1) applying at least one document quality measure to each of the plurality of electronic documents, (2) recognizing the references to entities in the plurality of electronic documents, (3) using at least one reference quality measure for each of the references to entities, (4) computing at least one topical category associated with each of the references to entities, (5) finding at least one co-occurring term associated with each of the references to entities, and (6) characterizing each of the references to entities by at least one characteristic category.

Claims

1. A method of extracting information about references to entities from a plurality of electronic documents, the method comprising:

applying at least one document quality measure to each of the plurality of electronic documents;

recognizing the references to entities in the plurality of electronic documents;

using at least one reference quality measure for each of the references to entities;

computing at least one topical category associated with each of the references to entities;

finding at least one co-occurring term associated with each of the references to entities; and

characterizing each of the references to entities by at least one characteristic category.

2. The method of claim 1 wherein the applying comprises assigning at least one quality score to each of the plurality of electronic documents.

3. The method of claim 2 wherein the assigning comprises assigning the quality score based on the source of the electronic document.

4. The method of claim 2 wherein the assigning comprises assigning the quality score based on the amount of text in the electronic document.

5. The method of claim 2 wherein the assigning comprises assigning the quality score based on whether the electronic document is a duplicate of other electronic documents in the plurality of electronic documents.

6. The method of claim 2 wherein the assigning comprises assigning the quality score based on whether the electronic document is a near duplicate of other electronic documents in the plurality of electronic documents.

7. The method of claim 2 wherein the assigning comprises assigning the quality score based on whether the electronic document contains unwanted text.

8. The method of claim 2 wherein the assigning comprises assigning the quality score based on the rank of the electronic document, wherein the rank is selected from the group consisting of pagerank, hostrank, and eyeball count.

9. The method of claim 2 further comprising, if the quality score of the electronic document is less than a threshold, eliminating the electronic document.

10. The method of claim 1 wherein the recognizing comprises identifying candidate references to entities in the plurality of electronic documents from a set of entity names.

11. The method of claim 10 wherein the identifying comprises identifying the candidate references to entities by an identifying technique, wherein the identifying technique is selected from the group consisting of direct spotting, index-based retrieval, and named entity recognition.

12. The method of claim 10 further comprising disambiguating the candidate references to entities, thereby identifying the references to entities.

13. The method of claim 1 wherein the using comprises assigning at least one quality score to each of the references to entities.

14. The method of claim 13 wherein the assigning comprises assigning the quality score based on whether the snippet of text in which the reference to entities occurs is unique.

15. The method of claim 13 wherein the assigning comprises assigning the quality score based on the running text quality of the reference to entities.

16. The method of claim 13 wherein the assigning comprises assigning the quality score based on whether the snippet of text in which the reference to entities occurs can be parsed by natural language parsing to yield a subject and a verb.

17. The method of claim 13 wherein the assigning comprises assigning the quality score based on whether the snippet of text in which the reference to entities occurs can be parsed by natural language parsing to yield a valid sentence.

18. The method of claim 13 wherein the assigning comprises assigning the quality score based on whether the snippet of text in which the reference to entities occurs satisfies a set of heuristic rules based on the textual properties of the snippet.

19. The method of claim 13 wherein the assigning comprises assigning the quality score based on the document markup properties of the snippet of text in which the reference to entities occurs.

20. The method of claim 13 wherein the assigning comprises assigning the quality score based on whether the snippet of text in which the reference to entities occurs comprises content text.

21. The method of claim 13 further comprising, if the quality score of the reference to entities is less than a threshold, eliminating the reference to entities.

22. The method of claim 1 wherein the computing comprises identifying specified words and phrases that co-occur with the references to entities.

23. The method of claim 1 wherein the finding comprises finding unspecified words or phrases that co-occur with the references to entities.

24. The method of claim 1 wherein the characterizing comprises assigning at least one characteristic to each of the references to entities.

25. The method of claim 24 wherein the assigning comprises assigning the date of the electronic document in which the reference to entities occurs as the characteristic.

26. The method of claim 24 wherein the assigning comprises assigning the source type of the electronic document in which the reference to entities occurs as the characteristic.

27. The method of claim 24 wherein the assigning comprises assigning the geographic location associated with the electronic document in which the reference to entities occurs as the characteristic.

28. The method of claim 24 wherein the assigning comprises assigning the language of the snippet of text in which the reference to entities occurs as the characteristic.

29. The method of claim 24 wherein the assigning comprises assigning the sentiment of the snippet of text in which the reference to entities occurs as the characteristic.

30. The method of claim 24 wherein the assigning comprises assigning the author of the snippet of text in which the reference to entities occurs as the characteristic.

31. The method of claim 24 wherein the assigning comprises assigning the rank of the electronic document in which the reference to entities occurs as the characteristic, wherein the rank is selected from the group consisting of pagerank, hostrank, and eyeball count.

32. The method of claim 1 further comprising storing the extracted information about the references to entities.

33. The method of claim 1 further comprising allowing for the input of feedback on the extracting.

34. A system of extracting information about references to entities from a plurality of electronic documents, the system comprising:

an applying module configured to apply at least one document quality measure to each of the plurality of electronic documents;

a recognizing module configured to recognize the references to entities in the plurality of electronic documents;

a using module configured to use at least one reference quality measure for each of the references to entities;

a computing module configured to compute at least one topical category associated with each of the references to entities;

a finding module configured to find at least one co-occurring term associated with each of the references to entities; and

a characterizing module configured to characterize each of the references to entities by at least one characteristic category.

35. A computer program product usable with a programmable computer having readable program code embodied therein of extracting information about references to entities from a plurality of electronic documents, the computer program product comprising:

computer readable code for applying at least one document quality measure to each of the plurality of electronic documents;

computer readable code for recognizing the references to entities in the plurality of electronic documents;

computer readable code for using at least one reference quality measure for each of the references to entities;

computer readable code for computing at least one topical category associated with each of the references to entities;

computer readable code for finding at least one co-occurring term associated with each of the references to entities; and

computer readable code for characterizing each of the references to entities by at least one characteristic category.