Method and system for determining relevance of terms in text documents

• US 8,321,398 B2
• Filed: 07/01/2009
• Issued: 11/27/2012
• Est. Priority Date: 07/01/2009
• Status: Active Grant

• Current Assignee: Refinitiv US Organization, LLC (Refinitiv US Holdings, Inc.)
• Original Assignee: Thomson Reuters LLC (The Woodbridge Co. Ltd.)
• Inventors: Aumann, Yehonatan, Keller, Orgad, Shlivinski, Ran
• Primary Examiner(s): NGUYEN, KIM T
• Application Number:
  US12/459,475
• Publication Number:
  US 20110004606A1
• Time in Patent Office: 1,245 Days
• Field of Search: 707/706, 707/749
• US Class Current: 707/706
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence

Abstract

The present invention provides a corpus-independent method for determining relevancy of terms to content of text appearing in a document by analyzing the document itself. Conventional information extraction, or other methods, may be applied to a document to generate a list of terms. The invention analyzes the document using relevancy scoring algorithms to determine a term relevancy score representing the term'"'"'s relevance to the text contained in the document. The scores, including an aggregate score, may be normalized in the process. Based on relevancy scoring, terms are then ranked and further processed. In this manner relevancy is determined based on the subject document itself and by analyzing the occurrences and locations of the terms within the document. Additional techniques may be applied to relate the relevancy scores generated by the present invention to a corpus or collection of documents.

Claims

1. A computer implemented method comprising:

receiving a list comprising an entity, the entity having been identified as being associated with an electronic document;

based solely upon a set of characteristics of the document, determining a relevancy score associated with the entity with respect to the document, wherein the set of characteristics includes at least one characteristic from the group consisting of;

a first number representing a number of sentences occurring in the document prior to a first sentence in which the entity is named;

a second number representing a number of sentences between first and last occurrences of the entity within the document; and

a third number representing a uniformity with which the entity occurs within the document; and

storing the relevancy score.

2. The method of claim 1, wherein the list comprises a second entity associated with the document, the method further comprising:

based solely upon the set of characteristics, determining a second relevancy score associated with the second entity with respect to the document;

storing the second relevancy score; and

generating a ranking based at least in part upon the first relevancy score and the second relevancy score.

3. The method of claim 2 further comprising:

tagging the document based at least in part on the ranking of the assigned score.

4. The method of claim 1, wherein the set of characteristics includes at least one characteristic from the group consisting of:

a first number representing a proximity of a first occurrence of the entity to the beginning of the document;

a second number representing a proximity of a first occurrence of the entity to the last occurrence of the entity within the document; and

a third number representing overall changes in a rate of occurrences of the entity throughout the document.

5. The method of claim 1 further comprising:

applying a Gaussian function to normalize the relevancy score.

6. The method of claim 5 further comprising determining whether the relevancy score meets a threshold prior to normalizing the relevancy score.

7. The method of claim 1, wherein the document is unstructured.

8. The method of claim 1 further comprising:

performing information extraction to generate the list.

9. The method of claim 1 wherein the step of determining includes using two or more relevancy algorithms to determine one or more of the following relevancy scores:

occurrence count score;

intro proximity score;

first and last occurrence range score; and

gap deviation score.

10. The method of claim 1, wherein the set of characteristics includes a number representing a number of times the entity is mentioned in the document.

11. The method of claim 1 further comprising determining a combined relevancy score associated with the entity based on the relevancy score and at least one additional factor.

12. The method of claim 11, wherein the at least one additional factor comprises TFIDF.

13. The method of claim 1 further comprising determining a plurality of relevancy scores for a plurality of entities;

determining a ranking of the plurality of entities based at least in part on the plurality of relevancy scores and at least one additional factor.

14. The method of claim 13, wherein the at least one additional factor comprises TFIDF.

15. A computer-implemented method comprising:

receiving terms extracted from a first electronic document;

scoring the extracted terms using two or more term relevancy algorithms based solely upon the first electronic document, wherein the two or more relevancy algorithms are from the group consisting of;

determining a first number representing a number of times the term is mentioned in the document;

determining a second number representing a proximity of a first occurrence of the term to the beginning of the document;

determining a third number representing a proximity of a first occurrence of the term to a last occurrence of the term within the document; and

determining a fourth number representing overall changes in a rate of occurrences of the term throughout the document;

aggregating for each of the extracted terms the relevancy scores generated by the two or more term relevancy algorithms to produce a term aggregate relevancy score for each of the extracted terms; and

ranking each of the extracted terms based on the term aggregate relevancy score assigned to the extracted terms to determine a relevance ranking of the extracted terms to the first electronic document.

16. The method of claim 15, wherein the two or more relevancy algorithms are used to determine one or more of the following relevancy scores:

occurrence count score;

intro proximity score;

first and last occurrence range score; and

gap deviation score.

17. The method of claim 15, wherein the ranking to determine a relevance ranking comprises normalizing at least one term aggregate relevancy score by a Gaussian function.

18. The method of claim 15 further comprising determining whether the term aggregate relevancy score meets a threshold and, if the threshold is met, normalizing the term aggregate relevancy score.

19. The method of claim 15, wherein each of the received extracted terms represents an entity.

20. The method of claim 15 further comprising:

tagging the first electronic document based at least in part on the ranking of the extracted terms.

21. The method of claim 15, wherein the first electronic document is unstructured.

22. The method of claim 15 further comprising:

performing information extraction to generate the extracted terms.

23. The method of claim 15 further comprising determining a combined relevancy score associated with the entity based on the term aggregate relevancy score and at least one additional factor.

24. The method of claim 23, wherein the at least one additional factor comprises TFIDF.

25. The method of claim 15 further comprising determining a plurality of term aggregate relevancy scores for a plurality of entities;

determining a ranking of the plurality of entities based at least in part on the plurality of term aggregate relevancy scores and at least one additional factor.

26. The method of claim 25, wherein the at least one additional factor comprises TFIDF.

27. An article comprising a non-transitory machine-readable medium, the medium having stored thereon instructions to be executed by a machine to perform operations, the article comprising instructions for:

receiving a list comprising a term, the term having been identified as being associated with an electronic document;

based solely upon a set of characteristics of the document, determining and assigning a relevancy score to the term with respect to the document, wherein the set of characteristics includes at least one characteristic from the group consisting of;

a first number representing a number of sentences occurring in the document prior to a first sentence in which the term appears;

a second number representing a number of sentences between first and last occurrences of the term within the document; and

a third number representing a uniformity with which the term occurs within the document; and

storing the relevancy score.

28. The article of claim 27, wherein the list comprises a second term associated with the document, the article further comprising instructions adapted to perform the following operations:

based solely upon the set of characteristics, determining and assigning a second relevancy score associated with the second term with respect to the document;

storing the second relevancy score; and

generating a ranking based at least in part upon a comparison of the first relevancy score and the second relevancy score.

29. The article of claim 27, wherein the instructions for determining and assigning a relevancy score comprise two or more relevancy algorithms that are used to determine one or more of the following relevancy scores:

occurrence count score;

intro proximity score;

first and last occurrence range score; and

gap deviation score.

30. The article of claim 29, wherein the two or more relevancy algorithms are from the group consisting of:

determining a first number representing a number of times the term is mentioned in the document;

determining a second number representing a proximity of a first occurrence of the term to the beginning of the document;

determining a third number representing a proximity of a first occurrence of the term to a last occurrence of the term within the document; and

determining a fourth number representing overall changes in a rate of occurrences of the term throughout the document.

31. The article of claim 27, wherein the instructions for determining and assigning a relevancy score comprise instructions adapted to normalize the score using a Gaussian function.

32. The article of claim 27, wherein the set of characteristics includes a number representing a number of times the term is mentioned in the document.

33. The article of claim 27 further comprising instructions for determining a combined relevancy score associated with the entity based on the relevancy score and at least one additional factor.

34. The article of claim 33, wherein the at least one additional factor comprises TFIDF.

35. The article of claim 27 further comprising instructions for determining a plurality of relevancy scores for a plurality of entities and instructions for determining a ranking of the plurality of entities based at least in part on the plurality of relevancy scores and at least one additional factor.

36. The article of claim 35, wherein the at least one additional factor comprises TFIDF.

37. A computer-based system comprising memory and a processor for executing instructions to perform operations, the system comprising:

input adapted to receive a list comprising a term, the term having been identified as being associated with an electronic document;

relevancy scoring module adapted to determine a relevancy score associated with the term with respect to the document, the relevancy score being based solely upon a set of characteristics of the document, wherein the set of characteristics includes at least one characteristic from the group consisting of;

a first number representing a number of sentences occurring in the document prior to a first sentence in which the term appears;

a second number representing a number of sentences between first and last occurrences of the term within the document; and

a third number representing a uniformity with which the term occurs within the document; and

memory for storing the relevancy score.

38. The system of claim 37, wherein the list comprises a second term associated with the document and wherein the relevancy scoring module is further adapted to determine a second relevancy score associated with the second term with respect to the document, the second relevancy score being based solely upon the set of characteristics of the document, and wherein the memory stores the second relevancy score.

39. The system of claim 38, wherein the relevancy scoring module further comprises instructions for generating a ranking based at least in part upon a comparison of the first relevancy score and the second relevancy score.

40. The system of claim 37, wherein the relevancy scoring module comprises instructions associated with two or more relevancy algorithms that are executed by the processor to determine one or more of the following relevancy scores:

occurrence count score;

intro proximity score;

first and last occurrence range score; and

gap deviation score.

41. The system of claim 40, wherein the two or more relevancy algorithms are from the group consisting of:

determining a number representing a number of times the term is mentioned in the document;

determining a number representing a proximity of a first occurrence of the term to the beginning of the document;

determining a number representing a proximity of a first occurrence of the term to a last occurrence of the term within the document; and

determining a number representing overall changes in a rate of occurrences of the term throughout the document.

42. The system of claim 37, wherein the instructions for determining and assigning a relevancy score comprise instructions adapted to normalize the score using a Gaussian function.

43. The system of claim 37 further comprising:

an information extraction module adapted to identify and extract terms from the document and wherein the list comprises the identified and extracted terms.

44. The system of claim 37, wherein the relevancy scoring module comprises instructions adapted to determine whether the relevancy score meets a threshold and, if the threshold is met, normalizing the relevancy score.

45. The system of claim 37, wherein the set of characteristics includes a number representing a number of times the term is mentioned in the document.

46. The system of claim 37 further comprising instructions for determining a combined relevancy score associated with the term based on the relevancy score and at least one additional factor.

47. The system of claim 46, wherein the at least one additional factor comprises TFIDF.

48. The system of claim 37, wherein the list comprises a plurality of terms and further comprising instructions for determining a plurality of relevancy scores for the plurality of terms and instructions for determining a ranking of the plurality of terms based at least in part on the plurality of relevancy scores and at least one additional factor.

49. The system of claim 48, wherein the at least one additional factor comprises TFIDF.