Semantic document profiling

• US 20070073678A1
• Filed: 09/23/2005
• Published: 03/29/2007
• Est. Priority Date: 09/23/2005
• Status: Abandoned Application

• Current Assignee: Applied Linguistics LLC
• Original Assignee: Applied Linguistics LLC
• Inventors: Timofeyev, Maksim, Speers, d'Armond, Scott, Bernard
• Application Number:
  US11/232,898
• Publication Number:
  US 20070073678A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/374 Thesaurus

Abstract

A method of semantic profiling of documents comprises receiving a document to be profiled, the document comprising a plurality of terms, for each of at least a portion of the plurality of terms in the document determining a part of speech and a grammatical function of the term, obtaining senses of the term, selecting a sense as a most likely meaning of the term, and calculating an information value of the term, and generating a semantic profile of the document comprising at least some of the calculated information values.

Claims

1. A method of semantic profiling of documents comprising:

receiving a document to be profiled, the document comprising a plurality of terms;

for each of at least a portion of the plurality of terms in the document;

determining a part of speech and a grammatical function of the term,

obtaining senses of the term,

selecting a sense as a most likely meaning of the term, and

calculating an information value of the term; and

generating a semantic profile of the document comprising at least some of the calculated information values.

2. The method of claim 1, wherein a sense is selected as a most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document.

3. The method of claim 2, wherein a sense is selected as a most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document by:

determining at what level of a semantic taxonomy hierarchy the senses of each term are related to the senses of the other terms; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to senses of the other terms.

4. The method of claim 3, wherein the lower the level at which each sense of the term is related to senses of the other terms, the higher the weighting assigned to the relation.

5. The method of claim 3, wherein the determination of taxonomic relationships of the senses of the term with senses of other terms in the document is done using a moving window based on the term.

6. The method of claim 3, wherein a sense is selected as a most likely meaning of the term based on statistics of co-occurrence of senses at a selected taxonomic level.

7. The method of claim 6, wherein a sense is selected as a most likely meaning of the term based on a most probable meaning of each sense based on statistics of occurrences of meaning.

8. The method of claim 7, wherein a sense is selected as a most likely meaning of the term by:

selecting as the most likely meaning of the term either;

the sense determined to be the most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document;

or

the sense determined to be the most likely meaning of the term based on statistics of co-occurrence of senses at a selected taxonomic level;

or

the sense determined to be the most likely meaning of the term based on the most probable meaning of each sense based on statistics of occurrences of meaning.

9. The method of claim 1, wherein the information value is calculated by:

obtaining information relating to the selected sense of the term;

weighting the information relating to the selected sense of the term; and

calculating the information value based on the weighted information relating to the selected sense of the term.

10. The method of claim 9, wherein the information relating to the selected sense of the term comprises semantic and syntactic information relating to the term and to the selected sense of the term.

11. The method of claim 9, wherein the information relating to the selected sense of the term comprises at least one of:

information identifying a part of speech of the term;

information identifying a grammatical function of the term;

information identifying a semantico-syntactic class of the term; and

information identifying a polysemy count of the term.

12. The method of claim 11, wherein the information relating to the selected sense of the term further comprises:

a level of a semantic taxonomy hierarchy at which the senses of each term are related to selected senses of other terms in the document.

13. The method of claim 12, wherein the information relating to the selected sense of the term is weighted by:

applying a weighting to the at least one of information identifying a part of speech of the term, information identifying a grammatical function of the term, information identifying a semantico-syntactic class of the term, and information identifying a polysemy count of the term; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to a sense of at least one other term.

14. The method of claim 9, wherein the information relating to the selected sense of the term comprises:

information identifying a part of speech of the term;

information identifying a grammatical function of the term;

information identifying a semantico-syntactic class of the term;

information identifying a polysemy count of the term; and

a level of a semantic taxonomy hierarchy the senses of each term are related.

15. The method of claim 14, wherein the information relating to the selected sense of the term is weighted by:

applying a weighting to the information identifying a part of speech of the term, information identifying a grammatical function of the term, information identifying a semantico-syntactic class of the term, and information identifying a polysemy count of the term; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to a sense of at least one other term.

16. The method of claim 15, wherein the information value is calculated according to:

$\mathrm{PICW}=\mathrm{wPOS}<br><br>\frac{\left(\mathrm{wPCT}+\mathrm{wSSC}\right)}{2}<br><br>\underset{i=N}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{wGF},<br><br>\mathrm{ICW}=\mathrm{PICW}⨯<br><br>\mathrm{TW},\mathrm{and}$ $\mathrm{InfoVal}=\frac{\underset{i=N}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{ICW}}_i}{\sum <br><br>\mathrm{ICW}};<br><br>$

wherein wPOS is the weighted information identifying a part of speech of the term, wGF is the weighted information identifying a grammatical function of the term, wSSC is the weighted information identifying a semantico-syntactic class of the term, wPCT is the weighted information identifying a polysemy count of the term, and TW is the taxonomic weight.

17. A method for performing document-based searching comprising:

receiving a search document, the search document comprising a plurality of terms;

for each of at least a portion of the plurality of terms in the search document;

determining a part of speech and a grammatical function of the term,

obtaining senses of the term,

selecting a sense as a most likely meaning of the term, and

calculating an information value of the term; and

generating a semantic profile of the search document comprising at least some of the calculated information values; and

accessing a database comprising a plurality of semantic profiles of documents to retrieve documents having semantic profiles that are similar to the semantic profile of the search documents, each semantic profile in the database comprising a plurality of information values of terms included in the document.

18. The method of claim 17, wherein a sense is selected as a most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document.

19. The method of claim 18, wherein a sense is selected as a most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document by:

determining at what level of a semantic taxonomy hierarchy the senses of each term are related to the senses of the other terms; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to senses of the other terms.

20. The method of claim 19, wherein the lower the level at which each sense of the term is related to senses of the other terms, the higher the weighting assigned to the relation.

21. The method of claim 19, wherein the determination of taxonomic relationships of the senses of the term with senses of other terms in the document is done using a moving window based on the term.

22. The method of claim 19, wherein a sense is selected as a most likely meaning of the term based on statistics of co-occurrence of senses at a selected taxonomic level.

23. The method of claim 22, wherein a sense is selected as a most likely meaning of the term based on a most probable meaning of each sense based on statistics of occurrences of meaning.

24. The method of claim 23, wherein a sense is selected as a most likely meaning of the term by:

selecting as the most likely meaning of the term either;

the sense determined to be the most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document;

or

the sense determined to be the most likely meaning of the term based on statistics of co-occurrence of senses at a selected taxonomic level;

or

the sense determined to be the most likely meaning of the term based on the most probable meaning of each sense based on statistics of occurrences of meaning.

25. The method of claim 17, wherein the information value is calculated by:

obtaining information relating to the selected sense of the term;

weighting the information relating to the selected sense of the term; and

calculating the information value based on the weighted information relating to the selected sense of the term.

26. The method of claim 25, wherein the information relating to the selected sense of the term comprises semantic and syntactic information relating to the term and to the selected sense of the term.

27. The method of claim 25, wherein the information relating to the selected sense of the term comprises at least one of:

information identifying a part of speech of the term;

information identifying a grammatical function of the term;

information identifying a semantico-syntactic class of the term; and

information identifying a polysemy count of the term.

28. The method of claim 27, wherein the information relating to the selected sense of the term further comprises:

a level of a semantic taxonomy hierarchy at which the senses of each term are related to selected senses of other terms in the document.

29. The method of claim 28, wherein the information relating to the selected sense of the term is weighted by:

applying a weighting to the at least one of information identifying a part of speech of the term, information identifying a grammatical function of the term, information identifying a semantico-syntactic class of the term, and information identifying a polysemy count of the term; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to a sense of at least one other term.

30. The method of claim 25, wherein the information relating to the selected sense of the term comprises:

information identifying a part of speech of the term;

information identifying a grammatical function of the term;

information identifying a semantico-syntactic class of the term;

information identifying a polysemy count of the term; and

a level of a semantic taxonomy hierarchy the senses of each term are related.

31. The method of claim 30, wherein the information relating to the selected sense of the term is weighted by:

applying a weighting to the information identifying a part of speech of the term, information identifying a grammatical function of the term, information identifying a semantico-syntactic class of the term, and information identifying a polysemy count of the term; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to a sense of at least one other term.

32. The method of claim 31, wherein the information value is calculated according to:

$\mathrm{PICW}=\mathrm{wPOS}<br><br>\frac{\left(\mathrm{wPCT}+\mathrm{wSSC}\right)}{2}<br><br>\underset{i=N}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{wGF},<br><br>\mathrm{ICW}=\mathrm{PICW}⨯<br><br>\mathrm{TW},\mathrm{and}$ $\mathrm{InfoVal}=\frac{\underset{i=N}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{ICW}}_i}{\sum <br><br>\mathrm{ICW}};<br><br>$

wherein wPOS is the weighted information identifying a part of speech of the term, wGF is the weighted information identifying a grammatical function of the term, wSSC is the weighted information identifying a semantico-syntactic class of the term, wPCT is the weighted information identifying a polysemy count of the term, and TW is the taxonomic weight.

33. The method of claim 17, further comprising:

ranking the retrieved documents based on similarity to the search document.

34. The method of claim 33, wherein the retrieved documents are ranked by:

comparing semantic profiles of the retrieved documents to the semantic profile of the search document.

35. The method of claim 17, further comprising:

generating a summary of each of at least some of the retrieved documents.

36. The method of claim 35, wherein the summary of each of at least some of the retrieved documents is generated by:

calculating an information value for each sentence in the document;

deleting sentences having an information value below a threshold value;

deleting non-main clauses having an information value below a threshold value;

normalizing to declarative form each remaining sentence and clause;

deleting noun phrases and verb phrases having an information value below a threshold value from each remaining sentence and clause;

selecting at least a portion of the remaining sentences and clauses as kernel phrases; and

replacing terms present in the kernel phrases with terms relating to similar concepts selected from a taxonomic hierarchy to form the summary of the document.

37. The method of claim 35, wherein terms present in the kernel phrases are replaced with terms relating to similar concepts selected from a taxonomic hierarchy by:

identifying a subject, verb, and object terms of each kernel phrase;

determining intersections of the subject, verb, and object terms at a level of the taxonomic hierarchy;

obtaining concept labels of the level of the taxonomic hierarchy at which the intersections; and

combining the obtained concept labels to form sentences that summarize the kernel phrases.

38. A method of summarizing a textual document comprising:

calculating an information value for each sentence of the document;

deleting from consideration for a summary sentences having an information value below a first threshold value to form retained sentences;

deleting from the retained sentences non-main clauses having information values below a second threshold value to form retained clauses;

normalizing the retained clauses to declarative form;

deleting modifiers having information values below a third threshold value from the normalized retained clauses to from kernel phrases;

selecting at least a portion of the kernel phrases; and

replacing at least portions of the kernel phrases with terms relating to similar concepts selected from a taxonomic hierarchy.

39. The method of claim 38, wherein the information value for a sentence is calculated by:

calculating an information value for at least some terms in the sentence;

summing the calculated information values for the terms in the sentence; and

dividing the sum by a number of the terms for which the information values were summed.

40. The method of claim 38, wherein the information value for a term is calculated by:

determining a part of speech and a grammatical function of the term,

obtaining senses of the term,

selecting a sense as a most likely meaning of the term, and

calculating an information value of the term.

41. The method of claim 40, wherein a sense is selected as a most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document.

42. The method of claim 41, wherein a sense is selected as a most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document by:

determining at what level of a semantic taxonomy hierarchy the senses of each term are related to the senses of the other terms; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to senses of the other terms.

43. The method of claim 42, wherein the lower the level at which each sense of the term is related to senses of the other terms, the higher the weighting assigned to the relation.

44. The method of claim 42, wherein the determination of taxonomic relationships of the senses of the term with senses of other terms in the document is done using a moving window based on the term.

45. The method of claim 42, wherein a sense is selected as a most likely meaning of the term based on statistics of co-occurrence of senses at a selected taxonomic level.

46. The method of claim 45, wherein a sense is selected as a most likely meaning of the term based on a most probable meaning of each sense based on statistics of occurrences of meaning.

47. The method of claim 46, wherein a sense is selected as a most likely meaning of the term by:

selecting as the most likely meaning of the term either;

the sense determined to be the most likely meaning of the term based on taxonomic relationships of the senses of the term with senses of other terms in the document;

or

the sense determined to be the most likely meaning of the term based on statistics of co-occurrence of senses at a selected taxonomic level;

or

the sense determined to be the most likely meaning of the term based on the most probable meaning of each sense based on statistics of occurrences of meaning.

48. The method of claim 40, wherein the information value is calculated by:

obtaining information relating to the selected sense of the term;

weighting the information relating to the selected sense of the term; and

calculating the information value based on the weighted information relating to the selected sense of the term.

49. The method of claim 48, wherein the information relating to the selected sense of the term comprises semantic and syntactic information relating to the term and to the selected sense of the term.

50. The method of claim 48, wherein the information relating to the selected sense of the term comprises at least one of:

information identifying a part of speech of the term;

information identifying a grammatical function of the term;

information identifying a semantico-syntactic class of the term; and

information identifying a polysemy count of the term.

51. The method of claim 50, wherein the information relating to the selected sense of the term further comprises:

a level of a semantic taxonomy hierarchy at which the senses of each term are related to selected senses of other terms in the document.

52. The method of claim 51, wherein the information relating to the selected sense of the term is weighted by:

applying a weighting to the at least one of information identifying a part of speech of the term, information identifying a grammatical function of the term, information identifying a semantico-syntactic class of the term, and information identifying a polysemy count of the term; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to a sense of at least one other term.

53. The method of claim 48, wherein the information relating to the selected sense of the term comprises:

information identifying a part of speech of the term;

information identifying a grammatical function of the term;

information identifying a semantico-syntactic class of the term;

information identifying a polysemy count of the term; and

a level of a semantic taxonomy hierarchy the senses of each term are related.

54. The method of claim 53, wherein the information relating to the selected sense of the term is weighted by:

applying a weighting to the information identifying a part of speech of the term, information identifying a grammatical function of the term, information identifying a semantico-syntactic class of the term, and information identifying a polysemy count of the term; and

assigning a taxonomic weight to relationships among the senses based on the level of the semantic taxonomy at which each sense of a term is related to a sense of at least one other term.

55. The method of claim 54, wherein the information value is calculated according to:

$\mathrm{PICW}=\mathrm{wPOS}<br><br>\frac{\left(\mathrm{wPCT}+\mathrm{wSSC}\right)}{2}<br><br>\underset{i=N}{\overset{N}{\sum <br><br>}}<br><br>\mathrm{wGF},<br><br>\mathrm{ICW}=\mathrm{PICW}⨯<br><br>\mathrm{TW},\mathrm{and}$ $\mathrm{InfoVal}=\frac{\underset{i=N}{\overset{N}{\sum <br><br>}}<br><br>{\mathrm{ICW}}_i}{\sum <br><br>\mathrm{ICW}};<br><br>$

wherein wPOS is the weighted information identifying a part of speech of the term, wGF is the weighted information identifying a grammatical function of the term, wSSC is the weighted information identifying a semantico-syntactic class of the term, wPCT is the weighted information identifying a polysemy count of the term, and TW is the taxonomic weight.

56. The method of claim 38, wherein the information value for a clause is calculated by:

calculating an information value for at least some terms in the clause;

summing the calculated information values for the terms in the clause; and

dividing the sum by a number of the terms for which the information values were summed.

57. The method of claim 38, wherein the information value for a modifier is calculated by:

calculating an information value for at least some terms in the modifier;

summing the calculated information values for the terms in the modifier; and

dividing the sum by a number of the terms for which the information values were summed.

58. The method of claim 38, wherein the kernel phrases are selected by:

selecting as the selected kernel phrases that portion of the kernel phrases having higher information values than kernel phrases that are not selected.

59. The method of claim 58, wherein the portion of kernel phrases that are selected is a fraction or percentage of the kernel phrases.

60. The method of claim 58, wherein the portion of kernel phrases that are selected is determined so as to form a summary of a selected size.

61. The method of claim 38, wherein the kernel phrases are replaced with terms relating to similar concepts selected from a taxonomic hierarchy by:

identifying subject, verb, and object terms for each kernel phrase;

determining intersections of the identified subject, verb, and object terms at a level of the taxonomic hierarchy;

combining concept labels of the level of the taxonomic hierarchy at which the identified subject, verb, and object terms intersect to form sentences that summarize each kernel phrase; and

replacing the kernel phrases with the summary sentences.