IDENTIFYING CONTENT OF INTEREST

• US 20070294784A1
• Filed: 06/08/2007
• Published: 12/20/2007
• Est. Priority Date: 06/08/2006
• Status: Active Grant

• Current Assignee: Arthur R. Crivella
• Original Assignee: Ase Edge, Inc.
• Inventors: Darwin, Clayton, Crivella, Arthur, West, Wayne
• Granted Patent: US 7,779,007 B2
• Time in Patent Office: Days
• US Class Current: 800/278
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/353 into predefined classes

Abstract

Methods of identifying content of interest within a corpus are disclosed. The methods may comprise the step of applying a first marker set to the corpus, where the first marker set comprises at least one marker identifying a first type of text. For a first textual unit included in the corpus, the methods may comprise generating a score for the first marker set and comparing the score to a reference score. The score may indicate a number of instances of the at least one marker in the first textual unit.

Claims

1. A method of generating a marker set comprising markers that identify a desired type of text, the method comprising:

selecting a seed marker set comprising at least one seed marker;

generating a seed corpus from a first reference corpus, wherein the seed corpus comprises a plurality of textual units, and wherein each of the plurality of textual units included in the seed corpus comprises at least one instance of a seed marker included in the seed marker set;

generating a statistical value describing the seed marker set and the seed corpus; and

generating a revised seed marker set.

2. The method of claim 1, further comprising analyzing a plurality of instances of markers included in the seed marker set in context.

3. The method of claim 1, further comprising:

generating a revised seed corpus from the seed corpus, wherein the revised seed corpus comprises a second plurality of textual units, and wherein each of the second plurality of textual units included in the revised seed corpus comprises at least one instance of a second seed marker included in the revised seed marker set; and

generating a second statistical value describing the revised seed marker set and the revised seed corpus.

4. The method of claim 1, further comprising applying the revised marker set to the first reference corpus.

5. The method of claim 1, wherein the statistical value indicates a rarity of a seed marker in the seed corpus.

6. The method of claim 1, wherein the statistical value indicates a saliency of a seed marker in the seed corpus.

7. The method of claim 1, wherein generating the statistical value comprises:

generating a first frequency of instances of a seed marker in the seed corpus; and

comparing the first frequency to an expected first frequency of instances of the seed marker in the seed corpus.

8. The method of claim 7, wherein the expected first frequency is based on a frequency of instances of the seed marker in a reference corpus.

9. The method of claim 7, further comprising scaling the first frequency to a frequency per token corpus norm.

10. The method of claim 7, wherein the statistical value is a z-score.

11. The method of claim 1, wherein the seed marker is at least one of the group consisting of a token, an n-gram and a collocate.

12. The method of claim 1, further comprising applying the seed marker set to a test corpus.

13. The method of claim 1, further comprising displaying a plurality of tokens forming collocates with the at least one instance of the seed marker.

14. A system comprising a processor, wherein the processor is programmed to perform the steps of:

selecting a seed marker set comprising at least one seed marker;

generating a seed corpus from a first reference corpus, wherein the seed corpus comprises a plurality of textual units, and wherein each of the plurality of textual units included in the seed corpus comprises at least one instance of a seed marker included in the seed marker set;

generating a statistical value describing the seed marker set and the seed corpus; and

generating a revised seed marker set.

15. A computer readable medium comprising instructions that when executed by a processor, cause the processor to perform the steps of:

selecting a seed marker set comprising at least one seed marker;

generating a seed corpus from a first reference corpus, wherein the seed corpus comprises a plurality of textual units, and wherein each of the plurality of textual units included in the seed corpus comprises at least one instance of a seed marker included in the seed marker set;

generating a statistical value describing the seed marker set and the seed corpus; and

generating a revised seed marker set.

16. A method of identifying content of interest within a corpus, the method comprising:

applying a first marker set to the corpus, wherein the first marker set comprises at least one marker identifying a first type of text;

for a first textual unit included in the corpus, generating a score for the first marker set, wherein the score indicates a number of instances of the at least one marker in the first textual unit; and

comparing the score to a reference score.

17. The method of claim 16, wherein the reference score represents an average score for the first marker set in a reference corpus.

18. The method of claim 16, further comprising generating a z-score indicating a difference between the scores and the reference scores.

19. The method of claim 16, further comprising:

applying a second marker set to the corpus, wherein the second marker set comprises at least one marker identifying a second type of text;

for the first textual unit, generating a second score for the second marker set, wherein the second score indicates a number of instances of the second marker set in the first textual unit; and

comparing the second score to a second reference score.

20. The method of claim 19, further comprising weighting the score for the first marker set relative to the second score for the second marker set.

21. A system comprising a processor, wherein the processor is programmed to perform the steps of:

applying a first marker set to the corpus, wherein the first marker set comprises at least one marker identifying a first type of text;

for a first textual unit included in the corpus, generating a score for the first marker set, wherein the score indicates a number of instances of the at least one marker in the first textual unit; and

comparing the score to a reference score.

22. A computer readable medium comprising instructions that when executed by a processor, cause the processor to perform the steps of:

applying a first marker set to the corpus, wherein the first marker set comprises at least one marker identifying a first type of text;

for a first textual unit included in the corpus, generating a score for the first marker set, wherein the score indicates a number of instances of the at least one marker in the first textual unit; and

comparing the score to a reference score.

23. A method of identifying content of interest within a corpus, the method comprising:

identifying a textual unit in the corpus that includes an instance of an anchor marker set;

generating a plurality of scores for the textual unit, wherein each of the plurality of scores indicates a number of instances in the textual unit of one of a plurality of marker sets;

comparing the plurality of scores to a plurality of reference scores;

calculating an offset between the instance of the anchor marker set and an instance of an instance of one of the plurality of marker sets; and

determining whether the textual unit comprises content of interest considering the comparing and the offset.

24. The method of claim 23, further comprising identifying at least one textual unit in the corpus that has external metadata consistent with a first set of criteria.

25. The method of claim 24, wherein the first set of criteria includes at least one criterion selected from the group consisting of:

an owner, a type, a creation date and a modification date.

26. The method of claim 23, further comprising identifying at least one textual unit in the corpus that has language metadata consistent with a second set of criteria.

27. The method of claim 23, wherein the second set of criteria includes at least criterion selected from the group consisting of:

a total number of tokens, an average number of characters per token, a number of tokens in a particular category.

28. The method of claim 23, further comprising generating a second plurality of scores for the textual unit, wherein each of the second plurality of scores indicates a number of instances in the textual unit of one of a second plurality of marker sets.

29. The method of claim 28, wherein at least one of the second plurality of marker sets includes markers that indicate a first language.

30. A system comprising a processor, wherein the processor is programmed to perform the steps of:

identifying a textual unit in the corpus that includes an instance of an anchor marker set;

generating a plurality of scores for the textual unit, wherein each of the plurality of scores indicates a number of instances in the textual unit of one of a plurality of marker sets;

comparing the plurality of scores to a plurality of reference scores;

calculating an offset between the instance of the anchor marker set and an instance of an instance of one of the plurality of marker sets; and

determining whether the textual unit comprises content of interest considering the comparing and the offset.

31. A computer readable medium comprising instructions that when executed by a processor, cause the processor to perform the steps of:

identifying a textual unit in the corpus that includes an instance of an anchor marker set;

generating a plurality of scores for the textual unit, wherein each of the plurality of scores indicates a number of instances in the textual unit of one of a plurality of marker sets;

comparing the plurality of scores to a plurality of reference scores;

calculating an offset between the instance of the anchor marker set and an instance of an instance of one of the plurality of marker sets; and

determining whether the textual unit comprises content of interest considering the comparing and the offset.

32. A method of evaluating textual content, the method comprising:

identifying instances of a marker in a corpus;

identifying instances of a second marker set in the corpus; and

for an instance of the second marker set that occurs within a predetermined range of an instance of the marker, displaying tokens comprising the instance of the second marker set, tokens comprising the instance of the marker and an intervening token.

33. The method of claim 32, wherein the instance of the second marker set occurs within the predetermined range of the instance of the marker when the instance of the second marker set and the instance of the marker occur within a predetermined number of tokens.

34. The method of claim 32, wherein the instance of the second marker set occurs within the predetermined range of the instance of the marker when the instance of the second marker set and the instance of the marker occur within a common textual unit.

35. The method of claim 32, further comprising:

identifying instances of a third marker set in the corpus; and

for an instance of the third marker set that occurs within a predetermined range of an instance of the marker, displaying tokens comprising the instance of the second marker set, tokens comprising the instance of the marker and an intervening token.

36. A system comprising a processor, wherein the processor is programmed to perform the steps of:

identifying instances of a marker in a corpus;

identifying instances of a second marker set in the corpus; and

for an instance of the second marker set that occurs within a predetermined range of an instance of the marker, displaying tokens comprising the instance of the second marker set, tokens comprising the instance of the marker and an intervening token.

37. A computer readable medium comprising instructions that when executed by a processor, cause the processor to perform the steps of:

identifying instances of a marker in a corpus;

identifying instances of a second marker set in the corpus; and

for an instance of the second marker set that occurs within a predetermined range of an instance of the marker, displaying tokens comprising the instance of the second marker set, tokens comprising the instance of the marker and an intervening token.