Detecting content-rich text

• US 20060161537A1
• Filed: 01/19/2005
• Published: 07/20/2006
• Est. Priority Date: 01/19/2005
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Amitay, Einat, Har'el, Nadav
• Application Number:
  US11/038,370
• Publication Number:
  US 20060161537A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

A method includes finding content-rich text in a document by identifying areas of narrative in the document. An apparatus includes a detector and a content-rich text indicator. The detector detects linguistic parameters which characterize narrative text in an input document and the content-rich text indicator provides the locations of narrative text in the input document.

Claims

1. A method comprising:

finding content-rich text in a document by identifying areas of narrative in said document.

2. The method according to claim 1 and wherein said identifying comprises analyzing the document for linguistic parameters which characterize narrative text.

3. The method according to claim 2 and wherein said linguistic parameters in English are closed class words.

4. The method according to claim 2 and wherein said linguistic parameters separate between semantic/content words and functional/syntactic words.

5. The method according to claim 2 and wherein said linguistic parameters are search engine stopwords.

6. The method according to claim 5 and wherein said finding comprises:

for each word, determining a weighted average as a function of the number of stopwords in a window around said word; and

selecting those words whose weighted average is above a threshold as part of said areas of narrative.

7. The method according to claim 6 and wherein said threshold is the midpoint between a minimum value and a maximum value for said weighted average.

8. The method according to claim 6 and wherein said threshold is a function of at least one of the following:

a maximum score, the type of text being analyzed and the language of said document.

9. The method according to claim 6 and wherein said threshold comprises more than one threshold.

10. The method according to claim 1 and wherein said document is at least one of the following types of documents:

an email, a support document containing bits of code, a journal, a web page, transcribed speech, a transcribed videoed lecture, a slide and a newspaper.

11. The method according to claim 1 and wherein said document is in English.

12. The method according to claim 1 and wherein said document is in a non-English language.

13. An apparatus comprising:

a detector to detect linguistic parameters which characterize narrative text in an input document; and

a content-rich text indicator to provide the locations of narrative text in said input document.

14. The apparatus according to claim 13 and wherein said linguistic parameters in English are closed class words.

15. The apparatus according to claim 13 and wherein said linguistic parameters separate between semantic/content words and functional/syntactic words.

16. The apparatus according to claim 13 and wherein said linguistic parameters are search engine stopwords.

17. The apparatus according to claim 16 and wherein said detector comprises an averager to determiner for each word, a weighted average as a function of the number of stopwords in a window around said word.

18. The apparatus according to claim 17 and wherein said indicator comprises a demapper to select those words whose weighted average is above a threshold as part of said areas of narrative.

19. The apparatus according to claim 18 and wherein said threshold is the midpoint between a minimum value and a maximum value for said weighted average.

20. The apparatus according to claim 18 and wherein said threshold is a function of at least one of the following:

a maximum score, the type of text being analyzed and the language of said document.

21. The apparatus according to claim 18 and wherein said threshold comprises more than one threshold.

22. The apparatus according to claim 13 and wherein said document is at least one of the following types of documents:

an email, a support document containing bits of code, a journal, a web page, transcribed speech, a transcribed videoed lecture, a slide and a newspaper.

23. The apparatus according to claim 13 and wherein said document is in English.

24. The apparatus according to claim 13 and wherein said document is in a non-English language.

25. A computer product readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps, said method steps comprising:

finding content-rich text in a document by identifying areas of narrative in said document.

26. The product according to claim 25 and wherein said identifying comprises analyzing the document for linguistic parameters which characterize narrative text.

27. The product according to claim 26 and wherein said linguistic parameters in English are closed class words.

28. The product according to claim 26 and wherein said linguistic parameters separate between semantic/content words and functional/syntactic words.

29. The product according to claim 26 and wherein said linguistic parameters are search engine stopwords.

30. The product according to claim 29 and wherein said finding comprises:

for each word, determining a weighted average as a function of the number of stopwords in a window around said word; and

selecting those words whose weighted average is above a threshold as part of said areas of narrative.

31. The product according to claim 30 and wherein said threshold is the midpoint between a minimum value and a maximum value for said weighted average.

32. The product according to claim 30 and wherein said threshold is a function of at least one of the following:

a maximum score, the type of text being analyzed and the language of said document.

33. The product according to claim 30 and wherein said threshold comprises more than one threshold.

34. The product according to claim 25 and wherein said document is at least one of the following types of documents:

an email, a support document containing bits of code, a journal, a web page, transcribed speech, a transcribed videoed lecture, a slide and a newspaper.

35. The product according to claim 25 and wherein said document is in English.

36. The product according to claim 25 and wherein said document is in a non-English language.