Text summarization method & apparatus using a multidimensional subspace

• US 20070118506A1
• Filed: 11/18/2005
• Published: 05/24/2007
• Est. Priority Date: 11/18/2005
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Kao, Anne, Wu, Yuan-Jye, Ferng, William, Cranfill, Robert, Poteet, Stephen
• Granted Patent: US 7,752,204 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: Y10S 707/917 Text

Abstract

A text summarizer identifies relevant terms in a document, weights the terms and extracts one or more segments to produce a summary or abstract. The various terms in a particular are weighted in relation to an existing document collection. A term weight computer computes term weights for terms in the document, and a threshold comparator compares the term weights to determine if the corresponding terms are relevant to the document collection. Next, a term weight summer adds the term weights for each occurrence of each relevant term in the various segments of the document, and a summation comparator compares the summations to identify a text summarization segment representative of the document. Optionally, relevant terms can be highlighted in the term summarization segment.

Claims

1. A computer-implemented method of summarizing a first unit of text data with relation to an existing document collection, comprising:

computing a term weight that is representative of the relevance of a term to a second unit of text data with relation to the document collection;

comparing the computed term weight to a predetermined threshold; and

returning a relevant term based at least in part on a result of the comparison.

2. The computer-implemented method of claim 1, further comprising:

summing a plurality of relevant term weights based on a number of occurrences of a plurality of corresponding relevant terms in a segment of the first unit of text data;

comparing a plurality of summations based on a plurality of corresponding segments of the first unit of text data to identify a text summarization segment; and

returning the text summarization segment.

3. The computer-implemented method of claim 2, further comprising highlighting the relevant term in the returned text summarization segment.

4. The computer-implemented method of claim 1, wherein the first unit of text data includes a document.

5. The computer-implemented method of claim 1, wherein the second unit of text data is the same as the first unit of text data.

6. The computer-implemented method of claim 1, wherein the second unit of text data is a query.

7. The computer-implemented method of claim 6, further comprising selecting as the first unit of text data a relevant document from the document collection based at least in part on the query.

8. The computer-implemented method of claim 1, wherein the step of computing further comprises calculating a relationship value indicative of a relationship between a query term and the document collection.

9. The computer-implemented method of claim 1, wherein the step of computing further comprises recomposing a vector representation in an original term space, the vector representation being based at least in part on a projection of an original vector representation in a predetermined vector subspace.

10. The computer-implemented method of claim 9, wherein the original vector representation is based at least in part on the first unit of text data.

11. The computer-implemented method of claim 9, wherein the original vector representation is based at least in part on a query.

12. The computer-implemented method of claim 9, wherein the subspace is determined in part by a decomposed term matrix representation of the document collection.

13. The computer-implemented method of claim 1, wherein the step of computing further comprises computing an original vector representation in an original term space and transforming the original vector representation into a projection in a predetermined vector subspace.

14. The computer-implemented method of claim 13, wherein the step of computing the original vector representation further comprises performing at least one selected from the following:

tokenizing a first term according to a predetermined tokenizing policy, removing a second term according to a predetermined stopword policy, removing a third term according to a predetermined low frequency term policy, normalizing a fourth term according to a predetermined normalization policy, and stemming a fifth term according to a predetermined stemming policy.

15. The computer-implemented method of claim 13, wherein the step of computing the original vector representation further comprises counting a number of occurrences of a term to determine a term frequency, and statistically transforming the term frequency according to a predetermined statistical transformation policy.

16. A computer program product for summarizing a first unit of text data with relation to an existing document collection, including a computer-readable medium encoded with instructions configured to be executed by a processor in order to perform predetermined operations comprising:

computing a term weight that is representative of the relevance of a term to a second unit of text data with relation to the document collection;

comparing the computed term weight to a predetermined threshold; and

returning a relevant term based at least in part on a result of the comparison.

17. The computer program product of claim 16, wherein the step of computing further comprises:

summing a plurality of relevant term weights based on a number of occurrences of a plurality of corresponding relevant terms in a segment of the first unit of text data;

comparing a plurality of summations based on a plurality of corresponding segments of the first unit of text data to identify a text summarization segment; and

returning the text summarization segment.

18. The computer program product of claim 17, wherein the predetermined operations further comprise highlighting the relevant term in the returned text summarization segment.

19. The computer program product of claim 16, wherein the first unit of text data includes a document.

20. The computer program product of claim 16, wherein the second unit of text data is the same as the first unit of text data.

21. The computer program product of claim 16, wherein the second unit of text data is a query

22. The computer program product of claim 21, wherein the predetermined operations further include selecting as the first unit of text data a relevant document from the document collection based at least in part on the query.

23. The computer program product of claim 16, wherein the step of computing further comprises calculating a relationship value indicative of a relationship between a query term and the document collection.

24. The computer program product of claim 16, wherein the step of computing further comprises recomposing a vector representation in an original term space, the vector representation being based at least in part on a projection of an original vector representation in a predetermined vector subspace.

25. The computer program product of claim 24, wherein the original vector representation is based at least in part on the first unit of text data.

26. The computer program product of claim 24, wherein the original vector representation is based at least in part on a query.

27. The computer program product of claim 24, wherein the subspace is determined in part by a decomposed term matrix representation of the document collection.

28. The computer program product of claim 16, wherein the step of computing further comprises computing an original vector representation in an original term space and transforming the original vector representation into a projection in a predetermined vector subspace.

29. The computer program product of claim 28, wherein the step of computing the original vector representation further comprises performing at least one selected from the following:

tokenizing a first term according to a predetermined tokenizing policy, removing a second term according to a predetermined stopword policy, removing a third term according to a predetermined low frequency term policy, normalizing a fourth term according to a predetermined normalization policy, and stemming a fifth term according to a predetermined stemming policy.

30. The computer program product of claim 28, wherein the step of computing the original vector representation further comprises counting a number of occurrences of a term to determine a term frequency, and statistically transforming the term frequency according to a predetermined statistical transformation policy.

31. A text summarizer, comprising:

a term weight computer configured to compute a term weight that is representative of the relevance of a term to the document collection;

a threshold comparator configured to compare the computed term weight to a predetermined threshold, wherein the text summarizer is configured to return a relevant term based at least in part on a result of the comparison.

32. The text summarizer of claim 31, further comprising:

a term weight summer configured to sum a plurality of relevant term weights based on a number of occurrences of a plurality of corresponding relevant terms in a segment of text data;

a summation comparator configured to compare a plurality of summations based on a plurality of corresponding segments of text data to identify a text summarization segment, wherein the text summarizer is configured to return the text summarization segment.

33. A computer-implemented method for creating a summary with relation to an existing document collection having one or more documents based on a query, comprising:

receiving query information from a user;

identifying a first document segment of a first document of the document collection, wherein the first document segment is substantially optimized to represent a summary of the first document in relation to the query information based on a weighting process of terms within the document, the weighting process being based on a subspace transformation of the query information, the subspace being based on a number of occurrences of terms in the documents of the document collection; and

returning the first document segment of the first document to the user.

34. The computer-implemented method of claim 33, wherein the step of identifying a text summarization segment includes computing one or more term weights for each of the document terms, each weight being representative of the relevance of a respective term to the query information with relation to the document collection.

35. The computer-implemented method of claim 34, wherein the step of identifying a text summarization segment further includes:

identifying a plurality of different document segments of the first document;

calculating a respective sum for each document segment to create a plurality of sums, each sum being based on the query term weights and on the occurrences of the corresponding query terms in a respective document segment; and

identifying a high-scoring document segment based on the plurality of sums.

36. The computer-implemented method of claim 34, wherein the step of identifying a text summarization segment further includes:

identifying one or more secondary terms having a known relation to at least one of the query terms; and

computing a secondary term weights for each of the secondary terms, each secondary weight being representative of the relevance of a secondary term to the document collection.

37. The computer-implemented method of claim 36, wherein the step of identifying a text summarization segment further includes:

identifying a plurality of different document segments of the first document;

calculating a respective sum for each document segment to create a plurality of sums, each sum being based on the query term weights, the secondary term weights and on the occurrences of the corresponding query terms and secondary terms in a respective document segment; and

identifying one or more high-scoring document segment based on the plurality of sums.

38. The computer-implemented method of claim 35, further comprising highlighting one or more terms related to the query in the returned text summarization segment.

39. The computer-implemented method of claim 38, wherein the form of highlighting of each highlighted term is based upon the respective computed weight.

40. The computer-implemented method of claim 37, further comprising highlighting at least one of a query term and a secondary term in the returned text summarization segment.

41. The computer-implemented method of claim 40, wherein the form of highlighting of each highlighted term is based upon the respective computed weight.

42. The computer-implemented method of claim 34, wherein the step of computing one or more term weights for the query terms includes recomposing a vector representation in an original term space, the vector representation being based at least in part on a projection of an original vector representation in a predetermined vector subspace.

43. The computer-implemented method of claim 33, wherein the query information is at least one of a sample document and a set of one or more distinct and separate terms provided by a user.

44. The computer-implemented method of claim 33, wherein the query information is derived from a specific document in the set of documents.

45. The computer-implemented method of claim 33, wherein there is a plurality of documents in the set of one or more documents.