Streaming text data mining method & apparatus using multidimensional subspaces

• US 20070083509A1
• Filed: 10/11/2005
• Published: 04/12/2007
• Est. Priority Date: 10/11/2005
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Kao, Anne, Wu, Yuan-Jye, Ferng, William, Cranfill, Robert, Poteet, Stephen
• Granted Patent: US 8,234,279 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

A streaming text data comparator performs real-time text data mining on streaming text data. The comparator receives a streaming text data document and generates a vector representation of the term frequencies relating to an existing document collection. The comparator then transforms the term frequency vector into a projection in a precomputed multidimensional subspace that represents the original document collection. The comparator further calculates a relationship value representing the similarities or differences between the vector representation and the subspace, and compares the relationship value to a predetermined threshold to determine whether the streaming text data document is related to the original document collection. If the streaming text data document is related, the streaming text data comparator intercalates the new document into the document collection. If the new document is not related, the comparator may store or delete the unrelated document.

Claims

1. A method of comparing a unit of streaming text data to an existing document collection, comprising:

computing a vector representation of the unit of streaming text data;

transforming the vector representation into a projection in a predetermined vector subspace; and

calculating a relationship value indicative of a relationship between the vector representation and the subspace.

2. The method of claim 1, wherein the unit of streaming text data includes a document.

3. The method of claim 1, wherein the step of computing further comprises performing at least one selected from the following on the unit of streaming text data:

tokenizing a first term according to a predetermined tokenizing policy, removing a second term according to a predetermined stopword policy, removing a third term according to a predetermined low frequency term policy, normalizing a fourth term according to a predetermined normalization policy, and stemming a fifth term according to a predetermined stemming policy.

4. The method of claim 1, wherein the step of computing further comprises counting a number of occurrences of a term in the unit of streaming text data to determine a term frequency.

5. The method of claim 4, wherein the step of computing further comprises statistically transforming the term frequency according to a predetermined statistical transformation policy.

6. The method of claim 1, wherein the subspace comprises a decomposed term matrix representation of the document collection.

7. The method of claim 1, wherein the relationship value is based at least in part on a distance between the projection and the vector representation.

8. The method of claim 1, wherein the relationship value is based at least in part on an angle between the projection and the vector representation.

9. The method of claim 1, wherein the relationship value is based at least in part on a magnitude of the projection.

10. The method of claim 1, further comprising receiving the unit of streaming text data over a communication network.

11. The method of claim 10, wherein the step of receiving further comprises receiving the unit of streaming text data by way of one selected from the following:

a hypertext transfer protocol (HTTP), a secure hypertext transfer protocol (HTTPS), a network news transfer protocol (NNTP), a post office protocol (POP), an Internet message access protocol (IMAP), a file transfer protocol (FTP), an Internet relay chat protocol (IRC) and a proprietary protocol.

12. The method of claim 1, further comprising comparing the relationship value to a predetermined threshold value.

13. The method of claim 12, further comprising intercalating the unit of streaming text data into the document collection based at least in part on a result of the comparison.

14. A computer program product for comparing a unit of streaming text data to an existing document collection, including a computer-readable medium encoded with instructions configured to be executed by a processor in order to perform predetermined operations comprising:

computing a vector representation of the unit of streaming text data;

transforming the vector representation into a projection in a predetermined vector subspace; and

calculating a relationship value indicative of the relationship between the vector representation and the subspace.

15. The computer program product of claim 14, wherein the unit of streaming text data includes a document.

16. The computer program product of claim 14, wherein the step of computing further comprises performing at least one selected from the following on the unit of streaming text data:

tokenizing a first term according to a predetermined tokenizing policy, removing a second term according to a predetermined stopword policy, removing a third term according to a predetermined low frequency term policy, normalizing a fourth term according to a predetermined normalization policy, and stemming a fifth term according to a predetermined stemming policy.

17. The computer program product of claim 14, wherein the step of computing further comprises counting a number of occurrences of a term in the unit of streaming text data to determine a term frequency.

18. The computer program product of claim 17, wherein the step of computing further comprises statistically transforming the term frequency according to a predetermined statistical transformation policy.

19. The computer program product of claim 14, wherein the subspace comprises a decomposed term matrix representation of the document collection.

20. The computer program product of claim 14, wherein the relationship value is based at least in part on a distance between the projection and the vector representation.

21. The computer program product of claim 14, wherein the relationship value is based at least in part on an angle between the projection and the vector representation.

22. The computer program product of claim 14, wherein the relationship value is based at least in part on a magnitude of the projection.

23. The computer program product of claim 14, wherein the predetermined operations further comprise receiving the unit of streaming text data over a communication network.

24. The computer program product of claim 23, wherein the step of receiving further comprises receiving the unit of streaming text data by way of one selected from the following:

a hypertext transfer protocol (HTTP), a secure hypertext transfer protocol (HTTPS), a network news transfer protocol (NNTP), a post office protocol (POP), an Internet message access protocol (IMAP), a file transfer protocol (FTP), an Internet relay chat protocol (IRC) and a proprietary protocol.

25. The computer program product of claim 14, wherein the predetermined operations further comprise comparing the relationship value to a predetermined threshold value.

26. The computer program product of claim 25, wherein the predetermined operations further comprise intercalating the unit of streaming text data into the document collection based at least in part on a result of the comparison.

27. A streaming text data comparator, comprising:

a streaming text data vectorizer configured to compute a vector representation of a unit of streaming text data;

a vector projector configured to transform the vector representation into a projection in a predetermined vector subspace; and

a relationship calculator configured to calculate a relationship value between the vector representation and the subspace.

28. The streaming text data comparator of claim 27, wherein the unit of streaming text data includes a document.

29. The streaming text data comparator of claim 27, wherein the streaming text data vectorizer is further configured to perform at least one selected from the following on the unit of streaming text data:

tokenize a first term according to a predetermined tokenizing policy, remove a second term according to a predetermined stopword policy, remove a third term according to a predetermined low frequency term policy, normalize a fourth term according to a predetermined normalization policy, and stem a fifth term according to a predetermined stemming policy.

30. The streaming text data comparator of claim 27, wherein the streaming text data vectorizer is further configured to count a number of occurrences of a term in the unit of streaming text data to determine a term frequency.

31. The streaming text data comparator of claim 30, wherein the streaming text data vectorizer is further configured to statistically transform the term frequency according to a predetermined statistical transformation policy.

32. The streaming text data comparator of claim 27, wherein the subspace comprises a decomposed matrix representation of the document collection.

33. The streaming text data comparator of claim 27, wherein the relationship value is based at least in part on a distance between the projection and the vector representation.

34. The streaming text data comparator of claim 27, wherein the relationship value is based at least in part on an angle between the projection and the vector representation.

35. The streaming text data comparator of claim 27, wherein the relationship value is based at least in part on a magnitude of the projection.

36. The streaming text data comparator of claim 27, further comprising a streaming text data receiver configured to receive the streaming text data over a communication network.

37. The streaming text data comparator of claim 36, wherein the streaming text data receiver is further configured to receive the unit of streaming text data by way of one selected from the following:

a hypertext transfer protocol (HTTP), a secure hypertext transfer protocol (HTTPS), a network news transfer protocol (NNTP), a post office protocol (POP), an Internet message access protocol (IMAP), a file transfer protocol (FTP), an Internet relay chat protocol (IRC) and a proprietary protocol.

38. The streaming text data comparator of claim 27, further comprising a threshold comparator configured to compare the relationship value to a predetermined threshold value.

39. The streaming text data comparator of claim 38, further comprising a streaming text data intercalator configured to intercalate the unit of streaming text data into the document collection based at least in part on a result of the comparison.