METHOD FOR DOMAIN IDENTIFICATION OF DOCUMENTS IN A DOCUMENT DATABASE

• US 20060206483A1
• Filed: 05/05/2006
• Published: 09/14/2006
• Est. Priority Date: 10/27/2004
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Inventors: Frieder, Ophir, Knepper, Margaret M., Fox, Kevin Lee
• Granted Patent: US 7,814,105 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

A method for processing documents in a document database includes determining vocabulary words for each document, and determining a respective relevancy for each vocabulary word based upon occurrences thereof in all of the documents. Similarities are determined between the documents based upon the vocabulary words and their respective relevancies. At least one domain identification is determined for the documents based upon the determined similarities.

Claims

1. A method for processing a plurality of documents in a document database comprising:

determining vocabulary words for each document of the plurality thereof;

determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents;

determining similarities between the plurality of documents based upon the vocabulary words and their respective relevancies; and

determining at least one domain identification for documents based upon the determined similarities.

2. A method according to claim 1 wherein determining similarities between the plurality of documents comprises selecting a portion of the vocabulary words based on their respective relevancies for defining a superset of vocabulary words, with the superset of vocabulary words illustrating similarities between the plurality of documents and having an overall domain identification associated therewith.

3. A method according to claim 2 further comprising dividing the overall domain identification into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification;

and wherein relevancies of vocabulary words associated with each lower level domain identification changes so that similar documents are grouped together for each lower level domain identification.

4. A method according to claim 1 wherein determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

5. A method according to claim 4 further comprising generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities between the plurality of documents.

6. A method according to claim 1 further comprising permitting user setting of the respective relevancy of at least one of the vocabulary words.

7. A method according to claim 1 further comprising determining a relevancy feedback factor;

and wherein determining the respective relevancies for each vocabulary word is further based upon the user feedback factor; and

wherein determining the similarities is further based on the respective relevancies with respect to the feedback factor.

8. A method according to claim 7 wherein the relevancy feedback factor is based upon at least one of a user search query and an algorithm.

9. A method according to claim 1 wherein determining the similarities is based on identifying patterns of words in the vocabulary words.

10. A method according to claim 9 wherein the pattern of words comprises n-word phrases, where n≧

2.

11. A method according to claim 1 further comprising

determining vocabulary words corresponding to a predetermined domain identification;

determining similarities between the vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identification; and

assigning the predetermined domain identification to at least one of the documents based on the determined similarities.

12. A method according to claim 11 wherein the vocabulary words corresponding to the predetermined domain identification are user definable.

13. A method according to claim 1 wherein the domain identification for similar documents are user determinable.

14. A method according to claim 1 wherein the domain identification for similar documents is determined by a computer based upon overlapping vocabulary words.

15. A method according to claim 1 wherein the plurality of documents comprises at least one web site document.

16. A method according to claim 1 wherein the plurality of documents comprises at least one electronic mail message.

17. A method according to claim 1 wherein the plurality of documents comprises at least one non-text document having metadata associated therewith.

18. A method according to claim 17 wherein the at least one non-text documents comprises at least one of a video image file, a photograph file, and an audio file.

19. A method for processing a plurality of documents in a document database comprising:

determining vocabulary words for each document of the plurality thereof;

determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents using a computer;

selecting a portion of the vocabulary words based on their respective relevancies for defining a superset of vocabulary words;

determining similarities between the documents associated with the vocabulary words in the superset of vocabulary words and their respective relevancies; and

determining at least one domain identification for documents based upon the determined similarities.

20. A method according to claim 19 wherein determining the vocabulary words and selecting the portion of the vocabulary words are performed using a computer.

21. A method according to claim 19 wherein the at least one domain identification corresponds to an overall domain identification;

and further comprising dividing the overall domain identification into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification; and

wherein relevancies of vocabulary words associated with each lower level domain identification changes so that similar documents are grouped together for each lower level domain identification.

22. A method according to claim 19 wherein determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

23. A method according to claim 22 further comprising generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities between the plurality of documents.

24. A method according to claim 23 wherein determining the similarities is based on identifying patterns of words in the vocabulary words.

25. A method according to claim 19 further comprising:

determining vocabulary words corresponding to a predetermined domain identification;

determining similarities between the superset of vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identification; and

assigning the predetermined domain identification to at least one of the documents based on the determined similarities.

26. A method according to claim 19 wherein the plurality of documents comprises at least one of a web site document, an electronic mail message, and a non-text document having metadata associated therewith.

27. A computer-readable medium having computer-executable instructions for causing a computer to perform steps comprising:

determining vocabulary words for each document of the plurality thereof;

determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents;

determining similarities between the plurality of documents based upon the vocabulary words and their respective relevancies; and

determining at least one domain identification for documents based upon the determined similarities.

28. A computer-readable medium according to claim 27 wherein determining similarities between the plurality of documents comprises selecting a portion of the vocabulary words based on their respective relevancies for defining a superset of vocabulary words, with the superset of vocabulary words illustrating similarities between the plurality of documents and having an overall domain identification associated therewith.

29. A computer-readable medium according to claim 28 further comprising dividing the overall domain identification into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification;

and wherein relevancies of vocabulary words associated with each lower level domain identification changes so that similar documents are grouped together for each lower level domain identification.

30. A computer-readable medium according to claim 27 wherein determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

31. A computer-readable medium according to claim 30 further comprising generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities between the plurality of documents.

32. A computer-readable medium according to claim 27 wherein determining the similarities is based on identifying patterns of words in the vocabulary words.

33. A computer-readable medium according to claim 27 further comprising:

determining vocabulary words corresponding to a predetermined domain identification;

determining similarities between the vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identification; and

assigning the predetermined domain identification to at least one of the documents based on the determined similarities.

34. A computer-readable medium according to claim 27 wherein the plurality of documents comprises at least one of a web site document, an electronic mail message and non-text document having metadata associated therewith.

35. A computer-implemented system for processing documents in a document database comprising:

a first module for determining vocabulary words for each document of the plurality thereof;

a second module for determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents;

a third module for determining similarities between the plurality of documents based upon the vocabulary words and their respective relevancies; and

a fourth module for determining at least one domain identification for documents based upon the determined similarities.

36. A computer-implemented system according to claim 35 wherein said third module for determining similarities between the plurality of documents comprises selecting a portion of the vocabulary words based on their respective relevancies for defining a superset of vocabulary words, with the superset of vocabulary words illustrating similarities between the plurality of documents and having an overall domain identification associated therewith.

37. A computer-implemented system according to claim 36 further comprising a fifth module dividing the overall domain identification into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification;

and wherein relevancies of vocabulary words associated with each lower level domain identification changes so that similar documents are grouped together for each lower level domain identification.

38. A computer-implemented system according to claim 35 wherein said second module for determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

39. A computer-implemented system according to claim 38 wherein said second module further comprises generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities between the plurality of documents.

40. A computer-implemented system according to claim 35 wherein said third module for determining the similarities is based on identifying patterns of words in the vocabulary words.

41. A computer-implemented system according to claim 35 further comprising a sixth module for:

determining vocabulary words corresponding to a predetermined domain identification;

determining similarities between the superset of vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identification; and

assigning the predetermined domain identification to at least one of the documents based on the determined similarities.

42. A computer-implemented system according to claim 35 wherein the plurality of documents comprises at least one of a web site document, an electronic mail message and non-text document having metadata associated therewith.