Method for domain identification of documents in a document database

• US 7,814,105 B2
• Filed: 05/05/2006
• Issued: 10/12/2010
• Est. Priority Date: 10/27/2004
• Status: Expired due to Fees

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Inventors: Frieder, Ophir, Fox, Kevin Lee, Knepper, Margaret M.
• Primary Examiner(s): Mofiz, Apu M
• Assistant Examiner(s): Le, Hung D
• Application Number:
  US11/381,832
• Publication Number:
  US 20060206483A1
• Time in Patent Office: 1,621 Days
• Field of Search: None
• US Class Current: 707/738
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

A method for processing documents in a document database includes determining vocabulary words for each document, and determining a respective relevancy for each vocabulary word based upon occurrences thereof in all of the documents. Similarities are determined between the documents based upon the vocabulary words and their respective relevancies. At least one domain identification is determined for the documents based upon the determined similarities.

Claims

1. A method for processing a plurality of documents in a document database using a computer-implemented system comprising a processor and a display operatively coupled to the processor, the method comprising:

operating the processor to perform the following without requiring pre-computations

a) determining vocabulary words for each document of the plurality thereof;

b) determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents;

c) determining similarities and differences between the plurality of documents based upon the vocabulary words and their respective relevancies;

d) defining supersets of vocabulary words based on the determined similarities and differences; and

e) determining domain identifications for the supersets of vocabulary words using results of a) through d) and only after a) through d); and

operating the display to display the defined supersets of vocabulary words and their domain identifications, including display of the vocabulary words as being relevant and irrelevant based on occurrences thereof in the plurality of documents.

2. A method according to claim 1 further comprising dividing the domain identifications into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification;

and wherein relevancies of vocabulary words associated with each lower level domain identification changes so that similar documents are grouped together for each lower level domain identification.

3. A method according to claim 1 wherein determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

4. A method according to claim 3 further comprising generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities and differences between the plurality of documents.

5. A method according to claim 1 further comprising permitting user setting of the respective relevancy of at least one of the vocabulary words.

6. A method according to claim 1 further comprising determining a relevancy feedback factor;

and wherein determining the respective relevancies for each vocabulary word is further based upon the user feedback factor; and

wherein determining the similarities and differences is further based on the respective relevancies with respect to the feedback factor.

7. A method according to claim 6 wherein the relevancy feedback factor is based upon at least one of a user search query and an algorithm.

8. A method according to claim 1 wherein determining the similarities and differences are based on identifying patterns of words in the vocabulary words.

9. A method according to claim 8 wherein the pattern of words comprises n-word phrases, where n is greater than or equal to 2.

10. A method according to claim 1 further comprising:

determining vocabulary words corresponding to a predetermined domain identifications;

determining similarities and differences between the vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identifications; and

assigning the predetermined domain identifications to the plurality of documents based on the determined similarities and differences of the vocabulary words.

11. A method according to claim 10 wherein the vocabulary words corresponding to the predetermined domain identifications are user definable.

12. A method according to claim 1 wherein the domain identifications are user determinable.

13. A method according to claim 1 wherein the domain identifications are determined based upon overlapping vocabulary words.

14. A method according to claim 1 wherein the plurality of documents comprises at least one web site document.

15. A method according to claim 1 wherein the plurality of documents comprises at least one electronic mail message.

16. A method according to claim 1 wherein the plurality of documents comprises at least one non-text document having metadata associated therewith.

17. A method according to claim 16 wherein the at least one non-text documents comprises at least one of a video image file, a photograph file, and an audio file.

18. A method for processing a plurality of documents in a document database using a computer-implemented system comprising a processor and a display coupled to the processor, the method comprising:

operating the processor to perform the following without requiring pre-computations

a) determining vocabulary words for each document of the plurality thereof;

b) determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents using a computer;

c) determining similarities and differences between the plurality of documents based upon the vocabulary words and their respective relevancies;

d) defining supersets of vocabulary words based on the determined similarities and differences;

e) determining domain identifications for the supersets of vocabulary words based upon the determined similarities and differences using results of a) through d) and only after a) through d); and

dividing the domain identifications into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification, with relevancies of vocabulary words associated with each lower level domain identification changing so that similar documents are grouped together for each lower level domain identification; and

operating the display to display the defined supersets of vocabulary words and their domain identifications including the lower level domain identifications, including display of the vocabulary words as being relevant and irrelevant based on occurrences thereof in the plurality of documents.

19. A method according to claim 18 wherein determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

20. A method according to claim 19 further comprising generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities and differences between the plurality of documents.

21. A method according to claim 20 wherein determining the similarities and differences are based on identifying patterns of words in the vocabulary words.

22. A method according to claim 18 further comprising:

determining vocabulary words corresponding to predetermined domain identifications;

determining similarities and differences between the vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identifications; and

assigning the predetermined domain identifications to the plurality of documents based on the determined similarities and differences of the vocabulary words.

23. A method according to claim 18 wherein the plurality of documents comprises at least one of a web site document, an electronic mail message, and a non-text document having metadata associated therewith.

24. A computer-readable storage medium having computer-executable instructions stored thereon for causing a computer-implemented system comprising a processor and a display coupled to the processor to perform steps comprising:

operating the processor to perform the following without requiring pre-computations

a) determining vocabulary words for each document of the plurality thereof;

b) determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents;

c) determining similarities and differences between the plurality of documents based upon the vocabulary words and their respective relevancies;

d) defining supersets of vocabulary words based on the determined similarities and differences; and

e) determining domain identifications for the supersets of vocabulary words using results of a) through d) and only after a) through d); and

operating the display to display the defined supersets of vocabulary words and their domain identifications, including display of the vocabulary words as being relevant and irrelevant based on occurrences thereof in the plurality of documents.

25. A computer-readable medium according to claim 24 further comprising dividing the domain identifications into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification;

and wherein relevancies of vocabulary words associated with each lower level domain identification changes so that similar documents are grouped together for each lower level domain identification.

26. A computer-readable medium according to claim 24 wherein determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

27. A computer-readable medium according to claim 26 further comprising generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities and differences between the plurality of documents.

28. A computer-readable medium according to claim 24 wherein determining the similarities and differences are based on identifying patterns of words in the vocabulary words.

29. A computer-readable medium according to claim 24 further comprising:

determining vocabulary words corresponding to predetermined domain identifications;

determining similarities and differences between the vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identifications; and

assigning the predetermined domain identifications to the plurality of documents based on the determined similarities and differences of the vocabulary words.

30. A computer-readable medium according to claim 24 wherein the plurality of documents comprises at least one of a web site document, an electronic mail message and non-text document having metadata associated therewith.

31. A computer-implemented system comprising:

a processor for processing documents in a document database, said processor configured to perform the following without requiring pre-computations

a) determining vocabulary words for each document of the plurality thereof;

b) determining a respective relevancy for each vocabulary word based upon occurrences thereof in the plurality of documents;

c) determining similarities and differences between the plurality of documents based upon the vocabulary words and their respective relevancies;

d) defining supersets of vocabulary words based on the determined similarities and differences; and

e) determining domain identifications for the supersets of vocabulary words based upon the determined similarities and differences using results of a) through d) and only after a) through d); and

a display coupled to said processor for displaying the defined supersets of vocabulary words and their domain identifications, including display of the vocabulary words as being relevant and irrelevant based on occurrences thereof in the plurality of documents.

32. A computer-implemented system according to claim 31 wherein said processor is further configured for dividing the domain identifications into lower level domain identifications based upon selecting vocabulary words associated with each respective lower level domain identification;

and wherein relevancies of vocabulary words associated with each lower level domain identification changes so that similar documents are grouped together for each lower level domain identification.

33. A computer-implemented system according to claim 31 wherein determining the respective relevancies of the vocabulary words comprises:

counting how many times each vocabulary word is used in the plurality of documents; and

counting how many of the plurality of documents uses each of the vocabulary words.

34. A computer-implemented system according to claim 33 wherein said processor is further configured for generating a word/document ratio for each respective vocabulary word based upon the counting, and if the word/document ratio is less than a threshold, then the relevancy of the word is not used when determining similarities and differences between the plurality of documents.

35. A computer-implemented system according to claim 31 wherein determining the similarities and differences are based on identifying patterns of words in the vocabulary words.

36. A computer-implemented system according to claim 31 wherein said processor is further configured for:

determining vocabulary words corresponding to predetermined domain identifications;

determining similarities and differences between the vocabulary words for the plurality of documents and the vocabulary words corresponding to the predetermined domain identifications; and

assigning the predetermined domain identifications to the plurality of documents based on the determined similarities and differences of the vocabulary words.

37. A computer-implemented system according to claim 31 wherein the plurality of documents comprises at least one of a web site document, an electronic mail message and non-text document having metadata associated therewith.