Techniques for organizing data to support efficient review and analysis

• US 7,945,600 B1
• Filed: 03/04/2005
• Issued: 05/17/2011
• Est. Priority Date: 05/18/2001
• Status: Expired due to Fees

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Stratify, Inc. (Open Text Corporation)
• Inventors: Lakhamraju, Mohana Krishna, Ramana, Gollakota Venkata, Thomas, Joy Aloysius, Mathew, George Manianghat, Nayak, Pangal Pandurang, Lamping, John O.
• Primary Examiner(s): Breene; John
• Assistant Examiner(s): Kerzhner; Aleksandr
• Application Number:
  US11/073,338
• Time in Patent Office: 2,265 Days
• Field of Search: 707/804, 707/999.102
• US Class Current: 707/804
• CPC Class Codes: G06F 16/353 into predefined classes

Abstract

Techniques for organizing a corpus of electronic documents. The electronic documents are organized in a manner that facilitates review of the documents. The documents are organized into a concept-based hierarchical collection of folders based upon contents of the documents.

Claims

1. A method comprising:

generating, by a computer system, a summary representation for each electronic document in a plurality of electronic documents, the summary representation representing a summary of content of the electronic document;

filtering, by the computer system, the plurality of electronic documents based upon their summary representations to generate a filtered subset;

organizing, by the computer system, the electronic documents in the filtered subset into a hierarchical collection of folders, the organizing comprising;

determining similarity metrics between the electronic documents in the filtered subset based upon their summary representations;

determining concepts associated with the electronic documents in the filtered subset based upon their summary representations; and

grouping the electronic documents in the filtered subset into the hierarchical collection of folders based upon the similarity metrics and the concepts; and

assigning, by the computer system based upon a set of rules pertaining to document similarity, a number to each electronic document in the filtered subset by traversing the hierarchical collection of folders, such that when the electronic documents in the filtered subset are sorted based upon the assigned numbers to generate a sorted list of electronic documents, related electronic documents occur consecutively in the sorted list of electronic documents.

2. The method of claim 1 further comprising:

displaying a graphical user interface (GUI) to facilitate review of the electronic documents in the filtered subset, the GUI comprising;

a first region displaying the hierarchical collection of folders;

a second region displaying a list of electronic documents grouped in a selected folder from the hierarchical collection of folders;

a third region displaying an electronic document selected from the list of electronic documents.

3. The method of claim 2 further comprising:

identifying at least one set of near duplicate documents from the filtered subset, where the similarity metric between any two electronic documents in the set of near duplicate documents satisfies a threshold.

4. The method of claim 3 wherein, for any one electronic document from the at least one set of near duplicate documents, the GUI provides a control for displaying information identifying the other electronic documents in the at least one set of near duplicate documents.

5. The method of claim 2 further comprising:

enabling, using the GUI, association of one or more electronic documents from the filtered subset with one or more tags from a plurality of tags.

6. The method of claim 2 further comprising:

enabling, using the GUI, assignment of one or more electronic documents from the filtered subset to one or more users from a plurality of users.

7. The method of claim 2 further comprising:

enabling, using the GUI, an action to be performed on multiple electronic documents from the filtered subset.

8. The method of claim 1 further comprising:

for at least one folder in the hierarchical collection of folders, organizing electronic documents grouped in the at least one folder into an organized list of electronic documents such that related electronic documents occur next to each other in the organized list.

9. The method of claim 8 wherein organizing the electronic documents grouped in the at least one folder comprises:

assigning a number to each electronic document grouped in the at least one folder based upon the set of rules pertaining to document similarity, such that when the electronic documents grouped in the at least one folder are sorted based upon the assigned numbers to generate the organized list of electronic documents, related electronic documents occur consecutively in the organized list of electronic documents.

10. The method of claim 1 wherein grouping the electronic documents in the filtered subset comprises:

generating the hierarchical collection of folders using an existing folder organization comprising a set of folders, the hierarchical collection of folders comprising at least the set of folders, wherein a first document from the filtered subset that is related to a first concept is placed in a folder from the existing folder organization if the existing folder organization comprises a folder representing the first concept, and wherein a new folder is created to store the first document if the existing folder organization does not comprise a folder representing the first concept.

11. The method of claim 10 wherein the existing folder organization is generated based upon a set of keywords or a set of rules.

12. The method of claim 1 wherein the summary representation for each electronic document is a high-dimensional space vector that includes pairs of words and scores, each score representing a number of occurrences of a corresponding word in the electronic document.

13. A system comprising:

a memory configured to store a plurality of electronic documents; and

a processor;

wherein the processor is configured to;

generate a summary representation of each electronic document in the plurality of electronic documents, the summary representation representing a summary of content of the electronic document;

filter the plurality of electronic documents based upon their summary representations to generate a filtered subset;

organize the electronic documents in the filtered subset into a hierarchical collection of folders, the organizing comprising;

determining similarity metrics between the electronic documents in the filtered subset based upon their summary representations;

determining concepts associated with the electronic documents in the filtered subset based upon their summary representations; and

grouping the electronic documents in the filtered subset into the hierarchical collection of folders based upon the similarity metrics and the concepts; and

assign, based upon a set of rules pertaining to document similarity, a number to each electronic document in the filtered subset by traversing the hierarchical collection of folders, such that when the electronic documents in the filtered subset are sorted based upon the assigned numbers to generate a sorted list of electronic documents, related electronic documents occur consecutively in the sorted list of electronic documents.

14. The system of claim 13 further comprising an output device, wherein the processor is configured to:

display a graphical user interface (GUI) on the output device to facilitate review of the electronic documents in the filtered subset, the GUI comprising;

a first region displaying the hierarchical collection of folders;

a second region displaying a list of electronic documents grouped in a selected folder from the hierarchical collection of folders;

a third region displaying a first electronic document selected from the list of electronic documents.

15. The system of claim 14 wherein the processor is configured to:

identify at least one set of near duplicate documents from the filtered subset, where the similarity metric between any two electronic documents in the set of near duplicate documents satisfies a threshold.

16. The system of claim 15 wherein, for any one electronic document from the at least one set of near duplicate documents, the GUI provides a control for displaying information identifying the other electronic documents in the at least one set of near duplicate documents.

17. The system of claim 14 wherein the processor is configured to:

enable, using the GUI, association of one or more electronic documents from the filtered subset with one or more tags from a plurality of tags.

18. The system of claim 14 wherein the processor is configured to:

enable, using the GUI, assignment of one or more electronic documents from the filtered subset to one or more users from a plurality of users.

19. The system of claim 14 wherein the processor is configured to:

enable, using the GUI, an action to be performed on multiple electronic documents from the filtered subset.

20. The system of claim 13 wherein the processor is configured to:

for at least one folder in the hierarchical collection of folders, organize electronic documents grouped in the at least one folder into an organized list of electronic documents such that related electronic documents occur next to each other in the organized list.

21. The system of claim 20 wherein the processor is configured to:

assign a document number to each electronic document grouped in the at least one folder based upon the set of rules pertaining to document similarity, such that when the electronic documents grouped in the at least one folder are sorted based upon the assigned numbers to generate the organized list of electronic documents, related electronic documents occur consecutively in the organized list of electronic documents.

22. The system of claim 13 wherein the processor is configured to:

generate the hierarchical collection of folders using an existing folder organization comprising a set of folders, the hierarchical collection of folders comprising at least the set of folders, wherein a first document from the filtered subset that is related to a first concept is placed in a folder from the existing folder organization if the existing folder organization comprises a folder representing the first concept, and wherein a new folder is created to store the first document if the existing folder organization does not comprise a folder representing the first concept.

23. The system of claim 22 wherein the existing folder organization is generated based upon a set of keywords or a set of rules.

24. A computer-readable medium storing a plurality of instructions for controlling a data processor, the plurality of instructions comprising:

instructions that cause the data processor to generate a summary representation for each electronic document in a plurality of electronic documents, the summary representation representing a summary of content of the electronic document;

instructions that cause the data processor to filter the plurality of electronic documents based upon their summary representations to generate a filtered subset;

instructions that cause the data processor to organize the electronic documents in the filtered subset into a hierarchical collection of folders, the instructions that cause the data processor to organize comprising;

instructions that cause the data processor to determine similarity metrics between the electronic documents in the filtered subset based upon their summary representations;

instructions that cause the data processor to determine concepts associated with the electronic documents in the filtered subset based upon their summary representations; and

instructions that cause the data processor to group the electronic documents in the filtered subset into the hierarchical collection of folders based upon the similarity metrics and the concepts; and

instructions that cause the data processor to assign, based upon a set of rules pertaining to document similarity, a number to each electronic document in the filtered subset by traversing the hierarchical collection of folders, such that when the electronic documents in the filtered subset are sorted based upon the assigned numbers to generate a sorted list of electronic documents, related electronic documents occur consecutively in the sorted list of electronic documents.

25. The computer-readable medium of claim 24 wherein the plurality of instructions comprises:

instructions that cause the data processor to generate a graphical user interface (GUI) to facilitate review of the electronic documents in the filtered subset, the GUI comprising;

a first region displaying the hierarchical collection of folders;

a second region displaying a list of electronic documents grouped in a selected folder from the hierarchical collection of folders;

a third region displaying a first electronic document selected from the list of electronic documents.

26. The computer-readable medium of claim 25 wherein the plurality of instructions comprises:

instructions that cause the data processor to identify at least one set of near duplicate documents from the filtered subset, where the similarity metric between any two electronic documents in the set of near duplicate documents satisfies a threshold; and

wherein, for any one electronic document from the at least one set of near duplicate documents, the GUI provides a control for displaying information identifying the other electronic documents in the at least one set of near duplicate documents.

27. The computer-readable medium of claim 25 wherein the plurality of instructions comprises:

instructions that cause the data processor to enable association of one or more electronic documents from the filtered subset with one or more tags from a plurality of tags; and

instructions that cause the data processor to enable assignment of one or more electronic documents from the filtered subset to one or more users from a plurality of users.

28. The computer-readable medium of claim 25 wherein the plurality of instructions comprises:

instructions that cause the data processor to enable performing an action on multiple electronic documents from the filtered subset.

29. The computer-readable medium of claim 24 wherein the plurality of instructions comprises:

instructions that cause the data processor to, for at least one folder in the hierarchical collection of folders, organize electronic documents grouped in the at least one folder into an organized list of electronic documents such that related electronic documents occur next to each other in the organized list.

30. The computer-readable medium of claim 29 wherein the plurality of instructions comprises:

instructions that cause the data processor to assign a number to each electronic document grouped in the at least one folder based upon the set of rules pertaining to document similarity, such that when the electronic documents grouped in the at least one folder are sorted based upon the assigned numbers to generate the organized list of electronic documents, related electronic documents occur consecutively in the organized list of electronic documents.

31. The computer-readable medium of claim 24 wherein the instructions that cause the data processor to group the electronic documents in the filtered subset comprise instructions that cause the processor to generate the hierarchical collection of folders using an existing folder organization comprising a set of folders, the hierarchical collection of folders comprising at least the set of folders, wherein a first document from the filtered subset that is related to a first concept is placed in a folder from the existing folder organization if the existing folder organization comprises a folder representing the first concept, and wherein a new folder is created to store the first document if the existing folder organization does not comprise a folder representing the first concept.

32. The computer-readable medium of claim 31 wherein the existing folder organization is generated based upon a set of keywords or a set of rules.