Document Classification and Characterization

• US 20130246426A1
• Filed: 03/11/2013
• Published: 09/19/2013
• Est. Priority Date: 01/26/2011
• Status: Active Grant

• Current Assignee: Consilio LLC
• Original Assignee: DiscoverReady LLC (Consilio LLC)
• Inventors: Barsony, Stephen John, Messing, Yerachmiel Tzvi, Shub, David Matthew, Wagner, James Kenneth Jr.
• Granted Patent: US 9,703,863 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/358 Browsing; Visualisation therefor
• CPC Class Codes: G06Q 10/00 Administration; Management

Abstract

Data is received that characterizes each of a plurality of documents within a document set. Based on this data, the plurality of documents are grouped into a plurality of stacks using one or more grouping algorithms. A prime document is identified for each stack that includes attributes representative of the entire stack. Subsequently, provision of data is provided that characterizes documents for each stack including at least the identified prime document to at least one human reviewer. User-generated input from the human reviewer is later received that categorized each provided document and data characterizing the user-generated input can then be provided. Related apparatus, systems, techniques and articles are also described.

Claims

1. A method comprising:

receiving data characterizing each of a plurality of documents within a document set;

grouping the plurality of documents into a plurality of stacks using one or more grouping algorithms;

identifying a prime document for each stack, the prime document including attributes representative of the entire stack;

providing data characterizing documents for each stack including at least the identified prime document to at least one human reviewer;

receiving user-generated input from the human reviewer categorizing each provided document;

sending data characterizing supplemental documents within a stack other than the provided documents to at least one human reviewer for quality control; and

selecting randomized and stratified supplemental documents whose data is sent to at least one human reviewer for quality control based on an algorithm designed to select documents based on their likelihood to require remediation.

2. A method as in claim 1, further comprising:

providing data characterizing the user-generated input.

3. A method as in claim 1, wherein categorization of each provided document by the user-generated input is propagated to all documents within the corresponding stack.

4. A method as in claim 1, wherein the human reviewer categorizes each provided document in a group of document review categories.

5. A method as in claim 4, wherein the document review categories are selected from a group comprising:

relevance, responsiveness, and privilege.

6. A method as in claim 1, further comprising:

defining tiers of documents within each stack, and wherein the supplemental documents comprise documents from one or more tiers.

7. A method as in claim 6, wherein the tiers are based on one or more of:

document similarity relative to the corresponding prime document, document type, document author, document sender, document recipient.

8. A method as in claim 3, wherein documents are incrementally added to the document set after the grouping of the plurality of documents into the plurality of stacks, and wherein the method further comprises:

associating the incrementally added documents to one of the plurality of stacks; and

for each stack;

if the stack has already been categorized, adding the corresponding incrementally added documents to the stack and propagating the categorization to the incrementally added documents in such stack;

or

if the stack has not been categorized, adding the incrementally added documents to the stack.

9. A method as in claim 3, wherein at least one documents is incrementally added to the document set after the grouping of the plurality of documents into the plurality of stacks, and wherein the method further comprises:

determining that the at least one incrementally added document is not associated with a previously defined stack; and

defining a new stack including the at least one incrementally added document.

10. A method as in claim 3, further comprising:

defining hierarchical relationships among the plurality of documents within the set of documents; and

wherein the grouping algorithms take into account the relationships between documents when grouping the plurality of documents into the plurality of stacks.

11. A method as in claim 1, wherein the documents in the stacks are disjoint.

12. A method as in claim 2, wherein providing the data comprises one or more of:

displaying the data, transmitting the data to a remote computing system, and persisting the data.

13. A method as in claim 1, wherein the data characterizing documents for each stack provided to the at least one human reviewer comprise reference numbers for such documents.

14. A method as in claim 1, wherein the data characterizing documents for each stack provided to the at least one human reviewer comprise digitally scanned versions of such documents.

15. An article of manufacture comprising:

computer executable instructions stored on non-transitory computer readable media, which, when executed by a computer, causes the computer to perform operations comprising;

receiving data characterizing each of a plurality of documents within a document set;

grouping the plurality of documents into a plurality of stacks using one or more grouping algorithms;

identifying a prime document for each stack, the prime document including attributes representative of the entire stack;

providing data characterizing documents for each stack including at least the identified prime document to at least one human reviewer;

receiving user-generated input from the human reviewer categorizing each provided document;

sending data characterizing supplemental documents within a stack other than the provided documents to at least one human reviewer for quality control;

an

selecting randomized and stratified supplemental documents whose data is sent to at least one human reviewer for quality control based on an algorithm designed to select documents based on their likelihood to require remediation.

16. An article as in claim 15, wherein categorization of each provided document by the user-generated input is propagated to all documents within the corresponding stack.

17. An article as in claim 15, wherein the human reviewer categorizes each provided document in a group of document review categories.

18. An article as in claim 17, wherein the document review categories are selected from a group comprising:

relevance, responsiveness, and privilege.

19. An article as in claim 15, wherein the operations further comprise:

defining tiers of documents within each stack, and wherein the supplemental documents comprise documents from two or more tie;

wherein the tiers are based on one or more of;

document similarity relative to the corresponding prime document, document type, document author, document sender, document recipient.

20. A system comprising:

at least one data processor; and

memory storing instructions, which when executed by the at least one data processor, result in operations comprising;

receiving data characterizing each of a plurality of documents within a document set;

grouping the plurality of documents into a plurality of stacks using one or more grouping algorithms;

identifying a prime document for each stack, the prime document including attributes representative of the entire stack;

providing data characterizing documents for each stack including at least the identified prime document to at least one human reviewer;

receiving user-generated input from the human reviewer categorizing each provided document;

sending data characterizing supplemental documents within a stack other than the provided documents to at least one human reviewer for quality control;

an

selecting randomized and stratified supplemental documents whose data is sent to at least one human reviewer for quality control based on an algorithm designed to select documents based on their likelihood to require remediation.