System and method for propagating classification decisions

• US 8,296,290 B2
• Filed: 02/04/2011
• Issued: 10/23/2012
• Est. Priority Date: 02/05/2010
• Status: Expired due to Fees

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: FTI Consulting Incorporated
• Inventors: Robinson, Eric Michael, Gabriel, Manfred J.
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US13/021,618
• Publication Number:
  US 20110196879A1
• Time in Patent Office: 627 Days
• Field of Search: None
• US Class Current: 707/722
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G06Q 50/18 Legal services

Abstract

A system and method for propagating classification decisions is provided. Text marked within one or more unclassified documents that is determined to be responsive to a predetermined issue is received from a user. The unclassified documents are selected from a corpus. A search query is generated from the responsive text. Same result documents are identified by applying inclusive search parameters to the query, applying the search query to the corpus, and identifying the documents that satisfy the query. Similar result documents are identified by adjusting a breadth of the query by applying less inclusive search parameters and identifying documents from the corpus that satisfy the query. A responsive classification code is automatically assigned to each same result document for classification as responsive documents. The similar documents are provided to the user. A responsive classification decision is received from the user for classification as the responsive documents.

Claims

1. A system for propagating classification decisions, comprising:

a quantity module to determine a quantity of unclassified documents for review by a user, wherein the quantity of the unclassified documents is determined in accordance with the equation;

2. A system according to claim 1, further comprising:

a selection module to select further unclassified documents from the corpus based on a similarity of each further unclassified document to the responsive text.

3. A system according to claim 1, further comprising:

a document calculation module to determine a total expected number of the responsive documents in the corpus and to provide the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.

4. A system according to claim 3, wherein the total expected number of responsive documents is calculated based on a number of the responsive documents and a total number of documents in the corpus.

5. A system according to claim 1, wherein the marked text is identified by a marking indicator comprising at least one of a text box, highlighting, underlining, font style, and font size.

6. A system according to claim 1, wherein the query generator generates at least one further search query based on a further text selection that is responsive to the predetermined issue and identifies further responsive documents as those that satisfy the further search query.

7. A system according to claim 1, further comprising:

a validator to select a sample of the unclassified documents remaining in the corpus, to analyze the selected unclassified documents for responsive text, and to transmit further unclassified documents from the corpus to the user for review when at least one of the selected unclassified documents includes responsive text.

8. A system according to claim 7, further comprising:

a document calculation module to calculate a number of the unclassified documents for inclusion in the sample in accordance with the equation;

9. A system according to claim 1, further comprising:

a quality control module to edit the search query based on feedback from the user.

10. A method for propagating classification decisions, comprising:

determining a quantity of unclassified documents for review by a user, wherein the quantity of the unclassified documents is determined in accordance with the equation;

11. A method according to claim 10, further comprising:

selecting further unclassified documents from the corpus based on a similarity of each of the further unclassified documents to the responsive text.

12. A method according to claim 10, further comprising:

determining a total expected number of the responsive documents in the corpus; and

providing the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.

13. A method according to claim 12, wherein the total expected number of responsive documents is calculated based on a a number of the responsive documents and a total number of documents in the corpus.

14. A method according to claim 10, wherein the marked text is identified by a marking indicator comprising at least one of a text box, highlighting, underlining, font style, and font size.

15. A method according to claim 10, further comprising:

identifying further responsive documents for the predetermined issue, comprising;

generating at least one further search query based on a further text selection that is responsive to the predetermined issue; and

selecting the further responsive documents as those that satisfy the further search query.

16. A method according to claim 10, further comprising:

selecting a sample of the unclassified documents remaining in the corpus;

analyzing the selected unclassified documents for responsive text; and

transmitting further unclassified documents from the corpus to the user for review when at least one of the selected unclassified documents includes responsive text.

17. A method according to claim 16, further comprising:

calculating a number of unclassified documents for inclusion in the sample in accordance with the equation;

18. A method according to claim 10, further comprising:

editing the search query based on feedback from the user.

19. A system for propagating classification decisions, comprising:

text marked within one or more unclassified documents that is responsive to a predetermined issue, wherein the one or more unclassified documents are selected from a corpus of unclassified documents;

a query generator to generate a search query from the responsive text and to identify result documents, comprising;

a same search module to identify same result documents by applying inclusive search parameters to the query, by applying the search query to the corpus, and by identifying the documents that satisfy the query as the same result documents; and

a similar search module to identify similar result documents from the corpus by adjusting a breath of the query by applying less inclusive search parameters and by identifying documents from the corpus that satisfy the query as the similar result documents;

a propagator to automatically assign a responsive classification code to the same result documents for classification as responsive documents;

a document feeder to provide the similar documents to the user and to receive a responsive classification decision from the user for at least one of the similar documents for classification as the responsive documents;

a validation module to provide a number of the unclassified documents remaining in the corpus for further review, wherein the number of remaining unclassified documents is determined in accordance with the equation;

20. A system according to claim 19, further comprising:

a document selection module to randomly select the M remaining unclassified documents for review.

21. A system according to claim 19, further comprising:

a responsive document determination module to perform one of determining that all the responsive documents in the corpus have been identified when none of the M remaining unclassified documents include responsive material and selecting further remaining unclassified documents for review when at least one of the remaining unclassified documents includes responsive material.

22. A system according to claim 19, wherein the document feeder further provides the one or more unclassified documents to the user for review via at least one of randomly selecting each of the one or more unclassified documents, selecting each of the one or more unclassified documents based on a similarity of that unclassified document to the responsive text, and randomly selecting a first set of documents from the one or more unclassified documents and selecting the remaining unclassified documents from the one or more unclassified documents based on the similarity of that unclassified document to the responsive text.

23. A system according to claim 19, further comprising:

a document calculation module to determine a total expected number of the responsive documents in the corpus, wherein the total expected number of responsive documents is calculated based on a number of the responsive documents and a total number of documents in the corpus, and to further provide the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.

24. A method for propagating classification decisions, comprising:

receiving from a user, text marked within one or more unclassified documents that is responsive to a predetermined issue, wherein the one or more unclassified documents are selected from a corpus of unclassified documents;

generating a search query from the responsive text;

identifying same result documents, comprising;

applying inclusive search parameters to the query;

applying the search query to the corpus; and

identifying the documents that satisfy the query as the same result documents;

identifying similar result documents from the corpus, comprising;

adjusting a breath of the query by applying less inclusive search parameters; and

identifying documents from the corpus that satisfy the query as the similar result documents;

automatically assigning a responsive classification code to the same result documents for classification as responsive documents;

providing the similar documents to the user and receiving a responsive classification decision from the user for at least one of the similar documents for classification as the responsive documents; and

providing a number of the unclassified documents remaining in the corpus for further review, wherein the number of remaining unclassified documents for further review is determined in accordance with the equation;

25. A method according to claim 24, further comprising:

randomly selecting the M remaining unclassified documents for review.

26. A method according to claim 24, further comprising at least one of:

determining that all the responsive documents in the corpus have been identified when none of the M remaining unclassified documents include responsive text; and

selecting further remaining unclassified documents for review when at least one of the remaining unclassified documents includes responsive text.

27. A method according to claim 24, further comprising:

providing the one or more unclassified documents to the user for review, comprising at least one of;

randomly selecting each of the one or more unclassified documents;

selecting each of the one or more unclassified documents based on a similarity of that unclassified document to the responsive text; and

randomly selecting a first set of documents from the one or more unclassified documents and selecting the remaining documents from the one or more unclassified documents based on the similarity of that unclassified document to the responsive text.

28. A method according to claim 24, further comprising:

determining a total expected number of the responsive documents in the corpus based on a number of the responsive documents and a total number of documents in the corpus; and

providing the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.