Methods for enhancing efficiency and cost effectiveness of first pass review of documents

• US 8,150,827 B2
• Filed: 06/07/2006
• Issued: 04/03/2012
• Est. Priority Date: 06/07/2006
• Status: Active Grant

• Current Assignee: Kldiscovery Ontrack, LLC (KLDiscovery, Inc.)
• Original Assignee: Renew Data Corporation (LDiscovery LLC)
• Inventors: Kraftsow, Andrew
• Primary Examiner(s): Ali, Mohammad
• Assistant Examiner(s): Corbo, Griselle
• Application Number:
  US11/449,400
• Publication Number:
  US 20070288445A1
• Time in Patent Office: 2,127 Days
• Field of Search: 707/3
• US Class Current: 707/706
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 16/3346 using probabilistic model

Abstract

Methods for reviewing a collection of documents to identify relevant documents from the collection are provided. A search of the collection can be run based on query terms, to return a subset of responsive documents. A probability of relevancy can be determined for a document in the returned subset, and the document is removed from the subset if it does not reach a threshold probability of relevancy. Documents in a thread of a correspondence (for example, an e-mail) in the responsive documents subset can be added to the responsive documents subset. Further, an attachment to a document in the responsive documents subset can be added to the responsive documents subset. A statistical technique can be applied to determine whether remaining documents in the collection meet a predetermined acceptance level.

Claims

1. A method utilizing a computer system including a computer processor, the method for reviewing a collection of documents in electronic form stored in one or more memories wherein the computer system is configured to communicate with the one or more memories to identify relevant documents from the collection of documents, the method comprising the steps of:

receiving a plurality of query terms in a relevant language string first defined and indicated by a reviewer, via an interface controller of the computer system;

automatically creating a Boolean query by the system and without user interaction, based on the plurality of query terms in the relevant language string, by applying criteria to refine the relevant language string;

running a search of the collection of documents in the one or more memories, the search being based on the plurality of query terms of the relevant language string first defined and indicated by the reviewer, the computer processor using the Boolean query to isolate substantially all documents from the collection of documents stored in memory that are relevant to the Boolean query and to return the substantially all documents in a subset of responsive documents from the collection of documents stored in the memory;

establishing a threshold probability of relevancy for a particular reviewing operation in relation to the collection of documents;

determining a corresponding probability of relevancy for each document in the subset of responsive documents;

removing from the subset of responsive documents, documents that do not reach a threshold probability of relevancy established for the particular reviewing operation;

randomly selecting a predetermined number of documents from a remaining subset of the collection of documents not in the subset of responsive documents, wherein if the randomly selected documents includes one or more additional relevant documents, the query terms are expanded and the search is re-run with the expanded query terms; and

determining whether the randomly selected documents include additional relevant documents by further refining the Boolean query; and

comparing a ratio of the additional relevant documents and the randomly selected documents to a predetermined acceptance level, to determine whether to apply a refined set of query terms.

2. The method of claim 1, wherein the probability of relevancy of a document is scaled according to a measure of obscurity of the query terms found in the document.

3. The method of claim 1, wherein the search is applied through a search engine operated by the computer processor.

4. The method of claim 1, wherein the search using the Boolean query is a concept search, and the concept search is applied through a concept search engine operated by the computer processor.

5. The method of claim 1, further comprising the steps of:

identifying one or more specific query terms in the additional relevant documents that render the documents relevant;

expanding the query terms to include the specific terms; and

re-running at least the search with the expanded query terms.

6. The method of claim 1, further comprising the steps of:

selecting two or more search terms;

identifying synonyms of the two or more search terms; and

forming the query terms based on the search terms and synonyms.

7. The method of claim 1, further comprising the steps of:

identifying a correspondence between a sender and a recipient, in the responsive documents subset;

automatically determining one or more additional documents which are in a thread of the correspondence, the additional documents not being in the responsive documents subset; and

adding the additional documents to the responsive documents subset.

8. The method of claim 1, further comprising the steps of:

determining whether any of the documents in the responsive documents subset includes an attachment that is not in the responsive documents subset; and

adding the attachment to the responsive documents subset.

9. The method of claim 1, further comprising the step of:

applying a statistical technique to determine whether remaining documents not in the responsive documents set meet a predetermined acceptance level.

10. A method utilizing a computer system including a computer processor and a memory for reviewing a collection of documents in electronic form stored in the memory to identify relevant documents from the collection, the method comprising the steps of:

running a search of the collection of documents, the search being based on a plurality of query terms of a relevant language string first defined and indicated by a reviewer to the computer system, and subsequently utilized by the computer processor to automatically create a Boolean query by the system and without user interaction, based on applying criteria to refine the relevant language string, the computer processor using the Boolean query to isolate substantially all documents from the collection of documents stored in memory that are relevant to the Boolean query and to return the substantially all documents in a subset of responsive documents from the collection of documents stored in the memory;

establishing a threshold probability of relevancy for a particular reviewing operation;

determining a corresponding probability of relevancy for each document in the subset of responsive documents; and

removing from the subset of responsive documents, documents that do not reach a threshold probability of relevancy established for the particular reviewing operation, wherein the search includes (a) a precision Boolean search of the collection of documents based on the Boolean query utilizing the plurality of query terms, the Boolean search returning a first subset of responsive documents from the collection, and (b) a second search by applying a recall query based on the plurality of query terms to remaining ones of the collection of documents which were not returned by the Boolean search, the second search returning a second subset of responsive documents in the collection, and wherein the responsive documents subset is composed of the first and second subsets.

11. The method of claim 10, wherein the first Boolean search applies a measurable precision query based on the plurality of query terms.

12. The method of claim 11, further comprising the steps of:

automatically tagging each document in the first subset with a precision tag;

reviewing the document bearing the precision tag to determine whether the document is properly tagged with the precision tag; and

determining whether to narrow the precision query and rerun the Boolean search with the narrowed query terms.

13. The method of claim 10, further comprising the steps of:

automatically tagging each document in the second subset with a recall tag;

reviewing the document bearing the recall tag to determine whether the document is properly tagged with the recall tag; and

determining whether to narrow the recall query and rerun the second search with the narrowed query terms.

14. The method of claim 10, further comprising the step of:

reviewing the first and second subsets to determine whether to modify the query terms and rerun the Boolean search and second search with modified query term.

15. A method utilizing a computer system including a computer processor and one or more memories, the method for reviewing a collection of documents stored in electronic form in the one or more memories to identify substantially all relevant documents from the collection, the method comprising the steps of:

running a first search of the collection of documents, based on a plurality of query terms of a relevant language string first formulated by a reviewer and indicated to the computer system and used by the computer processor of the computer system to automatically create an automatic Boolean query without user interaction based on applying criteria to refine the relevant language string, the computer processor using the automatic Boolean query to isolate substantially all documents from the collection of documents stored in memory that are relevant to the Boolean query, the search returning a subset of responsive documents from the collection stored in memory including all documents relevant to the Boolean query;

utilizing the computer processor to automatically identify a correspondence between a sender of a particular document in the collection of documents stored in memory and a recipient of a particular document in the collection of documents stored in the memory, which are included in the subset of responsive documents;

automatically determining additional documents which are determined to be in a thread of the correspondence between the sender and the recipient, but not included in the subset of responsive documents;

adding the additional documents in the thread of correspondence to the subset of responsive documents;

randomly selecting a predetermined number of documents from a remainder of the collection of documents not in the responsive documents subset;

determining whether the randomly selected documents include additional relevant documents;

comparing a ratio of the additional relevant documents and the randomly selected documents to a predetermined acceptance level; and

expanding the query terms and rerunning the search with the expanded query terms, if the ratio does not meet the predetermined acceptance level.

16. The method of claim 15, wherein the search is applied through a search engine operated by the computer processor.

17. The method of claim 15, wherein the search using the Boolean query includes a concept search, and the concept search is applied through a concept search engine operated by the computer processor.

18. The method of claim 15, further comprising the steps of:

determining for each document in the subset of responsive documents, a corresponding probability of relevancy; and

removing from the subset of responsive documents those documents that do not reach a threshold probability of relevancy.

19. The method of claim 18, wherein the probability of relevancy of a document is scaled according to a measure of obscurity of the query terms found in the document.

20. The method of claim 15, further comprising the step of:

applying a statistical technique to determine whether a remaining subset of the collection of documents not in the responsive documents subset meets a predetermined acceptance level.

21. The method of claim 15, further comprising the step of:

randomly selecting a predetermined number of documents from a remainder of the collection of documents not in the responsive documents subset;

determining whether the randomly selected documents include additional relevant documents;

identifying one or more specific terms in the additional relevant documents that render the documents relevant;

expanding the query terms to include the specific terms; and

re-running the search with the expanded query terms.

22. The method of claim 15, further comprising the steps of:

selecting two or more search terms;

identifying synonyms of the search terms; and

forming the query terms based on the search terms and synonyms.

23. The method of claim 15, further comprising the step of:

determining whether any of the responsive documents in the responsive documents subset includes an attachment that is not in the subset; and

adding the attachment to the subset.