Methods for enhancing efficiency and cost effectiveness of first pass review of documents

• US 20070288445A1
• Filed: 06/07/2006
• Published: 12/13/2007
• Est. Priority Date: 06/07/2006
• Status: Active Grant

• Current Assignee: Kldiscovery Ontrack, LLC (KLDiscovery, Inc.)
• Original Assignee: Digital Mandate LLC (LDiscovery LLC)
• Inventors: Kraftsow, Andrew
• Granted Patent: US 8,150,827 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 16/3346 using probabilistic model

Abstract

Methods for reviewing a collection of documents to identify relevant documents from the collection are provided. A search of the collection can be run based on query terms, to return a subset of responsive documents. A probability of relevancy can be determined for a document in the returned subset, and the document is removed from the subset if it does not reach a threshold probability of relevancy. Documents in a thread of a correspondence (for example, an e-mail) in the responsive documents subset can be added to the responsive documents subset. Further, an attachment to a document in the responsive documents subset can be added to the responsive documents subset. A statistical technique can be applied to determine whether remaining documents in the collection meet a predetermined acceptance level.

Claims

1. A method for reviewing a collection of documents to identify relevant documents from the collection, the method comprising:

running a search of the collection of documents, the search being based on a plurality of query terms and returning a subset of responsive documents from the collection;

determining a corresponding probability of relevancy for each document in the responsive documents subset; and

removing from the responsive documents subset, documents that do not reach a threshold probability of relevancy.

2. The method of claim 1, wherein the probability of relevancy of a document is scaled according to a measure of obscurity of the search terms found in the document.

3. The method of claim 1, wherein the search is applied through a search engine.

4. The method of claim 1, wherein the search includes a concept search, and the concept search is applied through a concept search engine.

5. The method of claim 1, further comprising:

randomly selecting a predetermined number of documents from a remaining subset of the collection of documents not in the responsive documents subset; and

determining whether the randomly selected documents include additional relevant documents.

6. The method of claim 5, further comprising:

identifying one or more specific terms in the additional relevant documents that render the documents relevant;

expanding the query terms with the specific terms; and

re-running at least the search with the expanded query terms.

7. The method of claim 5, wherein if the randomly selected documents includes one or more additional relevant documents, the query terms are expanded and the search is re-run with the expanded query terms.

8. The method of claim 7, further comprising comparing a ratio of the additional relevant documents and the randomly selected documents to a predetermined acceptance level, to determine whether to apply a refined set of query terms.

9. The method of claim 1, further comprising:

selecting two or more search terms;

identifying synonyms of the search terms; and

forming the query terms based on the search terms and synonyms.

10. The method of claim 1, further comprising:

identifying a correspondence between a sender and a recipient, in the responsive documents subset;

automatically determining one or more additional documents which are in a thread of the correspondence, the additional documents not being in the responsive documents subset; and

adding the additional documents to the responsive documents subset.

11. The method of claim 1, further comprising:

determining whether any of the documents in the responsive documents subset includes an attachment that is not in the responsive documents subset; and

adding the attachment to the responsive documents subset.

12. The method of claim 1, further comprising applying a statistical technique to determine whether remaining documents not in the responsive documents set meets a predetermined acceptance level.

13. The method of claim 1, wherein the search includes (a) a Boolean search of the collection of documents based on the plurality of query terms, the Boolean search returning a first subset of responsive documents from the collection, and (b) a second search by applying a recall query based on the plurality of query terms to remaining ones of the collection of documents which were not returned by the Boolean search, the second search returning a second subset of responsive documents in the collection, and wherein the responsive documents subset is constituted by the first and second subsets.

14. The method of claim 13, wherein the first Boolean search applies a measurable precision query based on the plurality of query terms.

15. The method of claim 14, further comprising:

automatically tagging each document in the first subset with a precision tag;

reviewing the document bearing the precision tag to determine whether the document is properly tagged with the precision tag; and

determining whether to narrow the precision query and rerun the Boolean search with the narrowed query terms.

16. The method of claim 13, further comprising:

automatically tagging each document in the second subset with a recall tag;

reviewing the document bearing the recall tag to determine whether the document is properly tagged with the recall tag; and

determining whether to narrow the recall query and rerun the second search with the narrowed query terms.

17. The method of claim 13, further comprising reviewing the first and second subsets to determine whether to modify the query terms and rerun the Boolean search and second search with modified query terms.

18. A method for reviewing a collection of documents to identify relevant documents from the collection, the method comprising:

running a search of the collection of documents, based on a plurality of query terms, the search returning a subset of responsive documents in the collection;

automatically identifying a correspondence between a sender and a recipient, in the responsive documents subset;

automatically determining one or more additional documents which are in a thread of the correspondence, the additional documents not being in the responsive documents subset; and

adding the additional documents to the responsive documents subset.

19. The method of claim 18, wherein the search is applied through a search engine.

20. The method of claim 18, wherein the search includes a concept search, and the concept search is applied through a concept search engine.

21. The method of claim 18, further comprising:

determining for each document in the responsive documents subset, a corresponding probability of relevancy; and

removing from the responsive documents subset documents that do not reach a threshold probability of relevancy.

22. The method of claim 21, wherein the probability of relevancy of a document is scaled according to a measure of obscurity of the search terms found in the document.

23. The method of claim 18, further comprising applying a statistical technique to determine whether a remaining subset of the collection of documents not in the responsive documents subset meets a predetermined acceptance level.

24. The method of claim 18, further comprising:

randomly selecting a predetermined number of documents from a remainder of the collection of documents not in the responsive documents subset;

determining whether the randomly selected documents include additional relevant documents;

identifying one or more specific terms in the additional relevant documents that render the documents relevant;

expanding the query terms with the specific terms; and

re-running the search with the expanded query terms.

25. The method of claim 18, further comprising:

randomly selecting a predetermined number of documents from a remainder of the collection of documents not in the responsive documents subset;

determining whether the randomly selected documents include additional relevant documents;

comparing a ratio of the additional relevant documents and the randomly selected documents to a predetermined acceptance level; and

expanding the query terms and rerunning the search with the expanded query terms, if the ratio does not meet the predetermined acceptance level.

26. The method of claim 18, further comprising:

selecting two or more search terms;

identifying synonyms of the search terms; and

forming the query terms based on the search terms and synonyms.

27. The method of claim 18, further comprising:

determining whether any of the responsive documents in the responsive documents subset includes an attachment that is not in the subset; and

adding the attachment to the subset.

28. A method for reviewing a collection of documents to identify relevant documents from the collection, the method comprising:

running a search of the collection of documents, based on a plurality of query terms, the search returning a subset of responsive documents in the collection;

automatically determining whether any of the responsive documents in the responsive documents subset includes an attachment that is not in the subset; and

adding the attachment to the responsive documents subset.

29. The method of claim 28, wherein the search is applied through a search engine.

30. The method of claim 28, wherein the search includes a concept search, and the concept search is applied through a concept search engine.

31. The method of claim 28, further comprising:

determining for each document in the responsive documents subset, a corresponding probability of relevancy; and

removing from the responsive documents subset documents that do not reach a threshold probability of relevancy.

32. The method of claim 31, wherein the probability of relevancy of a document is scaled according to a measure of obscurity of the search terms found in the document.

33. The method of claim 28, further comprising applying a statistical technique to determine whether a remaining subset of the collection of documents not in the responsive documents subset meets a predetermined acceptance level.

34. The method of claim 28, further comprising:

randomly selecting a predetermined number of documents from a remainder of the collection of documents not in the responsive documents subset;

determining whether the randomly selected documents include additional relevant documents;

identifying one or more specific terms in the additional responsive documents that render the documents relevant;

expanding the query terms with the specific terms; and

re-running the search with the expanded query terms.

35. The method of claim 28, further comprising:

selecting two or more search terms;

identifying synonyms of the search terms; and

forming the query terms based on the search terms and synonyms.

36. The method of claim 28, further comprising:

identifying a correspondence between a sender and a recipient, in the responsive documents subset;

automatically determining one or more additional documents which are in a thread of the correspondence, the additional documents not being in the responsive documents subset; and

adding the additional documents to the responsive documents subset.

37. A method for reviewing a collection of documents to identify relevant documents from the collection, the method comprising:

running a search of the collection of documents, based on a plurality of query terms, the search returning a subset of responsive documents from the collection;

randomly selecting a predetermined number of documents from a remainder of the collection of documents not in the responsive documents subset;

determining whether the randomly selected documents include additional relevant documents;

identifying one or more specific terms in the additional responsive documents that render the documents relevant;

expanding the query terms with the specific terms; and

re-running the search with the expanded query terms.