System and method for high precision and high recall relevancy searching

• US 8,296,309 B2
• Filed: 05/28/2010
• Issued: 10/23/2012
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: H5, Inc.
• Original Assignee: H5, Inc.
• Inventors: Brassil, Daniel Paul, Hogan, Christopher Michael, Hedin, Bruce, Jade, Teresa Sunshine, Bauer, Robert S.
• Primary Examiner(s): Ly, Cheyne
• Application Number:
  US12/789,948
• Publication Number:
  US 20100306206A1
• Time in Patent Office: 879 Days
• Field of Search: 704/251, 707/748, 707/754, 707/758
• US Class Current: 707/758
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536

Abstract

A method and system for performing high precision and high recall relevancy searching is provided. According to embodiments of the present invention, a relevance rule is generated based on a user model and language from within one or more relevant and non-relevant documents. A query is created based on the relevance rule wherein the query may be applied to a corpus to identify relevant and non-relevant documents. The relevance rule may be iteratively refined in order to increase the accuracy of the query. The resulting query may be used by a litigator during the discovery phase of a litigation to respond to a request for production.

Claims

1. A computer-implemented method comprising:

generating, by a processing device, a filter for identifying a relevant document based on an initial relevance rule related to a set of documents;

applying, by the processing device, the filter to the set of documents thereby identifying a subset of relevant documents;

receiving, by the processing device from an assessor, the subset of relevant documents comprising an identification of key information;

generating, by the processing device, an updated relevance rule based on the key information and the initial relevance rule, wherein the generating comprises identifying, by the computer, a conflict between the key information and the initial relevance rule, wherein one of the key information and the initial relevance rule identifies at least one document within the set of documents to be relevant and the other of the key information and the initial relevance rule identifies the at least one document within the set of documents to be non-relevant;

generating, by the processing device, a query based on the updated relevance rule for identifying the relevant documents within the set of documents; and

outputting, by the processing device, the set of documents within which the relevant documents have been identified.

2. The computer-implemented method of claim 1, wherein generating the query for searching the set of documents comprises:

selecting one or more words from a plurality of words included in the updated relevance rule, and

generating a Boolean query based on the one or more selected words.

3. The computer-implemented method of claim 1, wherein applying the filter to the set of documents to identify the subset of relevant documents comprises:

grouping the set of documents into a plurality of subsets, wherein each subset corresponds to a topic, and

identifying the subset of relevant documents, wherein the topic corresponding to the subset of relevant documents matches a relevant word included in the filter.

4. The computer-implemented method of claim 1, wherein generating the updated relevance rule comprises:

providing the key information and the initial relevance rule to an assessor to resolve the conflict, and

receiving an updated relevance rule wherein the initial relevance rule has been altered to resolve the conflict between the initial relevance rule and the key information.

5. The computer-implemented method of claim 1, wherein the initial relevance rule comprises a list of relevant words.

6. The computer-implemented method of claim 1, further comprising:

applying the query to a second set of documents to identify a set of relevant documents and a set of non-relevant documents; and

generating an accuracy rating reflecting the accuracy with which the query correctly identified the set of relevant documents and the set of non-relevant documents.

13. The computer-implemented method of claim 1, further comprising creating a user model based on information received from a user, wherein the user model including at least four elements.

14. The computer-implemented method of claim 13 wherein the four elements comprise use case, scope, nuance and linguistic variability.

15. The computer-implemented method of claim 13 wherein the user model represents the relevant and non-relevant documents interpreted by the user.

16. The computer-implemented method of claim 13 wherein the initial relevance rule is generated based on the user model.

7. A system comprising:

a processing device;

a database coupled to the processing device;

an assessment module coupled to the processing device and the database, wherein the assessment module configured to;

generate a filter for identifying a relevant document based on an initial relevance rule related to a set of documents,

apply the filter to the set of documents thereby identifying a subset of relevant documents,

receive from an assessor, the subset of relevant documents comprising an identification of key information,

generate an updated relevance rule based on the key information and the initial relevance rule, wherein the generate comprising identify a conflict between the key information and the initial relevance rule, wherein one of the key information and the initial relevance rule identifies at least one document within the set of documents to be relevant and the other of the key information and the initial relevance rule identifies the at least one document within the set of documents to be non-relevant;

generate a query based on the updated relevance rule for identifying relevant documents within the set of documents,

and

a classification module coupled to the processing device and the database, the classification module configured to output the set of documents wherein the relevant documents have been identified.

8. The system of claim 7, wherein the assessment module is further configured to:

select one or more words from a plurality of words included in the initial relevance rule, and

generate a Boolean query including the one or more selected words.

9. The system of claim 7, wherein the assessment module is further configured to:

group the set of documents into a plurality of subsets, wherein each subset corresponds to a topic, and

identify the subset of relevant documents, wherein the topic corresponding to the subset of relevant documents matches a relevant word included in the filter.

10. The system of claim 7, wherein the assessment module is further configured to:

provide the key information and the initial relevance rule to an assessor to resolve the conflict, and

receive an updated relevance rule wherein the initial relevance rule has been altered to resolve the conflict between the initial relevance rule and the key information.

11. The system of claim 7, wherein the initial relevance rule comprises a list of relevant words.

12. The system of claim 7, further comprising a measurement module coupled to the processing device and the database, wherein the measurement module is configured to:

apply the query to a second set of documents to identify a set of relevant documents and a set of non-relevant documents; and

receive an accuracy rating reflecting the accuracy with which the query correctly identified the set of relevant documents and the set of non-relevant documents.

17. The system of claim 7 further comprising a user modeling module coupled to the processing device and the database, wherein the user modeling module is configured to:

create a user model based on information received from a user, wherein the user model including at least four elements.

18. The system of claim 17 wherein the four elements comprise use case, scope, nuance and linguistic variability.

19. The system of claim 17 wherein the user model represents the relevant and non-relevant documents interpreted by the user.

20. The system of claim 17 wherein the initial relevance rule is generated based on the user model.