SYSTEM AND METHOD FOR ASSISTED DOCUMENT REVIEW
First Claim
1. A method of reviewing documents comprising:
- partitioning a collection of documents into sets of documents for review by a plurality of reviewers;
for each set;
organizing the documents in the set into a plurality of groups,displaying documents in the set on a display device for review by a reviewer,receiving the reviewer'"'"'s labels for the displayed documents,based on the reviewer'"'"'s labels, assigning a class from a plurality of classes to each of the reviewed documents,progressively training a classifier model stored in computer memory based on features extracted from the reviewed documents in the set and their assigned classes; and
prior to review of all documents in the set, identifying documents in the set for which the classifier model assigns a class different from the one assigned based on the reviewer'"'"'s label and returning a subset of the identified documents for a second review by a reviewer.
7 Assignments
0 Petitions
Accused Products
Abstract
A system and method for reviewing documents are provided. A collection of documents is portioned into sets of documents for review by a plurality of reviewers. For each set, documents in the set are displayed on a display device for review by a reviewer and temporarily organized through grouping and sorting. The reviewer'"'"'s labels for the displayed documents are received. Based on the reviewer'"'"'s labels, a class from a plurality of classes is assigned to each of the reviewed documents. A classifier model stored in computer memory is progressively trained, based on features extracted from the reviewed documents in the set and their assigned classes. Prior to review of all documents in the set, a calculated subset of documents for which the classifier model assigns a class different from the one assigned based on the reviewer'"'"'s label is returned for a second review by a reviewer. Models generated from one or more other document sets can be used to assess the review of a first of the sets.
-
Citations
23 Claims
-
1. A method of reviewing documents comprising:
-
partitioning a collection of documents into sets of documents for review by a plurality of reviewers; for each set; organizing the documents in the set into a plurality of groups, displaying documents in the set on a display device for review by a reviewer, receiving the reviewer'"'"'s labels for the displayed documents, based on the reviewer'"'"'s labels, assigning a class from a plurality of classes to each of the reviewed documents, progressively training a classifier model stored in computer memory based on features extracted from the reviewed documents in the set and their assigned classes; and prior to review of all documents in the set, identifying documents in the set for which the classifier model assigns a class different from the one assigned based on the reviewer'"'"'s label and returning a subset of the identified documents for a second review by a reviewer. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20)
-
-
21. A reviewing system comprising:
-
memory which stores a set of documents for review; a display which displays documents in the set; a user input device for receiving information from which class labels applied to the documents by a reviewer are determined; a classifier model which is progressively trained based on the applied class labels; a document reviewing application stored in memory which, prior to all documents in the set being labeled, compares a class assigned by the classifier for a labeled document to the label applied by the reviewer and identifies labeled documents where the class label applied by the reviewer does not match the class assigned by the reviewer and returns, for a second review, a subset of the identified labeled documents. - View Dependent Claims (22)
-
-
23. A method of reviewing documents comprising:
-
partitioning a collection of documents into sets of documents; for a first of the sets of documents, generating a first classifier model based on reviewer-applied labels for documents in the first set; for a second of the sets of documents, generating a second classifier model based on reviewer-applied labels for documents in the second set; and assessing a quality of the labels applied to the first set of documents including; assigning classes to the documents from the first set of documents with the second classifier model, and comparing the reviewer-applied labels of the documents in the first set with the classes assigned by the second classifier model to identify documents for which the reviewer-applied labels do not match the assigned classes.
-
Specification