System for enhancing expert-based computerized analysis of a set of digital documents and methods useful in conjunction therewith
First Claim
1. An electronic document analysis method receiving N electronic documents pertaining to a case encompassing a set of issues including at least one issue and establishing relevance of at least the N electronic documents to at least one individual issue in the set of issues, the method performed with a processor, the method comprising, for at least one individual issue from among said set of issues:
- i. receiving an output of a categorization process applied to documents in at least control subsets of said at least N electronic documents, said output including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a nonrelevant-to-said-individual issue indication;
ii. seeking an input as to whether or not to initiate a new iteration I;
if a new iteration is initiated, perform steps iv-x;
iii. if a new iteration is not initiated, go to step xi;
iv. receiving an output of a categorization process applied to the m electronic documents;
v. adding m electronic documents to an existing training subset and building a text classifier simulating said categorization process using said output for all documents in said training subset of documents;
vi. evaluating said text classifier'"'"'s quality using said output for documents in said control subset;
vii. selecting a cut-off point for binarizing said rankings of said documents in said control subset;
viii. using said cut-off point, computing and storing at least one quality criterion characterizing said binarizing of said rankings of said documents in said control subset, thereby to define a quality of performance indication of a current iteration I;
ix. displaying a comparison of the quality of performance indication of the current iteration I to quality of performance indications of previous iterations;
x. returning to step ii; and
xi. generating a computer display of said output of said categorization process received in step i as most recently performed, including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication.
2 Assignments
0 Petitions
Accused Products
Abstract
An electronic document analysis method receiving N electronic documents pertaining to a case encompassing a set of issues including at least one issue and establishing relevance of at least the N documents to at least one individual issue in the set of issues, the method comprising, for at least one individual issue from among the set of issues, receiving an output of a categorization process applied to each document in training and control subsets of the at least N documents, the output including, for each document in the subsets, one of a relevant-to-the-individual issue indication and a non-relevant-to-the-individual issue indication; building a text classifier simulating the categorization process using the output for all documents in the training subset of documents; and running the text classifier on the at least N documents thereby to obtain a ranking of the extent of relevance of each of the at least N documents to the individual issue. The method may also comprise evaluating the text classifier'"'"'s quality using the output for all documents in the control subset.
-
Citations
20 Claims
-
1. An electronic document analysis method receiving N electronic documents pertaining to a case encompassing a set of issues including at least one issue and establishing relevance of at least the N electronic documents to at least one individual issue in the set of issues, the method performed with a processor, the method comprising, for at least one individual issue from among said set of issues:
-
i. receiving an output of a categorization process applied to documents in at least control subsets of said at least N electronic documents, said output including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a nonrelevant-to-said-individual issue indication; ii. seeking an input as to whether or not to initiate a new iteration I;
if a new iteration is initiated, perform steps iv-x;iii. if a new iteration is not initiated, go to step xi; iv. receiving an output of a categorization process applied to the m electronic documents; v. adding m electronic documents to an existing training subset and building a text classifier simulating said categorization process using said output for all documents in said training subset of documents; vi. evaluating said text classifier'"'"'s quality using said output for documents in said control subset; vii. selecting a cut-off point for binarizing said rankings of said documents in said control subset; viii. using said cut-off point, computing and storing at least one quality criterion characterizing said binarizing of said rankings of said documents in said control subset, thereby to define a quality of performance indication of a current iteration I; ix. displaying a comparison of the quality of performance indication of the current iteration I to quality of performance indications of previous iterations; x. returning to step ii; and xi. generating a computer display of said output of said categorization process received in step i as most recently performed, including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
-
-
20. A method according to claim I comprising:
-
generating a computer display of at least one user-selected document within said set of documents, wherein at least some words in said user-selected document are differentially presented depending on their contribution to said classification of the document as relevant or irrelevant by said text classifier; sequentially removing certain sets of words from each individual document in the set of documents and using the text classifier to classify said document'"'"'s relevance assuming said words are removed, thereby to obtain a relevance output for each set of words; and comparing said relevance output to an output obtained by using the text classifier to classify said individual document without removing any words, thereby to obtain an indication of the contribution of each set of words to the relevance of the document.
-
Specification