Method of tuning a computer system

• US 10,558,713 B2
• Filed: 10/02/2018
• Issued: 02/11/2020
• Est. Priority Date: 07/13/2018
• Status: Active Grant

• Current Assignee: Matthew Lavy
• Original Assignee: ResponsiML Ltd.
• Inventors: Lavy, Matthew, Turnbull, Alistair
• Primary Examiner(s): Hershley, Mark E
• Application Number:
  US16/149,523
• Publication Number:
  US 20200019615A1
• Time in Patent Office: 497 Days
• Field of Search: 707737
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Disclosed herein is a method for tuning a computer system suitable for classifying documents or to locating texts satisfying a criterion in documents. Examples of the computer system may be used for electronic discovery or to exercise due diligence in a transaction. The method includes obtaining a target of accuracy of the computer system and tuning the accuracy of the computer system by adjusting a characteristic of the computer system based on the target.

Claims

1. A method comprising:

obtaining a target of accuracy of a computer system configured to classify documents or to locate texts satisfying a criterion in documents;

determining classification of a set of documents or location of the texts satisfying the criterion in the set of documents thereby labeling the set of documents, using the computer system, wherein the set of documents are previously unlabeled;

performing validation of the classification or the location of the texts, in which the classification or the location is either upheld or overturned; and

tuning the accuracy of the computer system by adjusting an amount of the validation based on the target;

wherein adjusting the amount of the validation comprises;

for each of the set of documents and each of a plurality of hypotheses on a largest value and a smallest value of scores for the set of documents, the scores respectively representing probabilities of the classification or location of the texts of the set of documents, generating a hypothesized score as a function of the largest value and the smallest value;

initializing credibilities respectively for the hypotheses as an equal value, the credibilities respectively representing probabilities of the hypotheses correctly estimating their respective largest value and smallest value; and

adjusting the credibilities using Bayesian inference, based on the hypothesized scores for the set of documents and the hypotheses and results of the validation.

2. The method of claim 1, wherein obtaining the target is based on a measure of risks caused by errors of the computer system.

3. The method of claim 1, further comprising determining a measure of the accuracy;

wherein tuning the accuracy of the computer system further comprises comparing the measure with the target.

4. The method of claim 1, wherein tuning the accuracy of the computer system further comprises adjusting an amount of training of the computer system.

5. The method of claim 1, wherein the computer system is configured to iteratively receive the validation.

6. The method of claim 1, wherein tuning the accuracy of the computer system further comprises adjusting a bias of the computer system.

7. The method of claim 1, wherein the computer system comprises a plurality of nodes, the nodes comprising one or more processors;

wherein tuning the accuracy of the computer system further comprises adjusting an attribute of couplings among the nodes.

8. The method of claim 1, wherein the computer system is configured to classify the documents into a plurality of classes.

9. The method of claim 1, wherein the computer system is probabilistic.

10. The method of claim 1, wherein the accuracy is a function of true positives, false positives, true negatives or false negatives of the computer system.

11. The method of claim 1, wherein the accuracy is a function of precision, recall or F-score of the computer system with respect to a class.

12. The method of claim 1, further comprising:

calculating a cumulative probability of the target of accuracy having been met for each of the hypotheses;

calculating an aggregate probability of the target of accuracy based on the cumulative probability and the credibilities; and

ceasing tuning of the computer system when the aggregate probability of the target of accuracy reaches a confidence threshold.

13. The method of claim 1, wherein the function is a harmonic function of the largest value and the smallest value.

14. A computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing a method comprising:

obtaining a target of accuracy of a computer system configured to classify documents or to locate texts satisfying a criterion in documents;

determining classification of a set of documents or location of the texts satisfying the criterion in the set of documents thereby labeling the set of documents, using the computer system, wherein the set of documents are previously unlabeled;

performing validation of the classification or the location of the texts, in which the classification or the location is either upheld or overturned; and

tuning the accuracy of the computer system by adjusting an amount of the validation based on the target;

wherein adjusting the amount of the validation comprises;

for each of the set of documents and each of a plurality of hypotheses on a largest value and a smallest value of scores for the set of documents, the scores respectively representing probabilities of the classification or location of the texts of the set of documents, generating a hypothesized score as a function of the largest value and the smallest value;

initializing credibilities respectively for the hypotheses as an equal value, the credibilities respectively representing probabilities of the hypotheses correctly estimating their respective largest value and smallest value; and

adjusting the credibilities using Bayesian inference, based on the hypothesized scores for the set of documents and the hypotheses and results of the validation.

15. The computer program product of claim 14, wherein obtaining the target is based on a measure of risks caused by errors of the computer system.

16. The computer program product of claim 14, wherein the method further comprises determining a measure of the accuracy;

wherein tuning the accuracy of the computer system further comprises comparing the measure with the target.

17. The computer program product of claim 14, wherein tuning the accuracy of the computer system further comprises adjusting an amount of training of the computer system.

18. The computer program product of claim 14, wherein the computer system is configured to iteratively receive the validation.

19. The computer program product of claim 14, wherein tuning the accuracy of the computer system further comprises adjusting a bias of the computer system.

20. The computer program product of claim 14, wherein the computer system comprises a plurality of nodes, the nodes comprising one or more processors;

wherein tuning the accuracy of the computer system further comprises adjusting an attribute of couplings among the nodes.

21. The computer program product of claim 14, wherein the computer system is configured to classify the documents into a plurality of classes.

22. The computer program product of claim 14, wherein the computer system is probabilistic.

23. The computer program product of claim 14, wherein the accuracy is a function of true positives, false positives, true negatives or false negatives of the computer system.

24. The computer program product of claim 14, wherein the accuracy is a function of precision, recall or F-score of the computer system with respect to a class.

25. The computer program product of claim 14, wherein the method further comprises:

calculating a cumulative probability of the target of accuracy having been met for each of the hypotheses;

calculating an aggregate probability of the target of accuracy based on the cumulative probability and the set of credibilities; and

ceasing tuning of the computer system when the aggregate probability of the target of accuracy reaches a confidence threshold.

26. The computer program product of claim 14, wherein the function is a harmonic function of the largest value and the smallest value.