Systems, methods, and software for classifying text from judicial opinions and other documents

• US 7,062,498 B2
• Filed: 12/21/2001
• Issued: 06/13/2006
• Est. Priority Date: 11/02/2001
• Status: Expired due to Term

• Current Assignee: Thomson Reuters Enterprise Centre GmbH (The Woodbridge Co. Ltd.)
• Original Assignee: Thomson Legal Regulatory Global AG
• Inventors: Jackson, Peter, Al-Kofahi, Khalid, Tyrell, Alex, Travers, Timothy Earl
• Primary Examiner(s): LE, UYEN T
• Application Number:
  US10/027,914
• Publication Number:
  US 20030101181A1
• Time in Patent Office: 1,635 Days
• Field of Search: 707/100, 707/101, 715/530, 715/531
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

To reduce cost and improve accuracy, the inventors devised systems, methods, and software to aid classification of text, such as headnotes and other documents, to target classes in a target classification system. For example, one system computes composite scores based on: similarity of input text to text assigned to each of the target classes; similarity of non-target classes assigned to the input text and target classes; probability of a target class given a set of one or more non-target classes assigned to the input text; and/or probability of the input text given text assigned to the target classes. The exemplary system then evaluates the composite scores using class-specific decision criteria, such as thresholds, ultimately assigning or recommending assignment of the input text to one or more of the target classes. The exemplary system is particularly suitable for classification systems having thousands of classes.

Claims

1. An automated method of classifying input text to a target classification system having two or more target classes, the method comprising:

for each target class;

providing at least first and second class-specific weights and a class-specific decision threshold;

using at least first and second classification methods to determine respective first and second scores based on the input text and the target class;

determining a composite score based on the first score scaled by the first class-specific weight for the target class and the second score scaled by the second class-specific weight for the target class; and

classifying or recommending classification of the input text to the target class based on the composite score and the class-specific decision threshold.

2. The method of claim 1, wherein at least one of the first and second scores is based on a set of one or more noun-word pairs associated with the input text and a set of one or more noun-word pairs associated with the target class, with at least one noun-word pair in each set including a noun and a non-adjacent word.

3. The method of claim 1, wherein providing each first and second class-specific weight and class-specific decision thresholds comprises searching for a combination of first and second class-specific weights and class-specific decision threshold that yield a predetermined level of precision at a predetermined level of recall based on text classified to the target classification system.

4. The method of claim 1, wherein a non-target classification system includes two or more non-target classes, and at least one of the first and second scores is based on one or more of the non-target classes that are associated with the input text and one or more of the non-target classes that are associated with the target class.

5. The method of claim 4:

wherein the input text is a headnote for a legal document; and

wherein the target classification system and the non-target classification system are legal classification systems.

6. The method of claim 1, wherein the target classification system includes over 1000 target classes.

7. Tho method of claim 1, further comprising:

displaying a graphical user interface including first and second regions, with the first region displaying or identifying at least a portion of the input text and the second region displaying information regarding the target classification system and at least one target class for which the input text was recommended for classification; and

displaying a selectable feature on the graphical user interface, wherein selecting the feature initiates classification of the input text to the one target class.

8. A machine-readable medium comprising instructions for performing an automated method of classifying input text to a target classification system having two or more target classes, the medium including instructions that for each target class can cause:

providing at least first and second class-specific weights and a class-specific decision threshold;

using at least first and second classification methods to determine respective first and second scores based on the input text and the target class;

determining a composite score based on the first score scaled by the first class-specific weight for the target class and the second score scaled by the second class-specific weight for the target class; and

classifying or recommending classification of the input text to the target class based on the composite score and the class-specific decision threshold.

9. An automated method of classifying input text to a target classification system having two or more target classes, the method comprising:

for each target class;

determining first and second scores based on the input text and the target class and respective first and second classification methods;

determining a composite score based on the first score scaled by a first class-specific weight for the target class and the second score scaled by a second class-specific weight for the target class; and

determining whether to identify the input text for classification to the target class based on the composite score and a class-specific decision threshold for the target class.

10. The method of claim 9, wherein at least one of the first and second scores is based on a set of one or more noun-word pairs associated with the input text and a set of one or more noun-word pairs associated with the target class, with at least one noun-word pair in each set including a noun and a non-adjacent word.

11. The method of claim 9, wherein determining the first and second scores comprises determining any two of:

a score based on similarity of at least one or more portions of the input text to text associated with the target class;

a score based on similarity of a set of one or more non-target classes associated with the input text and a set of one or more non-target classes associated with the target class;

a score based on probability of the target class given a set of one or more non-target classes associated with the input text; and

a score based on probability of the target class given at least a portion of the input text.

12. The method of claim 11, wherein each target class is a document and the text associated with the target class comprises text of the document or text of another document associated with the target class.

13. The method of claim 9:

wherein determining the first and second scores for each target class comprises;

determining the first score based on similarity of at least one or more portions of the input text to text associated with the target class; and

determining the second score based on similarity of a set of one or more non-target classes associated with the input text and a set of one or more non-target classes associated with the target class;

wherein the method further comprises determining for each target class;

a third score based on probability of the target class given a set of one or more non-target classes associated with the input text; and

a fourth score based on probability of the target class given at least a portion of the input text; and

wherein the composite score is further based on the third score scaled by a third class-specific weight for the target class and the fourth score scaled by a fourth class-specific weight for the target class.

14. The method of claim 9:

wherein the input text is associated with first meta-data and each target class is associated with second meta-data; and

wherein at least one of the first and second scores is based on the first meta-data and the second meta-data.

15. The method of claim 14, wherein the first meta-data comprises a first set of non-target classes that are associated with the input text and the second meta-data comprises a second set of non-target classes that are associated with the target class.

16. A machine-readable medium comprising instructions for performing an automated method of classifying input text to a target classification system having two or more target classes, the medium including instructions for:

determining for each target class first and second scores based on the input text and the target class and respective first and second classification methods;

determining for each target class a composite score based on the first score scaled by a first class-specific weight for the target class and the second score scaled by a second class-specific weight for the target class; and

determining for each target class whether to identify the input text for classification to the target class based on the composite score and a class-specific decision threshold for the target class.

17. A system for classifying input text to a target classification system having two or more target classes, the system comprising:

means for determining for each of the target classes at least first and second scores based on the input text and the target class and respective first and second classification methods;

means for determining for each of the target classes a corresponding composite score based on the first score scaled by a first class-specific weight for the target class and the second score scaled by a second class-specific weight for the target class; and

means for determining for each of the target classes whether to classify or recommend classification of the input text to the target class based on the corresponding composite score and a class-specific decision threshold for the target class.