Systems, methods, and software for classifying text from judicial opinions and other documents

• US 20060010145A1
• Filed: 08/30/2005
• Published: 01/12/2006
• Est. Priority Date: 11/02/2001
• Status: Active Grant

• Current Assignee: Thomson Reuters Enterprise Centre GmbH (The Woodbridge Co. Ltd.)
• Original Assignee: Thomson Global Resources AG
• Inventors: Jackson, Peter, Al-Kofahi, Khalid, Tyrell, Alex, Travers, Timothy Earl
• Granted Patent: US 7,580,939 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

To reduce cost and improve accuracy, the inventors devised systems, methods, and software to aid classification of text, such as headnotes and other documents, to target classes in a target classification system. For example, one system computes composite scores based on: similarity of input text to text assigned to each of the target classes; similarity of non-target classes assigned to the input text and target classes; probability of a target class given a set of one or more non-target classes assigned to the input text; and/or probability of the input text given text assigned to the target classes. The exemplary system then evaluates the composite scores using class-specific decision criteria, such as thresholds, ultimately assigning or recommending assignment of the input text to one or more of the target classes. The exemplary system is particularly suitable for classification systems having thousands of classes.

Claims

1. An automated method of classifying input text according to a target classification system having two or more target classes, the method comprising:

for each target class, determining a composite score based on a first score scaled by a first class-specific weight for the target class and a second score scaled by a second class-specific weight for the target class, with the first and second scores based on an input text and text associated with the target class; and

for each target class, classifying or recommending classification of the input text to the target class based on the composite score and a class-specific decision threshold for the target class.

2. The method of claim 1, wherein the first and second scores are based on at least one of:

a score based on similarity of at least one or more portions of the input text to text associated with the target class;

a score based on similarity of a set of one or more non-target classes associated with the input text and a set of one or more non-target classes associated with the target class;

a score based on probability of the target class given a set of one or more non-target classes associated with the input text; and

a score based on probability of the target class given at least a portion of the input text.

3. The method of claim 1, further comprising:

updating the class-specific threshold for one of the target classes based on acceptance or rejection of recommended classifications of the input text.

4. An automated method of classifying text to one or more target classes in a target classification system, the method comprising:

identifying one or more noun-word pairs in a portion of text; and

determining one or more scores based on frequencies of one or more of the identified noun-word pairs in the portion of text and one or more noun-word pairs in text associated with one of the target classes.

5. The method of claim 4, wherein identifying one or more noun-word pairs in the portion of text comprises:

identifying a first noun in the portion of text; and

identifying one or more words within a predetermined number of words of the first noun.

6. The method of claim 5, wherein identifying one or more words within a predetermined number of words of the first noun comprises excluding a set of one or more stop words.

7. The method of claim 4, wherein the portion of text is a paragraph.

8. The method of claim 4, wherein the one or more scores include:

at least one score based on similarity of at least one or more portions of the input text to text associated with the target class;

at least one score based on similarity of a set of one or more non-target classes associated with the input text and a set of one or more non-target classes associated with the target class;

at least one score based on probability of the target class given a set of one or more non-target classes associated with the input text; and

at least one score based on probability of the target class given at least a portion of the input text.

9. The method of claim 4, wherein determining one or more scores based on one or more identified noun-word pairs and one or more noun-word pairs in other text associated with one of the target classes, comprises:

determining a respective weight for each identified noun-word pair, with the respective weight based on a product of a term frequency of the identified word-noun pair in the text and an inverse document frequency of the noun-word pairs in the other text associated with one of the target classes.

10. An automated method of classifying input text to one or more target classes in a target classification system, the method comprising:

identifying a first set of noun-word pairs in the input text, with the first set including at least one noun-word pair formed from a noun and non-adjacent word in the input text;

identifying two or more second sets of noun-word pairs, with each second set including at least one noun-word pair formed from a noun and non-adjacent word in text associated with a respective one of the target classes;

determining a set of scores based on the first and second sets of noun-word pairs; and

classifying or recommending classification of the input text to one or more of the target classes based on the set of scores.

11. A system for classifying input text to a target classification system having two or more target classes, the system comprising:

a scoring module for determining for each of the target classes at least first and second scores based on the input text and the target class;

a composite scoring module for determining for each of the target classes a corresponding composite score based on the first score scaled by a first class-specific weight for the target class and the second score scaled by a second class-specific weight for the target class; and

a classification module for determining for each of the target classes whether to classify or recommend classification of the input text to the target class based on the corresponding composite score and a class-specific decision threshold for the target class.

12. The system of claim 11:

wherein the scoring module comprises means for determining for each of the target classes at least first and second scores based on the input text and the target class;

wherein the a composite scoring module comprises means for determining for each of the target classes a corresponding composite score based on the first score scaled by a first class-specific weight for the target class and the second score scaled by a second class-specific weight for the target class; and

wherein the classification module comprises means for determining for each of the target classes whether to classify or recommend classification of the input text to the target class based on the corresponding composite score and a class-specific decision threshold for the target class.

13. A machine-readable medium comprising instructions related to classifying input text to a target classification system having two or more target classes, the instructions comprising:

a first set of instructions for determining first and second scores based on the input text and one of the target classes,

wherein the first score is based on;

similarity of at least one or more portions of the input text to text associated with the one target class;

or

similarity of a set of one or more non-target classes associated with the input text and a set of one or more non-target classes associated with the one target class; and

wherein the second score is based on;

probability of the one target class given a set of one or more non-target classes associated with the input text;

or

probability of the one target class given at least a portion of the input text;

a second set of instructions for determining a composite score based on the first and second scores; and

a third set of instructions for comparing the composite score to a decision threshold.

14. The medium of claim 13, wherein the second set of instructions for determining the composite score based on the first and second scores comprises instructions for

weighting the first and second scores by respective first and second class-specific weights associated with the one target class;

and

adding the weighted first score to the, second weighted scores.

15. The medium of claim 13, wherein the first score is based on a set of one or more noun-word pairs associated with the input text and a set of one or more noun-word pairs associated with the one target class, with at least one noun-word pair in each set including a noun and a non-adjacent word.

16. The medium of claim 15, wherein the noun and the non-adjacent word are no more than 32 words apart, excluding stop words.

17. The computer readable medium of claim 13, wherein each target class is a document and the text associated with the one target class comprises text of the document or text of another document associated with the target class.

18. A machine-readable medium comprising instructions for classifying input text to a target classification system having two or more target classes, the instructions comprising:

a first set of instructions for determining first and second scores based on the input text and one of the target classes,

wherein the first score is based on similarity of a set of one or more non-target classes associated with the input text and a set of one or more non-target classes associated with the one target class; and

wherein the second score is based on probability of the one target class given at least a portion of the input text;

a second set of instructions for determining a composite score based on a linear combination of the first and second scores; and

a third set of instructions for comparing the composite score to a decision threshold.

19. The medium of claim 18, wherein the first score is based on a set of one or more noun-word pairs associated with the input text and a set of one or more noun-word pairs associated with the one target class, with at least one noun-word pair in each set including a noun and a non-adjacent word.

20. The medium of claim 19, wherein the noun and the non-adjacent word are no more than 32 words apart, excluding stop words.

21. The medium of claim 18, wherein each target class is a document and the text associated with the one target class comprises text of the document or text of another document associated with the target class.