Sub-Model Generation to Improve Classification Accuracy

• US 20080312904A1
• Filed: 06/18/2007
• Published: 12/18/2008
• Est. Priority Date: 06/18/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Balchandran, Rajesh, Boyer, Linda M., Purdy, Gregory
• Granted Patent: US 9,058,319 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/216 using statistical methods

Abstract

A method of classifying text input for use with a natural language understanding system can include determining classification information including a primary classification and one or more secondary classifications for a received text input using a statistical classification model (statistical model). A statistical classification sub-model (statistical sub-model) can be selectively built according to a model generation criterion applied to the classification information. The method further can include selecting the primary classification or the secondary classification for the text input as a final classification according to the statistical sub-model and outputting the final classification for the text input.

Claims

1. A method of classifying text input for use with a natural language understanding system, the method comprising:

determining classification information comprising a primary classification and at least one secondary classification for a received text input using a statistical classification model (statistical model);

selectively building a statistical classification sub-model (statistical sub-model) according to whether the classification information conforms to an accuracy requirement;

selecting the primary classification or the at least one secondary classification for the text input as a final classification according to the statistical sub-model; and

outputting the final classification for the text input.

2. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

comparing a confidence score of the primary classification with a minimum threshold level; and

building the statistical sub-model when the confidence score does not exceed the minimum threshold level.

3. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

calculating a difference between a confidence score of the primary classification and a confidence score of the at least one secondary classification;

comparing the difference with a difference threshold level; and

building the statistical sub-model when the difference does not exceed the difference threshold level.

4. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

determining that the primary classification and the at least one secondary classification match a predetermined set of classifications; and

building the statistical sub-model when a match is determined.

5. The method of claim 1, wherein the statistical model comprises a plurality of classes, wherein selectively building a statistical sub-model further comprises generating the statistical sub-model only for a subset of the plurality of classes of the statistical model.

6. The method of claim 1, further comprising:

selecting features associated with the primary classification and the at least one secondary classification from a plurality of features from training data used to create the statistical model; and

building the statistical sub-model using the selected features.

7. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

selecting training data associated with the primary classification and the at least one secondary classification from a corpus of training data used to create the statistical model; and

building the statistical sub-model using the selected training data.

8. The method of claim 1, further comprising storing the statistical sub-model for subsequent recall in processing further text input.

9. The method of claim 1, further comprising:

determining a usage frequency of the statistical sub-model;

comparing the usage frequency with a minimum frequency threshold level; and

merging the primary classification and the at least one secondary classification into a single, merged class when the usage frequency exceeds the minimum frequency threshold level.

10. A method of improving classification accuracy of text input for use with a natural language understanding system, the method comprising:

processing a plurality of text inputs using a statistical classification model (statistical model) and a statistical classification sub-model (statistical sub-model), wherein the statistical model comprises a plurality of classes and the statistical sub-model comprises a subset of the plurality of classes;

determining a usage frequency of the statistical sub-model;

comparing the usage frequency with a minimum frequency threshold level;

merging the subset of the plurality of classes into a single, merged class within the statistical model when the usage frequency exceeds the minimum frequency threshold level; and

outputting an updated statistical model specifying the merged class.

11. The method of claim 10, wherein outputting further comprises:

selecting training data corresponding to the subset of the plurality of classes from training data used to generate the statistical model;

updating the training data by associating the selected training data with the merged class; and

generating the updated statistical model from the updated training data.

12. The method of claim 10, wherein outputting further comprises mapping each class of the subset of the plurality of classes to the merged class within the statistical model, wherein a text input belonging to any of the classes of the subset of the plurality of classes is classified to the merged class.

13. A computer program product comprising:

a computer-usable medium comprising computer-usable program code that classifies text input for use with a natural language understanding system, the computer program product comprising;

computer-usable program code that determines classification information comprising a primary classification and at least one secondary classification for a received text input using a statistical classification model (statistical model);

computer-usable program code that selectively builds a statistical classification sub-model (statistical sub-model) according to whether the classification information conforms to an accuracy requirement

computer-usable program code that selects the primary classification or the at least one secondary classification for the text input as a final classification according to the statistical sub-model; and

computer-usable program code that outputs the final classification for the text input.

14. The computer program product of claim 13, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that compares a confidence score of the primary classification with a minimum threshold level; and

computer-usable program code that builds the statistical sub-model when the confidence score does not exceed the minimum threshold level.

15. The computer program product of claim 13, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that calculates a difference between a confidence score of the primary classification and a confidence score of the at least one secondary classification;

computer-usable program code that compares the difference with a difference threshold level; and

computer-usable program code that builds the statistical sub-model when the difference does not exceed the difference threshold level.

16. The computer program product of claim 13, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that determines that the primary classification and the at least one secondary classification match a predetermined set of classifications; and

computer-usable program code that builds the statistical sub-model when a match is determined.

17. The computer program product of claim 13, wherein the statistical model comprises a plurality of classes, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises computer-usable program code that generates the statistical sub-model only for a subset of the plurality of classes of the statistical model.

18. The computer program product of claim 13, wherein the computer-usable medium further comprises:

computer-usable program code that selects features associated with the primary classification and the at least one secondary classification from a plurality of features from training data used to create the statistical model; and

computer-usable program code that builds the statistical sub-model using the selected features.

19. The computer program product of claim 13, wherein the computer usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that selects training data associated with the primary classification and the at least one secondary classification from a corpus of training data used to create the statistical model; and

computer-usable program code that builds the statistical sub-model using the selected training data.

20. The computer program product of claim 13, wherein the computer-usable medium further comprises computer-usable program code that stores the generated statistical sub-model for subsequent recall in processing further text input.