Sub-model generation to improve classification accuracy

• US 9,058,319 B2
• Filed: 06/18/2007
• Issued: 06/16/2015
• Est. Priority Date: 06/18/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boyer, Linda M., Purdy, Gregory, Balchandran, Rajesh
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US11/764,274
• Publication Number:
  US 20080312904A1
• Time in Patent Office: 2,920 Days
• Field of Search: 704/10, 704/9, 704/255, 704/257, 704/251, 704/235, 704/1, 707/999.101, 703/22, 341/28
• US Class Current: 1/1
• CPC Class Codes: G06F 40/216 using statistical methods

Abstract

A method of classifying text input for use with a natural language understanding system can include determining classification information including a primary classification and one or more secondary classifications for a received text input using a statistical classification model (statistical model). A statistical classification sub-model (statistical sub-model) can be selectively built according to a model generation criterion applied to the classification information. The method further can include selecting the primary classification or the secondary classification for the text input as a final classification according to the statistical sub-model and outputting the final classification for the text input.

Claims

1. A method of classifying text input for use with a natural language understanding system, the method comprising:

via a processor, determining classification information comprising a primary classification and at least one secondary classification for a received text input using a statistical classification model (statistical model);

via the processor, selectively building a statistical classification sub-model (statistical sub-model) according to whether the classification information conforms to an accuracy requirement;

via the processor, selecting the primary classification or the at least one secondary classification for the text input as a final classification according to the statistical sub-model; and

via the processor, outputting the final classification for the text input.

2. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

via the processor, comparing a confidence score of the primary classification with a minimum threshold level; and

via the processor, building the statistical sub-model when the confidence score does not exceed the minimum threshold level.

3. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

via the processor, calculating a difference between a confidence score of the primary classification and a confidence score of the at least one secondary classification;

via the processor, comparing the difference with a difference threshold level; and

via the processor, building the statistical sub-model when the difference does not exceed the difference threshold level.

4. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

via the processor, determining that the primary classification and the at least one secondary classification match a predetermined set of classifications; and

via the processor, building the statistical sub-model when a match is determined.

5. The method of claim 1, wherein the statistical model comprises a plurality of classes, wherein selectively building a statistical sub-model further comprises, via the processor, generating the statistical sub-model only for a subset of the plurality of classes of the statistical model.

6. The method of claim 1, further comprising:

via the processor, selecting features associated with the primary classification and the at least one secondary classification from a plurality of features from training data used to create the statistical model; and

via the processor, building the statistical sub-model using the selected features.

7. The method of claim 1, wherein selectively building a statistical sub-model further comprises:

via the processor, selecting training data associated with the primary classification and the at least one secondary classification from a corpus of training data used to create the statistical model; and

via the processor, building the statistical sub-model using the selected training data.

8. The method of claim 1, further comprising storing the statistical sub-model for subsequent recall in processing further text input.

9. The method of claim 1, further comprising, via the processor:

via the processor, determining a usage frequency of the statistical sub-model;

via the processor, comparing the usage frequency with a minimum frequency threshold level; and

via the processor, merging the primary classification and the at least one secondary classification into a single, merged class when the usage frequency exceeds the minimum frequency threshold level.

10. A method of improving classification accuracy of text input for use with a natural language understanding system, the method comprising:

via a processor, processing a plurality of text inputs using a statistical classification model (statistical model) and a statistical classification sub-model (statistical sub-model), wherein the statistical model comprises a plurality of classes and the statistical sub-model comprises a subset of the plurality of classes;

via the processor, determining a usage frequency of the statistical sub-model;

via the processor, comparing the usage frequency with a minimum frequency threshold level;

via the processor, merging the subset of the plurality of classes into a single, merged class within the statistical model when the usage frequency exceeds the minimum frequency threshold level; and

via the processor, outputting an updated statistical model specifying the merged class.

11. The method of claim 10, wherein outputting further comprises:

via the processor, selecting training data corresponding to the subset of the plurality of classes from training data used to generate the statistical model;

via the processor, updating the training data by associating the selected training data with the merged class; and

via the processor, generating the updated statistical model from the updated training data.

12. The method of claim 10, wherein outputting further comprises, via the processor, mapping each class of the subset of the plurality of classes to the merged class within the statistical model, wherein a text input belonging to any of the classes of the subset of the plurality of classes is classified to the merged class.

13. A computer program product comprising:

a computer-readable storage comprising computer-usable program code stored thereon that classifies text input for use with a natural language understanding system, the computer program product comprising;

computer-usable program code that determines classification information comprising a primary classification and at least one secondary classification for a received text input using a statistical classification model (statistical model);

computer-usable program code that selectively builds a statistical classification sub-model (statistical sub-model) according to whether the classification information conforms to an accuracy requirement;

computer-usable program code that selects the primary classification or the at least one secondary classification for the text input as a final classification according to the statistical sub-model; and

computer-usable program code that outputs the final classification for the text input, wherein

the computer-readable storage is not a transitory, propagating signal per se.

14. The computer program product of claim 13, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that compares a confidence score of the primary classification with a minimum threshold level; and

computer-usable program code that builds the statistical sub-model when the confidence score does not exceed the minimum threshold level.

15. The computer program product of claim 13, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that calculates a difference between a confidence score of the primary classification and a confidence score of the at least one secondary classification;

computer-usable program code that compares the difference with a difference threshold level; and

computer-usable program code that builds the statistical sub-model when the difference does not exceed the difference threshold level.

16. The computer program product of claim 13, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that determines that the primary classification and the at least one secondary classification match a predetermined set of classifications; and

computer-usable program code that builds the statistical sub-model when a match is determined.

17. The computer program product of claim 13, wherein the statistical model comprises a plurality of classes, wherein the computer-usable program code that selectively builds a statistical sub-model further comprises computer-usable program code that generates the statistical sub-model only for a subset of the plurality of classes of the statistical model.

18. The computer program product of claim 13, wherein the computer-readable storage further comprises:

computer-usable program code that selects features associated with the primary classification and the at least one secondary classification from a plurality of features from training data used to create the statistical model; and

computer-usable program code that builds the statistical sub-model using the selected features.

19. The computer program product of claim 13, wherein the computer usable program code that selectively builds a statistical sub-model further comprises:

computer-usable program code that selects training data associated with the primary classification and the at least one secondary classification from a corpus of training data used to create the statistical model; and

computer-usable program code that builds the statistical sub-model using the selected training data.

20. The computer program product of claim 13, wherein the computer-readable storage further comprises computer-usable program code that stores the generated statistical sub-model for subsequent recall in processing further text input.