Reclassification of Training Data to Improve Classifier Accuracy

• US 20080312906A1
• Filed: 06/18/2007
• Published: 12/18/2008
• Est. Priority Date: 06/18/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Balchandran, Rajesh, Boyer, Linda M., Purdy, Gregory
• Granted Patent: US 9,342,588 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

A method of creating a statistical classification model for a classifier within a natural language understanding system can include processing training data using an existing statistical classification model. Sentences of the training data correctly classified into a selected class of the statistical classification model can be selected. The selected sentences of the training data can be assigned to a fringe group or a core group according to confidence score. The training data can be updated by associating the fringe group with a fringe subclass of the selected class and the core group with a core subclass of the selected class. A new statistical classification model can be built from the updated training data. The new statistical classification model can be output.

Claims

1. A method of creating a statistical classification model for use with a natural language understanding system, the method comprising:

processing training data using an existing statistical classification model;

selecting sentences of the training data correctly classified into a selected class of the existing statistical classification model;

assigning each selected sentence of the training data to a fringe group or a core group according to confidence score;

updating the training data by associating the fringe group with a fringe subclass of the selected class and the core group with a core subclass of the selected class;

building a new statistical classification model from the updated training data; and

outputting the new statistical classification model.

2. The method of claim 1, wherein at runtime the method further comprises:

classifying a text input into the fringe subclass or the core subclass of the selected class according to the new statistical classification model; and

outputting an indication that the text input belongs to the selected class.

3. The method of claim 2, further comprising outputting a measure of accuracy for the indication that depends upon which subclass of the selected class into which the text input is classified.

4. The method of claim 1, wherein assigning the selected sentences further comprises, for each selected sentence, assigning the selected sentence to the fringe group or the core group according to which range of a plurality of ranges comprises a confidence score of the selected sentence.

5. The method of claim 1, wherein assigning the selected sentences further comprises:

determining a distribution of confidence scores for the selected sentences; and

for each selected sentence, assigning the selected sentence to the fringe group or the core group according to a distance between the confidence score of the selected sentence and a mean confidence score on the distribution.

6. The method of claim 1, wherein assigning the selected sentences further comprises, for each selected sentence, assigning the selected sentence to the fringe group or the core group according to a length of the selected sentence.

7. The method of claim 1, wherein assigning the selected sentences further comprises:

for each selected sentence, assigning the selected sentence to one of a plurality of fringe groups or one of a plurality of core groups,

wherein updating the training data further comprises associating each of the plurality of fringe groups with one of a plurality of fringe subclasses and each of the plurality of core groups with one of a plurality of core subclasses.

8. The method of claim 7, wherein assigning the selected sentences further comprises:

identifying a plurality of confidence score ranges according to confidence scores of the selected sentences, wherein each of the plurality of confidence score ranges defines one of the plurality of fringe groups or one of the plurality of core groups; and

for each selected sentence, assigning the selected sentence to one of the plurality of fringe groups or one of the plurality of core groups according to the confidence score range comprising the confidence score of the selected sentence.

9. The method of claim 7, wherein assigning the selected sentences further comprises:

determining a distribution of confidence scores for the selected sentences; and

for each selected sentence, assigning the selected sentence to one of the plurality of fringe groups or one of the plurality of core groups according to a distance between the confidence score of the selected sentence and a mean confidence score of the distribution.

10. The method of claim 7, wherein assigning the selected sentences further comprises:

identifying a plurality of sentence length ranges, wherein each of the plurality of sentence length ranges defines one of the plurality of fringe groups or one of the plurality of core groups; and

for each selected sentence, assigning the selected sentence to one of the plurality of fringe groups or one of the plurality of core groups according to which of the plurality of sentence length ranges comprises a length of the selected sentence.

11. A method of creating a statistical classification model for use with a natural language understanding system, the method comprising:

processing training data using an existing statistical classification model;

receiving a user input specifying at least one parameter for assigning sentences of the training data correctly classified into a selected class to a fringe group or a core group;

updating the training data by associating each group with a different subclass;

building a new statistical classification model from the updated training data; and

outputting the new statistical classification model.

12. The method of claim 11, further comprising receiving a user input specifying at least one of a number of fringe groups or a number of core groups.

13. The method of claim 11, wherein receiving a user input further comprises receiving a user input specifying ranges of confidence scores that define the fringe group and the core group.

14. A computer program product comprising:

a computer-usable medium comprising computer-usable program code that creates a statistical classification model for a classifier within a natural language understanding system, the computer-usable medium comprising;

computer-usable program code that processes training data using an existing statistical classification model;

computer-usable program code that selects sentences of the training data correctly classified into a selected class of the existing statistical classification model;

computer-usable program code that assigns each selected sentence of the training data to a fringe group or a core group according to confidence score;

computer-usable program code that updates the training data by associating the fringe group with a fringe subclass of the selected class and the core group with a core subclass of the selected class;

computer-usable program code that builds a new statistical classification model from the updated training data; and

computer-usable program code that outputs the new statistical classification model.

15. The computer program product of claim 14, wherein the computer-usable medium further comprises:

computer-usable program code that, at runtime of the classifier, classifies a text input into the fringe subclass or the core subclass of the selected class according to the new statistical classification model; and

computer-usable program code that outputs an indication that the text input belongs to the selected class.

16. The computer program product of claim 15, wherein the computer-usable medium further comprises computer-usable program code that outputs a measure of accuracy for the indication that depends upon which subclass of the selected class into which the text input is classified.

17. The computer program product of claim 14, wherein the computer-usable program code that assigns the selected sentences further comprises computer-usable program code that, for each selected sentence, assigns the selected sentence to the fringe group or the core group according to which range of a plurality of ranges comprises a confidence score of the selected sentence.

18. The computer program product of claim 14, wherein the computer-usable program code that assigns selected sentences further comprises:

computer-usable program code that determines a distribution of confidence scores for the selected sentences; and

computer-usable program code that, for each selected sentence, assigns the selected sentence to the fringe group or the core group according to a distance between the confidence score of the selected sentence and a mean confidence score on the distribution.

19. The computer program product of claim 14, wherein the computer-usable program code that assigns the selected sentences further comprises computer-usable program code that, for each selected sentence, assigns the selected sentence to the fringe group or the core group according to a length of the selected sentence.

20. The computer program product of claim 14, wherein the computer-usable program code that assigns the selected sentences further comprises:

computer-usable program code that, for each selected sentence, assigns the selected sentence to one of a plurality of fringe groups or to one of a plurality of core groups,

wherein the computer-usable program code that updates the training data further comprises computer-usable program code that associates each of the plurality of fringe groups with one of a plurality of fringe subclasses and each of the plurality of core groups with one of a plurality of core subclasses.