Reclassification of training data to improve classifier accuracy

• US 9,342,588 B2
• Filed: 06/18/2007
• Issued: 05/17/2016
• Est. Priority Date: 06/18/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Balchandran, Rajesh, Boyer, Linda M., Purdy, Gregory
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US11/764,291
• Publication Number:
  US 20080312906A1
• Time in Patent Office: 3,256 Days
• Field of Search: 704/1, 704/9, 704/256, 704/257, 715/256, 715/273, 702/182
• US Class Current: 1/1
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

A method of creating a statistical classification model for a classifier within a natural language understanding system can include processing training data using an existing statistical classification model. Sentences of the training data correctly classified into a selected class of the statistical classification model can be selected. The selected sentences of the training data can be assigned to a fringe group or a core group according to confidence score. The training data can be updated by associating the fringe group with a fringe subclass of the selected class and the core group with a core subclass of the selected class. A new statistical classification model can be built from the updated training data. The new statistical classification model can be output.

Claims

1. A method of creating a statistical classification model for use with a natural language understanding system, the method comprising:

via a processor, processing training data using an existing statistical classification model;

via the processor, selecting sentences of the training data correctly classified into a selected class of the existing statistical classification model;

via the processor, assigning each selected sentence of the training data to a fringe group or a core group according to confidence score;

via the processor, updating the training data by associating the fringe group with a fringe subclass of the selected class and the core group with a core subclass of the selected class;

via the processor, building a new statistical classification model from the updated training data; and

via the processor, outputting the new statistical classification model.

2. The method of claim 1, wherein at runtime the method further comprises:

via the processor, classifying a text input into the fringe subclass or the core subclass of the selected class according to the new statistical classification model; and

via the processor, outputting an indication that the text input belongs to the selected class.

3. The method of claim 2, further comprising, via the processor, outputting a measure of accuracy for the indication that depends upon which subclass of the selected class into which the text input is classified.

4. The method of claim 1, wherein assigning the selected sentences further comprises, for each selected sentence, assigning the selected sentence to the fringe group or the core group according to which range of a plurality of ranges comprises a confidence score of the selected sentence.

5. The method of claim 1, wherein assigning the selected sentences further comprises:

determining a distribution of confidence scores for the selected sentences; and

for each selected sentence, assigning the selected sentence to the fringe group or the core group according to a distance between the confidence score of the selected sentence and a mean confidence score on the distribution.

6. The method of claim 1, wherein assigning the selected sentences further comprises, for each selected sentence, assigning the selected sentence to the fringe group or the core group according to a length of the selected sentence.

7. The method of claim 1, wherein assigning the selected sentences further comprises:

for each selected sentence, assigning the selected sentence to one of a plurality of fringe groups or one of a plurality of core groups, wherein updating the training data further comprises associating each of the plurality of fringe groups with one of a plurality of fringe subclasses and each of the plurality of core groups with one of a plurality of core subclasses.

8. The method of claim 7, wherein assigning the selected sentences further comprises:

identifying a plurality of confidence score ranges according to confidence scores of the selected sentences, wherein each of the plurality of confidence score ranges defines one of the plurality of fringe groups or one of the plurality of core groups; and

for each selected sentence, assigning the selected sentence to one of the plurality of fringe groups or one of the plurality of core groups according to the confidence score range comprising the confidence score of the selected sentence.

9. The method of claim 7, wherein assigning the selected sentences further comprises:

determining a distribution of confidence scores for the selected sentences; and

for each selected sentence, assigning the selected sentence to one of the plurality of fringe groups or one of the plurality of core groups according to a distance between the confidence score of the selected sentence and a mean confidence score of the distribution.

10. The method of claim 7, wherein assigning the selected sentences further comprises:

identifying a plurality of sentence length ranges, wherein each of the plurality of sentence length ranges defines one of the plurality of fringe groups or one of the plurality of core groups; and

for each selected sentence, assigning the selected sentence to one of the plurality of fringe groups or one of the plurality of core groups according to which of the plurality of sentence length ranges comprises a length of the selected sentence.

11. A computer-readable storage comprising

computer-usable program code that creates a statistical classification model for a classifier within a natural language understanding system, the computer-readable storage comprising:

computer-usable program code that processes training data using an existing statistical classification model;

computer-usable program code that selects sentences of the training data correctly classified into a selected class of the existing statistical classification model;

computer-usable program code that assigns each selected sentence of the training data to a fringe group or a core group according to confidence score;

computer-usable program code that updates the training data by associating the fringe group with a fringe subclass of the selected class and the core group with a core subclass of the selected class;

computer-usable program code that builds a new statistical classification model from the updated training data; and

computer-usable program code that outputs the new statistical classification model, wherein

the computer-readable storage is not a transitory, propagating signal per se.

12. The computer-readable storage of claim 11, wherein the computer-usable medium further comprises:

computer-usable program code that, at runtime of the classifier, classifies a text input into the fringe subclass or the core subclass of the selected class according to the new statistical classification model; and

computer-usable program code that outputs an indication that the text input belongs to the selected class.

13. The computer-readable storage of claim 12, wherein the computer-readable storage further comprises computer-usable program code that outputs a measure of accuracy for the indication that depends upon which subclass of the selected class into which the text input is classified.

14. The computer-readable storage of claim 11, wherein the computer-usable program code that assigns the selected sentences further comprises computer-usable program code that, for each selected sentence, assigns the selected sentence to the fringe group or the core group according to which range of a plurality of ranges comprises a confidence score of the selected sentence.

15. The computer-readable storage of claim 11, wherein the computer-usable program code that assigns selected sentences further comprises:

computer-usable program code that determines a distribution of confidence scores for the selected sentences; and

computer-usable program code that, for each selected sentence, assigns the selected sentence to the fringe group or the core group according to a distance between the confidence score of the selected sentence and a mean confidence score on the distribution.

16. The computer-readable storage of claim 11, wherein the computer-usable program code that assigns the selected sentences further comprises computer-usable program code that, for each selected sentence, assigns the selected sentence to the fringe group or the core group according to a length of the selected sentence.

17. The computer-readable storage of claim 11, wherein the computer-usable program code that assigns the selected sentences further comprises:

computer-usable program code that, for each selected sentence, assigns the selected sentence to one of a plurality of fringe groups or to one of a plurality of core groups, wherein the computer-usable program code that updates the training data further comprises computer-usable program code that associates each of the plurality of fringe groups with one of a plurality of fringe subclasses and each of the plurality of core groups with one of a plurality of core subclasses.