Method and apparatus of providing semi-automated classifier adaptation for natural language processing

• US 8,892,437 B1
• Filed: 11/13/2013
• Issued: 11/18/2014
• Est. Priority Date: 09/06/2011
• Status: Active Grant

• Current Assignee: Open Invention Network LLC
• Original Assignee: West Corporation
• Inventors: Witt-ehsani, Silke
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US14/078,749
• Time in Patent Office: 370 Days
• Field of Search: 704/257, 704/240, 704/243, 704/277, 704/1, 704/4, 704/10, 704/235
• US Class Current: 704/235
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/0638 Interactive procedures

Abstract

Example embodiments of the present invention may include a method that provides transcribing spoken utterances occurring during a call and assigning each of the spoken utterances with a corresponding set of first classifications. The method may also include determining a confidence rating associated with each of the spoken utterances and the assigned set of first classifications, and performing at least one of reclassifying the spoken utterances with new classifications based on at least one additional classification operation, and adding the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.

Claims

1. A method, comprising:

transcribing via a processor at least a portion of a plurality of spoken utterances occurring during a call;

assigning each of the plurality of spoken utterances with a corresponding set of first classifications;

determining a confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and

performing at least one of;

reclassifying the plurality of spoken utterances with new classifications based on at least one additional classification operation, and

adding the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.

2. The method of claim 1, wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.

3. The method of claim 1, wherein the transcribed at least a portion of the plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.

4. The method of claim 1, wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then adding the labeled plurality of spoken utterances to the training data set.

5. The method of claim 1, wherein if determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then reclassifying the plurality of spoken utterances with new classifications based on at least two different classification operations.

6. The method of claim 5, wherein the at least two different classification operations comprise at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naï

ve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.

7. The method of claim 1, further comprising:

updating a production classifier used to perform classification operations by adding the new classifications to a database; and

repeating the transcribing, via the processor, of the at least a portion of the plurality of spoken utterances occurring during the call;

repeating the assigning of each of the plurality of spoken utterances with the corresponding set of first classifications;

repeating the determining of the confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and

repeating the performing of the at least one of the

reclassifying the plurality of spoken utterances with the new classifications based on the at least one additional classification operation, and

adding the assigned first classifications and the corresponding plurality of spoken utterances to the training data set.

8. An apparatus, comprising:

a receiver configured to receive a plurality of spoken utterances and record them in a database memory; and

a processor configured to

transcribe at least a portion of the plurality of spoken utterances occurring during a call,

assign each of the plurality of spoken utterances with a corresponding set of first classifications,

determine a confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications, and

perform at least one of

reclassify the plurality of spoken utterances with new classifications based on at least one additional classification operation, and

add the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.

9. The apparatus of claim 8, wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.

10. The apparatus of claim 8, wherein the transcribed at least a portion of the plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.

11. The apparatus of claim 8, wherein if the determined confidence ratings associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then the processor is configured to add the labeled plurality of spoken utterances to the training data set.

12. The apparatus of claim 8, wherein if the determined confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then the processor is configured to reclassify the plurality of spoken utterances with new classifications based on at least two different classification operations.

13. The apparatus of claim 12, wherein the at least two different classification operations comprise the process performing at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naï

ve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.

14. A non-transitory computer readable storage medium configured to store instructions that when executed cause a processor to perform:

transcribing via a processor at least a portion of a plurality of spoken utterances occurring during a call;

assigning each of the plurality of spoken utterances with a corresponding set of first classifications;

determining a confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and

performing at least one of

reclassifying the plurality of spoken utterances with new classifications based on at least one additional classification operation, and

adding the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.

15. The non-transitory computer readable storage medium of claim 14, wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.

16. The non-transitory computer readable storage medium of claim 14, wherein the transcribed at least a portion of the plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.

17. The non-transitory computer readable storage medium of claim 14, wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then adding the labeled plurality of spoken utterances to the training data set.

18. The non-transitory computer readable storage medium of claim 14, wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then reclassifying the plurality of spoken utterances with new classifications based on at least two different classification operations.

19. The non-transitory computer readable storage medium of claim 18, wherein the at least two different classification operations comprise at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naï

ve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.

20. The non-transitory computer readable storage medium of claim 14, wherein the processor is further configured to perform:

updating a production classifier used to perform classification operations by adding the new classifications to a training database; and

repeating the transcribing, via the processor, of the at least a portion of the plurality of spoken utterances occurring during the call;

repeating the assigning of each of the plurality of spoken utterances with the corresponding set of first classifications;

repeating the determining of the confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and

repeating the performing of the at least one of the

reclassifying the plurality of spoken utterances with the new classifications based on the at least one additional classification operation, and

adding the assigned first classifications and the corresponding plurality of spoken utterances to the training data set.