Method and system for generating and correcting classification models

• US 10,599,953 B2
• Filed: 08/27/2014
• Issued: 03/24/2020
• Est. Priority Date: 08/27/2014
• Status: Active Grant

• Current Assignee: Verint Americas Incorporated (Verint Systems Incorporated)
• Original Assignee: Verint Americas Incorporated (Verint Systems Incorporated)
• Inventors: Brown, Fred A, Miller, Tanya M, Wooters, Charles C, Brown, Megan, Brown, Molly Q
• Primary Examiner(s): Jami, Hares
• Application Number:
  US14/470,908
• Publication Number:
  US 20160063097A1
• Time in Patent Office: 2,036 Days
• Field of Search: 707737, 707791, 707802, 707803
• CPC Class Codes: G06F 16/358 Browsing; Visualisation therefor
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher

Abstract

Data having some similarities and some dissimilarities may be clustered or grouped according to the similarities and dissimilarities. The data may be clustered using agglomerative clustering techniques. The clusters may be used as suggestions for generating groups where a user may demonstrate certain criteria for grouping. The system may learn from the criteria and extrapolate the groupings to readily sort data into appropriate groups. The system may be easily refined as the user gains an understanding of the data.

Claims

1. One or more non-transitory computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

generating a first classification model and a second classification model based on at least one received indication regarding a data set of natural language inputs from conversations of calls or chats;

importing data in form of structure, unstructured, partially structured, or a combination of thereof;

wherein when importing a markup language data, preserving or leveraging information contained in the markup language data;

comparing, using a comparator, (1) a first score using test results from the first classification model with respect to a portion of data of the data set and (2) a second score using test results from the second classification model with respect to the portion of the data set;

when the first score agrees with the second score within a threshold range, validating, by the one or more processors, the first classification model;

when the first score is different from the second score with respect to the threshold range, generating, by the one or more processors, a clarification question for a user to select and displaying, on a display of a user device, the clarification question for the user;

when an indication is received from a user, in response to the clarification question, that the first classification model is correct, updating, by the one or more processors, the second classification model based at least in part on the indication from the user; and

when an indication is received from a user, in response to the clarification question, that the second classification model is correct, updating, by the one or more processors, the first classification model based at least in part on the indication from the user.

2. The one or more non-transitory computer-readable storage media of claim 1, wherein the first classification model comprises a symbolic language model;

and the second classification model comprises a statistical language model.

3. The one or more non-transitory computer-readable storage media of claim 1, wherein the clarification question comprises a question asking a user to select an appropriate answer to an input statement.

4. One or more non-transitory computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

generating a first classification model and a second classification model based on at least one received indication regarding a data set of natural language inputs from conversations of calls or chats;

importing data in form of structure, unstructured, partially structured, or a combination of thereof;

wherein when importing a markup language data, preserving or leveraging information contained in the markup language data;

comparing, by a comparator, (1) a first score using test results from the first classification model with respect to a portion of data of the data set and (2) a second score using test results from the second classification model with respect to the portion of the data set;

when the first score agrees with the second score within a threshold range, validating, by the one or more processors, the first classification model for use;

when the first score is different from the second score within a first threshold difference, generating, by the one or more processors, a first clarification question for a user to select, displaying, on a display of a user device, the first clarification question for the user, and updating, by the one or more processors, the first classification model based at least in part on a selection of an answer to the first clarification question; and

when the first score is different from the second score within a second threshold difference and without the first threshold difference, generating, by the one or more processors, a second clarification question for the user to select, displaying, on the display of the user device, the second clarification question for the user, and updating, by the one or more processors, the first classification model based at least in part on a selection of an answer to the second clarification question.

5. The one or more non-transitory computer-readable storage media of claim 4, wherein the first classification model comprises a symbolic language model;

and the second classification model comprises a statistical language model.

6. The one or more non-transitory computer-readable storage media of claim 4, wherein the first clarification question comprises a question asking a user to select an appropriate answer to an input statement.

7. The one or more non-transitory computer-readable storage media of claim 4, wherein the second clarification question comprises a question asking a user to select an appropriate restatement of an input statement, the appropriate restatement of the input being a leading restatement of the input statement.

8. A system comprising:

one or more non-transitory computer-readable storage media;

computer-readable instructions stored on the one or more computer-readable storage media which, when executed by one or more processors, configure the one or more processors to;

generate a first classification model and a second classification model based on at least one received indication regarding a data set of natural language inputs from conversations of calls or chats;

import data in form of structure, unstructured, partially structured, or a combination of thereof;

wherein when importing a markup language data, preserving or leveraging information contained in the markup language data;

compare, using a comparator, (1) a first score using test results from the first classification model with respect to a portion of data of the data set and (2) a second score using test results from the second classification model with respect to the portion of the data set;

when the first score agrees with the second score within a threshold range, validate, by the one or more processors, the first classification model;

when the first score is different from the second score with respect to the threshold range, generate, by the one or more processors, a clarification question for a user to select, and display, on a display of a user device, the clarification question for the user;

when an indication is received from a user, in response to the clarification question that the first classification model is correct, update, by the one or more processors, the second classification model based at least in part on the indication from the user; and

when an indication is received from a user, in response to the clarification question that the second classification model is correct, update, by the one or more processors, the first classification model based at least in part on the indication from the user.

9. The system of claim 8, wherein the first classification model comprises a symbolic language model;

and the second classification model comprises a statistical language model.

10. The system of claim 8, wherein the clarification question comprises a question asking a user to select an appropriate answer to an input statement.

11. A system comprising:

one or more non-transitory computer-readable storage media;

computer-readable instructions stored on the one or more computer-readable storage media which, when executed by one or more processors, configure the one or more processors to;

generate a first classification model and a second classification model based on at least one received indication regarding a data set of natural language inputs from conversations of calls or chats;

import data in form of structure, unstructured, partially structured, or a combination of thereof;

wherein when importing a markup language data, preserving or leveraging information contained in the markup language data;

compare, by a comparator, (1) a first score using test results from the first classification model with respect to a portion of data of the data set and (2) a second score using test results from the second classification model with respect to the portion of the data set;

when the first score agrees with the second score within a threshold range, validate, by the one or more processors, the first classification model for use;

when the first score is different from the second score within a first threshold difference, generate, by the one or more processors, a first clarification question for a user to select, display, on a display of a user device, the first clarification question for the user and update, by the one or more processors, the first classification model based at least in part on a selection of an answer to the first clarification question; and

when the first score is different from the second score within a second threshold difference and outside of the first threshold difference, generate, by the one or more processors, a second clarification question for the user to select, display, on the display of the user device, the second clarification question for the user, and update, by the one or more processors, the first classification model based at least in part on a selection of an answer to the second clarification question.

12. The system of claim 11, wherein the first classification model comprises a symbolic language model;

and the second classification model comprises a statistical language model.

13. The system of claim 11, wherein the first clarification question comprises a question asking a user to select an appropriate answer to an input statement.

14. The system of claim 11, wherein the second clarification question comprises a question asking a user to select an appropriate restatement of an input statement, the appropriate restatement of the input being a leading restatement of the input statement.