Training systems for pseudo labeling natural language

• US 10,635,751 B1
• Filed: 05/23/2019
• Issued: 04/28/2020
• Est. Priority Date: 05/23/2019
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Relangi, Aditya, Langford, Michael
• Primary Examiner(s): Serrou, Abdelali
• Application Number:
  US16/421,026
• Time in Patent Office: 341 Days
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Examples of the present disclosure can comprise systems and methods for creating and modifying named entity recognition models. The system can use two or more existing named entity recognition models to output responses to natural language queries for which the models have not yet been trained. When the output from the two or more models match, the query and the resulting output can be stored as training data for a new named entity recognition model. If the output from the two or models do not match, the query and the outputs can be stored in an exceptions file for additional review. In some embodiments, the system can comprise one or more processors and a display for providing a user interface (UI).

Claims

1. A system for providing natural language processing and interactive responses, the system comprising:

one or more processors;

a display for providing a user interface (UI), the UI comprising;

an input field for receiving an input by a user; and

a display window for displaying the user input and one or more responses;

a natural language processing (NLP) device comprising two or more trained named entity recognition models; and

a memory in communication with at least the one or more processors, the display, and the NLP device, the memory storing instructions that, when executed by the one or more processors, are configured to cause the one or more processors to;

receive, via the input field, a first input comprising a first natural language request;

provide, to the NLP device, the first input;

process, using the two or more trained named entity recognition models, the first input to apply pseudo labels to named entities in the first input;

receive, from each of the two or more trained entity recognition models, a response, each response comprising two or more named entities identified in the first input with corresponding pseudo labels;

determine, that the responses from the two or more trained named entity recognition models do not match store, in the memory, the user input and the two or more responses in an entry in an exceptions file;

determine that the exceptions file includes at least one entry;

display, on the user interface, a notification to a reviewer to review the exceptions file;

provide, to the reviewer via the UI, the exceptions file;

receive, via the UI, a corrected response from the reviewer;

store, in the memory, the corrected response in a training data set; and

train a first named entity recognition model using the training data set.

2. The system of claim 1, wherein the instructions further cause the one or more processors to:

retrieve, from the memory, the training data set comprising corrected responses provided by the reviewer in reviewing the exceptions file;

train a second named entity recognition model, using the training data set, to apply pseudo labels to the first input when the two or more responses do not match; and

store, in the memory, the second named entity recognition model.

3. The system of claim 2, wherein the second named entity recognition model is added to the two or more named entity recognition models.

4. The system of claim 1, wherein the determining comprises:

receiving a confidence level for each of the two or more responses from each respective model of the two or more trained named entity recognition models;

averaging a value of the confidence levels; and

determining that the average value is less than a predetermined threshold value.

5. The system of claim 1, wherein the instructions further cause the one or more processors to:

provide the first input and the two or more responses to a different trained named entity recognition model that is separate from the two or more trained named entity recognition models; and

receive, from the different trained named entity recognition model, a corrected response to the request.

6. A system for providing natural language processing and interactive responses, the system comprising:

a display comprising a user interface (UI), the UI including an input field for receiving input from a user;

one or more processors a natural language processing (NLP) device comprising two or more trained named entity recognition models; and

a memory in communication with at least the one or more processors, the display, and the NLP device, the memory storing instructions that, when executed by the one or more processors, are configured to cause the system to;

train a first named entity recognition model using a training data set, the training data set comprising previous natural language requests labeled by the two or more trained named entity recognition models;

add the first named entity recognition model to the two or more trained named entity recognition models;

receive, via the input field, a first input comprising a first natural language request;

provide, to the NLP device, the first input;

process, using the two or more trained named entity recognition models, the first input to apply pseudo labels to named entities in the first input;

receive, from each of the two or more trained named entity recognition models, a response, each response comprising two or more named entities with corresponding pseudo labels;

determine, with the one or more processors, that the two or more responses do not match;

store, in the memory, the first input and the two or more responses in an entry in an exceptions file;

determine that the exceptions file includes at least one entry;

display, on the user interface, a notification to a reviewer to review the exceptions file;

provide, to the reviewer via the UI, the exceptions file;

receive, via the UI, a corrected response from the reviewer;

store, the memory, the corrected response in a training data set; and

train a second named entity recognition model using the training data set.

7. The system of claim 6, the UI further comprising:

a display window for displaying the first input and one or more responses.

8. The system of claim 6, wherein the determining comprises:

receiving a confidence level for the two or more responses from each respective model of the two or more trained named entity recognition models;

averaging a value of the confidence levels; and

determining that the average value is less than a predetermined threshold value.

9. The system of claim 6, wherein the instructions further cause the one or more processors to:

determine that the exceptions file includes at least one entry;

provide the request and the two or more responses to a different trained named entity recognition model that is separate from the two or more trained named entity recognition models; and

receive, from the different trained named entity recognition model, a corrected response to the request.

10. The system of claim 9, wherein the instructions further cause the one or more processors to:

store, in the memory, the corrected response in the training data set; and

train the second named entity recognition model using the training data set.

11. A system for providing natural language processing and interactive responses, the system comprising:

one or more processors;

a display for providing a user interface (UI), the UI comprising;

an input field for receiving an input by a user; and

a display window for displaying the user input and one of more responses;

a natural language processing (NLP) device comprising two or more trained named entity recognition models; and

a memory in communication with at least the one or more processors, the display, and the NLP device, the memory storing instructions that, when executed by the one or more processors, are configured to cause the system to;

receive, via the input field, a first input comprising a first natural language request;

provide, to the NLP device, the first input;

process, using the two or more trained named entity recognition models, the first input to apply pseudo labels to named entities in the first input;

receive, from each of the two of more trained entity recognition models, a response, each response comprising two or more named entities with corresponding pseudo labels;

determine, that the two or more responses do not match;

store, in the memory, the user input and the two or more responses in an entry in an exceptions file;

display, on the user interface, a notification to a reviewer to review the exceptions file;

provide, to the user via the UI, the exceptions file;

receive, via the UI, a corrected response from the user;

store, the memory, the corrected response in a new training data set;

retrieve, from the memory, the new training data set;

train a new trained entity recognition model using the new training data set; and

store, in the memory, the new trained entity recognition model.

12. The system of claim 11, wherein the instructions further cause the one or more processors to:

provide the request and the two or more responses to a different trained named entity recognition model that is separate from the two or more trained named entity recognition models; and

receive, from the different trained named entity recognition model, a corrected response to the request.

13. The system of claim 11, further comprising an audio sensor.

14. The system of claim 13, wherein the instructions further cause the one or more processors to:

receive a spoken word request via the audio sensor;

process the spoken word request to obtain an audio natural language request; and

wherein the first natural language request comprises the audio natural language request.

15. The system of claim 11, wherein the determining comprises:

receiving a confidence level for each of the responses from the two or more trained named entity recognition models;

averaging the confidence levels; and

determining that the average is less than a predetermined threshold value.