Classifying data with deep learning neural records incrementally refined through expert input

• US 9,324,022 B2
• Filed: 03/04/2015
• Issued: 04/26/2016
• Est. Priority Date: 03/04/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Signal/Sense Incorporated
• Inventors: Williams, David Russell Jr., Gutzwiller, Luke Robert, Hazen, Megan Ursula, Anderson, Brigham Sterling, McIntyre, Alan, Abeles, Tom
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Nilsson, Eric
• Application Number:
  US14/639,005
• Publication Number:
  US 20150254555A1
• Time in Patent Office: 419 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/086 using evolutionary algorithms, e.g. genetic algorithms or genetic programming

Abstract

Embodiments are directed towards classifying data using machine learning that may be incrementally refined based on expert input. Data provided to a deep learning model that may be trained based on a plurality of classifiers and sets of training data and/or testing data. If the number of classification errors exceeds a defined threshold classifiers may be modified based on data corresponding to observed classification errors. A fast learning model may be trained based on the modified classifiers, the data, and the data corresponding to the observed classification errors. And, another confidence value may be generated and associated with the classification of the data by the fast learning model. Report information may be generated based on a comparison result of the confidence value associated with the fast learning model and the confidence value associated with the deep learning model.

Claims

1. A method for classifying information over a network using a computer that includes one or more hardware processors, where each action of the method is performed by the one or more hardware processors, comprising:

providing data to a deep learning model, wherein the deep learning model was previously trained based on a plurality of classifiers and one or more sets of training data;

classifying the data using the deep learning model and the one or more classifiers, wherein a confidence value is associated with the deep learning model classification of the data;

when a number of deep learning model classification errors exceeds a defined threshold, performing further actions, including;

modifying one or more classifiers of the plurality of classifiers based on the data corresponding to one or more of the deep learning model classification errors;

employing the one or more modified classifiers and that portion of the data that corresponds to the one or more deep learning model classification errors to train a fast learning model;

employing the fast learning model and the one or more modified classifiers to also classify the data, wherein another confidence value is associated with the classification of the data by the fast learning model; and

generating report information based on a comparison result of the other confidence value that is associated with the fast learning model and the confidence value that is associated with the deep learning model.

2. The method of claim 1, further comprising:

retraining the deep learning model using the one or more modified classifiers; and

discarding the trained fast learning model.

3. The method of claim 1, wherein the data includes one or more of real-time network information or captured network information.

4. The method of claim 1, further comprising, when the data is classified as associated with anomalous activity, generating one or more notifications based on a type of the anomalous activity.

5. The method of claim 1, further comprising:

when the data is classified as associated with a new network entity, performing further actions, including;

associating historical network information with the new network entity based on a type of the new network entity; and

buffering real-time network information that is associated with the new network entity.

6. The method of claim 1, further comprising, buffering the data in real-time using a sensor computer, wherein the data is network information.

7. The method of claim 1, wherein exceeding the defined threshold, further comprises, exceeding one or more different thresholds that are defined for different types of deep learning model classification errors, wherein the deep learning model classification errors related to dangerous events have a lower defined threshold than the deep learning model classification errors related to safe events.

8. The method of claim 1, further comprising, retraining the deep learning model based on a defined schedule.

9. A system for classifying information over a network, comprising:

a network computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

a processor device that executes instructions that perform actions, including;

providing data to a deep learning model, wherein the deep learning model was previously trained based on a plurality of classifiers and one or more sets of training data;

classifying the data using the deep learning model and the one or more classifiers, wherein a confidence value is associated with the deep learning model classification of the data; and

when a number of deep learning model classification errors exceeds a defined threshold, performing further actions, including;

modifying one or more classifiers of the plurality of classifiers based on the data corresponding to one or more of the deep learning model classification errors;

employing the one or more modified classifiers and that portion of the data that corresponds to the one or more deep learning model classification errors to train a fast learning model;

employing the fast learning model and the one or more modified classifiers to also classify the data, wherein another confidence value is associated with the classification of the data by the fast learning model; and

generating report information based on a comparison result of the other confidence value that is associated with the fast learning model and the confidence value that is associated with the deep learning model; and

a client computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

a processor device that executes instructions that perform actions, including;

providing at least a portion of the data to the deep learning model.

10. The system of claim 9, wherein the network computer processor device executes instructions that perform actions, further comprising:

retraining the deep learning model using the one or more modified classifiers; and

discarding the trained fast learning model.

11. The system of claim 9, wherein the data includes one or more of real-time network information or captured network information.

12. The system of claim 9, wherein the network computer processor device executes instructions that perform actions, further comprising, when the data is classified as associated with anomalous activity, generating one or more notifications based on a type of the anomalous activity.

13. The system of claim 9, wherein the network computer processor device executes instructions that perform actions, further comprising:

when the data is classified as associated with a new network entity, performing further actions, including;

associating historical network information with the new network entity based on a type of the new network entity; and

buffering real-time network information that is associated with the new network entity.

14. The system of claim 9, wherein the network computer processor device executes instructions that perform actions, further comprising, buffering the data in real-time using a sensor computer, wherein the data is network information.

15. The system of claim 9, wherein exceeding the defined threshold, further comprises, exceeding one or more different thresholds that are defined for different types of deep learning model classification errors, wherein the deep learning model classification errors related to dangerous events have a lower defined threshold than the deep learning model classification errors related to safe events.

16. The system of claim 9, wherein the network computer processor device executes instructions that perform actions, further comprising, retraining the deep learning model based on a defined schedule.

17. A processor readable non-transitory storage media that includes instructions for classifying information, wherein execution of the instructions by a processor device performs actions, comprising:

providing data to a deep learning model, wherein the deep learning model was previously trained based on a plurality of classifiers and one or more sets of training data;

classifying the data using the deep learning model and the one or more classifiers, wherein a confidence value is associated with the deep learning model classification of the data;

when a number of deep learning model classification errors exceeds a defined threshold, performing further actions, including;

modifying one or more classifiers of the plurality of classifiers based on the data corresponding to one or more of the deep learning model classification errors;

employing the one or more modified classifiers and that portion of the data that corresponds to the one or more deep learning model classification errors to train a fast learning model;

employing the fast learning model and the one or more modified classifiers to also classify the data, wherein another confidence value is associated with the classification of the data by the fast learning model; and

generating report information based on a comparison result of the other confidence value that is associated with the fast learning model and the confidence value that is associated with the deep learning model.

18. The media of claim 17, further comprising:

retraining the deep learning model using the one or more modified classifiers; and

discarding the trained fast learning model.

19. The media of claim 17, wherein the data includes one or more of real-time network information or captured network information.

20. The media of claim 17, further comprising, when the data is classified as associated with anomalous activity, generating one or more notifications based on a type of the anomalous activity.

21. The media of claim 17, further comprising:

when the data is classified as associated with a new network entity, performing further actions, including;

associating historical network information with the new network entity based on a type of the new network entity; and

buffering real-time network information that is associated with the new network entity.

22. The media of claim 17, further comprising, buffering the data in real-time using a sensor computer, wherein the data is network information.

23. The media of claim 17, wherein exceeding the defined threshold, further comprises, exceeding one or more different thresholds that are defined for different types of deep learning model classification errors, wherein the deep learning model classification errors related to dangerous events have a lower defined threshold than the deep learning model classification errors related to safe events.

24. A network computer for classifying information, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

a processor device that executes instructions that perform actions, including;

providing data to a deep learning model, wherein the deep learning model was previously trained based on a plurality of classifiers and one or more sets of training data;

classifying the data using the deep learning model and the one or more classifiers, wherein a confidence value is associated with the deep learning model classification of the data;

when a number of deep learning model classification errors exceeds a defined threshold, performing further actions, including;

modifying one or more classifiers of the plurality of classifiers based on the data corresponding to one or more of the deep learning model classification errors;

employing the one or more modified classifiers and that portion of the data that corresponds to the one or more deep learning model classification errors to train a fast learning model;

employing the fast learning model and the one or more modified classifiers to also classify the data, wherein another confidence value is associated with the classification of the data by the fast learning model; and

generating report information based on a comparison result of the other confidence value that is associated with the fast learning model and the confidence value that is associated with the deep learning model.

25. The network computer of claim 24, further comprising:

retraining the deep learning model using the one or more modified classifiers; and

discarding the trained fast learning model.

26. The network computer of claim 24, wherein the data includes one or more of real-time network information or captured network information.

27. The network computer of claim 24, further comprising, when the data is classified as associated with anomalous activity, generating one or more notifications based on a type of the anomalous activity.

28. The network computer of claim 24, further comprising:

when the data is classified as associated with a new network entity, performing further actions, including;

associating historical network information with the new network entity based on a type of the new network entity; and

buffering real-time network information that is associated with the new network entity.

29. The network computer of claim 24, further comprising, buffering the data in real-time using a sensor computer, wherein the data is network information.

30. The network computer of claim 24, wherein exceeding the defined threshold, further comprises, exceeding one or more different thresholds that are defined for different types of deep learning model classification errors, wherein the deep learning model classification errors related to dangerous events have a lower defined threshold than the deep learning model classification errors related to safe events.