AUTOMATED DATA-GENERATION FOR EVENT-BASED SYSTEM

• US 20180032861A1
• Filed: 07/29/2016
• Published: 02/01/2018
• Est. Priority Date: 07/29/2016
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Oliner, Adam, Yang, Zidong, Sreshta, Sinduja
• Granted Patent: US 11,227,208 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06Q 10/0637 Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals

Abstract

Described herein is a technology that facilitates the production of and the use of automated datagens for event-based. A datagen (i.e., data-generator or data generation system) is a component, module, or subsystem of computer systems that searches, monitors, and analyzes machine data. A datagen produces events that are further processed in various ways for subsequent use (such as searching, monitoring, and analysis).

Claims

1. A computer-implemented method comprising:

obtaining a training corpus that contains example structured events, each structured event having at least one data field configured to contain data derived from machine-generated data;

training a datagen that is configured to generate new structured events in accordance with the example structured events of the training corpus, the training includes employment of deep-learning technique to train the datagen using the training corpus.

2. A method as recited in claim 1, wherein the deep-learning technique includes a character-based recurrent neural network.

3. A method as recited in claim 1 further comprising producing new events by the trained datagen.

4. A method as recited in claim 1, wherein the example structured events of the training corpus is a sequence of textual characters and the training of the datagen includes calculating statistical predictions based upon the sequence of the textual characters of the example structured events of the training corpus.

5. A method as recited in claim 1, wherein the example structured events of the training corpus is a sequence of textual characters and the training of the datagen includes calculating statistical predictions based upon the sequence of the textual characters of the example structured events of the training corpus, the method further comprising:

producing new events by the trained datagen, the producing includes generating a sequence of characters of the new events based upon the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus.

6. A method as recited in claim 1, wherein the example structured events of the training corpus is a sequence of textual characters and the training of the datagen includes calculating statistical predictions based upon the sequence of the textual characters of the example structured events of the training corpus, the method further comprising:

producing new events by the trained datagen, the producing includes generating a sequence of characters of the new events based upon the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus,

wherein the generating includes selecting a next character of a new event based upon greatest likelihood of the selected character appearing next in the generated sequence as determined by the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus.

7. A method as recited in claim 1 further comprising:

obtaining a dataset of machine-generated data;

generating, by the trained datagen and in accordance with the example structured events of the training corpus, multiple events from the obtained dataset of machine-generated data.

8. A method as recited in claim 1 further comprising:

obtaining a dataset of machine-generated data, wherein the dataset of machine-generated data includes a sequence of textual characters;

processing the obtained dataset of machine-generated data by the trained datagen in order of the sequence of textual characters;

during the processing of the obtained dataset, calculating a statistical prediction of a next yet-to-be-processed group of one or more textual characters of the sequence of the textual characters;

generating, by the trained datagen, an event from the obtained dataset of machine-generated data one textual character at a time, wherein each generated character is generated based on the calculated statistical prediction of the next yet-to-be-processed group of one or more textual characters.

9. One or more computer-readable media storing instructions thereon that, when executed by one or more processors, direct the one or more processors to perform operations comprising:

obtaining a training corpus that contains example structured events, each structured event having at least one data field configured to contain data derived from machine-generated data;

training a datagen that is configured to generate new structured events in accordance with the example structured events of the training corpus, the training includes employment of deep-learning technique to train the datagen using the training corpus.

10. One or more computer-readable media as recited in claim 9, wherein the deep-learning technique includes a character-based recurrent neural network.

11. One or more computer-readable media as recited in claim 9, wherein the operations further comprise producing new events by the trained datagen.

12. One or more computer-readable media as recited in claim 9, wherein the example structured events of the training corpus is a sequence of textual characters and the training of the datagen includes calculating statistical predictions based upon the sequence of the textual characters of the example structured events of the training corpus, the operations further comprise:

producing new events by the trained datagen, the producing includes generating a sequence of characters of the new events based upon the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus.

13. One or more computer-readable media as recited in claim 9, wherein the example structured events of the training corpus is a sequence of textual characters and the training of the datagen includes calculating statistical predictions based upon the sequence of the textual characters of the example structured events of the training corpus, the operations further comprise:

producing new events by the trained datagen, the producing includes generating a sequence of characters of the new events based upon the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus,

wherein the generating includes selecting a next character of a new event based upon greatest likelihood of the selected character appearing next in the generated sequence as determined by the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus.

14. One or more computer-readable media as recited in claim 9, the operations further comprise:

obtaining a dataset of machine-generated data;

generating, by the trained datagen and in accordance with the example structured events of the training corpus, multiple events from the obtained dataset of machine-generated data.

15. One or more computer-readable media as recited in claim 9, the operations further comprise:

obtaining a dataset of machine-generated data, wherein the dataset of machine-generated data includes a sequence of textual characters;

processing the obtained dataset of machine-generated data by the trained datagen in order of the sequence of textual characters;

during the processing of the obtained dataset, calculating a statistical prediction of a next yet-to-be-processed group of one or more textual characters of the sequence of the textual characters;

generating, by the trained datagen, an event from the obtained dataset of machine-generated data one textual character at a time, wherein each generated character is generated based on the calculated statistical prediction of the next yet-to-be-processed group of one or more textual characters.

16. An automated data generation system comprising:

a training data handler configured to obtain a training corpus that contains example structured events, each structured event having at least one data field configured to contain data derived from machine-generated data;

a deep-learning engine configured to receive the obtained training corpus from the training data handler, the deep-learning engine further configured to train a datagen to generate new structured events in accordance with the example structured events of the training corpus, the training includes employment of deep-learning engine to train the datagen using the training corpus.

17. A system as recited in claim 16, wherein the deep-leaning engine is configured to employ a character-based recurrent neural network.

18. A system as recited in claim 16, wherein the trained datagen is configured to generate new structured events.

19. A system as recited in claim 16, wherein:

the example structured events of the training corpus is a sequence of textual characters;

the deep-learning engine is further configured to;

calcualte statistical predictions based upon the sequence of the textual characters of the example structured events of the training corpus;

produce new events by generating a sequence of characters of the new events based upon the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus, wherein the generating includes selecting a next character of a new event based upon greatest likelihood of the selected character appearing next in the generated sequence as determined by the calculated statistical predictions of the sequence of the textual characters of the example structured events of the training corpus.