TRANSFORMING EVENT DATA USING REMOTE CAPTURE AGENTS AND TRANSFORMATION SERVERS

• US 20180006911A1
• Filed: 09/19/2017
• Published: 01/04/2018
• Est. Priority Date: 04/15/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Dickey, Michael
• Granted Patent: US 10,257,059 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0856 by backing up or archiving configuration information
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps

Abstract

The disclosed embodiments provide a method and system for processing network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network data obtained from network packets at the remote capture agent. The system then uses the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent.

Claims

1. A computer-implemented method performed by a configuration server coupled to a network, the method comprising:

receiving input specifying;

at least one first setting related to generation of an event stream by at least one remote capture agent, the event stream including timestamped event data generated from network packets to be monitored by the at least one remote capture agent,

at least one second setting related to transformation, by a transformation server that is separate from the at least one remote capture agent, of the timestamped event data of the event stream into transformed timestamped event data, and

at least one third setting instructing the at least one remote capture agent to send the timestamped event data of the event stream to the transformation server;

generating configuration information based on the at least one first setting, the at least one second setting, and the at least one third setting; and

sending at least a portion of the configuration information to the at least one remote capture agent, the configuration information causing the at least one remote capture agent to generate the event stream and to send the generated event stream to the transformation server.

2. The computer-implemented method of claim 1, wherein causing the at least one remote capture agent to generate the event stream includes causing the at least one remote capture agent to segment the monitored network packets into a plurality of events and to associate each event of the plurality of events with a timestamp.

3. The computer-implemented method of claim 1, wherein the configuration information causes the transformation server to transform the event stream by performing an operation involving data contained in at least one timestamped event of the timestamped event data.

4. The computer-implemented method of claim 1, wherein the transformation includes at least one of:

generating one or more new timestamped events based on data aggregated from two or more separate timestamped events, generating one or more new timestamped events from data contained in a single timestamped event, performing a hashing algorithm on one or more fields of the timestamped event data, storing the timestamped event data in one or more log files, and storing the timestamped event data into one or more database tables.

5. The computer-implemented method of claim 1, further comprising:

receiving additional input updating one or more settings of the configuration information; and

sending, based on the additional input, updated configuration information to the at least one remote capture agent and to the transformation server.

6. The computer-implemented method of claim 1, wherein the configuration information includes at least one of:

a description of the event stream, an event stream type identifier of the event stream, a custom field associated with generating the event stream, and an additional parameter associated with generating the event stream.

7. The computer-implemented method of claim 1, wherein the configuration information causes the at least one remote capture agent to transform the timestamped event data.

8. The computer-implemented method of claim 1, wherein the configuration information causes the at least one remote capture agent to transform the timestamped event data, the transformation including at least one of:

generating one or more new timestamped events based on data aggregated from two or more separate timestamped events, generating one or more new timestamped events from data contained in a single timestamped event, performing a hashing algorithm on one or more fields of the timestamped event data.

9. The computer-implemented method of claim 1, wherein transformation of the timestamped event data includes:

obtaining a time interval associated with the timestamped event data or the network packets, and

aggregating the timestamped event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

10. The computer-implemented method of claim 1, further comprising causing display of a graphical user interface (GUI) including user interface elements corresponding to the at least one first setting, the at least one second setting, and the at least one third setting, and wherein the input is received via the GUI.

11. The computer-implemented method of claim 1, wherein the entire configuration information is sent to the at least one remote capture agent and the to the transformation server.

12. The computer-implemented method of claim 1, wherein only a portion of the configuration information is sent to the at least one remote capture agent.

13. The computer-implemented method of claim 1, wherein only a portion of the configuration information is sent to the transformation server.

14. The computer-implemented method of claim 1, wherein the transformation server is one of a plurality of transformation servers.

15. The computer-implemented method of claim 1, wherein the transformation server is a first transformation server of a plurality of transformation servers, and wherein the first transformation server sends, based on the configuration information, the transformed event stream to a second transformation server of the plurality of transformation servers for subsequent transformation of the transformed event stream.

16. The computer-implemented method of claim 1, wherein the transformation server performs at least two different types of transformations to the event stream.

17. The computer-implemented method of claim 1, further comprising causing display of a GUI including a first graphical indication of the event stream, a second graphical indication of the transformation server, and a third graphical indication indicating a connection between the event stream and the transformation server.

18. The computer-implemented method of claim 1, further comprising causing display of a GUI including a first graphical indication of the event stream, a second graphical indication of the transformation server, and a third graphical indication indicating a connection between the event stream and the transformation server, wherein the at least one third setting is generated based on input received via the GUI creating a graphical connection between the first graphical indication of the event stream and the second graphical indication of the transformation server.

19. An apparatus, comprising:

one or more processors;

a non-transitory computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium storing instructions which, when executed by the one or more processors, causes the apparatus to;

receive input specifying;

at least one first setting related to generation of an event stream by at least one remote capture agent, the event stream including timestamped event data generated from network packets to be monitored by the at least one remote capture agent,

at least one second setting related to transformation, by a transformation server that is separate from the at least one remote capture agent, of the timestamped event data of the event stream into transformed timestamped event data, and

at least one third setting instructing the at least one remote capture agent to send the timestamped event data of the event stream to the transformation server;

generate configuration information based on the at least one first setting, the at least one second setting, and the at least one third setting; and

send at least a portion of the configuration information to the at least one remote capture agent, the configuration information causing the at least one remote capture agent to generate the event stream and to send the generated event stream to the transformation server.

20. The apparatus of claim 19, wherein causing the at least one remote capture agent to generate the event stream includes causing the at least one remote capture agent to segment the monitored network packets into a plurality of events and to associate each event of the plurality of events with a timestamp.

21. The apparatus of claim 19, wherein the configuration information causes the transformation server to transform the event stream by performing an operation involving data contained in at least one timestamped event of the timestamped event data.

22. The apparatus of claim 19, wherein the transformation includes at least one of:

generating one or more new timestamped events based on data aggregated from two or more separate timestamped events, generating one or more new timestamped events from data contained in a single timestamped event, performing a hashing algorithm on one or more fields of the timestamped event data, storing the timestamped event data in one or more log files, and storing the timestamped event data into one or more database tables.

23. The apparatus of claim 19, wherein the configuration information includes at least one of:

a description of the event stream, an event stream type identifier of the event stream, a custom field associated with generating the event stream, and an additional parameter associated with generating the event stream.

24. The apparatus of claim 19, wherein the configuration information causes the at least one remote capture agent to transform the timestamped event data, the transformation including at least one of:

generating one or more new timestamped events based on data aggregated from two or more separate timestamped events, generating one or more new timestamped events from data contained in a single timestamped event, performing a hashing algorithm on one or more fields of the timestamped event data.

25. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform operations comprising:

receiving input specifying;

at least one first setting related to generation of an event stream by at least one remote capture agent, the event stream including timestamped event data generated from network packets to be monitored by the at least one remote capture agent,

at least one second setting related to transformation, by a transformation server that is separate from the at least one remote capture agent, of the timestamped event data of the event stream into transformed timestamped event data, and

at least one third setting instructing the at least one remote capture agent to send the timestamped event data of the event stream to the transformation server;

generating configuration information based on the at least one first setting, the at least one second setting, and the at least one third setting; and

sending at least a portion of the configuration information to the at least one remote capture agent, the configuration information causing the at least one remote capture agent to generate the event stream and to send the generated event stream to the transformation server.

26. The non-transitory computer-readable storage medium of claim 25, claim wherein causing the at least one remote capture agent to generate the event stream includes causing the at least one remote capture agent to segment the monitored network packets into a plurality of events and to associate each event of the plurality of events with a timestamp.

27. The non-transitory computer-readable storage medium of claim 25, claim wherein the configuration information causes the transformation server to transform the event stream by performing an operation involving data contained in at least one timestamped event of the timestamped event data.

28. The non-transitory computer-readable storage medium of claim 25, claim wherein the transformation includes at least one of:

generating one or more new timestamped events based on data aggregated from two or more separate timestamped events, generating one or more new timestamped events from data contained in a single timestamped event, performing a hashing algorithm on one or more fields of the timestamped event data, storing the timestamped event data in one or more log files, and storing the timestamped event data into one or more database tables.

29. The non-transitory computer-readable storage medium of claim 25, claim wherein the configuration information includes at least one of:

a description of the event stream, an event stream type identifier of the event stream, a custom field associated with generating the event stream, and an additional parameter associated with generating the event stream.

30. The non-transitory computer-readable storage medium of claim 25, claim wherein the configuration information causes the at least one remote capture agent to transform the timestamped event data, the transformation including at least one of:

generating one or more new timestamped events based on data aggregated from two or more separate timestamped events, generating one or more new timestamped events from data contained in a single timestamped event, performing a hashing algorithm on one or more fields of the timestamped event data.