TRANSFORMATION OF NETWORK DATA AT REMOTE CAPTURE AGENTS

• US 20150295766A1
• Filed: 04/15/2014
• Published: 10/15/2015
• Est. Priority Date: 04/15/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Dickey, Michael
• Granted Patent: US 9,762,443 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0856 by backing up or archiving configuration information
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps

Abstract

The disclosed embodiments provide a method and system for processing network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network data obtained from network packets at the remote capture agent. The system then uses the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent.

Claims

1. A computer-implemented method for processing network data, comprising:

obtaining, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network;

using the configuration information to configure the generation of event data from network data obtained from network packets at the remote capture agent; and

using the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent;

wherein the transformation of the event data or the network data comprises one or more of,

normalizing different representations of the same data, and

performing a data-enrichment operation that uses an address from the event data or the network data to look up related data in a lookup table, and

include the related data in the transformed event data.

2. The computer-implemented method of claim 1, further comprising:

providing an event stream comprising the transformed event data to one or more transformation servers for further transformation of the transformed event data by the one or more transformation servers.

3. The computer-implemented method of claim 1, further comprising:

upon receiving an update to the configuration information from the configuration server, using the update to reconfigure the transformation of the event data or the network data by the remote capture agent during runtime of the remote capture agent.

4. The computer-implemented method of claim 1, wherein generation of the event data from the network data comprises:

upon identifying one or more network packets associated with an event, using the network data from the one or more network packets to generate the event data corresponding to the event.

5. The computer-implemented method of claim 1, wherein transformation of the event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

6. The computer-implemented method of claim 1, wherein transformation of the event data comprises:

obtaining a time interval associated with aggregation of the event data or the network data; and

aggregating the event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

7. The computer-implemented method of claim 1, wherein the configuration information is provided to the configuration server by an application used to access the transformed event data.

8. The computer-implemented method of claim 1, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

9. The computer-implemented method of claim 1, wherein the remote capture agent is installed in a virtual computing environment.

10. A system for processing network data, comprising:

a processor;

a memory, coupled to the processor, which stores a program module configured to be executed by the processor, the program module including;

instructions for obtaining, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network;

instructions for using the configuration information to configure the generation of event data from network data obtained from network packets at the remote capture agent; and

instructions for using the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent;

wherein the transformation of the event data or the network data comprises one or more of,

normalizing different representations of the same data,

performing a data-enrichment operation that uses an address from the event data or the network data to look up related data in a lookup table, and

include the related data in the transformed event data.

11. The system of claim 10, wherein the program module further includes:

instructions for providing an event stream comprising the transformed event data to one or more transformation servers for further transformation of the transformed event data by the one or more transformation servers.

12. The system of claim 10, wherein the program module further includes:

instructions for, upon identifying one or more network packets associated with an event, using the network data from the one or more network packets to generate the event data corresponding to the event.

13. The system of claim 10, wherein transformation of the event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

14. The system of claim 10, wherein transformation of the event data comprises:

obtaining a time interval associated with aggregation of the event data or the network data; and

aggregating the event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

15. The system of claim 10, wherein the configuration information is provided to the configuration server by an application used to access the transformed event data.

16. The system of claim 10, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

17. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for processing network data, the method comprising:

obtaining, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network;

using the configuration information to configure the generation of event data from the network data obtained from network packets at the remote capture agent; and

using the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent;

wherein the transformation of the event data or the network data comprises one or more of,

normalizing different representations of the same data,

performing a data-enrichment operation that uses an address from the event data or the network data to look up related data in a lookup table, and

include the related data in the transformed event data.

18. The non-transitory computer-readable storage medium of claim 17, the method further comprising:

providing an event stream comprising the transformed event data to one or more transformation servers for further transformation of the transformed event data by the one or more transformation servers.

19. The non-transitory computer-readable storage medium of claim 17, wherein transformation of the event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

20. The non-transitory computer-readable storage medium of claim 17, wherein transformation of the event data comprises:

obtaining a time interval associated with aggregation of the event data or the network data; and

aggregating the event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

21. The non-transitory computer-readable storage medium of claim 17, wherein the configuration information is provided to the configuration server by an application used to access the transformed event data.

22. The non-transitory computer-readable storage medium of claim 17, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.