Transformation of network data at remote capture agents

• US 9,762,443 B2
• Filed: 04/15/2014
• Issued: 09/12/2017
• Est. Priority Date: 04/15/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Dickey, Michael
• Primary Examiner(s): BAYARD, DJENANE M
• Application Number:
  US14/253,753
• Publication Number:
  US 20150295766A1
• Time in Patent Office: 1,246 Days
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0856 by backing up or archiving configuration information
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps

Abstract

The disclosed embodiments provide a method and system for processing network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network data obtained from network packets at the remote capture agent. The system then uses the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent.

Claims

1. A computer-implemented method performed by a remote capture agent coupled to a network, comprising:

obtaining configuration information from a configuration server over a network, wherein the configuration information is usable by the remote capture agent to generate timestamped event data from network packets and to transform the timestamped event data into transformed event data;

monitoring network traffic comprising a plurality of network packets;

generating, based on the configuration information, timestamped event data from at least one network packet of the plurality of network packets, wherein generating the timestamped event data includes segmenting the at least one network packet into a plurality of events and associating each event of the plurality of events with a respective timestamp; and

transforming, based on the same configuration information, the timestamped event data into transformed event data, wherein transforming the timestamped event data includes performing an operation involving data contained in at least one event of the plurality of events.

2. The computer-implemented method of claim 1, further comprising:

sending an event stream comprising the transformed event data to one or more stream servers for further transformation of the transformed event data by the one or more stream servers.

3. The computer-implemented method of claim 2, wherein the event stream includes a type of event data specified in the configuration information.

4. The computer-implemented method of claim 1, further comprising:

receiving an update to the configuration information from the configuration server; and

generating, based on the updated configuration information, timestamped event data from at least one network packet of the plurality of network packets.

5. The computer-implemented method of claim 1, wherein transformation of the timestamped event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

6. The computer-implemented method of claim 1, wherein transformation of the timestamped event data comprises:

obtaining a time interval associated with the timestamped event data or the network data; and

aggregating the timestamped event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

7. The computer-implemented method of claim 1, wherein the configuration server obtains the configuration information from an application used to access the transformed event data.

8. The computer-implemented method of claim 1, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

9. The computer-implemented method of claim 1, wherein the remote capture agent is installed in a virtual computing environment.

10. The computer-implemented method of claim 1, wherein the transformation of the timestamped event data comprises:

identifying a network address in the timestamped event data;

identifying related data in a lookup table based on the identified network address; and

including the related data in the transformed event data.

11. A remote capture agent, comprising:

a processor;

a memory storing instructions which, when executed by the processor, cause the remote

capture agent to;

obtain configuration information from a configuration server over a network,

wherein the configuration information is usable by the remote capture agent to generate timestamped event data from network packets and to transform the timestamped event data into transformed event data;

monitor network traffic comprising a plurality of network packets;

generate, based on the configuration information, timestamped event data based on data contained in at least one network packet of the plurality of network packets; and

transform, based on the same configuration information, the timestamped event data into transformed event data.

12. The remote capture agent of claim 11, wherein the instructions, when executed by the processor, further cause the remote capture agent to:

send an event stream comprising the transformed event data to one or more transformation servers for further transformation of the transformed event data by the one or more transformation servers.

13. The remote capture agent of claim 12, wherein the event stream includes a type of event data specified in the configuration information.

14. The remote capture agent of claim 11, wherein transformation of the timestamped event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

15. The remote capture agent of claim 11, wherein transformation of the timestamped event data comprises:

obtaining a time interval associated with the timestamped event data or the network data; and

aggregating the timestamped event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

16. The remote capture agent of claim 11, wherein the configuration server obtains the configuration information from an application used to access the transformed event data.

17. The remote capture agent of claim 11, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

18. The remote capture agent of claim 11, wherein the transformation of the event data or the network data comprises:

identifying a network address in the timestamped event data;

identifying related data in a lookup table based on the identified network address; and

including the related data in the transformed event data.

19. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause a remote capture agent coupled to a network to perform a method comprising:

obtaining configuration information from a configuration server over a network, wherein the configuration information is usable by the remote capture agent to generate timestamped event data from network packets and to transform the timestamped event data into transformed event data;

monitoring network traffic comprising a plurality of network packets;

generating, based on the configuration information, timestamped event data from at least one network packet of the plurality of network packets, wherein generating the timestamped event data includes segmenting the at least one network packet into a plurality of events and associating each event of the plurality of events with a respective timestamp; and

transforming, based on the same configuration information, the timestamped event data into transformed event data, wherein transforming the timestamped event data includes performing an operation involving data contained in at least one event of the plurality of events.

20. The non-transitory computer-readable storage medium of claim 19, the method further comprising:

sending an event stream comprising the transformed event data to one or more stream servers for further transformation of the transformed event data by the one or more stream servers.

21. The non-transitory computer-readable storage medium of claim 20, wherein the event stream includes a type of event data specified in the configuration information.

22. The non-transitory computer-readable storage medium of claim 19, wherein transformation of the timestamped event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

23. The non-transitory computer-readable storage medium of claim 19, wherein transformation of the timestamped event data comprises:

obtaining a time interval associated with the timestamped event data or the network data; and

aggregating the timestamped event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

24. The non-transitory computer-readable storage medium of claim 19, wherein the configuration server obtains the configuration information from an application used to access the transformed event data.

25. The non-transitory computer-readable storage medium of claim 19, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

26. The non-transitory computer-readable storage medium of claim 19, wherein the transformation of the event data or the network data comprises:

identifying a network address in the timestamped event data;

identifying related data in a lookup table based on the identified network address; and

including the related data in the transformed event data.