Dynamic configuration of remote capture agents for network data capture

• US 9,923,767 B2
• Filed: 04/15/2014
• Issued: 03/20/2018
• Est. Priority Date: 04/15/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Dickey, Michael
• Primary Examiner(s): Baturay, Alicia
• Application Number:
  US14/253,744
• Publication Number:
  US 20150295765A1
• Time in Patent Office: 1,435 Days
• Field of Search: 709224, 709206, 709202, 709219, 709226
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0856 by backing up or archiving configuration information

Abstract

The disclosed embodiments provide a method and system for facilitating the processing of network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network packets at the remote capture agent. Upon receiving an update to the configuration information from the configuration server, the system uses the update to reconfigure the generation of the event data by the remote capture agent during runtime of the remote capture agent.

Claims

1. A computer-implemented method performed by a remote capture agent coupled to a network, the method comprising:

obtaining configuration information from a configuration server over the network, the configuration information specifying a plurality of event streams to be generated by the remote capture agent and further specifying a respective event type associated with each event stream of the plurality of event streams;

monitoring network traffic comprising a plurality of network packets;

generating, based on the configuration information, a plurality of events from the network traffic, wherein generating an event of the plurality of events comprises;

extracting network packet data from at least one network packet of the plurality of network packets and associating the extracted network packet data with the event;

applying a filtering rule to the extracted network packet data to determine an event type associated with the event;

adding, based on the determined event type, the event to at least one event stream of the plurality of event streams;

for each event stream of the plurality of event streams;

selecting, based on the event type for the event stream specified in the configuration information, a component of a plurality of components on the network to which to send the event stream; and

sending the event stream to the selected component on the network for subsequent processing.

2. The computer-implemented method of claim 1, wherein the selected component is a transformation server used to further transform the event data.

3. The computer-implemented method of claim 1, further comprising:

transforming an event of the plurality of events into a transformed event; and

sending an event stream of the plurality of event streams containing the transformed event to one or more transformation servers.

4. The computer-implemented method of claim 1, wherein the configuration information specifies one or more transformations of the event data, and wherein the one or more transformations comprise at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

5. The computer-implemented method of claim 1, wherein the configuration information is obtained using at least one of a push mechanism and a pull mechanism.

6. The computer-implemented method of claim 1, wherein the configuration information comprises at least one of an identifier for an event stream, a description for the event stream, an event stream type for the event stream, a custom field for the event stream, and an additional parameter for the event stream.

7. The computer-implemented method of claim 1, wherein the event data comprises at least one of a transaction type, a timestamp, and an error indicator.

8. The computer-implemented method of claim 1, wherein the configuration information comprises an additional parameter, and wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, and an inclusion of a matching transaction or matching error in the event data.

9. The computer-implemented method of claim 1, wherein the remote capture agent is installed in a virtual computing environment.

10. The computer-implemented method of claim 1, wherein at least one event stream of the plurality of event streams corresponds to event data including one or more of the following:

clickstream events;

HTTP transactions;

business transactions;

errors;

alerts; and

classified transactions.

11. The computer-implemented method of claim 1, further comprising:

obtaining an update to the configuration information from the configuration server; and

generating, based on the updated configuration information, additional events from the network traffic.

12. A computer-implemented method performed by a configuration server coupled to a network, the method comprising:

obtaining configuration information for a set of remote capture agents on a set of networks, the configuration information specifying a plurality of event streams to be generated by each remote capture agent of the set of remote capture agents and further specifying a respective event type associated with each event stream of the plurality of event streams;

sending the configuration information to at least one remote capture agent of the set of remote capture agents, the configuration information causing the at least one remote capture agent of the set of remote capture agents to generate a plurality of events from network traffic, wherein generating an event of the plurality of events comprises;

extracting network packet data from at least one network packet and associating the extracted network packet data with the event;

applying a filtering rule to the extracted network packet data to determine an event type associated with the event; and

adding, based on the determined event type, the event to at least one event stream of the plurality of event streams;

wherein the configuration information further causes the at least one remote capture agent of the set of remote capture agents to, for each event stream of the plurality of event streams;

select, based on the event type for the event stream specified in the configuration information, a component of a plurality of components on the network to which to send the event stream; and

send the event stream to the selected component on the network for subsequent processing.

13. The computer-implemented method of claim 12, further comprising:

obtaining an update to the configuration information at the configuration server; and

sending the updated configuration information to the remote capture agents, wherein the update is used by the remote capture agents to reconfigure the generation of the event data during runtime of the remote capture agents.

14. The computer-implemented method of claim 12, wherein the configuration information is sent to the remote capture agents using at least one of a push mechanism and a pull mechanism.

15. The computer-implemented method of claim 12, wherein the configuration information is obtained from an application used to access the event data after the event data is generated.

16. The computer-implemented method of claim 12, wherein the method further comprises:

storing event data from the plurality of event streams in a data store; and

while subsequently processing a query,

employing a retrieval schema, which includes an extraction rule that indicates how to extract one or more values from an event, to extract values from event data stored in the data store; and

identifying responsive events based on the extracted values; and

wherein the retrieval schema is a late-binding retrieval schema that is applied during query execution.

17. The computer-implemented method of claim 12, wherein at least one event stream of the plurality of event streams corresponds to event data including one or more of the following:

clickstream events;

HTTP transactions;

business transactions;

errors;

alerts; and

classified transactions.

18. The computer-implemented method of claim 12, further comprising:

obtaining an update to the configuration information for the set of remote capture agents on the set of networks; and

sending the updated configuration information to at least one remote capture agent of the set of remote capture agents, the updated configuration information causing the at least one remote capture agent of the set of remote capture agents to generate additional events from the network traffic.

19. A remote capture agent coupled to a network, the remote capture agent comprising:

a processor;

a non-transitory computer readable storage medium storing instructions which, when executed by the processor, cause the remote capture agent to;

obtain configuration information from a configuration server over a network, the configuration information specifying a plurality of event streams to be generated by the remote capture agent and further specifying a respective event type associated with each event stream of the plurality of event streams;

monitor network traffic comprising a plurality of network packets;

generate, based on the configuration information, a plurality of events from the network traffic, wherein generating an event of the plurality of events comprises;

extracting network packet data from at least one network packet of the plurality of network packets and associating the extracted network packet data with the event;

applying a filtering rule to the extracted network packet data to determine an event type associated with the event;

adding, based on the determined event type, the event to at least one event stream of the plurality of event streams

for each event stream of the plurality of event streams;

select, based on the event type for the event stream specified in the configuration information, a component of a plurality of components on the network to which to send the event stream; and

send the event stream to the selected component on the network for subsequent processing.

20. The remote capture agent of claim 19, wherein the remote capture agent obtains the configuration information using at least one of a push mechanism and a pull mechanism.

21. The remote capture agent of claim 19, wherein the configuration information comprises at least one of an identifier for an event stream, a description for the event stream, an event stream type for the event stream, a custom field for the event stream, and an additional parameter for the event stream.

22. The remote capture agent of claim 19, wherein the configuration information comprises an additional parameter, and wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, an inclusion of a matching transaction or matching error in the event data, and a type of event used by the event stream.

23. The remote capture agent of claim 19, wherein at least one event stream of the plurality of event streams corresponds to event data including one or more of the following:

clickstream events;

HTTP transactions;

business transactions;

errors;

alerts; and

classified transactions.

24. The remote capture agent of claim 19, wherein the instructions which, when executed by the processor, further cause the remote capture agent to:

obtain an update to the configuration information from the configuration server; and

generate, based on the updated configuration information, additional events from the network traffic.

25. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause a remote capture agent to perform operations comprising:

obtaining configuration information from a configuration server over a network, the configuration information specifying a plurality of event streams to be generated by the remote capture agent and further specifying a respective event type associated with each event stream of the plurality of event streams;

monitoring network traffic comprising a plurality of network packets;

generating, based on the configuration information, a plurality of events from the network traffic, wherein generating an event of the plurality of events comprises;

extracting network packet data from at least one network packet of the plurality of network packets and associating the extracted network packet data with the event;

applying a filtering rule to the extracted network packet data to determine an event type associated with the event;

adding, based on the determined event type, the event to at least one event stream of the plurality of event streams;

for each event stream of the plurality of event streams;

selecting, based on the event type for the event stream specified in the configuration information, a component of a plurality of components on the network to which to send the event stream; and

sending the event stream to the selected component on the network for subsequent processing.

26. The non-transitory computer-readable storage medium of claim 25, wherein the selected component is a transformation server used to further transform the event data.

27. The non-transitory computer-readable storage medium of claim 25, wherein the configuration information is obtained using at least one of a push mechanism and a pull mechanism.

28. The non-transitory computer-readable storage medium of claim 25, wherein the configuration information comprises at least one of an identifier for an event stream of the plurality of event streams, a description of the event stream, an event stream type for the event stream, a custom field for the event stream, and an additional parameter for the event stream.

29. The non-transitory computer-readable storage medium of claim 25, wherein the instructions, when executed by the computer, further cause the remote capture agent to perform operations comprising:

storing event data from the plurality of event streams in a data store; and

while subsequently processing a query,

employing a retrieval schema, which includes an extraction rule that indicates how to extract one or more values from an event, to extract values from event data stored in the data store; and

identifying responsive events based on the extracted values; and

wherein the retrieval schema is a late-binding retrieval schema that is applied during query execution.

30. The non-transitory computer-readable storage medium of claim 25, wherein at least one event stream of the plurality of event streams corresponds to event data including one or more of the following:

clickstream events;

HTTP transactions;

business transactions;

errors;

alerts; and

classified transactions.

31. The non-transitory computer-readable storage medium of claim 25, wherein the instructions which, when executed by the computer, cause the remote capture agent to perform operations further comprising:

receiving an update to the configuration information from the configuration server; and

generating, based on the updated configuration information, additional events from the network traffic.