DYNAMIC CONFIGURATION OF REMOTE CAPTURE AGENTS FOR NETWORK DATA CAPTURE

• US 20150295765A1
• Filed: 04/15/2014
• Published: 10/15/2015
• Est. Priority Date: 04/15/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Dickey, Michael
• Granted Patent: US 9,923,767 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0856 by backing up or archiving configuration information

Abstract

The disclosed embodiments provide a method and system for facilitating the processing of network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network packets at the remote capture agent. Upon receiving an update to the configuration information from the configuration server, the system uses the update to reconfigure the generation of the event data by the remote capture agent during runtime of the remote capture agent.

Claims

1. A computer-implemented method for facilitating the processing of network data, comprising:

obtaining, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network;

using the configuration information to configure the generation of event data from network packets at the remote capture agent;

upon receiving an update to the configuration information from the configuration server, using the update to reconfigure the generation of the event data by the remote capture agent during runtime of the remote capture agent;

during runtime of the remote capture agent, generating the event data from network packets received at the remote capture agent;

storing the event data in a data store; and

while subsequently processing a query,

employing a retrieval schema, which includes an extraction rule that indicates how to extract one or more values from an event, to extract values from events in the event data obtained from the data store, and

identifying responsive events based on the extracted values.

2. The computer-implemented method of claim 1, further comprising:

providing an event stream comprising the event data to one or more transformation servers for further transformation of the event data by the one or more transformation servers.

3. The computer-implemented method of claim 1, further comprising:

using the configuration information to configure transformation of the event data into transformed event data at the remote capture agent; and

providing the transformed event data to one or more transformation servers.

4. The computer-implemented method of claim 1,

wherein the configuration information comprises one or more transformations of the event data, and

wherein the one or more transformations comprise at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

5. The computer-implemented method of claim 1, wherein the configuration information and the update are obtained by the remote capture agent using at least one of a push mechanism and a pull mechanism.

6. The computer-implemented method of claim 1, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

7. The computer-implemented method of claim 1, wherein the event data comprises at least one of a transaction type, a timestamp, and an error.

8. The computer-implemented method of claim 1,

wherein the configuration information comprises an additional parameter, and

wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, an inclusion of a matching transaction or matching error in the event data, and a type of event used by the event stream.

9. The computer-implemented method of claim 1, wherein the remote capture agent is installed in a virtual computing environment.

23. The computer-implemented method of claim 1, wherein the retrieval schema is a late-binding retrieval schema that is applied during query execution.

10. A computer-implemented method for facilitating network data capture, comprising:

obtaining, at a configuration server, configuration information for a set of remote capture agents on a set of networks; and

using the configuration server to provide the configuration information over the set of networks to the remote capture agents, wherein the configuration information is used by the remote capture agents to configure the generation of event data from network packets captured by the remote capture agents

during runtime of the remote capture agent, generating the event data from network packets received at the remote capture agent;

storing the event data in a data store; and

while subsequently processing a query,

employing a retrieval schema, which includes an extraction rule that indicates how to extract one or more values from an event, to extract values from events in the event data obtained from the data store, and

identifying responsive events based on the extracted values.

11. The computer-implemented method of claim 10, further comprising:

obtaining an update to the configuration information at the configuration server; and

using the configuration server to provide the update to the remote capture agents, wherein the update is used by the remote capture agents to reconfigure the generation of the event data during runtime of the remote capture agents.

12. The computer-implemented method of claim 10, wherein the configuration information is provided to the remote capture agents using at least one of a push mechanism and a pull mechanism.

13. The computer-implemented method of claim 10, wherein the configuration information is obtained from an application used to access the event data after the event data is generated.

24. The computer-implemented method of claim 10, wherein the retrieval schema is a late-binding retrieval schema that is applied during query execution.

14. A system for facilitating network data capture, comprising:

a remote-processing node comprising a computer system with a network connection;

a remote capture agent that executes on the remote-processing node, comprising;

a capture component configured to capture network packets from a network;

an events generator configured to generate event data from the network packets;

a configuration component configured to;

obtain configuration information from a configuration server over a network, and

use the configuration information to configure the generation of event data at the events generator, and

upon receiving an update to the configuration information, using the update to reconfigure the generation of the event data by the events generator during runtime of the remote capture agent; and

a data-store node comprising a computer system with a network connection;

a data store that executes on the data-store node and is configured to store the event data generated by the events generator; and

an indexer that processes a query by,

employing a retrieval schema, which includes an extraction rule that indicates how to extract one or more values from an event, to extract values from events in the event data obtained from the data store, and

identifying responsive events based on the extracted values.

15. The system of claim 14, further comprising:

the configuration server configured to provide the configuration information and the update to the remote capture agent.

16. The system of claim 14, wherein the configuration information and the update are obtained by the configuration component using at least one of a push mechanism and a pull mechanism.

17. The system of claim 14, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

18. The system of claim 14,

wherein the configuration information comprises an additional parameter, and

wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, an inclusion of a matching transaction or matching error in the event data, and a type of event used by the event stream.

19. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for processing network data, the method comprising:

obtaining, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network;

using the configuration information to configure the generation of event data from network packets at the remote capture agent; and

upon receiving an update to the configuration information from the configuration server, using the update to reconfigure the generation of the event data by the remote capture agent during runtime of the remote capture agent;

during runtime of the remote capture agent, generating the event data from network packets received at the remote capture agent;

storing the event data in a data store; and

while subsequently processing a query,

employing a late-binding schema, which includes an extraction rule that indicates how to extract one or more values from an event, to extract values from events in the event data obtained from the data store, and

identifying responsive events based on the extracted values.

20. The non-transitory computer-readable storage medium of claim 19, the method further comprising:

providing an event stream comprising the event data to one or more transformation servers for further transformation of the event data by the one or more transformation servers.

21. The non-transitory computer-readable storage medium of claim 19, wherein the configuration information and the update are obtained by the remote capture agent using at least one of a push mechanism and a pull mechanism.

22. The non-transitory computer-readable storage medium of claim 19, wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

26. The non-transitory computer-readable storage medium of claim 19, wherein the retrieval schema is a late-binding retrieval schema that is applied during query execution.

25. The system of claim 140, wherein the retrieval schema is a late-binding retrieval schema that is applied during query execution.