PROTOCOL-BASED CAPTURE OF NETWORK DATA USING REMOTE CAPTURE AGENTS

• US 20160127517A1
• Filed: 10/30/2014
• Published: 05/05/2016
• Est. Priority Date: 10/30/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Shcherbakov, Vladimir A., Dickey, Michael R.
• Granted Patent: US 9,838,512 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

The disclosed embodiments provide a system that processes network data. During operation, the system obtains, at a remote capture agent, a first protocol classification for a first packet flow captured by the remote capture agent. Next, the system uses configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent, wherein the first event stream comprises time-series event data generated from network packets in the first packet flow based on the first protocol classification. The system then transmits the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network.

Claims

1. A method for processing network data, comprising:

obtaining, at a remote capture agent, a first protocol classification for a first packet flow captured by the remote capture agent;

using configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent, wherein the first event stream comprises time-series event data generated from network packets in the first packet flow based on the first protocol classification; and

transmitting the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network.

2. The method of claim 1, further comprising:

obtaining, at the remote capture agent, the configuration information from a configuration server over the network; and

using the configuration information to configure the generation of the time-series event data from the network packets during runtime of the remote capture agent.

3. The method of claim 1, further comprising:

obtaining, at the remote capture agent, a second protocol classification for a second packet flow captured at the remote capture agent;

using configuration information associated with the second protocol classification to build a second event stream from the second packet flow at the remote capture agent, wherein the second event stream comprises time-series event data from network packets in the second packet flow based on the second protocol classification; and

transmitting the second event stream over the network.

4. The method of claim 1, further comprising:

identifying, at the remote capture agent, the network packets in the first packet flow based on control information in the network packets.

5. The method of claim 1, further comprising:

assembling the first packet flow from the network packets; and

upon detecting encryption of the network packets in the first packet flow, decrypting the network packets in the first packet flow prior to obtaining the first protocol classification for the first packet flow.

6. The method of claim 1, wherein the network packets in the first packet flow are associated with at least one of:

a source;

a destination;

a network address;

a port; and

a transport layer protocol.

7. The method of claim 1, wherein using the configuration information associated with the first protocol classification to build the first event stream from the first packet flow at the remote capture agent comprises:

identifying one or more event attributes associated with the first protocol classification from the configuration information;

extracting the one or more event attributes from the network packets in the first packet flow; and

including the extracted one or more event attributes in the first event stream.

8. The method of claim 1, wherein using the configuration information associated with the first protocol classification to build the first event stream from the first packet flow at the remote capture agent comprises:

identifying one or more event attributes associated with the first protocol classification from the configuration information;

extracting the one or more event attributes from the network packets in the first packet flow;

using the configuration information to transform the extracted one or more event attributes; and

including the transformed one or more event attributes in the first event stream.

9. The method of claim 1, wherein the first protocol classification comprises at least one of:

a transport layer protocol;

a session layer protocol;

a presentation layer protocol; and

an application layer protocol.

10. A system for processing network data, comprising:

a remote capture agent, comprising;

a capture component configured to capture network packets from a network;

an events generator configured to;

obtain a first protocol classification for a first packet flow captured at the remote capture agent; and

use configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent, wherein the first event stream comprises time-series event data generated from network packets in the first packet flow based on the first protocol classification; and

a communications component configured to transmit the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network.

11. The system of claim 10, further comprising:

the configuration server configured to provide the configuration information to the remote capture agent,

wherein the remote capture agent further comprises;

a configuration component configured to use the configuration information to configure the generation of the time-series event data from the network packets during runtime of the remote capture agent.

12. The system of claim 10,

wherein the events generator is further configured to:

obtain a second protocol classification for a second packet flow captured at the remote capture agent; and

use configuration information associated with the second protocol classification to build a second event stream from the second packet flow at the remote capture agent, wherein the second event stream comprises time-series event data from network packets in the second packet flow based on the second protocol classification, and

wherein the communications component is further configured to transmit the second event stream over the network.

13. The system of claim 10, wherein the capture component is further configured to:

identify the network packets in the first packet flow based on control information in the network packets;

assemble the first packet flow from the network packets; and

upon detecting encryption of the network packets in the first packet flow, decrypt the network packets in the first packet flow prior to obtaining the first protocol classification for the first packet flow.

14. The system of claim 10, wherein using the configuration information associated with the first protocol classification to build the first event stream from the first packet flow at the remote capture agent comprises:

identifying one or more event attributes associated with the first protocol classification from the configuration information;

extracting the one or more event attributes from the network packets in the first packet flow; and

including the extracted one or more event attributes in the first event stream.

15. The system of claim 10, wherein using the configuration information associated with the first protocol classification to build the first event stream from the first packet flow at the remote capture agent comprises:

identifying one or more event attributes associated with the first protocol classification from the configuration information;

extracting the one or more event attributes from the network packets in the first packet flow;

using the configuration information to transform the extracted one or more event attributes; and

including the transformed one or more event attributes in the first event stream.

16. The system of claim 10, wherein the first protocol classification comprises at least one of:

a transport layer protocol;

a session layer protocol;

a presentation layer protocol; and

an application layer protocol.

17. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for processing network data, the method comprising:

obtaining, at a remote capture agent, a first protocol classification for a first packet flow captured by the remote capture agent;

using configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent, wherein the first event stream comprises time-series event data generated from network packets in the first packet flow based on the first protocol classification; and

transmitting the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network.

18. The non-transitory computer-readable storage medium of claim 17, the method further comprising:

obtaining, at the remote capture agent, the configuration information from a configuration server over the network; and

using the configuration information to configure the generation of the time-series event data from the network packets during runtime of the remote capture agent.

19. The non-transitory computer-readable storage medium of claim 17, the method further comprising:

obtaining, at the remote capture agent, a second protocol classification for a second packet flow captured at the remote capture agent;

using configuration information associated with the second protocol classification to build a second event stream from the second packet flow at the remote capture agent, wherein the second event stream comprises time-series event data from network packets in the second packet flow based on the second protocol classification; and

transmitting the second event stream over the network.

20. The non-transitory computer-readable storage medium of claim 17, the method further comprising:

identifying, at the remote capture agent, the network packets in the first packet flow based on control information in the network packets;

assembling the first packet flow from the network packets; and

upon detecting encryption of the network packets in the first packet flow, decrypting the network packets in the first packet flow prior to obtaining the first protocol classification for the first packet flow.

21. The non-transitory computer-readable storage medium of claim 17, wherein using the configuration information associated with the first protocol classification to build the first event stream from the first packet flow at the remote capture agent comprises:

identifying one or more event attributes associated with the first protocol classification from the configuration information;

extracting the one or more event attributes from the network packets in the first packet flow; and

including the extracted one or more event attributes in the first event stream.

22. The non-transitory computer-readable storage medium of claim 17, wherein using the configuration information associated with the first protocol classification to build the first event stream from the first packet flow at the remote capture agent comprises:

identifying one or more event attributes associated with the first protocol classification from the configuration information;

extracting the one or more event attributes from the network packets in the first packet flow;

using the configuration information to transform the extracted one or more event attributes; and

including the transformed one or more event attributes in the first event stream.

23. The non-transitory computer-readable storage medium of claim 17, wherein the first protocol classification comprises at least one of:

a transport layer protocol;

a session layer protocol;

a presentation layer protocol; and

an application layer protocol.