Configuring generation of multiple event streams from a packet flow

• US 10,264,106 B2
• Filed: 10/31/2017
• Issued: 04/16/2019
• Est. Priority Date: 10/30/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Shcherbakov, Vladimir A., Dickey, Michael
• Primary Examiner(s): Zhao, Wei
• Application Number:
  US15/799,158
• Publication Number:
  US 20180048741A1
• Time in Patent Office: 532 Days
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

The disclosed embodiments provide a system that processes network data. During operation, the system obtains, at a remote capture agent, a first protocol classification for a first packet flow captured by the remote capture agent. Next, the system uses configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent, wherein the first event stream comprises time-series event data generated from network packets in the first packet flow based on the first protocol classification. The system then transmits the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network.

Claims

1. A computer-implemented method performed by a remote capture agent coupled to a network, the method comprising:

monitoring network traffic comprising a plurality of network packets on the network;

identifying a packet flow in the plurality of network packets, wherein the packet flow is associated with a communication path between a source and a destination;

identifying a first protocol and a second protocol associated with the packet flow;

generating, based on configuration information associated with the first protocol and the second protocol, a first event stream from the packet flow, wherein the first event stream comprises time-series event data created based on first data derived from network packets of the packet flow, and a second event stream from the packet flow, wherein the second event stream comprises time-series event data created based on second data derived from network packets of the packet flow; and

sending the first event stream and the second event stream to another component on the network.

2. The computer-implemented method of claim 1, further comprising:

obtaining, at the remote capture agent, the configuration information from a configuration server over the network; and

using the configuration information to configure the generation of the first event stream and the second event stream during runtime of the remote capture agent.

3. The computer-implemented method of claim 1, further comprising:

wherein the packet flow is a first packet flow and the plurality of network packets is a first plurality of network packets;

identifying a second packet flow including a second plurality of network packets;

identifying a third protocol associated with the second packet flow;

generating, based on configuration information associated with the third protocol, a third event stream from the second packet flow at the remote capture agent, wherein the third event stream comprises time-series event data created based on data derived from network packets of the second packet flow; and

transmitting the third event stream to another component on the network.

4. The computer-implemented method of claim 1, further comprising identifying the plurality of network packets of the packet flow based on control information in the network packets.

5. The computer-implemented method of claim 1, further comprising:

assembling the packet flow from the plurality of network packets; and

in response to detecting encryption of the network packets of the packet flow, decrypting the network packets in the packet flow prior to identifying the first protocol and the second protocol associated with the packet flow.

6. The computer-implemented method of claim 1, wherein the network packets of the packet flow are associated with at least one of:

the source, the destination, a network address, a port, and a transport layer protocol.

7. The computer-implemented method of claim 1, wherein generating the first event stream from the packet flow further comprises:

identifying one or more event attributes associated with the first protocol from the configuration information;

extracting the one or more event attributes from the plurality of network packets in the packet flow; and

including the extracted one or more event attributes in the first event stream.

8. The computer-implemented method of claim 1, wherein generating the event stream from the packet flow further comprises:

identifying one or more event attributes associated with the first protocol from the configuration information;

extracting the one or more event attributes from the plurality of network packets in the packet flow;

transforming, based on the configuration information, the extracted one or more event attributes; and

including the transformed one or more event attributes in the first event stream.

9. The computer-implemented method of claim 1, wherein the first protocol comprises at least one of:

a first transport layer protocol, a first session layer protocol, a first presentation layer protocol, and a first application layer protocol, and wherein the second protocol comprises at least one of;

a second transport layer protocol, a second session layer protocol, a second presentation lay protocol, and a second application layer protocol.

10. The computer-implemented method of claim 1, wherein the first event stream is sent to a first component on the network, and wherein the second event stream is sent to a second component on the network that is different from the first component.

11. A remote capture agent, comprising:

a processor;

a non-transitory computer readable storage medium storing instructions which, when executed by the processor, cause the remote capture agent to;

monitor network traffic comprising a plurality of network packets on the network;

identify a packet flow in the plurality of network packets, wherein the packet flow is associated with a communication path between a source and a destination;

identify a first protocol and a second protocol associated with the packet flow;

generate, based on configuration information associated with the first protocol and the second protocol, a first event stream from the packet flow, wherein the first event stream comprises time-series event data created based on first data derived from network packets of the packet flow, and a second event stream from the packet flow, wherein the second event stream comprises time-series event data created based on second data derived from network packets of the packet flow; and

send the first event stream and the second event stream to another component on the network.

12. The remote capture agent of claim 11, wherein the instructions, when executed by the processor, further cause the remote capture agent to:

obtain the configuration information from a configuration server over a network; and

use the configuration information to configure the generation of the first event stream and the second event stream during runtime of the remote capture agent.

13. The remote capture agent of claim 11, wherein the packet flow is a first packet flow and the plurality of network packets is a first plurality of network packets, and wherein the instructions, when executed, further cause the remote capture agent to:

identify a second packet flow including a second plurality of network packets;

identify a third protocol associated with the second packet flow;

generate, based on configuration information associated with the third protocol, a third event stream from the second packet flow at the remote capture agent, wherein the third event stream comprises time-series event data created based on data derived from network packets of the second packet flow; and

transmit the third event stream to another component on the network.

14. The remote capture agent of claim 11, wherein the instructions, when executed by the processor, further cause the remote capture agent to identify the plurality of network packets of the packet flow based on control information in the network packets.

15. The remote capture agent of claim 11, wherein the instructions, when executed by the processor, further cause the remote capture agent to:

assemble the packet flow from the plurality of network packets; and

in response to detecting encryption of the network packets of the packet flow, decrypt the network packets in the packet flow prior to identifying the first protocol and the second protocol associated with the packet flow.

16. The remote capture agent of claim 11, wherein the network packets of the packet flow are associated with at least one of:

the source, the destination, a network address, a port, and a transport layer protocol.

17. The remote capture agent of claim 11, wherein the instructions, when executed by the processor, further cause the remote capture agent to:

identify one or more event attributes associated with the first protocol from the configuration information;

extract the one or more event attributes from the plurality of network packets in the packet flow; and

include the extracted one or more event attributes in the first event stream.

18. The remote capture agent of claim 11, wherein the instructions, when executed by the processor, further cause the remote capture agent to:

identify one or more event attributes associated with the first protocol from the configuration information;

extract the one or more event attributes from the plurality of network packets in the packet flow;

transform, based on the configuration information, the extracted one or more event attributes; and

include the transformed one or more event attributes in the first event stream.

19. The remote capture agent of claim 11, wherein the first protocol comprises at least one of:

a first transport layer protocol, a first session layer protocol, a first presentation layer protocol, and a first application layer protocol, and wherein the second protocol comprises at least one of;

a second transport layer protocol, a second session layer protocol, a second presentation lay protocol, and a second application layer protocol.

20. The remote capture agent of claim 11, wherein the first event stream is sent to a first component on the network, and wherein the second event stream is sent to a second component on the network that is different from the first component.

21. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform operations comprising:

monitoring network traffic comprising a plurality of network packets on the network;

identifying a packet flow in the plurality of network packets, wherein the packet flow is associated with a communication path between a source and a destination;

identifying a first protocol and a second protocol associated with the packet flow;

generating, based on configuration information associated with the first protocol and the second protocol, a first event stream from the packet flow, wherein the first event stream comprises time-series event data created based on first data derived from network packets of the packet flow, and a second event stream from the packet flow, wherein the second event stream comprises time-series event data created based on second data derived from network packets of the packet flow; and

sending the first event stream and the second event stream to another component on the network.

22. The non-transitory computer-readable storage medium of claim 21, wherein the instructions, when executed by the computer, further cause operations comprising:

obtaining, at the remote capture agent, the configuration information from a configuration server over the network; and

using the configuration information to configure the generation of the first event stream and the second event stream during runtime of the remote capture agent.

23. The non-transitory computer-readable storage medium of claim 21, wherein the packet flow is a first packet flow and the plurality of network packets is a first plurality of network packets, and wherein the instructions, when executed by the computer, further cause operations comprising:

identifying a second packet flow including a second plurality of network packets;

identifying a third protocol associated with the second packet flow;

generating, based on configuration information associated with the third protocol, a third event stream from the second packet flow at the remote capture agent, wherein the third event stream comprises time-series event data created based on data derived from network packets of the second packet flow; and

transmitting the third event stream to another component on the network.

24. The non-transitory computer-readable storage medium of claim 21, wherein the instructions, when executed by the computer, further cause operations comprising identifying the plurality of network packets of the packet flow based on control information in the network packets.

25. The non-transitory computer-readable storage medium of claim 21, wherein the instructions, when executed by the computer, further cause operations comprising:

assembling the packet flow from the plurality of network packets; and

in response to detecting encryption of the network packets of the packet flow, decrypting the network packets in the packet flow prior to obtaining the protocol for the packet flow.

26. The non-transitory computer-readable storage medium of claim 21, wherein the network packets of the packet flow are associated with at least one of:

the source, the destination, a network address, a port, and a transport layer protocol.

27. The non-transitory computer-readable storage medium of claim 21, wherein generating the first event stream from the packet flow further comprises:

identifying one or more event attributes associated with the first protocol from the configuration information;

extracting the one or more event attributes from the plurality of network packets in the packet flow; and

including the extracted one or more event attributes in the first event stream.

28. The non-transitory computer-readable storage medium of claim 21, wherein generating the event stream from the packet flow further comprises:

identifying one or more event attributes associated with the first protocol from the configuration information;

extracting the one or more event attributes from the plurality of network packets in the packet flow;

transforming, based on the configuration information, the extracted one or more event attributes; and

including the transformed one or more event attributes in the first event stream.

29. The non-transitory computer-readable storage medium of claim 21, wherein the first protocol comprises at least one of:

a first transport layer protocol, a first session layer protocol, a first presentation layer protocol, and a first application layer protocol, and wherein the second protocol comprises at least one of;

a second transport layer protocol, a second session layer protocol, a second presentation lay protocol, and a second application layer protocol.

30. The non-transitory computer-readable storage medium of claim 21, wherein the first event stream is sent to a first component on the network, and wherein the second event stream is sent to a second component on the network that is different from the first component.