Facilitating custom content extraction from network packets

• US 10,334,085 B2
• Filed: 01/29/2015
• Issued: 06/25/2019
• Est. Priority Date: 01/29/2015
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Hsiao, Fang I, Ching, Clayton S., Dickey, Michael R., Shcherbakov, Vladimir A., Sharp, Clint
• Primary Examiner(s): Donabed, Ninos
• Application Number:
  US14/609,292
• Publication Number:
  US 20160226944A1
• Time in Patent Office: 1,608 Days
• Field of Search: 709224, 707758
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

The disclosed embodiments provide a system for extracting custom content from network packets. During operation, the system receives a stream of packets. The system then parses packets in the stream to determine a protocol for each packet. Next, the system applies a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content. Then, the system stores the extracted content in events in a data store to facilitate subsequent queries involving the extracted content.

Claims

1. A method performed by a remote capture agent coupled to a computer network, the method comprising:

monitoring a stream of network packets;

for each network packet of a plurality of network packets in the stream of network packets;

parsing the network packet to identify a structure of the network packet, the structure of the network packet used to determine a protocol associated with the network packet;

applying an extraction rule associated with the protocol to the network packet to obtain extracted content, wherein applying the extraction rule includes;

identifying at least one user-specified field in the network packet containing structured data from which the extracted content is to be obtained, and

extracting data from the structured data contained in the user-specified field of the network packet;

generating a timestamped event including a field storing the extracted content; and

sending the timestamped event including the extracted content to another component on the computer network for storage in a data store, the data store facilitating the querying of timestamped event data stored in the data store using late-binding schemas generated from received queries.

2. The method of claim 1, wherein the method further comprises:

receiving a query to be applied to timestamped events stored in the data store;

retrieving timestamped events from the data store satisfying the query;

using a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and

processing the query using the retrieved data values.

3. The method of claim 1, wherein a rule for determining the protocol is different than the extraction rule that is used to obtain the extracted content from the network packet.

4. The method of claim 1, wherein applying the extraction rule to the network packet includes applying a field-specific regular expression to the at least one user-specified field in the network packet.

5. The method of claim 1, wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for an XML data format to the at least one user-specified field in the network packet.

6. The method of claim 1, wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for a JSON data format to the at least one user-specified field in the network packet.

7. The method of claim 1, wherein parsing the network packet to determine a protocol associated with the network packet includes using a deep-packet inspection engine to determine the protocol.

8. The method of claim 1, wherein the extraction rule is specified by a user using a graphical user interface (GUI).

9. The method of claim 1, wherein the extraction rule is specified by a user using a graphical user interface (GUI);

and

wherein the GUI includes fields used to receive input from a user, wherein the input includes one or more of;

a source field identifier that identifies a source field in network packets of the stream of network packets from which to obtain the extracted content,

an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,

an extraction rule definition, and

an identifier that is used to identify the extracted content.

10. The method of claim 1, wherein the remote capture agent receives the extraction rule from a configuration server;

and

wherein sending the timestamped event includes streaming the timestamped event from the remote capture agent to the data store.

11. A non-transitory computer-readable storage medium storing instructions that when executed by a processor cause performance of operations comprising:

monitoring a stream of network packets;

for each network packet of a plurality of network packets in the stream of network packets;

parsing the network packet to identify a structure of the network packet, the structure of the network packed used to determine a protocol associated with the network packet;

applying an extraction rule associated with the protocol to the network packet to obtain extracted content, wherein applying the extraction rule includes;

identifying at least one user-specified field in the network packet containing structured data from which the extracted content is to be obtained,

extracting data from the structured data contained in the user-specified field of the network packet;

generating a timestamped event including a field storing the extracted content; and

sending the timestamped event including the extracted content to another component on a computer network for storage in a data store, the data store facilitating the querying of timestamped event data stored in the data store using late-binding schemas generated from received queries.

12. The non-transitory computer-readable storage medium of claim 11, further comprising instructions which, when executed by the processor, cause performance of operations comprising:

receiving a query to be applied to timestamped events stored in the data store;

retrieving timestamped events from the data store satisfying the query;

using a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and

processing the query using the retrieved data values.

13. The non-transitory computer-readable storage medium of claim 11, wherein a rule for determining the protocol is different than the extraction rule that is used to obtain the extracted content from the network packet that matches the determined protocol.

14. The non-transitory computer-readable storage medium of claim 11, wherein applying the extraction rule to the network packet includes applying a field-specific regular expression to the at least one user-specified field in the network packet.

15. The non-transitory computer-readable storage medium of claim 11, wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for an XML data format to the at least one user-specified field in the network packet.

16. The non-transitory computer-readable storage medium of claim 11, wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for a JSON data format to the at least one user-specified field in the network packet.

17. The non-transitory computer-readable storage medium of claim 11, wherein parsing the network packet to determine a protocol associated with the network packet includes using a deep-packet inspection engine to determine the protocol.

18. The non-transitory computer-readable storage medium of claim 11, wherein the extraction rule is specified by a user using a graphical user interface (GUI).

19. The non-transitory computer-readable storage medium of claim 11, wherein the extraction rule is specified by a user using a graphical user interface (GUI);

and

wherein the GUI includes fields used to receive input from a user, wherein the input includes one or more of;

a source field identifier that identifies a source field in network packets of the stream of network packets from which to obtain the extracted content,

an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,

an extraction rule definition, and

an identifier that is used to identify the extracted content.

20. The non-transitory computer-readable storage medium of claim 11,

wherein a remote capture agent receives the extraction rule from a configuration server;

and

wherein sending the extracted content includes streaming the timestamped events from the remote capture agent to the data store.

21. An apparatus, comprising:

one or more processors; and

a non-transitory computer-readable memory storing instructions that, when executed by the one or more processors, cause the apparatus to;

monitor a stream of network packets;

for network packet each of a plurality of network packets in the stream of network packets;

parse the network packet to identify a structure of the network packet, the structure of the network packet used to determine a protocol associated with the network packet;

apply an extraction rule associated with the protocol to the network packet to obtain extracted content, wherein applying the extraction rule includes;

identifying at least one user-specified field in the network packet containing structured data from which the extracted content is to be obtained, and

extracting data from the structured data contained in the user-specified field of the network packet;

generate a timestamped event including a field storing the extracted content; and

send the timestamped event including the extracted content to another component on a computer network for storage in a data store, the data store facilitating the querying of timestamped event data stored in the data store using late-binding schemas generated from received queries.

22. The apparatus of claim 21, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:

receive a query to be applied to timestamped events stored in the data store;

retrieve timestamped events from the data store satisfying the query;

use a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and

process the query using the retrieved data values.

23. The apparatus of claim 21, wherein a rule for determining the protocol is different than the extraction rule that is used to obtain the extracted content from the network packet that matches the determined protocol.

24. The apparatus of claim 21, wherein applying the extraction rule to the network packet includes applying a field-specific regular expression to at least one user-specified field in the network packet.

25. The apparatus of claim 21, wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for an XML data format to at least one user-specified field in the network packet.

26. The apparatus of claim 21, wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for a JSON data format to the at least one user-specified field in the network packet.

27. The apparatus of claim 21, wherein parsing the network packet to determine a protocol associated with the network packet includes using a deep-packet inspection engine to determine the protocol.

28. The apparatus of claim 21, wherein extraction rule is specified by a user using a graphical user interface (GUI).

29. The apparatus of claim 21, wherein the extraction rule is specified by a user using a graphical user interface (GUI);

and

wherein the GUI includes fields used to receive input from a user, wherein the input includes one or more of;

a source field identifier that identifies a source field in network packets of the stream of network packets from which to obtain the extracted content,

an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,

an extraction rule definition, and

an identifier that is used to identify the extracted content.

30. The apparatus of claim 21,

wherein a remote capture agent receives the extraction rule from a configuration server;

and

wherein sending the extracted content includes streaming the timestamped events from the remote capture agent to the data store.