FACILITATING CUSTOM CONTENT EXTRACTION FROM NETWORK PACKETS

• US 20160226944A1
• Filed: 01/29/2015
• Published: 08/04/2016
• Est. Priority Date: 01/29/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Hsiao, Fang I., Ching, Clayton S., Dickey, Michael R., Shcherbakov, Vladimir A., Sharp, Clint
• Granted Patent: US 10,334,085 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

The disclosed embodiments provide a system for extracting custom content from network packets. During operation, the system receives a stream of packets. The system then parses packets in the stream to determine a protocol for each packet. Next, the system applies a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content. Then, the system stores the extracted content in events in a data store to facilitate subsequent queries involving the extracted content.

Claims

11-1. The non-transitory computer-readable storage medium of claim 12,

wherein the method further comprises obtaining the custom-content-extraction rule from a user through a user interface;

and

wherein the user interface comprises a dialog box including fields for receiving input from a user, wherein the input includes one or more of;

a source field identifier that identifies a source field in each packet from which to obtain the extracted content,

an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,

an extraction rule entered by the user, and

an identifier that is used to identify the extracted content.

12. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for extracting custom content from packets, the method comprising:

receiving a stream of packets;

parsing packets in the received stream to determine a protocol for each packet;

applying a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content; and

storing the extracted content in events in a data store to facilitate subsequent queries involving the extracted content.

1. A method for extracting custom content from packets, comprising:

receiving a stream of packets;

parsing packets in the received stream to determine a protocol for each packet;

applying a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content; and

storing the extracted content in events in a data store to facilitate subsequent queries involving the extracted content.

2. The method of claim 1, wherein the method further comprises:

receiving a query to be applied to the events in the data store;

retrieving events from the data store;

using a late-binding schema generated from the query to retrieve data values from the events; and

processing the query using the retrieved data values.

3. The method of claim 1, wherein a rule for determining the target protocol is different than the custom-content-extraction rule that is used to obtain the extracted content from each packet that matches the target protocol.

4. The method of claim 1, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific regular expression to one or more target fields in the packet.

5. The method of claim 1, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for a structured data format to one or more target fields in the packet.

6. The method of claim 1, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for an XML data format to one or more target fields in the packet.

7. The method of claim 1, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for a JSON data format to one or more target fields in the packet.

8. The method of claim 1, wherein parsing the packets in the stream to determine a protocol for each packet includes using a deep-packet inspection engine to determine the protocol for each packet.

9. The method of claim 1, further comprising obtaining the custom-content-extraction rule from a user through a user interface.

10. The method of claim 1,

wherein the method further comprises obtaining the custom-content-extraction rule from a user through a user interface;

and

wherein the user interface comprises a dialog box including fields for receiving input from a user, wherein the input includes one or more of;

a source field identifier that identifies a source field in each packet from which to obtain the extracted content,

an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,

an extraction rule entered by the user, and

an identifier that is used to identify the extracted content.

11. The method of claim 1,

wherein the method is performed by a remote capture agent located on a network interface on a network;

wherein the custom-content-extraction rule is propagated from a configuration server the remote capture agent; and

wherein storing the extracted content in the data store includes streaming the events from the remote capture agent to the data store.

13. The non-transitory computer-readable storage medium of claim 12, wherein the method further comprises:

receiving a query to be applied to the events in the data store;

retrieving events from the data store;

using a late-binding schema generated from the query to retrieve data values from the events; and

processing the query using the retrieved data values.

14. The non-transitory computer-readable storage medium of claim 12, wherein a rule for determining the target protocol is different than the custom-content-extraction rule that is used to obtain the extracted content from each packet that matches the target protocol.

15. The non-transitory computer-readable storage medium of claim 12, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific regular expression to one or more target fields in the packet.

16. The non-transitory computer-readable storage medium of claim 12, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for a structured data format to one or more target fields in the packet.

17. The non-transitory computer-readable storage medium of claim 12, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for an XML data format to one or more target fields in the packet.

18. The non-transitory computer-readable storage medium of claim 12, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for a JSON data format to one or more target fields in the packet.

19. The non-transitory computer-readable storage medium of claim 12, wherein parsing packets in the stream to determine a protocol for each packet includes using a deep-packet inspection engine to determine the protocol for each packet.

20. The non-transitory computer-readable storage medium of claim 12, wherein the method further comprises obtaining the custom-content-extraction rule from a user through a user interface.

22. The non-transitory computer-readable storage medium of claim 12,

wherein the method is performed by a remote capture agent located on a network interface on a network;

wherein the custom-content-extraction rule is propagated from a configuration server the remote capture agent; and

wherein storing the extracted content in the data store includes streaming the events from the remote capture agent to the data store.

23. An apparatus, comprising:

a computing node comprising one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the computing node to;

receive a stream of packets;

parse packets in the received stream to determine a protocol for each packet;

apply a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content; and

store the extracted content in events in a data store to facilitate subsequent queries involving the extracted content.

24. The apparatus of claim 23, further comprising:

a second computing node comprising a second set of one or more processors; and

a second memory storing instructions that, when executed by the second set of one or more processors, cause the second computing node to;

receive a query to be applied to the events in the data store;

retrieve events from the data store;

use a late-binding schema generated from the query to retrieve data values from the events; and

process the query using the retrieved data values.

25. The apparatus of claim 23, wherein a rule for determining the target protocol is different than the custom-content-extraction rule that is used to obtain the extracted content from each packet that matches the target protocol.

26. The apparatus of claim 23, wherein while applying the custom-content-extraction rule to a given packet, the computing node is configured to apply a field-specific regular expression to one or more target fields in the packet.

27. The apparatus of claim 23, wherein while applying the custom-content-extraction rule to a given packet, the computing node is configured to apply a field-specific extraction rule for a structured data format to one or more target fields in the packet.

28. The apparatus of claim 23, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for an XML data format to one or more target fields in the packet.

29. The apparatus of claim 23, wherein applying the custom-content-extraction rule to a given packet includes applying a field-specific extraction rule for a JSON data format to one or more target fields in the packet.

30. The apparatus of claim 23, wherein while parsing packets in the stream to determine a protocol for each packet, the computing node is configured to use a deep-packet inspection engine to determine the protocol for each packet.

31. The apparatus of claim 23, wherein the memory also includes instructions that cause the computing node to obtain the custom-content-extraction rule from a user through a user interface.

32. The apparatus of claim 23,

wherein the memory also includes instructions that cause the computing node to obtain the custom-content-extraction rule from a user through a user interface;

and

wherein the user interface comprises a dialog box including fields for receiving input from a user, wherein the input includes one or more of;

a source field identifier that identifies a source field in each packet from which to obtain the extracted content,

an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,

an extraction rule entered by the user, and

an identifier that is used to identify the extracted content.

33. The apparatus of claim 23,

wherein the computing node comprises a remote capture agent located on a network interface on a network;

wherein the custom-content-extraction rule is propagated from a configuration server the remote capture agent; and

wherein while storing the extracted content in the data store, the computing node is configured to stream the events from the remote capture agent to the data store.