Storing network bidirectional flow data and metadata with efficient processing technique

• US 9,426,071 B1
• Filed: 08/19/2014
• Issued: 08/23/2016
• Est. Priority Date: 08/22/2013
• Status: Active Grant

• Current Assignee: Musarubra US LLC (Musarubra US SellCo LLC)
• Original Assignee: FireEye, Inc. (Alphabet Inc.)
• Inventors: Caldejon, Randy I., Edwards, Dennis Lee, Fauerbach, Christopher Hayes
• Primary Examiner(s): Wu, Jianye
• Application Number:
  US14/463,226
• Time in Patent Office: 735 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 45/7453 using hashing
• CPC Class Codes: H04L 61/457 containing identifiers of data entities on a computer, e.g. file names

Abstract

A processing technique provides an improved indexing arrangement that enables storage, filtering and querying of metadata used to retrieve packets captured from a network and persistently stored in a data repository. A packet capture engine records the packets in packet capture (PCAP) formats from a network link at a substantially high packet transfer rate to persistent storage of the data repository in a sustained manner. Efficient filtering and querying of the metadata to retrieve the stored packets may be achieved, in part, by organizing the metadata as one or more metadata repositories. The processing technique uses the Berkeley Packet Filter (BPF) language as an interface of a BPF engine to search or index the stored packets in response to queries. The BPF engine processes BPF expressions used as precursors to the indexing arrangement to enable access to the repositories when searching and locating stored packets matching the expressions.

Claims

1. A method comprising:

intercepting a first packet from a network link at a node coupled to the network link, the node including persistent storage devices organized as a plurality of volumes having a hierarchical file system;

computing a first hash value based on a network flow of the first packet;

recording the first packet in a packet capture (PCAP) format as a first PCAP record including the first hash value and first flow metadata appended to the first PCAP record;

copying the first PCAP record to a first metadata repository stored as a first file on a first volume of the hierarchical file system of the node, wherein the first metadata repository stores a plurality of second PCAP records having second hash values and second flow metadata; and

concurrently searching and retrieving one or more of the second PCAP records of the first metadata repository while copying the first packet to a data repository stored as a second file of a second volume of the hierarchical file system of the node to realize a substantially high sustained packet transfer rate of the network link.

2. The method of claim 1 further comprising:

obtaining a first offset of the first packet in the data repository and a first path to the data repository, wherein the data repository stores a plurality of second packets;

copying the first hash value along with the first path and first offset as a first entry into a second metadata repository, wherein the second metadata repository stores a plurality of second entries having second hash values along with second paths and second offsets; and

providing an indexing arrangement having multiple levels of indirection that enables concurrent searching, retrieval and copying of the first and second PCAP records, first and second entries, and first and second packets to accommodate the substantially high sustained packet transfer rate of the network link.

3. The method of claim 1 wherein concurrently searching and retrieving comprises:

issuing an expression as a query to the node;

searching the second flow metadata of the second PCAP records stored in the first metadata repository to match the expression;

in response to a match, using a second hash value retrieved from a matching second PCAP record to scan a second metadata repository to find a second entry matching the second hash value; and

in response to finding a matching second entry, retrieving a second packet from the data repository.

4. The method of claim 1 wherein the data repository is embodied as secondary storage and wherein recording the first packet comprises:

capturing the first packet and subsequent packets at the substantially high sustained packet transfer rate of the network link for a duration that yields an amount of captured packets that exceeds a memory capacity of the node such that at least a portion of the captured packets is written through to secondary storage without dropping the captured packets and while maintaining capture of the packets at the substantially high sustained packet transfer rate.

5. The method of claim 3 wherein the expression is an extended Berkeley Packet Filter (BPF) expression and wherein searching the second flow metadata comprises extending BPF syntax to include abbreviations for the second flow metadata of the second PCAP records.

6. The method of claim 5 wherein extending the BPF syntax comprises performing a mapping to replace keywords in the extended BPF expression with a standard BPF compatible statement.

7. The method of claim 1 wherein the substantially high sustained packet transfer rate is equal to or greater than 10 Gb/sec.

8. A system comprising:

one or more processors of a node coupled to a network link, the node including persistent storage devices organized as a plurality of volumes having a hierarchical file system;

a plurality of storage repositories organized as an indexing arrangement and coupled to the one or more processors, the storage repositories including (i) a data repository configured to store packets intercepted from the network link, (ii) a first metadata repository configured to store flow records having hash values and flow metadata, and (iii) a second metadata repository configured to store entries having the hash values along with paths and offsets to the packets stored in the data repository; and

a memory coupled to the one or more processors and configured to store one or more processes of an operating system, the one or more processes executable by the one or more processors to utilize the indexing arrangement to concurrently search and retrieve one or more of the flow records from the first metadata repository stored as a first file of a first volume of the hierarchical file system while copying one or more of the packets to the data repository for storage on a second file of a second volume of the hierarchical file system to realize a substantially high sustained packet transfer rate of the network link.

9. The system of claim 8 wherein the one or more processes comprises a Berkeley Packet Filter (BPF) engine that processes a BPF expression used as a precursor to the indexing arrangement to enable access to the repositories when searching and locating the stored packets matching the expression.

10. The system of claim 9 wherein the BPF engine performs filtering operations to match the BPF expression against headers of the flow records using the flow metadata of the flow records.

11. The system of claim 10 wherein, in response to a match, the BPF engine cooperates with a first process of the one or more processes to retrieve a hash value from a matching flow record and scan the second metadata repository to find one or more entries that match the hash value.

12. The system of claim 11 wherein, for entries that match the hash value, the first process uses the paths and offsets that reference locations in the data repository to retrieve and pass matching packets to the BPF engine.

13. The system of claim 12 wherein the BPF engine collects the matching packets in a time stamp order and process the matching packets to ensure matching of the BPF expression.

14. The system of claim 13 wherein the BPF engine processes the matching packets to perform a full verification on the headers.

15. The system of claim 8 wherein the one or more processes comprises a decoder of a packet capture engine the decodes and hashes a network flow of each packet intercepted from the network link.

16. The system of claim 15 wherein the network flow comprises source and destination Internet Protocol (IP) addresses, source and destination port numbers and protocol type.

17. The system of claim 15 wherein the decoder hashes the network flow using a hash function to calculate a hash value for each packet.

18. The system of claim 17 wherein the hash function is a CRC-32C function.

19. The system of claim 8 wherein the substantially high sustained packet transfer rate is approximately 10 Gb/sec full duplex.

20. A non-transitory computer readable medium including program instructions for execution on one or more processors, the program instructions when executed operable to:

intercept a first packet from a network link at a node coupled to the network link, the node including persistent storage devices organized as volumes having a hierarchical file system;

compute a first hash value based on a network flow of the first packet;

record the first packet in a packet capture (PCAP) format as a first PCAP record including the first hash value and first flow metadata appended to the first PCAP record;

copy the first PCAP record to a first metadata repository stored as a first file of a first volume of the hierarchical file system of the node, wherein the first metadata repository stores a plurality of second PCAP records having second hash values and second flow metadata; and

concurrently search and retrieve one or more of the second PCAP records of the first metadata repository while copying the first packet to a data repository for storage on a second file of a second volume of the hierarchical file system of the node to realize a substantially high sustained packet transfer rate of 10 Gb/sec full duplex of the network link.