Arrangement for efficient search and retrieval of indexes used to locate captured packets

• US 9,876,701 B1
• Filed: 07/20/2016
• Issued: 01/23/2018
• Est. Priority Date: 08/22/2013
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Caldejon, Randy I., Edwards, Dennis Lee, Fauerbach, Christopher Hayes
• Primary Examiner(s): Wu, Jianye
• Application Number:
  US15/215,277
• Time in Patent Office: 552 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 45/7453 using hashing
• CPC Class Codes: H04L 61/457 containing identifiers of data entities on a computer, e.g. file names

Abstract

An indexing arrangement enables efficient search and retrieval of indexes persistently stored in a metadata repository and used to locate packets captured from a network and persistently stored in a data repository. The packets are captured at a packet capture and retrieval system having persistent storage devices organized as files of the metadata and data repositories. Search and retrieval of the indexes within the files of the metadata repository occur at substantially a same time as one or more other captured packets is written to one or more files of the data repository to realize a substantially high sustained packet transfer rate of the network.

Claims

1. A method comprising:

capturing one or more packets from a network at a packet capture and retrieval system having persistent storage devices configured to store files;

writing each captured packet to a data repository stored as one or more first files on the persistent storage devices;

generating an index based on a network flow of each captured packet, the index configured to locate the captured packet written to the data repository;

copying the index to a metadata repository stored as one or more second files on the persistent storage devices; and

searching and retrieving the index within a second file of the metadata repository to enable location of the captured packet, wherein the searching and retrieving the index within the second file occur at substantially a same time as writing one or more other captured packets to the first file to realize a substantially high sustained packet transfer rate of the network.

2. The method of claim 1 wherein the index includes a path to a first file of the data repository and an offset within the first file.

3. The method of claim 1 wherein each first file has an associated second file.

4. The method of claim 3 further comprising:

identifying each first file by a path formed from a stream and a time stamp relating to a time of capture of a packet.

5. The method of claim 4 wherein the time of capture is in nanoseconds.

6. The method of claim 4 further comprising providing a naming convention for the first file based on the stream and the time stamp.

7. The method of claim 4 wherein the stream and the time stamp cooperate to provide a full path name to the first file containing the packet.

8. The method of claim 1 wherein writing each captured packet further comprises providing one or more streams to write the captured packet to the first file.

9. The method of claim 8 further comprising representing each stream by a directory of a file system of the packet capture and retrieval system.

10. The method of claim 9 further comprising mapping each stream to a central processing unit of the packet capture and retrieval system to enable concurrent processing of the data repository.

11. The method of claim 1 wherein the data repository and metadata repository are separately accessed.

12. The method of claim 1 wherein searching and retrieving comprises:

issuing an expression as a query to the packet capture and retrieval system to retrieve a first captured packet written to the data repository;

translating the expression to a first index based on a hash value of a first network flow of the first captured packet;

scanning the metadata repository to find an entry matching the hash value; and

in response to finding a matching entry, retrieving the first captured packet from the data repository using the first index.

13. A system comprising:

one or more processors of a packet capture and retrieval system having persistent storage devices configured to store files;

a plurality of storage repositories organized as an indexing arrangement and coupled to the one or more processors, the storage repositories including a data repository having one or more first files on the persistent storage devices configured to store packets captured from a network and a metadata repository having one or more second files on the persistent storage devices configured to store indexes used to locate the captured packets stored in the data repository; and

a memory coupled to the one or more processors and configured to store one or more processes of an operating system, the one or more processes executable by the one or more processors to utilize the indexing arrangement to search and retrieve the indexes from the metadata repository while copying one or more other packets to the data repository to realize a substantially high sustained packet transfer rate of the network.

14. The system of claim 13 wherein an index includes a path to a first file of the data repository and an offset within the first file.

15. The system of claim 13 wherein each first file has an associated second file.

16. The system of claim 15 wherein each first file is identified by a path formed from a stream and a time stamp relating to a time of capture of a packet.

17. The system of claim 16 wherein a naming convention is provided for the first file based on the stream and the time stamp.

18. The system of claim 16 wherein the stream and the time stamp cooperate to provide a full path name to the first file containing the packet.

19. The system of claim 13 wherein one or more streams is provided to store the captured packets in the one or more first files.

20. The system of claim 19 wherein each stream is represented by a directory of a file system of the packet capture and retrieval system.

21. The system of claim 20 wherein each stream is mapped to a central processing unit of the packet capture and retrieval system to enable concurrent processing of the data repository.

22. The system of claim 13 wherein the data repository and metadata repository are separately accessed.

23. A non-transitory computer readable medium including program instructions for execution on one or more processors, the program instructions configured to:

capture one or more packets from a network at a packet capture and retrieval system having persistent storage devices configured to store files;

write each captured packet to a data repository stored as one or more first files on the persistent storage devices;

generate an index based on a network flow of each captured packet, the index configured to locate the captured packet written to the data repository;

copy the index to a metadata repository stored as one or more second files on the persistent storage devices; and

search and retrieve the index within a second file of the metadata repository to enable location of the captured packet, wherein search and retrieve of the index within the second file occur at substantially a same time as one or more other captured packets is written to the first file to realize a substantially high sustained packet transfer rate of the network.