Hierarchically organizing network data collected from full time recording machines and efficiently filtering the same

• US 7,047,297 B2
• Filed: 07/17/2002
• Issued: 05/16/2006
• Est. Priority Date: 07/17/2001
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Major, John Darren, Covington, Stanley P., Rowley, Bevan S., Huntington, Stephen Glen
• Primary Examiner(s): Lin, Wen-Tai
• Application Number:
  US10/199,168
• Publication Number:
  US 20030135612A1
• Time in Patent Office: 1,399 Days
• Field of Search: 709/224, 370/253, 714/39, 700/1
• US Class Current: 709/224
• CPC Class Codes: H04L 41/0803 Configuration setting
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/06 Generation of reports
• CPC Class Codes: H04L 43/067 using time frame reporting
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/163 In-band adaptation of TCP data exchange; In-band control procedures
• CPC Class Codes: H04L 69/164 Adaptation or special uses of UDP protocol

Abstract

Included in the invention are systems and methods of full time recording network traffic to a hierarchical data storage. Also included in the invention are systems and methods of retrieval of recorded network traffic from a hierarchically organized network data repository. Additionally included in the invention are systems and methods of efficiently filtering data in a hierarchically organized network data repository. Systems and methods of displaying recorded network data utilizing the retrieval systems are also included in the invention. Further included in the invention are systems and methods of providing sliding time window selection user interfaces. Detailed information on various example embodiments of the inventions are provided in the Detailed Description below, and the inventions are defined by the appended claims.

Claims

1. A full time network traffic recording system, comprising

a network capture device, said network capture device being configurable to sample network packets on a network segment without regard to packet destinations;

a packet annotator, said packet annotator in accessible communication with said network capture device wherein sampled network packets may be referenced or read, said packet annotator annotating the sampled network packets with at least a time of receipt to form annotated packets,

a network data caching system, said network data caching system having a network data cache, said packet annotator providing access to said caching system of the annotated packets wherein the annotated packets may be referenced or read, said caching system forming a series of hierarchically organized finite logical storage units containing the annotated packets, said caching system further recording at least a start and an end time of the contained annotated packets to the finite logical storage units, said caching system further being configurable to initiate write commands to a storage interface;

and the storage interface operable to receive write commands from said caching system, said interface being configurable to forward write commands to a storage device wherein finite logical storage units may be mapped to media of a configured storage device;

wherein said system performs the functions of;

(i) indexing received packets using an indexable item database,

(ii) receiving a filter request having a filter expression composed of matching expressions linked by logical operators, the filter expression having at least one matching expression referencing an indexable item,

(iii) reading the indexable item database,

(iv) computing efficiency ratings for each matching expression and successively each logical operator,

and (v) filtering stored network traffic data, said filtering applying matching expressions in preferential order of efficiency.

2. The system of claim 1, further comprising:

a stream filter, said filter operating to apply at least one filter criteria to the network packets sampled by said network capture device such that only packets which meet a filter criteria are made available to said packet annotator.

3. The system of claim 1, further comprising:

at least one storage device connected to said storage interface, said storage device providing read and write access to media.

4. The system of claim 1, further comprising:

an archival system in communication with said network data caching system, said archival system operable to read storage units formed by said caching system, said archival system further operable to submit write commands to a storage device;

the write commands effective to record at least one formed storage unit;

and a removable storage device in communication with said archival system, said removable storage device capable of reading and writing data to media removable from said removable storage device, said removable storage device being further configured to receive write commands from said archival system.

5. The system of claim 4, wherein:

said archival system is operable to store the storage units formed by said caching system to multiple pieces of removable media, and said archival system is further operable to write tracking information to the multiple media pieces.

6. The system of claim 1, further comprising:

an administration interconnect, said interconnect operable to configure the behavior of said network data caching system.

7. The system of claim 6, further comprising:

an administration network connected to said administration interconnect;

and an administration console, said console providing user interfaces to control the behavior of said network data caching system through said administration interconnect and said administration network.

8. The system of claim 6, further comprising:

an administration network connected to said administration interconnect;

a network replay machine in communication with said network data caching system through said administration network and said administration interconnect;

and a hierarchical network traffic data repository readable by said network replay machine.

9. The system of claim 8, further comprising:

a packet interpreter in communication with said network replay machine wherein network traffic data may be supplied to said packet interpreter from said hierarchical network traffic data repository.

10. The system of claim 8, further comprising:

a control engine in communication with said network replay machine wherein network traffic data may be supplied to said control engine from said hierarchical network traffic data repository.

11. The system of claim 1, further comprising:

an indexer, said indexer in communication with said network data cache, said indexer operable to note at least one indexable item of packets, said indexer further operable to record the values of the noted items to a data structure.

12. The system of claim 11, wherein said indexer is operable to note the source address of packets.

13. The system of claim 11, wherein said indexer is operable to note the destination address of packets.

14. The system of claim 11, wherein said indexer is operable to note the port of packets.

15. A full time network traffic recording computer system, comprising:

a network canture device, said network capture device being configurable to sample network packets on a network segment;

at least one processor, said processor in operable communication with said network capture device to receive sampled network packets;

storage device operating to accent write commands from said processor;

computer readable instructions contained in memory, said memory readable by said processor, computer readable instructions enabling said processor to perform the functions of;

(i) receiving sampled network packets from said network capture device,

(ii) annotating the received packets with at least the time of receipt, said annotating forming annotated packets,

(iii) organizing the annotated packets in a hierarchical organization of finite logical storage units,

and (iv) storing the finite logical storage units to said storage device;

wherein said computer readable instructions enabling said processor to perform the functions of;

(i) indexing the received packets using an indexable item database,

(ii) receiving a filter request having a filter expression composed of matching expressions linked by logical operators, the filter expression having at least one matching expression referencing an indexable item,

(iii) reading the indexable item database,

(iv) computing efficiency ratings for each matching expression and successively each logical operator,

and (v) filtering stored network traffic data, said filtering applying matching expressions in preferential order of efficiency.

16. The system of claim 15, further comprising:

a recording system interconnect.

17. A full time network traffic recording computer system, comprising:

a network capture device, said network capture device being configurable to sample network packets on a network segment;

at least one processor, said processor in operable communication with said network capture device to receive sampled network packets;

storage device operating to accept write commands from said processor;

a recording system interconnect;

computer readable instructions contained in memory, said memory readable by said processor, computer readable instructions enabling said processor to perform the functions of;

(i) receiving sampled network packets from said network capture device,

(ii) annotating the received packets with at least the time of receipt, said annotating forming annotated packets,

(iii) organizing the annotated packets in a hierarchical organization of finite logical storage units,

and (iv) storing the finite logical storage units to said storage device;

wherein said computer readable instructions enabling said processor to perform the functions of;

periodically creating a packet profile;

exchanging packet profile information with configured distributed network recording machines using said recording system interconnect;

comparing packet profile information to detect and report inconsistencies and errors among the distributed network recording machines;

and synchronizing packet stream information with configured distributed network recording machines using said recording system interconnect for validating the integrity of the recorded packets.

18. A full time network traffic recording system, comprising:

a network capture means, said network capture means being configurable to sample network packets on a network segment;

means of annotating sampled packets with at least a time of receipt to form annotated packets, said annotating means in accessible communication with said network capture means wherein sampled network packets may be referenced or read;

means of forming finite logical storage units containing the annotated packets, said annotating means providing referential or read access of the annotated packets to said forming means, said forming means further recording at least a start and an end time of the contained annotated packets to the formed units;

means of storing a succession of formed units to storage;

wherein said system performs the functions of;

(i) indexing received packets using an indexable item database,

(ii) receiving a filter request having a filter expression composed of matching expressions linked by logical operators, the filter expression having at least one matching expression referencing an indexable item,

(iii) reading the indexable item database,

(iv) computing efficiency ratings for each matching expression and successively each logical operator,

and (v) filtering stored network traffic data, said filtering applying matching expressions in preferential order of efficiency.

19. A method of capturing a large quantity of network data, comprising:

non-intrusively sampling network packets from a network segment;

annotating sampled network packets with at least a time of receipt, organizing the annotated packets into a hierarchical data organization composed of finite logical storage units;

storing the finite logical storage units to storage;

periodically creating a packet profile;

exchanging packet profile information with configured distributed network recording machines using a recording system interconnect;

comparing packet profile information to detect and report inconsistencies and errors among a plurality of distributed network recording machines;

and synchronizing packet stream information with configured distributed network recording machines using said recording system interconnect for validating the integrity of the recorded packets.