Capturing data packets from external networks into high availability clusters while maintaining high availability of popular data packets

• US 20150092550A1
• Filed: 09/27/2013
• Published: 04/02/2015
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: Zettaset Incorporated
• Original Assignee: Zettaset Incorporated
• Inventors: Christian, Brian P., Dalton, Michael W., Demarest, Timothy, Ware, Michael A.
• Granted Patent: US 9,571,356 B2
• Time in Patent Office: Days
• US Class Current: 370/235
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 47/31 by tagging of packets, e.g. using discard eligibility [DE] bits

Abstract

A method and high availability cluster configuration for capture of data packets propagating in an external network into the cluster through a wired-tap connection between the cluster and at least one router of the external network. Captured packets are supplied with popularity tags and striped over high-access-rate non-transitory storage media belonging to a capture pool of serving nodes that are specially provisioned and selected by the master node. The captured packets are either devolved or demoted to low-access-rate non-transitory storage media of at least two serving nodes not including any backup master based on popularity metrics contained their tags. The demotion, devolution as well as determination of popularity metrics and setting of a popularity threshold are managed by a distributed management protocol.

Claims

1. A method for capturing data packets propagating in a network to a high availability cluster having a master node and serving nodes, said method comprising the steps of:

a) establishing a wired-tap connection between said high availability cluster and at least one router belonging to said network;

b) capturing through said wired-tap connection captured data packets from among routed data packets being routed through said at least one router;

c) attaching a packet popularity tag to each one of said captured data packets;

d) striping said captured data packets over high-access-rate non-transitory storage media belonging to a capture pool of said serving nodes selected by said master node;

e) writing said captured data packets from said high-access-rate non-transitory storage media to low-access-rate non-transitory storage media of at least two serving nodes of said high availability cluster; and

f) retaining in said high-access-rate non-transitory storage media captured data packets in which said packet popularity tag exceeds a set threshold.

2. The method of claim 1, further comprising the step of changing the selection of said serving nodes in said capture pool such that a ratio of said captured data packets to said routed data packets is substantially at 100%.

3. The method of claim 1, wherein said step of capturing said captured data packets is performed by a network packet capturing device operating at wire speed.

4. The method of claim 1, wherein said packet popularity tag comprises at least one popularity metric that is related to a capture time.

5. The method of claim 4, wherein said at least one packet popularity metric further comprises a popularity attribute assigned by a search-ranking algorithm.

6. The method of claim 4, wherein said at least one popularity metric is selected from the group of popularity metrics consisting of time, interface, source, destination, protocol, port, packet length, time-to-live and version.

7. The method of claim 1, further comprising the steps of:

a) combining predetermined sets of said captured data packets into captured files;

b) attaching a captured file popularity tag to each of said captured data files;

wherein said steps of striping, writing and retaining are applied to said captured data files based on said captured file popularity tags.

8. The method of claim 7, further comprising the step of reassembling an original file from said captured data files.

9. The method of claim 7, wherein said captured file popularity tag comprises at least one popularity metric that is related to a capture time.

10. The method of claim 9, wherein said at least one captured file popularity metric further comprises a popularity attribute assigned by a search-ranking algorithm.

11. The method of claim 9, wherein said at least one popularity metric is selected from the group of popularity metrics consisting of time, interface, source, destination, protocol, port, packet length, time-to-live and version.

12. The method of claim 1, further comprising the step of migrating from said low-access-rate non-transitory storage media to said high-access-rate non-transitory storage media captured data packets in which said popularity tag exceeds said set threshold.

13. The method of claim 12, further comprising the step of changing said set threshold thereby initiating said step of migrating.

14. The method of claim 1, further comprising re-balancing of captured data packets among said low-access-rate non-transitory storage media and said high-access-rate non-transitory storage media.

15. The method of claim 1, further comprising the step of assigning said captured data packets to predetermined block locations in a POSIX-compliant UNIX file system.

16. The method of claim 15, wherein said POSIX-compliant UNIX file system comprises Lustre.

17. The method of claim 15, further comprising the steps of:

a) establishing a connection from a non-POSIX-compliant Distributed File System to said POSIX-compliant UNIX file system; and

b) performing a batch job on said captured data packets within said non-POSIX-compliant Distributed File System.

18. The method of claim 17, further comprising the step of establishing an access window to said captured data packets in said POSIX-compliant UNIX file system for monitoring the performance of said batch job.

19. The method of claim 18, wherein said access window comprises a POSIX-compliant interface to said POSIX-compliant UNIX file system and a second interface to said non-POSIX-compliant Distributed File System.

20. The method of claim 17, wherein said non-POSIX-compliant Distributed File System comprises at least one system selected from the group consisting of, Hadoop Distributed File System, MPI, Beowulf, ZFS and MogileFS.

21. A high availability cluster configured to capture data packets from a network having at least one router, said high availability cluster comprising:

a) a master node;

b) serving nodes each having a high-access-rate non-transitory storage media and a low-access-rate non-transitory storage media;

c) a network packet capturing device for establishing a wired-tap connection to said at least one router to capture captured data packets from among routed data packets being routed through said at least one router;

d) a distributed management protocol for attaching a packet popularity tag to each one of said captured data packets;

e) a capture pool selected by said master node from among said serving nodes for receiving said captured data packets, said captured data packets being striped over said high-access-rate non-transitory storage media of said capture pool and being written from said high-access-rate non-transitory storage media to said low-access-rate non-transitory storage media of at least two of said server nodes;

whereby said distributed management protocol retains in said high-access-rate non-transitory storage media captured data packets in which said packet popularity tag exceeds a set threshold.

22. The high availability cluster of claim 21, wherein at least one of said at least two of said serving nodes to whose low-access-rate non-transitory storage media said captured data packets are written is not in said capture pool.

23. The high availability cluster of claim 21, wherein said network packet capturing device is a wire speed network packet capturing device.

24. The high availability cluster of claim 21, further comprising:

a) POSIX-compliant UNIX file system operating on said high availability cluster;

b) a non-POSIX-compliant Distributed File System with a connection to said POSIX-compliant UNIX file system via an access window, said non-POSIX-compliant Distributed File System being configured to perform a batch job on said captured data packets.

25. The high availability cluster of claim 24, wherein said access window comprises a POSIX-compliant interface to said POSIX-compliant UNIX file system and a second interface to said non-POSIX-compliant Distributed File System.

26. The high availability cluster of claim 24, wherein said non-POSIX-compliant Distributed File System comprises at least one system selected from the group consisting of, Hadoop Distributed File System, ZFS and MogileFS.