System and method for reducing netflow traffic in a network environment

• US 9,065,767 B2
• Filed: 04/03/2012
• Issued: 06/23/2015
• Est. Priority Date: 04/03/2012
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Dong, Xuanming, Cantu, Cesare, Wang, Jeffrey Y M
• Primary Examiner(s): Bates, Kevin
• Assistant Examiner(s): McCray, Clarence D
• Application Number:
  US13/438,113
• Publication Number:
  US 20130262703A1
• Time in Patent Office: 1,176 Days
• Field of Search: 707/693, 709/247
• US Class Current: 1/1
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 47/38 by adapting coding or compression rate
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

A an example method includes building a dictionary between an exporter and a collector by encoding a first data record of a flow according to a dictionary template and exporting the first data record to the collector via a network communication. The method can also include compressing a second data record of the flow using the dictionary, where the compressing comprises encoding the second data record according to an encoding template; and exporting the second data record to the collector to be decompressed using the dictionary.

Claims

1. A method, comprising:

building a dictionary between an exporter and a collector by encoding a first data record of a flow according to a dictionary template at the exporter and exporting the first data record to the collector via a network communication, wherein the exporter collects information about network packets flowing through a network, formats the information into flow records, and exports the collected information to the collector, each flow record comprising a template record and a data record whose format is indicated by the template record, wherein the dictionary template includes a plurality of fields indicative of the flow;

compressing a second data record of the flow using the dictionary, wherein the compressing comprises encoding the second data record according to an encoding template comprising fewer fields than the dictionary template; and

exporting the second data record to the collector to be decompressed using the dictionary.

2. The method of claim 1, wherein the dictionary includes a dictionary template comprising a plurality of key fields, a plurality of non-key fields, and an entry index field, and wherein the encoding template comprises the plurality of non-key fields and a key entry index field.

3. The method of claim 2, wherein the encoding of the first data record comprises:

populating the plurality of key fields, the plurality of non-key fields and the entry index field with a plurality of key bytes, a first plurality of non-key bytes, and an entry index value, respectively, and wherein the encoding of the second data record comprises populating the plurality of non-key fields and the key entry index value field with a second plurality of non-key bytes and a key entry index value.

4. The method of claim 2, wherein the entry index value and the key entry index value are calculated from a hash of selected key bytes such that the key entry index value is equal to the entry index value.

5. The method of claim 2, wherein the collector is configured for:

determining that the first data record is encoded according to the dictionary template;

extracting a device key from the first data record; and

updating the dictionary using a composite key comprising the device key, a template identification (ID) identifying the dictionary template, and the entry index value, wherein the updating comprises adding the plurality of key bytes to the dictionary.

6. The method of claim 5, wherein the collector is further configured for:

determining that the second data record is encoded according to the encoding template;

extracting the device key from the second data record;

accessing the dictionary using another composite key comprising the device key, a template ID identifying the encoding template, and the key entry index value; and

substituting the key entry index value with the plurality of key bytes from the dictionary.

7. The method of claim 2, further comprising:

checking if a first entry in an aging table is to be aged out according to an aging criterion, wherein the first entry comprises the plurality of key bytes, a first plurality of non-key bytes, and the entry index value;

if the first entry is to be aged out,

copying the first entry to an exporting table;

adding a first non-key field, a second non-key field, and a third non-key field to the first entry in the exporting table, wherein the first non-key field is set to 1, the second non-key field is set to 1, and the third non-key field is set to 0;

exporting the first entry to the collector; and

setting the second non-key field to 0 after exporting.

8. The method of claim 1, further comprising:

determining if the dictionary is to be updated at the collector; and

if the dictionary is to be updated at the collector, encoding a first entry in an aging table into the first data record before exporting.

9. The method of claim 1, further comprising:

checking if a first entry in an aging table is to be aged out according to aging criterion, wherein the first entry comprises the plurality of key bytes, a first plurality of non-key bytes, and the entry index value;

if the first entry is to be aged out,

checking if a peer entry having the entry index value exists in the exporting table;

if the peer entry exists,

copying the first plurality of non-key bytes from the first entry to the peer entry;

setting a first non-key field and a second non-key field in the peer entry to 0 and 1, respectively; and

exporting the peer entry to the collector.

10. The method of claim 9, further comprising:

encoding the peer entry into the second data record before exporting.

11. Logic encoded in non-transitory media that includes instructions for execution and when executed by a processor, is operable to perform operations comprising:

building a dictionary between an exporter and a collector by encoding a first data record of a flow according to a dictionary template at the exporter and exporting the first data record to the collector via a network communication, wherein the exporter collects information about network packets flowing through a network, formats the information into flow records, and exports the collected information to the collector, each flow record comprising a template record and a data record whose format is indicated by the template record, wherein the dictionary template includes a plurality of fields indicative of the flow;

compressing a second data record of the flow using the dictionary, wherein the compressing comprises encoding the second data record according to an encoding template comprising fewer fields than the dictionary template; and

exporting the second data record to the collector to be decompressed using the dictionary.

12. The logic of claim 11, wherein the dictionary includes a dictionary template comprising a plurality of key fields, a plurality of non-key fields, and an entry index field, and wherein the encoding template comprises the plurality of non-key fields and a key entry index field.

13. The logic of claim 12, wherein the encoding of the first data record comprises:

populating the plurality of key fields, the plurality of non-key fields and the entry index field with a plurality of key bytes, a first plurality of non-key bytes, and an entry index value, respectively, and wherein the encoding of the second data record comprises populating the plurality of non-key fields and the key entry index value field with a second plurality of non-key bytes and a key entry index value.

14. The logic of claim 13, wherein the collector is configured for:

determining that the first data record is encoded according to the dictionary template;

extracting a device key from the first data record; and

updating the dictionary using a composite key comprising the device key, a template identification (ID) identifying the dictionary template, and the entry index value, wherein the updating comprises adding the plurality of key bytes to the dictionary.

15. The logic of claim 14, wherein the collector is further configured for:

determining that the second data record is encoded according to the encoding template;

extracting the device key from the second data record;

accessing the dictionary using another composite key comprising the device key, a template ID identifying the encoding template, and the key entry index value; and

substituting the key entry index value with the plurality of key bytes from the dictionary.

16. An apparatus, comprising:

a memory element for storing data; and

a processor that executes instructions associated with the data, wherein the processor and the memory element cooperate, such that the apparatus is configured for;

building a dictionary between an exporter and a collector by encoding a first data record of a flow according to a dictionary template at the exporter and exporting the first data record to the collector via a network communication, wherein the exporter collects information about network packets flowing through a network, formats the information into flow records, and exports the collected information to the collector, each flow record comprising a template record and a data record whose format is indicated by the template record, wherein the dictionary template includes a plurality of fields indicative of the flow;

compressing a second data record of the flow using the dictionary, wherein the compressing comprises encoding the second data record according to an encoding template comprising fewer fields than the dictionary template; and

exporting the second data record to the collector to be decompressed using the dictionary.

17. The apparatus of claim 16, wherein the dictionary includes a dictionary template comprising a plurality of key fields, a plurality of non-key fields, and an entry index field, and wherein the encoding template comprises the plurality of non-key fields and a key entry index field.

18. The apparatus of claim 17, wherein the encoding of the first data record comprises:

populating the plurality of key fields, the plurality of non-key fields and the entry index field with a plurality of key bytes, a first plurality of non-key bytes, and an entry index value, respectively, and wherein the encoding of the second data record comprises populating the plurality of non-key fields and the key entry index value field with a second plurality of non-key bytes and a key entry index value.

19. The apparatus of claim 17, wherein the collector is configured for:

determining that the first data record is encoded according to the dictionary template;

extracting a device key from the first data record; and

updating the dictionary using a composite key comprising the device key, a template identification (ID) identifying the dictionary template, and the entry index value, wherein the updating comprises adding the plurality of key bytes to the dictionary.

20. The apparatus of claim 19, wherein the collector is further configured for:

determining that the second data record is encoded according to the encoding template;

extracting the device key from the second data record;

accessing the dictionary using another composite key comprising the device key, a template ID identifying the encoding template, and the key entry index value; and

substituting the key entry index value with the plurality of key bytes from the dictionary.