Flow de-duplication for network monitoring

• US 9,548,908 B2
• Filed: 08/21/2012
• Issued: 01/17/2017
• Est. Priority Date: 08/21/2012
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Nguyen, Khanh V., Tyagi, Naveen Kumar, Lau, Ray M.
• Primary Examiner(s): Vostal, O. C.
• Application Number:
  US13/591,080
• Publication Number:
  US 20140059200A1
• Time in Patent Office: 1,610 Days
• Field of Search: 709/224
• US Class Current: 1/1
• CPC Class Codes: H04L 43/026 using flow identification

Abstract

A method is provided in one example and includes receiving flow data associated with a traffic flow. The flow data can be tagged with a data source identifier identifying a data source exporting the flow data, a source site identifier identifying a site associated with a source device of the traffic flow, and a destination site identifier identifying a destination site associated with a destination device of the traffic flow. The method further includes determining at least one authoritative data source for each site and metric type using at least one selection rule. The method further includes receiving a query for de-duplicated flow data, and generating de-duplicated flow data based on the data source identifier, source site identifier, and destination site identifier and particular flow data associated with the determined at least one authoritative data source.

Claims

1. A method, comprising:

receiving, by a server having a processor and memory, flow data associated with a traffic flow, the flow data tagged with a data source identifier identifying a data source exporting the flow data, a source site identifier identifying a source site associated with a source device of the traffic flow wherein the source site identifier identifies a geographical location of the source device, and a destination site identifier identifying a destination site associated with a destination device of the traffic flow wherein the destination site identifier identifies a geographical location of the destination device;

determining, by the server, at least one authoritative data source for each site and metric type using at least one selection rule;

receiving, by the server, a query for de-duplicated flow data, wherein the query includes a designation of a particular metric type and a designation of one of a source site identifier associated with a particular source site and a destination site identifier associated with a particular destination site; and

generating, by the server, de-duplicated flow data based on the particular metric type, data source identifier, source site identifier, destination site identifier and particular flow data associated with the determined at least one authoritative data source.

2. The method of claim 1, wherein determining at least one authoritative data source including generating a plurality of candidate authoritative data sources and applying the at least one selection rule to generate the at least one authoritative data source.

3. The method of claim 2, wherein the at least one selection rule includes a rule in which a candidate authoritative data source that is local to a particular site is chosen as the authoritative data source of the particular site over a candidate authoritative data source that is remote to the particular site.

4. The method of claim 2, wherein the at least one selection rule includes a rule in which a candidate authoritative data source providing the most accurate data for a given metric is selected.

5. The method of claim 2, wherein the at least one selection rule includes a rule in which a candidate authoritative data source that reports a high level of traffic activities for a particular site is chosen by the server as the authoritative data source of the particular site.

6. The method of claim 1, wherein the at least one selection rule includes a rule including a user designation of one or more data sources as the authoritative data source of a site and metric type.

7. The method of claim 1, wherein the at least one authoritative data source includes a plurality of authoritative data sources, and wherein generating the de-duplicated flow data includes aggregating flow data from the plurality of authoritative data sources.

8. The method of claim 1, wherein flow data are deduplicated by the server at a time of the query by matching the data source identifier and site identifier tags to authoritative data source identifiers of respective site identifiers.

9. The method of claim 8, wherein upon receiving a deduplication query for a particular site and a particular metric type, an authoritative data source database is queried to find an identifier of an authoritative data source associated with that particular site and metric type, and wherein a particular authoritative data source identifier is used as a filter to query a tagged flow data database.

10. The method of claim 1, further comprising:

performing a SQL INNER JOIN of an authoritative data source database and a tagged flow data database in response to receiving a deduplication query for multiple sites.

11. Logic encoded in one or more non-transitory tangible media that includes code for execution and when executed by a processor operable to perform operations, comprising:

receiving, by a server having a processor and memory, flow data associated with a traffic flow, the flow data tagged with a data source identifier identifying a data source exporting the flow data, a source site identifier identifying a source site associated with a source device of the traffic flow wherein the source site identifier identifies a geographical location of the source device, and a destination site identifier identifying a destination site associated with a destination device of the traffic flow wherein the destination site identifier identifies a geographical location of the destination device;

determining, by the server, at least one authoritative data source for each site and metric type using at least one selection rule;

receiving, by the server, a query for de-duplicated flow data, wherein the query includes a designation of a particular metric type and a designation of one of a source site identifier associated with a particular source site and a destination site identifier associated with a particular destination site; and

generating, by the server, de-duplicated flow data based on the particular metric type, data source identifier, source site identifier, destination site identifier and particular flow data associated with the determined at least one authoritative data source.

12. The logic of claim 11, wherein determining at least one authoritative data source includes generating a plurality of candidate authoritative data sources and applying the at least one selection rule to generate the at least one authoritative data source.

13. The logic of claim 12, wherein the at least one selection rule includes a rule in which a candidate authoritative data source that is local to a particular site is chosen as the authoritative data source of the particular site over a candidate authoritative data source that is remote to the particular site.

14. The logic of claim 12, wherein the at least one selection rule includes a rule in which a candidate authoritative data source providing the most accurate data for a given metric is selected.

15. The logic of claim 12, wherein the at least one selection rule includes a rule in which a candidate authoritative data source that reports a high level of traffic activities for a particular site is chosen by the server as the authoritative data source of the particular site.

16. The logic of claim 11, wherein the at least one selection rule includes a rule including a user designation of one or more data sources as the authoritative data source of a site.

17. The logic of claim 11, wherein the at least one authoritative data source includes a plurality of authoritative data sources, and wherein generating the de-duplicated flow data includes aggregating flow data from the plurality of authoritative data sources.

18. The logic of claim 11, wherein flow data are deduplicated by the server at a time of the query by matching the data source identifier and site identifier tags to authoritative data source identifiers of respective site identifiers.

19. The logic of claim 18, wherein upon receiving a deduplication query for a particular site and a particular metric type, an authoritative data source database is queried to find an identifier of an authoritative data source associated with that particular site and metric type, and wherein a particular authoritative data source identifier is used as a filter to query a tagged flow data database.

20. The logic of claim 11, the operations further comprising:

performing a SQL INNER JOIN of an authoritative data source database and a tagged flow data database in response to receiving a deduplication query for multiple sites.

21. An apparatus, comprising:

a memory element configured to store data,

a processor operable to execute instructions associated with the data, and

an authoritative data source detection module, the apparatus being configured to;

receive, by a server, flow data associated with a traffic flow, the flow data tagged with a data source identifier identifying a data source exporting the flow data, a source site identifier identifying a source site associated with a source device of the traffic flow wherein the source site identifier identifies a geographical location of the source device, and a destination site identifier identifying a destination site associated with a destination device of the traffic flow wherein the destination site identifier identifies a geographical location of the destination device;

determine, by the server, at least one authoritative data source for each site and metric type using at least one selection rule;

receive, by the server having a processor and memory, a query for de-duplicated flow data, wherein the query includes a designation of a particular metric type and a designation of one of a source site identifier associated with a particular source site and a destination site identifier associated with a particular destination site; and

generate, by the server, de-duplicated flow data based on the particular metric type, data source identifier, source site identifier, destination site identifier and particular flow data associated with the determined at least one authoritative data source.

22. The apparatus of claim 21, wherein determining at least one authoritative data source includes generating a plurality of candidate authoritative data sources and applying the at least one selection rule to generate the at least one authoritative data source.

23. The apparatus of claim 22, wherein the at least one selection rule includes a rule in which a candidate authoritative data source that is local to a particular site is chosen as the authoritative data source of the particular site over a candidate authoritative data source that is remote to the particular site.

24. The apparatus of claim 22, wherein the at least one selection rule includes a rule in which a candidate authoritative data source providing the most accurate data for a given metric is selected.

25. The apparatus of claim 22, wherein the at least one selection rule includes a rule in which a candidate authoritative data source that reports a high level of traffic activities for a particular site is chosen by the server as the authoritative data source of the particular site.

26. The apparatus of claim 21, wherein the at least one selection rule includes a rule including a user designation of one or more data sources as the authoritative data source of a site.

27. The apparatus of claim 21, wherein the at least one authoritative data source includes a plurality of authoritative data sources, and wherein generating the de-duplicated flow data includes aggregating flow data from the plurality of authoritative data sources.

28. The apparatus of claim 21, wherein flow data are deduplicated by the server at a time of the query by matching the data source identifier and site identifier tags to authoritative data source identifiers of respective site identifiers.

29. The apparatus of claim 28, wherein upon receiving a deduplication query for a particular site and a particular metric type, an authoritative data source database is queried to find an identifier of an authoritative data source associated with that particular site and metric type, and wherein a particular authoritative data source identifier is used as a filter to query a tagged flow data database.

30. The apparatus of claim 21, the apparatus being further configured to:

perform a SQL INNER JOIN of an authoritative data source database and a tagged flow data database in response to receiving a deduplication query for multiple sites.