System and method for conditional analysis of network traffic

• US 10,084,876 B2
• Filed: 03/13/2017
• Issued: 09/25/2018
• Est. Priority Date: 10/30/2013
• Status: Active Grant

• Current Assignee: Cognyte Technologies Israel Ltd. (Cognyte Software Ltd.)
• Original Assignee: Verint Systems Limited (Verint Systems Incorporated)
• Inventors: Yishay, Yitshak, Goldfarb, Eithan
• Primary Examiner(s): Hussain, Tauqir
• Assistant Examiner(s): Moreau, Austin
• Application Number:
  US15/457,122
• Publication Number:
  US 20170251074A1
• Time in Patent Office: 561 Days
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/60 Details of cache memory
• CPC Class Codes: G06F 2212/62 Details of cache specific to multiprocessor cache arrangements
• CPC Class Codes: H04L 43/062 related to network traffic
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Embodiments that are described herein provide improved methods and systems for analyzing network traffic. The disclosed embodiments enable an analytics system to perform complex processing to only new, first occurrences of received content, while refraining from processing duplicate instances of that content. In an embodiment, the analytics results regarding the first occurring content are reported and cached in association with the content. For any duplicate instance of the content, the analytics results are retrieved from the cache without re-processing of the duplicate content. When using the disclosed techniques, the system still processes all first occurring content but not duplicate instances of content that was previously received and processed. In the embodiments described herein, input data comprises communication packets exchanged in a communication network.

Claims

1. A method, comprising:

receiving network traffic that carries content items for processing by a network analytics system;

extracting a content item from the network traffic, wherein extracting the content item includes deriving a respective unique identifier for the content item;

in response to finding that the content item is a duplicate of previous content that was already processed by the network analytics system and cached in a cache memory, retrieving and outputting a cached analytics outcome of the content item from the cache memory;

for a given content item, counting a number of matching occurrences of the given content item, and caching the number of matching occurrences in the cache memory in association with a unique identifier of the given content item, wherein the number of matching occurrences is multiplied by a weight factor that is based on a processing time of the content item to produce a weighted number of matching occurrences; and

deleting from the cache memory the unique identifier of the given content item if the weighted number of matching occurrences during a predefined duration is lower than a predefined threshold;

wherein the network analytics system produces analytics outcomes based on an analytics rule, and comprising, upon changing the analytics rule, updating cached analytics outcomes for the content items for which the analytics rule was applied.

2. The method according to claim 1 further comprising, in response to finding that the content item does not duplicate, causing the network analytics system to produce the analytics outcome for the content item, and caching the analytics outcome, wherein the step of finding that the content item does not duplicate includes validating that the unique identifier does not match any identifier in the cache memory that caches identifiers of previous content items that were processed by the network analytics system.

3. The method according to claim 2 further comprising, in response to finding that the content item does not duplicate, caching the unique identifier of the content item, and the analytics outcome produced for the content item by the network analytics system, in the cache memory in association with the unique identifier.

4. The method according to claim 2, wherein deriving the unique identifier comprises composing the unique identifier from at least part of a Uniform Resource Locator (URL) in which the content item resides.

5. The method according to claim 4, wherein the at least part of the URL is chosen to exclude a variable section of the URL.

6. The method according to claim 2, wherein extracting the content item comprises extracting a traffic transaction created by servers that are infected by malware, and wherein deriving the unique identifier comprises deriving a pattern of the traffic transaction.

7. The method according to claim 2, wherein deriving the unique identifier comprises calculating a digital signature over at least part of the content item.

8. The method according to claim 7, wherein calculating the digital signature comprises calculating the digital signature over only a predefined portion of the content item.

9. The method according to claim 8, wherein the predefined portion is chosen to exclude a section of the content item that varies among duplicates of the content item.

10. The method according to claim 2, wherein the unique identifier comprises a first signature and a second signature, which is stronger than the first signature, and wherein validating that the unique identifier does not match any identifier in the cache memory comprises checking the second signature only if checking the first signature is not sufficient for deciding that the identifier does not match.

11. The method according to claim 1, wherein extracting the content item comprises recognizing HTTP transactions in the network traffic and extracting the content item from the HTTP transactions.

12. The method according to claim 11, wherein the content item comprises a multimedia content.

13. The method according to claim 1, wherein changing the analytics rule comprises removing the analytics rule, and wherein updating the analytics outcomes comprises deleting the cached content items for which the analytics rule was applied.

14. The method according to claim 1, wherein changing the analytics rule comprises changing the analytics rule with respect to a given content type, and wherein updating the analytics outcomes comprises removing the cached content items of the given content type.

15. The method according to claim 1, wherein changing the analytics rule comprises replacing the analytics rule with a new analytics rule, which is different from the analytics rule, and wherein updating the analytics outcomes comprises producing new analytics outcomes by applying the new analytics rule to the content items and replacing the cached analytics outcomes with the new analytics outcomes.

16. An apparatus, comprising:

an input circuit, which is configured to receive network traffic that carries content items for processing by a network analytics system; and

a processor coupled to a memory, which is configured to;

extract a content item from the network traffic, wherein extracting the content item includes deriving a respective unique identifier for the content item;

retrieve and output a cached analytics outcome of the content item from a cache memory in response to finding that the content item is a duplicate of a previous content that was already processed by the network analytics system and cached in the cache memory;

count, for a given content item, a number of matching occurrences of the given content item, and caching the number of matching occurrences in the cache memory in association with a unique identifier of the given content item, wherein the number of matching occurrences is multiplied by a weight factor that is based on a processing time of the content item to produce a weighted number of matching occurrences; and

delete from the cache memory the unique identifier of the given content item if the weighted number of matching occurrences during a predefined duration is lower than a predefined threshold;

wherein the network analytics system produces analytics outcomes based on an analytics rule, and comprising, upon changing the analytics rule, updating cached analytics outcomes for the content items for which the analytics rule was applied.

17. The apparatus of claim 16, wherein the processor is further configured to in response to finding that the content item does not duplicate, cause the network analytics system to produce the analytics outcome for the content item, and to cache the analytics outcome, wherein the processor is configured to find whether the content item does not duplicate by at least validating that the unique identifier does not match any identifier in the cache memory that caches identifiers of previous content items that were processed by the network analytics system.

18. A non-transitory computer readable medium, having instructions stored thereon that, when executed by a computing system, cause the computing system to at least:

receive network traffic that carries content items for processing by a network analytics system;

extract a content item from the network traffic, wherein extracting the content item includes deriving a respective unique identifier for the content item;

retrieve and output a cached analytics outcome of the content item from a cache memory in response to finding that the content item is a duplicate of previous content that was already processed by the network analytics system and cached in the cache memory;

count, for a given content item, a number of matching occurrences of the given content item, and caching the number of matching occurrences in the cache memory in association with a unique identifier of the given content item, wherein the number of matching occurrences is multiplied by a weight factor that is based on a processing time of the content item to produce a weighted number of matching occurrences; and

delete from the cache memory the unique identifier of the given content item if the weighted number of matching occurrences during a predefined duration is lower than a predefined threshold;

wherein the network analytics system produces analytics outcomes based on an analytics rule, and comprising, upon changing the analytics rule, updating cached analytics outcomes for the content items for which the analytics rule was applied.