System and method for data mining and security policy management

• US 8,601,537 B2
• Filed: 03/19/2012
• Issued: 12/03/2013
• Est. Priority Date: 07/10/2008
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Liu, Weimin
• Primary Examiner(s): LEMMA, SAMSON B
• Application Number:
  US13/424,249
• Publication Number:
  US 20120179687A1
• Time in Patent Office: 624 Days
• Field of Search: 726/1, 713/165, 713/167, 713/176, 713/180, 713/193, 707/741, 707/E17.002, 707/E17.014
• US Class Current: 726/1
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A system and method to generate and maintain controlled growth DAG are described. The controlled growth DAG conveys information about objects captured by a capture system.

Claims

1. A method, comprising:

capturing, by an electronic computing device, an object traversing a network environment;

storing the captured object; and

indexing the stored object according to tags, keywords and metadata, wherein the tags include information about the object and the tags are placed into a tag index, wherein the keywords are generated from a content of the object and the keywords are placed into a keyword index, and wherein the metadata is generated from properties of the object and the metadata is placed into a metadata index;

wherein the indexing is performed utilizing an adaptive time-based dictionary granularity with new keyword indexes and metadata indexes being created over time prevent a specific index from growing unbounded and a temporal basis for creating the new keyword indexes and the metadata indexes is determined by a plurality of factors, including;

a) a number of keywords or metadata that have been inserted into the keyword index or the metadata index, respectively;

b) a number of captured objects listed in each of the keyword index and the metadata index;

c) an aggregate size of each of the keyword index and the metadata index; and

d) an aggregate size of captured content being indexed.

2. The method of claim 1, wherein the tags include at least one selection from a group comprising:

a MAC address, a source Internet Protocol (IP) address of the object, a destination IP address of the object, a source port number of the object, a destination port number of the object, a protocol that carried the object, an instance of the object, a content type of the object, an encoding used by the protocol, a size of the object, a timestamp corresponding to a time when the object was captured, an owner requesting the capture of the object, a configuration corresponding to a capture rule directing the capture of the object, a signature of the object, and a signature of the tags.

3. The method of claim 1, wherein each entry in the keyword index includes a keyword and a reference to a corresponding associated object, and wherein each entry in the metadata index includes a metadata and a reference to a corresponding associated object.

4. The method of claim 1, wherein the keyword index and the metadata index are part of a single file per time period.

5. The method of claim 1, further comprising:

querying the keyword index for search keywords;

retrieving a first tag information corresponding to a first set of objects having the search keywords;

querying the metadata index for search metadata;

retrieving a second tag information corresponding to a second set of objects having the search metadata; and

intersecting the first tag information and the second tag information to determine a third tag information corresponding to a third set of objects having the search keywords and the search metadata.

6. The method of claim 1, wherein the creation of new indexes is additionally controlled by employing heuristics to optimize search performance, wherein the heuristics are provided by a user or an administrator.

7. The method of claim 1, further comprising:

performing a first query, comprising querying a first keyword index for search keywords and a first metadata index for search metadata, wherein the first keyword index and the first metadata index were created during a first time period;

retrieving a first tag information corresponding to a first set of objects having the search keywords and the search metadata over the first time period;

performing a second query comprising querying a second keyword index for the search keywords and a second metadata index for the search metadata, wherein the second keyword index and the second metadata index were created during a second time period;

retrieving a second tag information corresponding to a second set of objects having the search keywords and the search metadata over the second time period; and

intersecting the first tag information and the second tag information to determine a third tag information corresponding to a third set of objects having the search keywords and the search metadata over the first time period and the second time period.

8. A non-transitory machine-readable storage medium including program code which, when executed by a processor, causes the processor to perform a method, the method comprising:

capturing an object traversing a network environment;

storing the captured object; and

indexing the stored object according to tags, keywords and metadata, wherein the tags include information about the object and the tags are placed into a tag index, wherein the keywords are generated from a content of the object and the keywords are placed into a keyword index, and wherein the metadata is generated from properties of the object and the metadata is placed into a metadata index;

wherein the indexing is performed utilizing an adaptive time-based dictionary granularity with new keyword indexes and metadata indexes being created over time prevent a specific index from growing unbounded and a temporal basis for creating the new keyword indexes and the metadata indexes is determined by a plurality of factors, including;

a) a number of keywords or metadata that have been inserted into the keyword index or the metadata index, respectively;

b) a number of captured objects listed in each of the keyword index and the metadata index;

c) an aggregate size of each of the keyword index and the metadata index; and

d) an aggregate size of captured content being indexed.

9. The non-transitory machine-readable storage medium of claim 8, wherein the tags include at least one selection from a group comprising:

a MAC address, a source Internet Protocol (IP) address of the object, a destination IP address of the object, a source port number of the object, a destination port number of the object, a protocol that carried the object, an instance of the object, a content type of the object, an encoding used by the protocol, a size of the object, a timestamp corresponding to a time when the object was captured, an owner requesting the capture of the object, a configuration corresponding to a capture rule directing the capture of the object, a signature of the object, and a signature of the tags.

10. The non-transitory machine-readable storage medium of claim 8, wherein the method further comprises:

querying the keyword index for search keywords;

retrieving a first tag information corresponding to a first set of objects having the search keywords;

querying the metadata index for search metadata;

retrieving a second tag information corresponding to a second set of objects having the search metadata; and

intersecting the first tag information and the second tag information to determine a third tag information corresponding to a third set of objects having the search keywords and the search metadata.

11. The non-transitory machine-readable storage medium of claim 8, wherein the creation of new indexes is additionally controlled by employing heuristics to optimize search performance, wherein the heuristics are provided by a user or an administrator.

12. The non-transitory machine-readable storage medium of claim 8, wherein the method further comprises:

performing a first query, comprising querying a first keyword index for search keywords and a first metadata index for search metadata, wherein the first keyword index and the first metadata index were created during a first time period;

retrieving a first tag information corresponding to a first set of objects having the search keywords and the search metadata over the first time period;

performing a second query comprising querying a second keyword index for the search keywords and a second metadata index for the search metadata, wherein the second keyword index and the second metadata index were created during a second time period;

retrieving a second tag information corresponding to a second set of objects having the search keywords and the search metadata over the second time period; and

intersecting the first tag information and the second tag information to determine a third tag information corresponding to a third set of objects having the search keywords and the search metadata over the first time period and the second time period.

13. An apparatus, comprising:

a memory for storing data; and

a processor operable to execute instructions associated with the data, wherein the apparatus is configured for;

capturing an object traversing a network environment;

storing the captured object; and

indexing the stored object according to tags, keywords and metadata, wherein the tags include information about the object and the tags are placed into a tag index, wherein the keywords are generated from a content of the object and the keywords are placed into a keyword index, and wherein the metadata is generated from properties of the object and the metadata is placed into a metadata index;

wherein the indexing is performed utilizing an adaptive time-based dictionary granularity with new keyword indexes and metadata indexes being created over time prevent a specific index from growing unbounded and a temporal basis for creating the new keyword indexes and the metadata indexes is determined by a plurality of factors, including;

a) a number of keywords or metadata that have been inserted into the keyword index or the metadata index, respectively;

b) a number of captured objects listed in each of the keyword index and the metadata index;

c) an aggregate size of each of the keyword index and the metadata index; and

d) an aggregate size of captured content being indexed.

14. The apparatus of claim 13, wherein the tags include at least one selection from a group comprising:

a MAC address, a source Internet Protocol (IP) address of the object, a destination IP address of the object, a source port number of the object, a destination port number of the object, a protocol that carried the object, an instance of the object, a content type of the object, an encoding used by the protocol, a size of the object, a timestamp corresponding to a time when the object was captured, an owner requesting the capture of the object, a configuration corresponding to a capture rule directing the capture of the object, a signature of the object, and a signature of the tags.

15. The apparatus of claim 13, further configured for:

querying the keyword index for search keywords;

retrieving a first tag information corresponding to a first set of objects having the search keywords;

querying the metadata index for search metadata;

retrieving a second tag information corresponding to a second set of objects having the search metadata; and

intersecting the first tag information and the second tag information to determine a third tag information corresponding to a third set of objects having the search keywords and the search metadata.

16. The apparatus of claim 13, wherein the creation of new indexes is additionally controlled by employing heuristics to optimize search performance, wherein the heuristics are provided by a user or an administrator.

17. The apparatus of claim 13, further configured for:

performing a first query, comprising querying a first keyword index for search keywords and a first metadata index for search metadata, wherein the first keyword index and the first metadata index were created during a first time period;

retrieving a first tag information corresponding to a first set of objects having the search keywords and the search metadata over the first time period;

performing a second query comprising querying a second keyword index for the search keywords and a second metadata index for the search metadata, wherein the second keyword index and the second metadata index were created during a second time period;

retrieving a second tag information corresponding to a second set of objects having the search keywords and the search metadata over the second time period; and

intersecting the first tag information and the second tag information to determine a third tag information corresponding to a third set of objects having the search keywords and the search metadata over the first time period and the second time period.