System and method for protecting specified data combinations

• US 8,806,615 B2
• Filed: 11/04/2010
• Issued: 08/12/2014
• Est. Priority Date: 11/04/2010
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Ahuja, Ratinder Paul Singh, Deninger, William J.
• Primary Examiner(s): Goldberg, Andrew
• Assistant Examiner(s): GRACIA, GARY S
• Application Number:
  US12/939,340
• Publication Number:
  US 20120114119A1
• Time in Patent Office: 1,377 Days
• Field of Search: 726/20
• US Class Current: 726/20
• CPC Class Codes: H04L 63/0853 using an additional device, e.g. smartcard, SIM or a different communication terminal cryptographic mechanisms or cryptographic arrangements for entity authentication involving additional secure or trusted devices H04L9/3234
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection

Abstract

A method in one example implementation includes extracting a plurality of data elements from a record of a data file, tokenizing the data elements into tokens, and storing the tokens in a first tuple of a registration list. The method further includes selecting one of the tokens as a token key for the first tuple, where the token is selected because it occurs less frequently in the registration list than each of the other tokens in the first tuple. In specific embodiments, at least one data element is an expression element having a character pattern matching a predefined expression pattern that represents at least two words and a separator between the words. In other embodiments, at least one data element is a word defined by a character pattern of one or more consecutive essential characters. Other specific embodiments include determining an end of the record by recognizing a predefined delimiter.

Claims

1. A method to be executed by a processor in a network environment, comprising:

extracting a plurality of data elements from a record of a data file;

tokenizing the plurality of data elements into a plurality of tokens;

storing the plurality of tokens in a first tuple of a registration list;

selecting one of the plurality of tokens as a token key for indexing the first tuple, wherein a total number of occurrences of the token key is less than a total number of occurrences of each other token of the first tuple, wherein the total number of occurrences of the token key and the total number of occurrences of each other token are determined based on a plurality of tuples in the registration list; and

generating an index table with a plurality of indexes each corresponding to a unique token key, the index table including a first index corresponding to the token key of the first tuple, wherein the generating the index table includes forcing the unique token keys into a boundary of memory with modulus, wherein the boundary is defined by a prime number,

wherein, when two or more tuples of the registration list are indexed by the token key, the first index includes two or more unique offsets indicating respective locations in the registration list of the two or more tuples, wherein each of the two or more tuples includes a respective plurality of tokens and each of the respective plurality of tokens includes the token key.

2. The method of claim 1, wherein at least one of the plurality of data elements is a word defined by a character pattern of one or more consecutive essential characters.

3. The method of claim 1, wherein at least one data element is an expression element defined by a character pattern matching a predefined expression pattern, the predefined expression pattern representing at least two words and a separator between the words.

4. The method of claim 1, wherein each of the tokens of the plurality of tokens is a numerical representation of a corresponding data element.

5. The method of claim 1, further comprising:

determining an end of the record by recognizing a predefined delimiter indicating the end of the record.

6. The method of claim 1, wherein each tuple of the registration list is associated with only one record of one or more data files.

7. The method of claim 1, wherein the index includes a document identifier associated with the first offset, the document identifier representing the data file containing the record.

8. The method of claim 1, wherein the selecting one of the plurality of tokens as a token key for the first tuple further includes:

generating a count table including a sum of occurrences of each token in the registration list; and

searching the count table to identify the one of the plurality of tokens having a lowest sum of occurrences relative to each other token in the first tuple.

9. An apparatus, comprising:

one or more registration modules operable to generate a registration list having a plurality of tuples, each tuple representing a specified combination of data elements; and

a processor operable to execute instructions associated with the one or more registration modules, including;

extracting a plurality of data elements from a record of a data file;

tokenizing the plurality of data elements into a plurality of tokens;

storing the plurality of tokens in a first tuple of the registration list;

selecting one of the plurality of tokens as a token key for indexing the first tuple, wherein a total number of occurrences of the token key is less than a total number of occurrences of each other token of the first tuple, wherein the total number of occurrences of the token key and the total number of occurrences of each other token are determined based on a plurality of tuples in the registration list; and

generating an index table with a plurality of indexes each corresponding to a unique token key, the index table including a first index corresponding to the token key of the first tuple, wherein the generating the index table includes forcing the unique token keys into a boundary of memory with modulus, wherein the boundary is defined by a prime number,

wherein, when two or more tuples of the registration list are indexed by the token key, the first index includes two or more unique offsets indicating respective locations in the registration list of the two or more tuples, wherein each of the two or more tuples includes a respective plurality of tokens and each of the respective plurality of tokens includes the token key.

10. The apparatus of claim 9, wherein at least one data element is an expression element defined by a character pattern matching a predefined expression pattern, the predefined expression pattern representing at least two words and a separator between the words.

11. The apparatus of claim 9, wherein the processor is operable to perform further instructions, comprising:

determining an end of the record by recognizing a predefined delimiter indicating the end of the record.

12. The apparatus of claim 11, wherein each tuple of the registration list is associated with only one record of one or more data files.

13. The apparatus of claim 9, wherein the selecting one of the plurality of tokens as a token key for the first tuple further includes:

generating a count table including a sum of occurrences for each token in the registration list; and

searching the count table to identify the one of the plurality of tokens having a lowest sum of occurrences relative to each other token in the first tuple.

14. A method to be executed by a processor in a network environment, comprising:

identifying a start of a first data element in a record of a data file;

determining the first data element is an expression element if a first string of characters beginning at the start of the first data element matches a predefined expression pattern, the predefined expression pattern representing at least two words and a separator between the two words;

extracting the expression element;

tokenizing the expression element into a first token;

storing the first token in a first tuple of a registration list;

selecting the first token as a token key for indexing the first tuple, wherein a total number of occurrences of the token key is less than a total number of occurrences of each one of one or more other tokens of the first tuple, wherein the total number of occurrences of the token key and the total number of occurrences of each one of the one or more other tokens are determined based on a plurality of tuples in the registration list; and

generating an index table with a plurality of indexes each corresponding to a unique token key, the index table including a first index corresponding to the token key of the first tuple, wherein the generating the index table includes forcing the unique token keys into a boundary of memory with modulus, wherein the boundary is defined by a prime number.

15. The method of claim 14, further comprising:

normalizing the expression element prior to the tokenizing the expression element.

16. The method of claim 15, wherein the normalizing includes eliminating the separator from the expression element.

17. The method of claim 14, wherein the predefined expression pattern is a longest one of a plurality of predefined expression patterns that matches any string of characters beginning at the start of the first data element.

18. The method of claim 14, further comprising:

identifying a start of a second data element in the record;

determining the second data element is a word, wherein none of a plurality of predefined expression patterns matches a second string of characters beginning at the start of the second data element;

extracting the word;

tokenizing the word into a second token; and

storing the second token in the first tuple of the registration list.

19. The method of claim 14, wherein a predefined delimiter separates each consecutive pair of records in the data file.

20. The method of claim 14, wherein a predefined delimiter is located at an end of the record in the data file, the predefined delimiter selected from a group consisting of a semi-colon, a line feed, a carriage return, a colon, and a period.

21. The method of claim 14, wherein each tuple of the registration list is associated with only one record of the data file.

22. The method of claim 14, wherein each of the at least two words is defined by a character pattern of one or more consecutive essential characters.

23. The method of claim 22, wherein the essential characters are selected from a group consisting of numbers, letters of a written language, and symbols representing speech segments of a written language.

24. The method of claim 14, wherein the separator is not equivalent to an essential character, and wherein the separator is differentiated from a predefined delimiter if the separator is equivalent to the predefined delimiter, the predefined delimiter indicating an end of the record.

25. An apparatus, comprising:

a registration list module; and

a processor operable to execute instructions associated with the registration list module, including;

identifying a start of a first data element in a record of a data file;

determining the first data element is an expression element if a first string of characters beginning at the start of the first data element matches a predefined expression pattern, the predefined expression pattern representing at least two words and a separator between the two words;

extracting the expression element;

tokenizing the expression element into a first token;

storing the first token in a first tuple of the registration list;

selecting the first token as a token key for indexing the first tuple, wherein a total number of occurrences of the token key is less than a total number of occurrences of each one of one or more other tokens of the first tuple, wherein the total number of occurrences of the token key and the total number of occurrences of each one of the one or more other tokens are determined based on a plurality of tuples in the registration list; and

generating an index table with a plurality of indexes each corresponding to a unique token key, the index table including a first index corresponding to the token key of the first tuple, wherein the generating the index table includes forcing the unique token keys into a boundary of memory with modulus, wherein the boundary is defined by a prime number.

26. The apparatus of claim 25, wherein the processor is operable to perform further instructions, comprising:

normalizing the expression element prior to the tokenizing the expression element, wherein the normalizing includes eliminating the separator from the expression element.

27. The apparatus of claim 25, wherein the predefined expression pattern is a longest one of a plurality of predefined expression patterns that matches any string of characters beginning at the start of the first data element.

28. The apparatus of claim 25, wherein the processor is operable to perform further instructions, comprising:

identifying a start of a second data element in the record;

determining the second data element is a word, wherein none of a plurality of predefined expression patterns matches a second string of characters beginning at the start of the second data element;

extracting the word;

tokenizing the word into a second token;

and storing the second token in the first tuple of the registration list.

29. The apparatus of claim 25, wherein a predefined delimiter separates each consecutive pair of records in the data file.

30. The apparatus of claim 25, wherein each tuple of the registration list is associated with only one record of the data file.

31. At least one non-transitory machine readable medium having instructions stored thereon that, when executed, cause one or more processors to:

extract a plurality of data elements from a record of a data file;

tokenize the plurality of data elements into a plurality of tokens;

store the plurality of tokens in a first tuple of a registration list;

select one of the plurality of tokens as a token key for indexing the first tuple, wherein a total number of occurrences of the token key is less than a total number of occurrences of each other token of the first tuple, wherein the total number of occurrences of the token key and the total number of occurrences of each other token are determined based on a plurality of tuples in the registration list; and

generate an index table with a plurality of indexes each corresponding to a unique token key, the index table including a first index corresponding to the token key of the first tuple, wherein the generating the index table includes forcing the unique token keys into a boundary of memory with modulus, wherein the boundary is defined by a prime number,

wherein, when two or more tuples of the registration list are indexed by the token key, the first index includes two or more unique offsets indicating respective locations in the registration list of the two or more tuples, wherein each of the two or more tuples includes a respective plurality of tokens and each of the respective plurality of tokens includes the token key.

32. The at least one machine readable medium of claim 31, wherein at least one of the plurality of data elements is a word defined by a character pattern of one or more consecutive essential characters.

33. The at least one machine readable medium of claim 31, wherein at least one data element is an expression element defined by a character pattern matching a predefined expression pattern, the predefined expression pattern representing at least two words and a separator between the words.

34. The at least one machine readable medium of claim 31, wherein the instructions, when executed, cause the one or more processors to:

determine an end of the record by recognizing a predefined delimiter indicating the end of the record.

35. The at least one machine readable medium of claim 31, wherein each tuple of the registration list is associated with only one record of one or more data files.

36. The at least one machine readable medium of claim 31, wherein the index includes a document identifier associated with the first offset, the document identifier representing the data file containing the record.

37. The at least one machine readable medium of claim 31, wherein to select the one of the plurality of tokens as a token key for the first tuple, the instructions, when executed, cause the one or more processors to:

generate a count table including a sum of occurrences of each token in the registration list; and

search the count table to identify the one of the plurality of tokens having a lowest sum of occurrences relative to each other token in the first tuple.

38. At least one non-transitory machine readable medium having instructions stored thereon that, when executed, cause one or more processors to:

identify a start of a first data element in a record of a data file;

determine the first data element is an expression element if a first string of characters beginning at the start of the first data element matches a predefined expression pattern, the predefined expression pattern representing at least two words and a separator between the two words;

extract the expression element;

tokenize the expression element into a first token;

store the first token in a first tuple of a registration list;

select the first token as a token key for indexing the first tuple, wherein a total number of occurrences of the token key is less than a total number of occurrences of each one of one or more other tokens of the first tuple, wherein the total number of occurrences of the token key and the total number of occurrences of each one of the one or more other tokens are determined based on a plurality of tuples in the registration list; and

generate an index table with a plurality of indexes each corresponding to a unique token key, the index table including a first index corresponding to the token key of the first tuple, wherein the generating the index table includes forcing the unique token keys into a boundary of memory with modulus, wherein the boundary is defined by a prime number.

39. The at least one machine readable medium of claim 38, wherein the instructions, when executed, cause the one or more processors to:

normalize the expression element prior to the tokenizing the expression element.

40. The at least one machine readable medium of claim 38, wherein the instructions, when executed, cause the one or more processors to:

identify a start of a second data element in the record;

determine the second data element is a word, wherein none of a plurality of predefined expression patterns matches a second string of characters beginning at the start of the second data element;

extract the word;

tokenize the word into a second token; and

store the second token in the first tuple of the registration list.

41. The at least one machine readable medium of claim 38, wherein a predefined delimiter separates each consecutive pair of records in the data file.

42. The at least one machine readable medium of claim 38, wherein a predefined delimiter is located at an end of the record in the data file, the predefined delimiter selected from a group consisting of a semi-colon, a line feed, a carriage return, a colon, and a period.

43. The at least one machine readable medium of claim 38, wherein each tuple of the registration list is associated with only one record of the data file.

44. The at least one machine readable medium of claim 38, wherein the separator is not equivalent to an essential character, and wherein the separator is differentiated from a predefined delimiter if the separator is equivalent to the predefined delimiter, the predefined delimiter indicating an end of the record.