Partial document content matching using sectional analysis

• US 7,954,151 B1
• Filed: 09/24/2004
• Issued: 05/31/2011
• Est. Priority Date: 10/28/2003
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Reizes, David Alexander, Wiese, James Christopher, Nisbet, James Donald, Hoyt, Stephen Crosby
• Primary Examiner(s): Moise; Emmanuel L
• Assistant Examiner(s): PHAM, LUU T
• Application Number:
  US10/949,552
• Time in Patent Office: 2,440 Days
• Field of Search: 726/22
• US Class Current: 726/22
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: H04L 63/101 Access control lists [ACL]
• CPC Class Codes: H04L 63/105 Multiple levels of security

Abstract

Monitored content is classified to determine partial matches with fragments of documents. A set of redundant keys, or sliding sectional fingerprints, are computed for every possible alignment of the documents with respect to the monitored content. The keys are stored in repositories according to the classification of the corresponding documents. Sectional fingerprints are computed for the monitored content, and the repositories are searched. If a match is found in a repository corresponding to public content, then the monitored data section is classified as public. If a match is found only in a repository corresponding to private content, then the data section is classified as private. Otherwise, the data section is classified as unknown. In a related aspect, a set of policies are searched for a first match in part according to the classifications of the monitored data sections, and a designated action taken if the first match is found.

Claims

1. A method comprising:

computing a set of reference sectional fingerprints corresponding to a reference document having a classification selected from a group comprising at least a public classification and a private classification, wherein successful access to the reference document by discovery agent software results in the reference document having the public classification and otherwise having a classification other than the public classification, and at least one of the reference sectional fingerprints is based at least in part on two or more tokens of the reference document, the tokens being selected from remaining words of the reference document after exclusion of a set of language-dependent words of the reference document;

associating the reference sectional fingerprints with the classification of the reference document;

after the associating, monitoring network traffic via a network interface;

computing a set of traffic sectional fingerprints corresponding to the monitored network traffic, wherein at least one of the traffic sectional fingerprints is based at least in part on two or more tokens of the monitored network traffic;

determining that at least one of the traffic sectional fingerprints matches at least one of the reference sectional fingerprints;

for each respective traffic sectional fingerprint matching at least one of the reference sectional fingerprints associated with the public classification, classifying the respective traffic sectional fingerprint as the public classification;

for each respective traffic sectional fingerprint matching none of the reference sectional fingerprints associated with the public classification and matching at least one of the reference sectional fingerprints associated with the private classification, classifying the respective traffic sectional fingerprint as the private classification;

wherein the act of associating, the acts of computing, and the act of determining are at least in part via one or more central processing units enabled to execute software;

wherein the reference sectional fingerprints and the traffic sectional fingerprints are sliding sectional fingerprints;

wherein the reference document is interpreted as groups of contiguous token strings and each reference sliding sectional fingerprint corresponds to one of the groups of reference document contiguous token strings; and

wherein the monitored network traffic is interpreted as groups of contiguous token strings and each traffic sliding sectional fingerprint corresponds to one of the groups of monitored traffic contiguous token strings.

2. The method of claim 1, wherein:

each group of reference document contiguous token strings being the same predetermined length of tokens.

3. The method of claim 2, wherein:

the groups of reference document contiguous token strings comprise one of every possible contiguous token string of the same predetermined length.

4. The method of claim 2, wherein:

the groups of reference document contiguous token strings do not comprise one of every possible contiguous token string of the same predetermined length.

5. The method of claim 1, wherein:

the reference sliding sectional fingerprints are non-overlapping reference sliding sectional fingerprints.

6. The method of claim 1, wherein:

each group of monitored traffic contiguous token strings being the same predetermined length of tokens.

7. The method of claim 6, wherein:

the groups of monitored traffic contiguous token strings comprise one of every possible contiguous token string of the same predetermined length.

8. The method of claim 6, wherein:

the groups of monitored traffic contiguous token strings do not comprise one of every possible contiguous token string of the same predetermined length.

9. The method of claim 1, wherein:

the traffic sliding sectional fingerprints are non-overlapping traffic sliding sectional fingerprints.

10. The method of claim 1, wherein:

each of the reference document groups and each of the monitored traffic groups are the same predetermined length of tokens.

11. The method of claim 10, wherein:

the groups of reference document contiguous token strings do not comprise one of every possible contiguous token string of the same predetermined length.

12. The method of claim 10, wherein:

the groups of monitored traffic contiguous token strings do not comprise one of every possible contiguous token string of the same predetermined length.

13. The method of claim 1, further comprising:

in response to the act of determining finding a match, performing, at least in part via the one or more central processing units, a first action, and otherwise performing a second action.

14. The method of claim 13, wherein:

the first action comprises a flagging action, and the second action is no action.

15. The method of claim 14, wherein:

the first action further comprises writing information to a log.

16. The method of claim 1, further comprising:

performing, at least in part via the one or more central processing units, an action selected from a group of actions comprising a first and a second action according to the classification of the at least one of the traffic sliding sectional fingerprints.

17. The method of claim 16, wherein:

the first action is no action.

18. The method of claim 17, wherein:

the second action comprises a flagging action.

19. The method of claim 18, wherein:

the second action further comprises writing information to a log.

20. The method of claim 1, further comprising:

in response to the act of determining finding no match, associating, at least in part via the one or more central processing units, the classifications of all of the traffic sliding sectional fingerprints as unknown.

21. The method of claim 20, further comprising:

performing an action, at least in part via the one or more central processing units, selected from a group of actions comprising a first, a second, and an unknown action according to the classification of the at least one of the traffic sectional fingerprints.

22. The method of claim 21, wherein:

the first action is no action.

23. The method of claim 22, wherein:

the second action comprises a first flagging action.

24. The method of claim 23, wherein:

the unknown action comprises a second flagging action.

25. The method of claim 1, wherein:

as a result of the discovery agent software not successfully accessing the reference document, the reference document has the private classification.

26. A method comprising:

computing document sliding sectional fingerprints corresponding to a first and a second document, wherein successful access to the first document by discovery agent software results in the first document having a public classification and unsuccessful access to the second document by the discovery agent software results in the second document having a private classification, and at least one of the document sliding sectional fingerprints is based at least in part on two or more tokens of at least one of the documents, the tokens being selected from remaining words of the at least one document after exclusion of a set of language-dependent words of the at least one document;

associating the document sliding sectional fingerprints with the classification of the corresponding document;

after the associating, monitoring network traffic via a network interface;

computing a set of traffic sliding sectional fingerprints corresponding to the monitored network traffic, wherein at least one of the traffic sliding sectional fingerprints is based at least in part on two or more tokens of the monitored network traffic;

for each of the traffic sliding sectional fingerprints

classifying the traffic sliding sectional fingerprint as the public classification in response to the traffic sliding sectional fingerprint matching at least one of the document sliding sectional fingerprints associated with the public classification, and

classifying the traffic sliding sectional fingerprint as the private classification in response to the traffic sliding sectional fingerprint matching none of the document sliding sectional fingerprints associated with the public classification and matching at least one of the document sliding sectional fingerprints associated with the private classification;

wherein the acts of computing, the act of associating, and the acts of classifying are at least in part via one or more central processing units enabled to execute software;

wherein each document is interpreted as groups of contiguous token sections and each document sliding sectional fingerprint corresponds to one of the groups of document contiguous token sections; and

wherein the monitored network traffic is interpreted as groups of contiguous token sections and each traffic sliding sectional fingerprint corresponds to one of the groups of monitored traffic contiguous token sections.

27. The method of claim 26, further comprising:

classifying the monitored network traffic as the private classification in response to any of the traffic sliding sectional fingerprints being classified as the private classification.

28. The method of claim 27, further comprising:

classifying, at least in part via the one or more central processing units, the monitored network traffic as the public classification in response to

none of the traffic sliding sectional fingerprints being classified as the private classification and

all of the traffic sliding sectional fingerprints being classified as the public classification.

29. The method of claim 28, further comprising:

classifying, at least in part via the one or more central processing units, the monitored network traffic as a third classification in response to

none of the traffic sliding sectional fingerprints being classified as the private classification and

at least one of the traffic sliding sectional fingerprints being not classified as the public classification.

30. The method of claim 29, wherein:

the third classification is an unknown classification.

31. The method of claim 26, further comprising:

for each of the traffic sliding sectional fingerprints

classifying, at least in part via the one or more central processing units, the traffic sliding sectional fingerprint as a third classification in response to the traffic sliding sectional fingerprint matching none of the document sliding sectional fingerprints associated with the public classification and matching none of the document sliding sectional fingerprints associated with the private classification.

32. The method of claim 26, wherein:

as a result of the discovery agent software not successfully accessing the first document, the first document has the private classification, and as a result of the discovery agent software successfully accessing the second document, the second document has the public classification.

33. A system comprising:

a sliding sectional fingerprint unit implemented at least in part via a hardware accelerator and enabled to compute a set of document sliding sectional fingerprints corresponding to a document having a classification selected from a group comprising at least a public classification and a private classification, wherein successful access to the document by discovery agent software results in the document having the public classification and otherwise having a classification other than the public classification, the sliding sectional fingerprint unit being further enabled to compute at least one of the document sliding sectional fingerprints based at least in part on two or more tokens of the document, the tokens being selected from remaining words of the document after exclusion of a set of language-dependent words of the document, and the sliding sectional fingerprint unit being further enabled to associate the document sliding sectional fingerprints with the classification of the document;

a network traffic monitoring and analyzing unit implemented at least in part via the hardware accelerator, coupled to the sliding sectional fingerprint unit, and enabled to compute a set of traffic sliding sectional fingerprints corresponding to monitored network traffic, at least one of the traffic sliding sectional fingerprints being based at least in part on two or more tokens of the monitored network traffic;

wherein the network traffic monitoring and analyzing unit is further enabled to determine that at least one of the traffic sliding sectional fingerprints matches at least one of the document sliding sectional fingerprints;

wherein for each respective traffic sliding sectional fingerprint, the network traffic monitoring and analyzing unit is further enabled to classify the respective traffic sliding sectional fingerprint as the public classification in response to the respective traffic sliding sectional fingerprint matching at least one of the document sliding sectional fingerprints associated with the public classification, and as the private classification in response to the respective traffic sliding sectional fingerprint matching none of the document sliding sectional fingerprints associated with the public classification and matching at least one of the document sliding sectional fingerprints associated with the private classification;

wherein the document is interpreted as groups of contiguous token sections and each document sliding sectional fingerprint corresponds to one of the groups of document contiguous token sections; and

wherein the monitored network traffic is interpreted as groups of contiguous token sections and each traffic sliding sectional fingerprint corresponds to one of the groups of monitored traffic contiguous token sections.

34. The system of claim 33, further comprising:

a repository unit coupled to the sliding sectional fingerprint unit and enabled to store the document sliding sectional fingerprints according to the classification of the document.

35. The system of claim 34, wherein:

the repository unit is further coupled to the traffic monitoring and analyzing unit; and

the traffic monitoring and analyzing unit is further enabled to search the repository unit to determine that at least one of the traffic sliding sectional fingerprints matches at least one of the document sliding sectional fingerprints.

36. The system of claim 33, wherein:

the traffic monitoring and analyzing unit is further enabled to perform a first action in response to the match being determined and otherwise to perform a second action.

37. The system of claim 36, wherein:

the first action comprises a flagging action, and the second action is no action.

38. The system of claim 37, wherein:

the traffic monitoring and analyzing unit further comprises a log, and the flagging action comprises writing information to the log.

39. The system of claim 36, wherein:

the first action comprises a blocking action, and the second action is no action.

40. The system of claim 39, wherein:

the traffic monitoring and analyzing unit further comprises a log, and the blocking action comprises writing information to the log.

41. The system of claim 33, wherein:

the sliding sectional fingerprint unit comprises a processor executing software functions enabling the document sliding sectional fingerprint computation.

42. The system of claim 33, wherein:

the traffic monitoring and analyzing unit comprises a processor executing software functions enabling the traffic sliding sectional fingerprint computation.

43. The system of claim 42, wherein:

the traffic monitoring and analyzing unit further comprises a network interface coupled to the processor; and

the software functions further enable collecting the monitored network traffic via the network interface.

44. The system of claim 33, wherein:

the document sliding sectional fingerprint computation comprises tokenizing the document.

45. The system of claim 44, wherein:

the tokenizing comprises

identifying significant words, and

discarding any combination of white space, punctuation, articles, and conjunctions.

46. The system of claim 44, wherein:

the document sliding sectional fingerprint computation further comprises splitting results of the tokenizing into overlapping sections.

47. The system of claim 46, wherein:

the document sliding sectional fingerprint computation comprises computing a hash function for each overlapping section, and the document sliding sectional fingerprints comprise results of the hash function computation.

48. The system of claim 33, wherein:

the traffic sliding sectional fingerprint computation comprises tokenizing the monitored network traffic.

49. The system of claim 48, wherein:

the tokenizing comprises

identifying significant words, and

discarding any combination of white space, punctuation, articles, and conjunctions.

50. The system of claim 49, wherein:

the traffic sliding sectional fingerprint computation further comprises splitting results of the tokenizing into contiguous non-overlapping sections.

51. The system of claim 50, wherein:

the traffic sliding sectional fingerprint computation comprises computing a hash function for each non-overlapping section, and the traffic sliding sectional fingerprints include results of the hash function computation.

52. The system of claim 49, wherein:

the traffic sliding sectional fingerprint computation further comprises splitting results of the tokenizing into contiguous overlapping sections.

53. The system of claim 52, wherein:

the traffic sliding sectional fingerprint computation comprises computing a hash function for each overlapping section, and the traffic sliding sectional fingerprints comprise results of the hash function computation.

54. The system of claim 33, wherein:

as a result of the discovery agent software not successfully accessing the document, the document has the private classification.

55. A system comprising:

a content appliance comprising

a repository having a first portion and a second portion corresponding respectively to a public classification and a private classification,

a processor coupled to the repository and enabled to execute content appliance software, and

a network interface coupled to the processor;

a computer coupled to the content appliance, the computer enabled to execute computer software;

wherein the content appliance software comprises functions enabling

receiving keys and corresponding file classification from the computer and storing the keys in a portion of the repository selected according to the file classification,

sampling network traffic via the network interface,

computing traffic sliding sectional fingerprints based on the sampled network traffic, at least one of the traffic sliding sectional fingerprints being based at least in part on two or more tokens of the sampled network traffic, and

classifying each traffic sliding sectional fingerprint computed, as the public classification in response to the traffic sliding sectional fingerprint matching any of the keys in the first portion of the repository, as the private classification in response to the traffic sliding sectional fingerprint not matching any of the keys in the first portion of the repository and matching any of the keys in the second portion of the repository, and as a third classification otherwise;

wherein the computer software comprises functions enabling

receiving a file and computing a corresponding set of file sliding sectional fingerprints, at least one of the file sliding sectional fingerprints being computed based at least in part on two or more tokens of the file, the tokens being selected from remaining words of the file after exclusion of a set of language-dependent words of the file, and

providing the file sliding sectional fingerprints and a classification of the file to the content appliance as a set of keys, the classification being the public classification as a result of the file being successfully accessed by discovery agent software and being a classification other than the public classification as a result of the file not being successfully accessed by the discovery agent software;

wherein the file is interpreted as groups of contiguous token sections and each file sliding sectional fingerprint corresponds to one of the groups of file contiguous token sections; and

wherein the sampled network traffic is interpreted as groups of contiguous token sections and each traffic sliding sectional fingerprint corresponds to one of the groups of sampled traffic contiguous token sections.

56. The system of claim 55, wherein:

the computer software comprises further functions enabling

crawling a designated plurality of file hierarchies and providing discovered files to enable receiving and computing file sliding sectional fingerprints.

57. The system of claim 55, wherein:

the receiving comprises parsing the discovered file according to a determined file type and converting content included in the discovered file to canonical information suitable for the computing file sliding sectional fingerprints.

58. The system of claim 57, wherein:

the determined file types comprise any combination of

a text file type;

an ASCII file type;

a Unicode file type;

an e-mail file type;

an e-mail attachment file type;

a compressed file type;

an HTML file type;

an XML file type;

a Microsoft Office application file type; and

an Adobe PDF file type.

59. The system of claim 55, wherein:

the computing file sliding sectional fingerprints and the computing traffic sliding sectional fingerprints comprise tokenizing.

60. The system of claim 59, wherein:

the tokenizing comprises

identifying significant words, and

discarding any combination of white space, punctuation, articles, and conjunctions.

61. The system of claim 55, wherein:

the file contiguous token sections are overlapping file contiguous token sections.

62. The system of claim 61, wherein:

each file sliding sectional fingerprint is based on a hash function computed on a corresponding one of the overlapping file contiguous token sections.

63. The system of claim 55, wherein:

the traffic contiguous token sections are non-overlapping traffic contiguous token sections.

64. The system of claim 63, wherein:

each traffic sliding sectional fingerprint is based on a hash function computed on a corresponding one of the non-overlapping traffic contiguous token sections.

65. The system of claim 55, wherein:

the traffic contiguous token sections are overlapping traffic contiguous token sections.

66. The system of claim 65, wherein:

each traffic sliding sectional fingerprint is based on a hash function computed on a corresponding one of the overlapping traffic contiguous token sections.

67. The system of claim 55, wherein:

the classification is the private classification as a result of the file not being successfully accessed by the discovery agent software.