Generating signatures over a document

• US 20050273614A1
• Filed: 06/07/2004
• Published: 12/08/2005
• Est. Priority Date: 06/07/2004
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: de la Iglesia, Erik, Ahuja, Ratinder Paul Singh, Lowe, Rick, Howard, Matthew
• Granted Patent: US 7,434,058 B2
• Time in Patent Office: Days
• US Class Current: 713/176
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/6263 during internet communication, e.g. revealing personal data from cookies
• CPC Class Codes: G06F 21/6272 by registering files or documents with a third party
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures

Abstract

A document accessible over a network can be registered. A registered document, and the content contained therein, cannot be transmitted undetected over and off of the network. In one embodiment, the invention includes maintaining a plurality of stored signatures over a registered document. In one embodiment, the plurality of stored signatures are generated by extracting content from the document, normalizing the extracted content, and generating the plurality of signatures using the normalized content.

Claims

1. A method for generating a plurality of signatures over a document, the method comprising:

extracting content from the document;

normalizing the extracted content; and

generating the plurality of signatures using the normalized content.

2. The method of claim 1, wherein extracting the content from the document comprises identifying a content type of the document, and applying an appropriate text extractor/decoder to the document based on the content type.

3. The method of claim 1, wherein extracting the content from the document further comprises decoding the content contained in the document.

4. The method of claim 1, wherein normalizing the extracted content comprises eliminating irrelevant text from the extracted content.

5. The method of claim 1, wherein normalizing the extracted content comprises comparing delimiting the extracted content into items by comparing strings of the extracted content to a plurality of normalization patterns.

6. The method of claim 1, further comprising tokenizing the normalized content prior to generating the plurality of signatures.

7. The method of claim 6, wherein tokenizing the normalized content comprises creating an ordered list of tokens by converting each delimited item in the normalized content into a token.

8. The method of claim 7, wherein each token comprises a fixed-length hash of a delimited item.

9. The method of claim 7, wherein generating the plurality of signatures comprises selecting a plurality of tokens from the ordered list of tokens and calculating a plurality of hashes using the selected tokens.

10. The method of claim 7, wherein generating the plurality of signatures comprises:

i) selecting M consecutive tokens from the ordered list of tokens, M being a positive integer;

ii) selecting N special tokens from the M consecutive tokens, N being a positive integer not greater than M;

iii) generating a signature by calculating a hash over the N special tokens;

iv) skipping ahead P tokens from the first of the M consecutive tokens, P being a positive integer; and

v) repeating i), ii), iii), and vi) until the plurality of signatures have been generated.

11. The method of claim 10, wherein selecting N special tokens comprises selecting N special tokens based, at least in part, on token priority.

12. The method of claim 11, wherein token priority is based on a type of delimited item used to create the token.

13. The method of claim 11, wherein token priority is based on the frequency of each token.

14. The method of claim 1, further comprising comparing the plurality of signatures with signatures stored in a signature database, and observing registered content contained in the document based on the comparison.

15. An apparatus comprising:

an extractor/decoder to extract content from a document;

a normalizer to normalize the extracted content; and

a signature generator to generate a plurality of signatures using the normalized content.

16. The apparatus of claim 15, further comprising a content classifier to determine a type of content contained in the document, wherein the extractor/decoder used to extract the content from the document is selected from a plurality of extractor/decoders based on the type of content.

17. The apparatus of claim 15, wherein the normalizer eliminates irrelevant text from the extracted content.

18. The apparatus of claim 15, further comprising a tokenizer configured to create an ordered list of tokens by converting each delimited item in the normalized content into a token.

19. The apparatus of claim 18, wherein each token comprises a fixed-length hash of a delimited item.

20. The apparatus of claim 18, wherein the signature generator selects a plurality of tokens from the ordered list of tokens, and calculates a plurality of hashes using the selected tokens.

21. The apparatus of claim 18, wherein the signature generator is configured to generate the plurality of signature by:

i) selecting M consecutive tokens from the ordered list of tokens, M being a positive integer;

ii) selecting N special tokens from the M consecutive tokens, N being a positive integer not greater than M;

iii) generating a signature by calculating a hash over the N special tokens;

iv) skipping ahead P tokens from the first of the M consecutive tokens, P being a positive integer; and

v) repeating i), ii), iii), and vi) until the plurality of signatures have been generated.

22. The apparatus of claim 21, wherein the signature generator selects the N special tokens based, at least in part, on token priority.

23. The apparatus of claim 22, wherein token priority is based on a type of delimited item used to create the token.

24. The apparatus of claim 22, wherein token priority is based on the frequency of each token.

25. The apparatus of claim 15, further comprising search engine to compare the plurality of signatures with signatures stored in a signature database, and a notification module to alert a user that registered content is contained in the document based on the comparison.

26. A machine-readable medium having stored thereon data representing instructions that, when executed by a processor, cause the processor to perform operations comprising:

extracting content from the document;

normalizing the extracted content;

generating a plurality of signatures using the normalized content.

27. The machine-readable medium of claim 26, wherein extracting the content from the document comprises identifying a content type of the document, and applying an appropriate text extractor/decoder to the document based on the content type.

28. The machine-readable medium of claim 26, wherein extracting the content from the document further comprises decoding the content contained in the document.

29. The machine-readable medium of claim 26, wherein normalizing the extracted content comprises eliminating irrelevant text from the extracted content.

30. The machine-readable medium of claim 26, wherein the instructions further cause the processor to tokenize the normalized content prior to generating the plurality of signatures.

31. The machine-readable medium of claim 30, wherein tokenizing the normalized content comprises creating an ordered list of tokens by converting each delimited item in the normalized content into a token.

32. The machine-readable medium of claim 31, wherein each token comprises a fixed-length hash of a delimited item.

33. The machine-readable medium of claim 31, wherein generating the plurality of signatures comprises selecting a plurality of tokens from the ordered list of tokens and calculating a plurality of hashes using the selected tokens.

34. The machine-readable medium of claim 31, wherein generating the plurality of signatures comprises:

i) selecting M consecutive tokens from the ordered list of tokens, M being a positive integer;

ii) selecting N special tokens from the M consecutive tokens, N being a positive integer not greater than M;

iii) generating a signature by calculating a hash over the N special tokens;

iv) skipping ahead P tokens from the first of the M consecutive tokens, P being a positive integer; and

v) repeating i), ii), iii), and vi) until the plurality of signatures have been generated.

35. The machine-readable medium of claim 26, wherein the instructions further cause the processor to compare the plurality of signatures with signatures stored in a signature database, and to observe registered content contained in the document based on the comparison.