Techniques for detecting encrypted data

• US 8,799,671 B2
• Filed: 05/06/2009
• Issued: 08/05/2014
• Est. Priority Date: 05/06/2009
• Status: Active Grant

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Conte, Thomas Martin, Wolfe, Andrew
• Primary Examiner(s): SHAW, YIN CHEN
• Application Number:
  US12/436,644
• Publication Number:
  US 20100287383A1
• Time in Patent Office: 1,917 Days
• Field of Search: 380/28, 713/189, 713/190, 713/193
• US Class Current: 713/189
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 2221/2107 File encryption

Abstract

Techniques are described that generally relate to methods for detecting encryption status of a data file or data stream and selectively encrypting the data file or data stream based on the encryption status of the data file or data stream are generally disclosed. Example methods may include one or more of reading the data file or data stream from a data source, calculating a value of a property of the data file or data stream, comparing the calculated value with a threshold value to determine whether the file is encrypted or unencrypted, and encrypting files that are determined to be unencrypted.

Claims

1. A method to detect an encryption status of a data file stored in a data source and to selectively encrypt the data file based on the encryption status, the method comprising:

reading the data file from the data source;

comparing a data file type of the data file to a set of data file types, wherein the data file types are included in the set according to a likelihood that the encryption status is known;

in response to a determination that the encryption status of the data file type of the data file is unknown, calculating a value of a property of the data file read from the data source, including calculating a distribution of frequencies of occurrence of a plurality of values in the data file read from the data source;

comparing the calculated value with a threshold value to determine whether the data file read from the data source is encrypted or unencrypted, including comparing the distribution of frequencies of occurrence of the plurality of values in the data file to an average distribution of frequencies for a known reference distribution to determine whether the distribution of frequencies of occurrence of the plurality of values in the data file differs significantly from the average distribution of frequencies for the known reference distribution, wherein the known reference distribution is associated with a text file type;

in response to determining that the data file read from the data source is unencrypted as a result of the comparing, encrypting the data file read from the data source and storing the encrypted data file in a cache; and

in response to determining that the data file read from the data source is encrypted as a result of the comparing, storing the data file read from the data source in the cache without further encryption.

2. The method of claim 1, further comprising determining a measured uniformity by measuring uniformity of the calculated frequencies of occurrence;

and

wherein comparing the calculated value with the threshold value comprises comparing the measured uniformity to the threshold value.

3. The method of claim 2, further comprising calculating a Shannon entropy of the frequencies of occurrence, wherein the measured uniformity is measured by the Shannon entropy.

4. The method of claim 3, wherein the Shannon entropy is defined as:

5. The method of claim 4, wherein the Shannon entropy is a value having a unit in bits.

6. The method of claim 5, wherein b is set at 2.

7. The method of claim 1, wherein reading the data file from the data source comprises:

determining file types of a plurality of data files within the data source;

selecting the data file from the plurality of data files within the data source based on the determined file types; and

reading the selected data file from the data source.

8. The method of claim 7, wherein calculating the value of the property of the data file read from the data source comprises calculating a measure of compressibility of the data file read from the data source;

comparing the calculated value with the threshold value comprises comparing the calculated measure of compressibility to a threshold compressibility value.

9. The method of claim 1, wherein

calculating the value of the property of the data file read from the data source comprises calculating a measure of compressibility of the data file read from the data source;

and

comparing the calculated value with the threshold value comprises comparing the calculated measure of compressibility to a threshold compressibility value.

10. The method of claim 7, wherein calculating the value of the property of the data file read from the data source comprises calculating a measure of compressibility of the data file read from the data source;

comparing the calculated value with the threshold value comprises comparing the calculated measure of compressibility to a threshold compressibility value.

20. The method of claim 1, wherein reading the data file from the data source includes reading the data file from the cache.

23. The method of claim 8, wherein the set of preselected file types is a table of preselected file types including a .rar file type.

11. A computing system arranged to detect an encryption status of a data file and selectively encrypt the data file based on the encryption status, the computing system comprising:

a data source including the data file stored therein on a non-transitory computer-readable medium; and

a data processor implemented in hardware and configured to;

read the data file from the data source into system memory;

calculate a value of a property of the read data file, by calculation of a distribution of frequencies of occurrence of a plurality of data values in the data file read from the data source;

compare the calculated value with a threshold value to determine whether the read data file is encrypted or unencrypted, by comparison of the distribution of frequencies of occurrence of the plurality of data values in the data file to an average distribution of frequencies for a known reference distribution to determine whether the distribution of frequencies of occurrence of the plurality of data values differs significantly from the average distribution of frequencies for the known reference distribution;

in response to a determination that the read data file is unencrypted based on the comparison of the calculated value with the threshold value;

determine a type of the data file;

compare the type of the data file to a table of file extensions indicating compressed files, wherein the table of file extensions includes a .rar file extension;

in response to the type of the data file being a match to a file extension in the table of file extensions, determine that the data file is encrypted;

in response to the type of the data file not being a match to file extensions in the table of file extensions, run a compression routine on the data file to generate a compressed data file;

compare a size of the data file to a size of the compressed data file;

based on the comparison of the size of the data file to the size of the compressed data file, determine whether the data file is compressible;

in response to a determination that the data file is compressible, determine that the data file is unencrypted;

in response to a determination that the data file is not compressible, determine that the data file is encrypted;

in response to a determination that the read data file is unencrypted based on the comparison of the calculated value with the threshold value and the comparison of the size of the data file to the size of the compressed data file, encrypt the read data file and store the encrypted data file in a cache; and

in response to a determination that the read data file is encrypted based on the comparison of the calculated value with the threshold value, the comparison of the type of the data file to the table of file extensions, or the comparison of the size of the data file to the size of the compressed data file, store the read data file in the cache without further encryption.

12. The computing system of claim 11, wherein

to calculate the value of the property of the read data file, the data processor is configured to:

calculate frequencies of occurrence of the plurality of values in the data file read from the data source, and

measure uniformity of the calculated frequencies of occurrence; and

to compare the calculated value with the threshold value, the data processor is configured to;

compare the measured uniformity against the threshold value.

13. The computing system of claim 11, wherein the known reference distribution is associated with a particular file type.

14. The computing system of claim 11,

wherein to read the data file from the data source, the data processor is configured to:

determine file types of a plurality of data files within the data source, and

select a subset of data files from the plurality of data files based on the determined file types;

wherein to calculate the value of the property of the read data file, the data processor is configured to calculate the values of the properties for each of the data files from the subset of data files; and

wherein to compare the calculated value with the threshold value, the data processor is configured to compare the calculated values of the properties for each of the data files from the subset of data files to the threshold value to determine whether each retrieved read data file is encrypted or unencrypted.

15. The computing system of claim 14, wherein to select the subset of data files from the plurality of data files based on the determined file types, the data processor is configured to

compare the determined file types of the plurality of data files to a set of preselected file types, wherein the set of preselected file types are chosen according to a likelihood that the encryption status of the file type is known;

and

select the subset of data files from the plurality of data files having file types different than the set of preselected file types.

16. The computing system of claim 13 wherein the known reference distribution is associated with a text file.

21. The computing system of claim 11, wherein the data source including the data file stored therein is stored in the cache.

24. The system of claim 12, wherein the data processor is configured to calculate a Shannon entropy of the frequencies of occurrence, wherein the measured uniformity is measured by the Shannon entropy and wherein the Shannon entropy is defined as:

17. A non-transitory computer accessible medium that includes computer executable instructions stored thereon to detect an encryption status of a data file stored in a data source and to selectively encrypt the data file based on the encryption status, when the computer executable instructions are executed by a processing unit the processing unit is configured to perform a procedure comprising:

reading the data file from the data source;

calculating a value of a property of the data file read from the data source;

comparing the calculated value with a threshold value to determine whether the data file read from the data source is encrypted or unencrypted, including comparing a distribution of frequencies of occurrence of the value of the property in the data file to an average distribution of frequencies for a known reference distribution, wherein the known reference distribution includes a distribution of frequencies of a text file;

in response to determining that the data file read from the data source is unencrypted as stored in the data source as a result of the comparing, encrypting the data file read from the data source and storing the encrypted data file in a cache; and

in response to determining that the data file read from the data source is encrypted as stored in the data source as a result of the comparing, storing the data file read from the data source in the cache without further encryption.

18. The computer accessible medium of claim 17,

wherein calculating the value of the property of the data file read from the data source comprises calculating frequencies of occurrence of a plurality of values in the data file read from the data source, and determining a measured uniformity by measuring uniformity of the calculated frequencies of occurrence;

and

wherein comparing the calculated value with the threshold value comprises comparing the measured uniformity to the threshold value.

19. The computer accessible medium of claim 17,

wherein calculating the value of the property of the data file read from the data source comprises calculating a measure of compressibility of the data file read from the data source;

and

wherein comparing the calculated value with the threshold value comprises comparing the calculated measure of compressibility to a threshold compressibility value.

22. The non-transitory computer accessible medium of claim 17, wherein reading the data file from the data source includes reading the data file from the cache.

25. The computer accessible medium of claim 18, wherein comparing the measured uniformity to the threshold value comprises comparing the distribution of frequencies of occurrence of the plurality of values in the data file to an average distribution of frequencies for the known reference distribution.

26. The computer accessible medium of claim 19, wherein calculating the measure of compressibility of the data file read from the data source comprises comparing a type of the data file read from the data source with file extensions contained in a table of file extensions including a .rar file extension.