Identifying and characterizing electronic files using a two-stage calculation

• US 9,239,924 B2
• Filed: 09/03/2012
• Issued: 01/19/2016
• Est. Priority Date: 04/30/1999
• Status: Expired due to Term

• Current Assignee: Intellectual Ventures I LLC (Intellectual Ventures LLC)
• Original Assignee: Intellectual Ventures I LLC (Intellectual Ventures LLC)
• Inventors: Shuster, Gary Stephen
• Primary Examiner(s): LEWIS, LISA C
• Application Number:
  US13/602,238
• Publication Number:
  US 20120331572A1
• Time in Patent Office: 1,233 Days
• Field of Search: 726 22- 25, 726/30
• US Class Current: 1/1
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 21/564 by virus signature recognition
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2221/2149 Restricted operating environment

Abstract

A computer system includes a server having a memory connected thereto. The server is adapted to be connected to a network to permit remote storage and retrieval of data files from the memory. A file identification application is operative with the server to identify errant files stored in the memory. The file identification application provides the functions of: (1) selecting a file stored in said memory; (2) generating a unique checksum corresponding to the stored fire; (3) comparing said unique checksum to each of a plurality of previously generated checksums, wherein the plurality of previously generated checksums correspond to known errant files; and (4) marking the file for deletion from the memory if the unique checksum matches one of the plurality of previously generated checksums.

Claims

1. A method comprising:

scanning a file directory to identify one or more suspect files stored in a file database;

calculating, by a first computing device, a first value corresponding to a file in the file directory, wherein the first value is calculated using a function on a first portion of the first file that is smaller than an entire size of the file;

the first computing device making an initial determination, based on the first value matching a particular value in a suspect files database, as to whether the file is a suspect file;

if the initial determination determines that the file is a suspect file;

the first computing device calculating a second value corresponding to the file, wherein the second value is calculated using the function on a second portion of the file that is larger than the first portion but smaller than the entire size of the file;

in response to the first computing device determining that the second value matches another particular value stored in the suspect files database;

the first computing device storing information identifying the file as a suspect file on a computer readable storage medium;

the first computing device generating a report that is based, at least in part, upon the identification of the file as a suspect file; and

the first computing device at least assisting in taking at least one remedial measure related to the file; and

if the initial determination determines that the file is not a suspect file;

the first computing device identifying the file as a suspect file based on a review of the file and storing the first value in the suspect files database.

2. The method of claim 1, wherein the suspect files database includes a plurality of values corresponding to a plurality of suspect files, wherein information identifying one or more particular characteristics of the file includes information indicating the file is suspect.

3. The method of claim 1, wherein calculating the first and second values includes calculating first and second checksums for the file.

4. The method of claim 1, wherein at least one of the first portion and the second portion of the file is a non-initial portion of the file.

5. The method of claim 1, further comprising determining, based at least in part on the entire size of the file exceeding a threshold size, that the first value should be calculated and compared to one or more values in the suspect files database.

6. The method of claim 1, further comprising determining, based at least in part on existence of data after an end-of-file marker in the file, that the first value should be calculated and compared to one or more values in the suspect files database.

7. The method of claim 1, wherein the first value is a first checksum and the particular value is a second checksum for a known file having one or more particular characteristics;

wherein making the initial determination that the file is a suspect file includes matching the first checksum to the second checksum.

8. The method of claim 1, wherein the first portion of the file is an initial portion of the file.

9. The method of claim 1, wherein the function used to calculate the first and second values corresponding to the file operates on each byte of data in a given file portion.

10. A non-transitory computer-readable medium having stored thereon instructions that are executable by a processor to cause a first computer system to perform operations comprising:

scanning a file directory to identify one or more suspect files stored in a file database;

calculating a first checksum of a file in the file directory based on a first portion of the file that is smaller than an entire size of the file;

making an initial determination, based on the calculated first checksum, as to whether the first portion of the file has data corresponding to one or more particular characteristics;

if the initial determination determines that the first portion of the file has data corresponding to one or more particular characteristics;

calculating a second checksum on the file, wherein the second checksum is calculated on a second portion of the file that is larger than the first portion but smaller than the entire size of the file;

in response to determining that the second calculated checksum matches a checksum stored in the suspect files database;

storing information indicating that the file is a suspect file;

generating a report that is based, at least in part, upon the identification of the file as a suspect file; and

assisting in taking at least one remedial measure related to the first file; and

if the initial determination determines that the first portion of the file does not have data corresponding to one or more particular characteristics;

identifying the file as a suspect file based on a review of the file and storing the calculated first checksum in the suspect files database.

11. The non-transitory computer-readable medium of claim 10, wherein identifying the file as a suspect file is based, at least in part, on the file having information indicating the file includes copyrighted content.

12. The non-transitory computer-readable medium of claim 10, wherein identifying the file as a suspect file is based, at least in part, on the entire size of the file.

13. The non-transitory computer-readable medium of claim 10, wherein calculating the first checksum and calculating the second checksum uses a single file read operation on the file.

14. The non-transitory computer-readable medium of claim 10, wherein the operations further comprise:

receiving checksum information from an external computer system that hosts at least a copy of a portion of the suspect files database; and

determining that the first and second checksums match known checksums based, at least in part, on, the received checksum information.

15. The non-transitory computer-readable medium of claim 10, wherein the operations further comprise identifying the file as being suspect based, at least in part, on information received from a human review of the file.

16. A computer system, comprising:

a processor; and

a computer-readable storage medium having stored thereon instructions that are executable by the processor to cause the computer system to perform operations comprising;

scanning a file directory to identify at least one suspect content stored in a content database;

calculating a first value corresponding to a first content in the file directory, wherein the first value is calculated based on a first portion of the first content that is smaller than an entire size of the first content;

making an initial determination, based on the first value matching a particular value in a suspect content database corresponding to known content, as to whether the first content may have one or more particular characteristics;

if the initial determination determines that the first content may have the one or more particular characteristics;

calculating a second value corresponding to the first content, wherein the second value is calculated based on a second portion of the first content that is larger than the first portion but smaller than the entire size of the first content;

in response to determining the second value matches an additional value in the suspect content database;

generating a report that is based, at least in part, upon the one or more particular characteristics; and

based on the second value matching the additional value corresponding to the known content, performing a corrective action on the first content; and

if the initial determination determines that the first content does not have the one or more particular characteristics;

identifying the first content as having the one or more particular characteristics based on a review of the first content.

17. The computer system of claim 16, wherein the corrective action is deleting the first content.

18. The computer system of claim 16, wherein the corrective action is preventing transmission of the first content from the computer system across a network.

19. The computer system of claim 16, wherein the operations further comprise:

determining a particular value and the additional value corresponding to the known prohibited content by performing respective calculations on first and second portions of the known prohibited content.

20. The computer system of claim 16, wherein the operations further comprise identifying second content as being prohibited based on a relationship of the second content to the first content.

21. The computer system of claim 16, wherein the operations further comprise:

if the initial determination is made;

storing information identifying the one or more particular characteristics of the first content in the suspect content database; and

if the initial determination is not made;

if the first content is identified as having the one or more particular characteristics based on the review of the first content, storing the one or more particular characteristics of the first content in the suspect content database.