Systems and methods for classifying files as candidates for deduplication

• US 8,849,768 B1
• Filed: 03/08/2011
• Issued: 09/30/2014
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Agrawal, Namita
• Primary Examiner(s): Ly, Anh
• Application Number:
  US13/043,441
• Time in Patent Office: 1,302 Days
• Field of Search: 707/692, 707/664, 707/687, 707/650, 707/E17.002, 707/E17.005, 707/E17.044, 707/E17.046, 707/E17.014, 707/E17.089, 707/E17.01, 707/699, 707/690, 707/769, 707/796, 707/803, 707/723, 707/705, 707/E17.007, 707/648, 707/640, 707/674, 715/777, 711/162, 711/104, 711/E12.001, 711/E12.002, 711/111, 711/165, 711/E12.103, 709/206, 709/217
• US Class Current: 707/664
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1737 for reducing power consumption or coping with limited storage space, e.g. in mobile devices saving storage space on storage devices G06F3/0608; power saving in storage systems G06F3/0625
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A computer-implemented method may include identifying at least one file and detecting an event that is suggestive of at least a portion of the file being duplicated in at least one additional file. The computer-implemented method may also include classifying the file as a candidate for deduplication in response to detecting the event. The computer-implemented method may further include maintaining the file'"'"'s candidate-for-deduplication classification for use in prompting a determination on whether the portion of the file is already stored within a storage device.

Claims

1. A computer-implemented method for classifying files as candidates for deduplication, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:

identifying at least a portion of a file;

detecting at least one event that is suggestive of a duplicate instance of the portion of the file already being stored within a storage device prior to determining whether the duplicate instance of the portion of the file is already stored within the storage device, the event comprising an attempt by a user of the computing device to mark the file to indicate that the file is likely to be duplicated on a plurality of computing devices that are backed up at the storage device;

in response to detecting the event, classifying the file as a candidate for deduplication such that the file'"'"'s candidate-for-deduplication classification indicates that the duplicate instance of the portion of the file is likely already stored within the storage device;

maintaining the file'"'"'s candidate-for-deduplication classification for use in prompting a determination on whether the duplicate instance of the portion of the file is already stored within the storage device by maintaining, within the file, an attribute associated with the file that indicates that the file is a candidate for deduplication;

during deduplication or backup of data within a storage system;

identifying, within the file, the attribute associated with the file;

determining, based on the attribute associated with the file, that the file is a candidate for deduplication;

in response to determining that the file is a candidate for deduplication, determining whether the portion of the file is already stored within the storage device.

2. The computer-implemented method of claim 1, wherein the event that is suggestive of the duplicate instance of the portion of the file already being stored within the storage device further comprises at least one of:

an attempt by a user of a computing system to mark the file to indicate that the portion of the file is duplicated in at least one additional file;

an attempt to copy the portion of the file into the additional file;

an attempt by a user of a computing system to send the file as an email attachment to a plurality of additional users whose computing devices are backed up by the same backup system;

after downloading the file to a computing system for a first time, an attempt to download the file to the computing system for a second time.

3. The computer-implemented method of claim 1, wherein the attribute associated with the file indicates that the portion of the file is duplicated in at least one additional file.

4. The computer-implemented method of claim 1, wherein maintaining the file'"'"'s candidate-for-deduplication classification comprises:

maintaining, within a database, the attribute associated with the file that indicates that the duplicate instance of the portion of the file is likely already stored within the storage device.

5. The computer-implemented method of claim 1, wherein maintaining the attribute associated with the file comprises setting at least one binary digit that indicates that the duplicate instance of the portion of the file is likely already stored within the storage device.

6. The computer-implemented method of claim 1, wherein the attribute associated with the file comprises:

metadata that identifies a number indicating how many additional files are expected to comprise a duplicate instance of the portion of the file.

7. The computer-implemented method of claim 1, further comprising:

detecting one or more attempts to modify the file;

determining that the number of detected attempts to modify the file is above a predetermined threshold;

in response to determining that the number of detected attempts to modify the file is above the predetermined threshold, removing the file'"'"'s candidate-for-deduplication classification.

8. The computer-implemented method of claim 1, wherein classifying the file as a candidate for deduplication comprises classifying the file as a candidate for source-side deduplication that is performed on the file prior to sending any portion of the file to the storage device.

9. The computer-implemented method of claim 1, wherein classifying the file as a candidate for deduplication comprises classifying the file as a candidate for in-band deduplication that is performed on the file while the file is transitioning to the storage device.

10. The computer-implemented method of claim 1, wherein classifying the file as a candidate for deduplication comprises classifying the file as a candidate for variable-block deduplication that facilitates deduplicating variable-sized portions of the file.

11. A computer-implemented method for determining whether files are candidates for deduplication, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:

identifying at least a portion of a file;

identifying a classification assigned to the file that is suggestive of a duplicate instance of the portion of the file already being stored within a storage device by identifying, within the file, an attribute associated with the file that indicates that the file is a candidate for deduplication, the classification having been assigned to the file in response to an attempt by a user of a computing system to mark the file to indicate that the file is likely to be duplicated on a plurality of computing devices that are backed up at the storage device;

determining, based on the classification assigned to the file, that the file is a candidate for deduplication prior to determining whether the duplicate instance of the portion of the file is already stored within the storage device;

in response to determining that the file is a candidate for deduplication,

determining whether the duplicate instance of the portion of the file is already stored within the storage device.

12. The computer-implemented method of claim 11, wherein identifying the portion of the file comprises identifying a request to copy the portion of the file to the storage device.

13. The computer-implemented method of claim 11, further comprising:

determining that the duplicate instance of the portion of the file is already stored within the storage device;

in response to determining that the duplicate instance of the portion of the file is already stored within the storage device, refraining from copying the portion of the file to the storage device.

14. The computer-implemented method of claim 11, further comprising:

determining that the duplicate instance of the portion of the file is not already stored within the storage device;

in response to determining that the duplicate instance of the portion of the file is not already stored within the storage device, copying the portion of the file to the storage device.

15. The computer-implemented method of claim 11, further comprising:

determining that the duplicate instance of the portion of the file is already stored within the storage device;

identifying the duplicate instance of the portion of the file stored within the storage device;

upon identifying the duplicate instance of the portion of the file, removing the duplicate instance of the portion of the file from the storage device.

16. The computer-implemented method of claim 11, further comprising:

detecting one or more attempts to modify the file;

determining that the number of detected attempts to modify the file is above a predetermined threshold;

in response to determining that the number of detected attempts to modify the file is above the predetermined threshold, removing the file'"'"'s candidate-for-deduplication classification.

17. A system for classifying files as candidates for deduplication, the system comprising:

at least one processor;

an identification module programmed to direct the processor to identify at least a portion of a file;

a detection module programmed to direct the processor to detect at least one event that is suggestive of a duplicate instance of the portion of the file already being stored within a storage device prior to a determination of whether the duplicate instance of the portion of the file is already stored within the storage device, the event comprising an attempt by a user of a computing device to mark the file to indicate that the file is likely to be duplicated on a plurality of computing devices that are backed up at the storage device;

a classification module programmed to direct the processor to;

classify, in response to detecting the event, the file as a candidate for deduplication such that the file'"'"'s candidate-for-deduplication classification indicates that the duplicate instance of the portion of the file is likely already stored within the storage device;

maintain the file'"'"'s candidate-for-deduplication classification for use in prompting an application to determine whether the duplicate instance of the portion of the file is already stored within the storage device by maintaining, within the file, an attribute associated with the file that indicates that the file is a candidate for deduplication;

a deduplication module programmed to, during deduplication or backup of data within a storage system;

identify, within the file, the attribute associated with the file;

determine, based on the attribute associated with the file, that the file is a candidate for deduplication;

in response to determining that the file is a candidate for deduplication, determine whether the portion of the file is already stored within the storage device.

18. The system of claim 17, wherein the event that is suggestive of the duplicate instance of the portion of the file already being stored within the storage device further comprises at least one of:

an attempt to copy the portion of the file into at least one additional file;

after downloading the file to a computing system for a first time, an attempt to download the file to the computing system for a second time.

19. The system of claim 17, wherein the event that is suggestive of the duplicate instance of the portion of the file already being stored within the storage device further comprises an attempt by a user of a computing system to send the file as an email attachment to a plurality of additional users whose computing devices are backed up by the same backup system.

20. The system of claim 17, wherein the classification module is further programmed to maintain the file'"'"'s candidate-for-deduplication classification by:

maintaining, within a database, the attribute associated with the file that indicates that the duplicate instance of the portion of the file is likely already stored within the storage device.