Systems and methods for deduplicating data transferred via physical storage media

• US 8,650,157 B1
• Filed: 02/10/2010
• Issued: 02/11/2014
• Est. Priority Date: 02/10/2010
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Jha, Mohit, Tanksale, Deepak
• Primary Examiner(s): Bhatia, Ajay
• Assistant Examiner(s): SHECHTMAN, CHERYL MARIA
• Application Number:
  US12/703,468
• Time in Patent Office: 1,462 Days
• Field of Search: 707/697
• US Class Current: 707/640
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A computer-implemented method for deduplicating data transferred via physical storage media may include: 1) identifying data stored at a first location that is to be transferred to a second location via a physical storage medium, 2) deduplicating the identified data, 3) storing the deduplicated data on the physical storage medium, and then, after the physical storage medium has been physically transported from the first location to the second location, 4) transferring the deduplicated data from the physical storage medium to a storage device at the second location.

Claims

1. A computer-implemented method for deduplicating data transported via physical storage media, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:

identifying data stored on a storage device at a first physical location that is to be;

transferred to a physical storage medium that is external to the storage device at the first physical location; and

physically transported from the first physical location to a second physical location via the physical storage medium;

identifying a deduplication agent preloaded onto the physical storage medium by a storage vendor;

deduplicating the identified data using the deduplication agent preloaded onto the physical storage medium by the storage vendor, wherein deduplicating the identified data comprises;

determining that an instance of at least one data segment within the identified data is stored on a storage device at the second physical location that is external to the physical storage medium;

creating, on the physical storage medium that is physically transported from the first location to the second location, a reference to the instance of the data segment that is stored on the storage device at the second physical location as opposed to storing the data segment on the physical storage medium;

storing the deduplicated data on the physical storage medium;

after the physical storage medium has been physically transported from the first physical location to the second physical location via the physical storage medium, transferring the deduplicated data from the physical storage medium to the storage device at the second physical location.

2. The method of claim 1, wherein identifying the data comprises at least one of:

identifying data stored on the storage device at the first physical location that is to be backed up to the storage device at the second physical location via the physical storage medium;

identifying data stored on the storage device at the first physical location that is to be restored to the storage device at the second physical location via the physical storage medium.

3. The method of claim 1, wherein deduplicating the identified data comprises one of:

deduplicating the identified data prior to storing the identified data on the physical storage medium;

deduplicating the identified data subsequent to storing the identified data on the physical storage medium.

4. The method of claim 1, wherein determining that an instance of the data segment is stored on the storage device at the second physical location comprises:

creating a hash of the data segment;

using the hash to query the storage device at the second physical location;

receiving a response from the storage device at the second physical location that indicates that an instance of the data segment is stored on the storage device at the second physical location.

5. The method of claim 1, wherein storing the deduplicated data on the physical storage medium comprises at least one of:

encrypting the deduplicated data using a symmetric encryption key;

encrypting the deduplicated data using an encryption key that is derived from the deduplicated data.

6. The method of claim 1, wherein transferring the deduplicated data from the physical storage medium to the storage device at the second physical location comprises storing the deduplicated data on the storage device at the second physical location in a deduplicated state.

7. The method of claim 1, wherein transferring the deduplicated data from the physical storage medium to the storage device at the second physical location comprises:

restoring the deduplicated data to a non-deduplicated state;

storing the non-deduplicated data on the storage device at the second physical location.

8. A system for deduplicating data transported via physical storage media, the system comprising:

an identification module programmed to identify data stored on a storage device at a first physical location that is to be;

transferred to a physical storage medium that is external to the storage device at the first physical location; and

physically transported from the first physical location to a second physical location via the physical storage medium;

a deduplication module preloaded onto the physical storage medium by a storage vendor, wherein the deduplication module is programmed to deduplicate the identified data at least in part by;

determining that an instance of at least one data segment within the identified data is stored on a storage device at the second physical location that is external to the physical storage medium;

creating, on the physical storage medium that is physically transported from the first location to the second location, a reference to the instance of the data segment that is stored on the storage device at the second physical location as opposed to storing the data segment on the physical storage medium;

a transfer module programmed to;

store the deduplicated data on the physical storage medium;

transfer the deduplicated data from the physical storage medium to the storage device at the second physical location after the physical storage medium has been physically transported from the first physical location to the second physical location via the physical storage medium;

at least one processor configured to execute the identification module, the deduplication module, and the transfer module.

9. The system of claim 8, wherein the identification module identifies the data by at least one of:

identifying data stored on the storage device at the first physical location that is to be backed up to the storage device at the second physical location via the physical storage medium;

identifying data stored on the storage device at the first physical location that is to be restored to the storage device at the second physical location via the physical storage medium.

10. The system of claim 8, wherein the deduplication module deduplicates the identified data prior to the transfer module storing the identified data on the physical storage medium.

11. The system of claim 8, wherein the deduplication module deduplicates the identified data subsequent to the transfer module storing the identified data on the physical storage medium.

12. The system of claim 8, wherein the deduplication module determines that an instance of the data segment is stored on the storage device at the second physical location by:

creating a hash of the data segment;

using the hash to query the storage device at the second physical location;

receiving a response from the storage device at the second physical location that indicates that an instance of the data segment is stored on the storage device at the second physical location.

13. The system of claim 8, wherein the transfer module stores the deduplicated data on the physical storage medium by at least one of:

encrypting the deduplicated data using a symmetric encryption key;

encrypting the deduplicated data using an encryption key that is derived from the deduplicated data.

14. The system of claim 8, wherein the transfer module transfers the deduplicated data from the physical storage medium to the storage device at the second physical location by storing the deduplicated data on the storage device at the second physical location in a deduplicated state.

15. The system of claim 8, wherein the transfer module transfers the deduplicated data from the physical storage medium to the storage device at the second physical location by:

restoring the deduplicated data to a non-deduplicated state;

storing the non-deduplicated data on the storage device at the second physical location.

16. A non-transitory computer-readable-storage medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

identify data stored on a storage device at a first physical location that is to be;

transferred to a physical storage medium that is external to the storage device at the first physical location; and

physically transported from the first physical location to a second physical location via the physical storage medium;

identify a deduplication agent preloaded onto the physical storage medium by a storage vendor;

deduplicate the identified data using the deduplication agent preloaded onto the physical storage medium by the storage vendor, wherein deduplicating the identified data comprises;

determining that an instance of at least one data segment within the identified data is stored on a storage device at the second physical location that is external to the physical storage medium;

creating, on the physical storage medium that is physically transported from the first location to the second location, a reference to the instance of the data segment that is stored on the storage device at the second physical location as opposed to storing the data segment on the physical storage medium;

store the deduplicated data on the physical storage medium;

transfer the deduplicated data from the physical storage medium to the storage device at the second physical location after the physical storage medium has been physically transported from the first physical location to the second physical location via the physical storage medium.