Eliminating duplicate data by sharing file system extents

• US 8,825,969 B2
• Filed: 11/29/2010
• Issued: 09/02/2014
• Est. Priority Date: 11/30/2009
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Gupta, Vivek, Pyati, Ameet, Singhal, Satish, Saxena, Pawan
• Primary Examiner(s): Bradley, Matthew
• Assistant Examiner(s): TSUI, DANIEL D
• Application Number:
  US12/955,770
• Publication Number:
  US 20110145523A1
• Time in Patent Office: 1,373 Days
• Field of Search: None
• US Class Current: 711/161
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0664 at device level, e.g. emulation of a storage device or system
• CPC Class Codes: G06F 3/0686 Libraries, e.g. tape libraries, jukebox

Abstract

A hardware and/or software facility to enable emulated storage devices to share data stored on physical storage resources of a storage system. The facility may be implemented on a virtual tape library (VTL) system configured to back up data sets that have a high level of redundancy on multiple virtual tapes. The facility organizes all or a portion of the physical storage resources according to a common store data layout. By enabling emulated storage devices to share data stored on physical storage resources, the facility enables deduplication across the emulated storage devices irrespective of the emulated storage device to which the data is or was originally written, thereby eliminating duplicate data on the physical storage resources and improving the storage consumption of the emulated storage devices on the physical storage resources.

Claims

1. A method performed by a computer system, comprising:

receiving a data set written to a virtual storage device;

partitioning the received data set into multiple sections of data, the partitioning including identifying at least one anchor point surrounded by a data region identified for potential deduplication within the received data set for partitioning into one or more variable sized sections of data, and partitioning remaining portions of the received data set that do not identify an anchor into one or more fixed sized sections of data;

for each section of data,

determining whether the section of data includes the duplicate data, wherein the duplicate data is data that was previously written to another virtual storage device and stored on a physical storage device at a storage location;

in response to determining that a section of data includes the duplicate data, performing deduplication in relation to the duplicate data; and

generating a descriptor, for the received data set, that identifies the storage location of each section of data previously written to another virtual storage device and stored on the physical storage device, rather than storing the corresponding duplicate data at another storage location on the physical storage device.

2. The method of claim 1 further comprising:

in response to determining that a section of data does not include duplicate data, allocating storage space on the physical storage device to store the section of data; and

including a reference to the allocated storage space in the generated descriptor.

3. The method of claim 1 wherein the virtual storage devices are virtual tapes of a virtual tape library system.

4. The method of claim 3 wherein the physical storage device is represented as a number of common store units, a portion of which are allocated to and shared between the virtual tapes of the virtual tape library system.

5. The method of claim 1 wherein the received data set is partitioned into 3 megabytes sections of uncompressed data.

6. The method of claim 1 wherein the deduplication is a sampling based deduplication technique.

7. The method of claim 1, wherein the descriptor is generated based on a data layout associated with the physical storage device that enables the received data set to be shared by the virtual storage device and another virtual storage device, wherein the storage location is represented in the data layout by a common store extent that includes one or more contiguous common store units that include a portion of each section of data previously written to another virtual storage device and stored on the physical storage device.

8. A virtual tape library (VTL) system comprising:

a processor;

a component coupled to the processor to;

receive a data set written to a virtual storage device;

partition the received data set into multiple sections of data, the partitioning including identifying at least one anchor point surrounded by a data region identified for potential deduplication within the received data set for partitioning into one or more variable sized sections of data, and partitioning remaining portions of the received data set that do not identify an anchor into one or more fixed sized sections of data;

for each section of data,

determine whether the section of data includes the duplicate data, wherein the duplicate data is data that was previously written to another virtual storage device and stored on a physical storage device at a storage location;

in response to determining that a section of data includes the duplicate data, perform deduplication in relation to the duplicate data; and

generate a descriptor, for the received data set, that identifies the storage location of each section of data previously written to another virtual storage device and stored on the physical storage device, rather than storing the corresponding duplicate data at another storage location on the physical storage device.

9. The VTL system of claim 8, wherein the component is further configured to:

in response to determining that a section of data does not include duplicate data, allocate storage space on the physical storage device to store the section of data; and

include a reference to the allocated storage space in the generated descriptor.

10. The VTL system of claim 8, wherein the virtual storage devices are virtual tapes of a virtual tape library system.

11. The VTL system of claim 8, wherein the physical storage device is represented as a number of common store units, a portion of which are allocated to and shared between the virtual tapes of the virtual tape library system.

12. The VTL system of claim 8, wherein the received data set is partitioned into 3 megabytes sections of uncompressed data.

13. The VTL system of claim 8, wherein the deduplication is a sampling based deduplication technique.

14. A system comprising:

a physical storage device to store a data set written to a virtual storage device; and

the virtual storage device coupled to the physical storage device to facilitate backup operations associated with the data set, the virtual storage device configured to;

receive a data set written to a virtual storage device;

partition the received data set into multiple sections of data, the partitioning including to identify at least one anchor point surrounded by a data region identified for potential deduplication within the received data set for partitioning into one or more variable sized sections of data, and to partition remaining portions of the received data set that do not identify an anchor into one or more fixed sized sections of data;

for each section of data,

determine whether the section of data includes the duplicate data, wherein the duplicate data is data that was previously written to another virtual storage device and stored on the physical storage device at a storage location;

in response to determining that a section of data includes the duplicate data, to perform deduplication in relation to the duplicate data; and

generate a descriptor, for the received data set, that identifies the storage location of each section of data previously written to another virtual storage device and stored on the physical storage device, rather than storing the corresponding duplicate data at another storage location on the physical storage device.

15. The system of claim 14, wherein the virtual storage device is further configured to:

in response to determining that a section of data does not include duplicate data, allocate storage space on the physical storage device to store the section of data; and

include a reference to the allocated storage space in the generated descriptor.

16. The system of claim 14, wherein the virtual storage devices are virtual tapes of a virtual tape library system.

17. The system of claim 14, wherein the physical storage device is represented as a number of common store units, a portion of which are allocated to and shared between the virtual tapes of the virtual tape library system.

18. The system of claim 14, wherein the received data set is partitioned into 3 megabytes sections of uncompressed data.

19. The system of claim 14, wherein the deduplication is a sampling based deduplication technique.