Methods and apparatus for deduplication in storage system

• US 7,870,105 B2
• Filed: 11/20/2007
• Issued: 01/11/2011
• Est. Priority Date: 11/20/2007
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Kano, Yoshiki, Arakawa, Hiroshi
• Primary Examiner(s): Vo; Tim T.
• Assistant Examiner(s): Ahn; Sangwoo
• Application Number:
  US11/943,494
• Publication Number:
  US 20090132619A1
• Time in Patent Office: 1,148 Days
• Field of Search: None
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 2211/1004 Adaptive RAID, i.e. RAID system adapts to changing circumstances, e.g. RAID1 becomes RAID5 as disks fill up
• CPC Class Codes: G06F 2211/1057 Parity-multiple bits-RAID6, i.e. RAID 6 implementations
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

In one implementation, a storage system comprises host computers, a management terminal and a storage system having block interface to communicate with the host computers/clients. The storage system also incorporates a deduplication capability using chunks (divided storage area). The storage system maintains a threshold (upper limit) with respect to the degree of deduplication (i.e. number of virtual data for one real data) specified by users or the management software. The storage system counts the number of links for each chunk and does not perform deduplication when the number of reduced data for a chunk exceeds the threshold, even if duplication is detected. In another implementation, the storage system additionally incorporates a data migration capability and migrates physical data to high reliability area such as area protected with double parity (i.e. RAID6) when the deduplication level for a chunk exceeds the threshold.

Claims

1. A computerized data storage system comprising:

a. At least one host computer;

b. A management terminal; and

c. A storage system comprising;

i. An interface operable to communicate with the at least one host computer;

ii. A storage device comprising a plurality of data objects; and

iii. A deduplication controller operable to perform a deduplication of data stored in the storage device, wherein the deduplication controller maintains a threshold with respect to allowed degree of deduplication, counts a number of links for each data object and does not perform deduplication when the counted number of links for the data object exceeds the threshold even if duplication is detected;

wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to one data object;

wherein the data object is stored without performing deduplication when the counted number of links for the data object exceeds the threshold even if duplication is detected.

2. The computerized data storage system of claim 1, wherein the interface is a block interface and wherein the data object is a chunk.

3. The computerized data storage system of claim 1, wherein the interface is a network interface and wherein the data object is a file.

4. The computerized data storage system of claim 1, wherein the interface is a tape interface and wherein the data object is a backup file.

5. The computerized data storage system of claim 1, wherein the deduplication controller is operable to perform the deduplication in response to receipt of a write command by the interface.

6. A computerized data storage system comprising:

a. At least one host computer;

b. A management terminal; and

c. A storage system comprising;

i. An interface operable to communicate with the at least one host computer;

ii. A normal reliability storage area;

iii. A high reliability storage area;

iv. A data migration controller operable to migrate data between the normal reliability storage area and the high reliability storage area; and

v. A deduplication controller operable to perform deduplication of data stored in the normal reliability storage area or the high reliability data storage area, wherein the deduplication controller maintains a threshold with respect to allowed degree of deduplication and counts a number of links for each object; and

wherein the deduplication controller is operable to cause the data migration controller to migrate a data object to the high reliability storage area when the counted number of links for the data object exceeds the threshold;

wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to one data object.

7. The computerized data storage system of claim 6, wherein the deduplication controller is further operable to cause the migration controller to migrate the data object to the normal reliability storage area from the high reliability storage area when the counted number of links for the data object falls below the threshold.

8. The computerized data storage system of claim 6, wherein the interface is a block interface and wherein the data object is a chunk.

9. The computerized data storage system of claim 6, wherein the interface is a network interface and wherein the data object is a file.

10. The computerized data storage system of claim 6, wherein the interface is a tape interface and wherein the data object is a backup file.

11. The computerized data storage system of claim 6, wherein the deduplication controller is operable to perform the deduplication in response to receipt of a write command by the interface.

12. A method performed by a storage system comprising an interface operable to communicate with at least one host computer and at least one storage device comprising a plurality of data objects, the method comprising:

a. Determining whether a first data is duplicated in at least one duplicate data object;

b. Maintaining a threshold with respect to allowed degree of deduplication;

c. Counting a number of links for the at least one duplicate data object;

d. If the first data is duplicated in the at least one duplicate data object and if the counted number of links does not exceed the threshold, performing deduplication of the data in the at least one duplicate data object; and

e. If the counted number of links exceeds the threshold, not performing the deduplication of the data in the at least one duplicate data object;

wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to one data object;

wherein the at least one duplicate data object is stored without performing deduplication when the counted number of links for the data object exceeds the threshold eve if deduplication is detected.

13. The method of claim 12, wherein the interface is a block interface and wherein the data object is a chunk.

14. The method of claim 12, wherein the interface is a network interface and wherein the data object is a file.

15. The method of claim 12, wherein the interface is a tape interface and wherein the data object is a backup file.

16. The method of claim 12, further comprising, before the determining, receiving a write request from the at least one host computer, the write request comprising the first data.

17. A method performed by a storage system comprising an interface operable to communicate with at least one host computer and at least one storage device comprising a plurality of data objects, the method comprising:

a. Determining whether a first data is duplicated in at least one duplicate data object of the plurality of data objects;

b. Maintaining a threshold with respect to allowed degree of deduplication;

c. Counting a number of links for the at least one duplicate data object;

d. If the first data is duplicated in the at least one duplicate data object, performing deduplication of the data in the at least one duplicate data object; and

e. If the counted number of links exceeds the threshold, migrating the at least one duplicate data object to a high reliability storage area;

wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to the at least one duplicate data object.

18. The method of claim 17, wherein the interface is a block interface and wherein the data object is a chunk.

19. The method of claim 17, wherein the interface is a network interface and wherein the data object is a file.

20. The method of claim 17, wherein the interface is a tape interface and wherein the data object is a backup file.

21. The method of claim 17, further comprising, before the determining, receiving a write request from the at least one host computer, the write request comprising the first data.