METHODS AND APPARATUS FOR DEDUPLICATION IN STORAGE SYSTEM

• US 20090132619A1
• Filed: 11/20/2007
• Published: 05/21/2009
• Est. Priority Date: 11/20/2007
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: KANO, Yoshiki, ARAKAWA, Hiroshi
• Granted Patent: US 7,870,105 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 2211/1004 Adaptive RAID, i.e. RAID system adapts to changing circumstances, e.g. RAID1 becomes RAID5 as disks fill up
• CPC Class Codes: G06F 2211/1057 Parity-multiple bits-RAID6, i.e. RAID 6 implementations
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

In one implementation, a storage system comprises host computers, a management terminal and a storage system having block interface to communicate with the host computers/clients. The storage system also incorporates a deduplication capability using chunks (divided storage area). The storage system maintains a threshold (upper limit) with respect to the degree of deduplication (i.e. number of virtual data for one real data) specified by users or the management software. The storage system counts the number of links for each chunk and does not perform deduplication when the number of reduced data for a chunk exceeds the threshold, even if duplication is detected. In another implementation, the storage system additionally incorporates a data migration capability and migrates physical data to high reliability area such as area protected with double parity (i.e. RAID6) when the deduplication level for a chunk exceeds the threshold.

Claims

1. A computerized data storage system comprising:

a. At least one host computer;

b. A management terminal; and

c. A storage system comprising;

i. An interface operable to communicate with the at least one host computer;

ii. A storage device comprising a plurality of data objects; and

a

iii. A deduplication controller operable to perform a deduplication of data stored in the storage device, wherein the deduplication controller maintains a threshold with respect to allowed degree of deduplication, counts a number of links for each data object and does not perform deduplication when the counted number of links for the data object exceeds the threshold even if duplication is detected.

2. The computerized data storage system of claim 1, wherein the interface is a block interface and wherein the data object is a chunk.

3. The computerized data storage system of claim 1, wherein the interface is a network interface and wherein the data object is a file.

4. The computerized data storage system of claim 1, wherein the interface is a tape interface and wherein the data object is a backup file.

5. The computerized data storage system of claim 1, wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to one data object.

6. The computerized data storage system of claim 1, wherein the deduplication controller is operable to perform the deduplication in response to receipt of a write command by the interface.

7. A computerized data storage system comprising:

a. At least one host computer;

b. A management terminal; and

c. A storage system comprising;

i. An interface operable to communicate with the at least one host computer;

ii. A normal reliability storage area;

iii. A high reliability data storage area;

iv. A data migration controller operable to migrate data between the normal reliability storage area and the high reliability data storage area; and

v. A deduplication controller operable to perform deduplication of data stored in the normal reliability data storage area or the high reliability data storage area, wherein the deduplication controller maintains a threshold with respect to allowed degree of deduplication and counts a number of links for each object; and

wherein the deduplication controller is operable to cause the data migration controller to migrate a data object to the high reliability storage area when the counted number of links for the data object exceeds the threshold.

8. The computerized data storage system of claim 7, wherein the deduplication controller is further operable to cause the migration controller to migrate the data object to the normal reliability storage area from the high reliability storage area when the counted number of links for the data object falls below the threshold.

9. The computerized data storage system of claim 7, wherein the interface is a block interface and wherein the data object is a chunk.

10. The computerized data storage system of claim 7, wherein the interface is a network interface and wherein the data object is a file.

11. The computerized data storage system of claim 7, wherein the interface is a tape interface and wherein the data object is a backup file.

12. The computerized data storage system of claim 7, wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to one data object.

13. The computerized data storage system of claim 7, wherein the deduplication controller is operable to perform the deduplication in response to receipt of a write command by the interface.

14. A method performed by a storage system comprising an interface operable to communicate with at least one host computer and at least one storage device comprising a plurality of data objects;

the method comprising;

a. Determining whether a first data is duplicated in the at least one duplicate data object;

b. Maintaining a threshold with respect to allowed degree of deduplication;

c. Counting a number of links for the at least one duplicate data object;

d. If the first data is duplicated in the at lest one duplicate data object and if the counted number of links does not exceed the threshold, performing deduplication of the data in the at least one duplicate data object; and

e. If the counted number of links exceeds the threshold, not performing the deduplication of the data in the at least one duplicate data object.

15. The method of claim 14, wherein the interface is a block interface and wherein the data object is a chunk.

16. The method of claim 14, wherein the interface is a network interface and wherein the data object is a file.

17. The method of claim 14, wherein the interface is a tape interface and wherein the data object is a backup file.

18. The method of claim 14, wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to the duplicate data object.

19. The method of claim 14, further comprising, before the determining, receiving a write request from the at least one host computer, the write request comprising the first data.

20. A method performed by a storage system comprising an interface operable to communicate with at least one host computer and at least one storage device comprising a plurality of data objects;

the method comprising;

a. Determining whether the a data is duplicated in the at least one duplicate data object of the plurality of data objects;

b. Maintaining a threshold with respect to allowed degree of deduplication;

c. Counting a number of links for the at least one duplicate data object;

d. If the first data is duplicated in the at lest one duplicate data object, performing deduplication of the data in the at least one duplicate data object; and

e. If the counted number of links exceeds the threshold, migrating the at least one duplicate data object to a high reliability storage area.

21. The method of claim 20, wherein the interface is a block interface and wherein the data object is a chunk.

22. The method of claim 20, wherein the interface is a network interface and wherein the data object is a file.

23. The method of claim 20, wherein the interface is a tape interface and wherein the data object is a backup file.

24. The method of claim 20, wherein performing a deduplication comprises creating a plurality of links pointing from a plurality of virtual objects to the duplicate data object.

25. The method of claim 20, further comprising, before the determining, receiving a write request from the at least one host computer, the write request comprising the first data.

26. A computer-readable medium storing a set of instruction, the set of instructions, when executed by a storage system comprising an interface operable to communicate with at least one host computer and at least one storage device comprising a plurality of data objects;

causing the storage system to;

a. Determine whether a first data is duplicated in the at least one duplicate data object;

b. Maintain a threshold with respect to allowed degree of deduplication;

c. Count a number of links for the at least one duplicate data object;

d. If the first data is duplicated in the at lest one duplicate data object and if the counted number of links does not exceed the threshold, perform deduplication of the data in the at least one duplicate data object; and

e. If the counted number of links exceeds the threshold, not perform the deduplication of the data in the at least one duplicate data object.

27. The computer-readable medium of claim 26, wherein the set of instructions is further operable to cause the storage system to receive, before the determining, a write request from the at least one host computer, the write request comprising the first data.

29. The computer-readable medium of claim 26, wherein the set of instructions is further operable to cause the storage system to receive, before the determining, a write request from the at least one host computer, the write request comprising the first data.

28. A computer-readable medium storing a set of instruction, the set of instructions, when executed by a storage system comprising an interface operable to communicate with at least one host computer and at least one storage device comprising a plurality of data objects;

causing the storage system to;

a. Determine whether a first data is duplicated in the at least one duplicate data object of the plurality of data objects;

b. Maintain a threshold with respect to allowed degree of deduplication;

c. Count a number of links for the at least one duplicate data object;

d. If the first data is duplicated in the at lest one duplicate data object, perform deduplication of the data in the at least one duplicate data object; and

e. If the counted number of links exceeds the threshold, migrate the at least one duplicate data object to a high reliability storage area.