Storage system and method for deduplicating data

• US 9,600,201 B2
• Filed: 03/27/2014
• Issued: 03/21/2017
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Yoshida, Misato, Hayasaka, Mitsuo
• Primary Examiner(s): Maung, Zarni
• Application Number:
  US14/424,177
• Publication Number:
  US 20160259590A1
• Time in Patent Office: 1,090 Days
• Field of Search: 709/203, 709223-229, 709/250
• US Class Current: 1/1
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 41/0213 Standardised network management protocols, e.g. simple network management protocol [SNMP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The present invention provides a storage system capable of performing deduplication among nodes in a cluster configuration with a low overhead. The respective nodes mutually monitor whether there is update of deduplication information in other nodes. When it is determined in its own node that there is update in another node, it reflects the update information in another node to the deduplication information of its own node. Then, each node uses the deduplication information having reflected the deduplication information of other nodes to perform deduplication of the storage data of its own node.

Claims

1. A storage system of a cluster configuration for deduplicating and storing a content from a superior device, wherein

the storage system has two or more nodes with a storage device for storing a content by dividing the same into one or more chunks having a fixed length or a variable length;

wherein each node includes;

a deduplication information including a storage information for storing the content in a given area within the storage device and managing the same, for managing a result of execution of the deduplication; and

a deduplication information update log for reflecting the result of deduplication in its own node to a deduplication information in another node;

when a deduplication information update log not yet reflected to the deduplication information of its own node exists in another node, updates the not-yet-reflected deduplication information update log to the deduplication information of its own node; and

based on the reflected deduplication information of its own node, performs deduplication by determining whether a duplicated chunk exists in a content to be stored in its own node.

2. The storage system according to claim 1, wherein

a unit of deduplication is either a whole content or a chunk.

3. The storage system according to claim 2, wherein

each node further has a log acquisition list for storing and managing the deduplication information update log name acquired from another node, and determine whether the deduplication information update log of another node is already reflected in the deduplication information of its own node or not, based on the log acquisition list of its own node.

4. The storage system according to claim 3, wherein

each node further has a lock file for storing and managing the deduplication information update log name being edited, and a deduplication information update log being edited stored in a lock file of another node will not be reflected in the deduplication information of its own node.

5. The storage system according to claim 4, wherein

a file name in the deduplication information update log includes a date and time information.

6. The storage system according to claim 3, wherein

each node deletes the deduplication information update log of its own node reflected in a deduplication information of another node.

7. The storage system according to claim 3, wherein

a determination on whether the deduplication information update log of its own node is reflected to a deduplication information of another node or not is performed based on a log acquisition list of another node.

8. The storage system according to claim 2, wherein

whether a portion or all of the chunks constituting a content received by the own node is stored in another node or not is determined by acquiring the deduplication information of another node, and

if it is already stored in another node, deduplication of a duplicated chunk is executed by the received content;

and

a content of update of the deduplication information of another node is stored in a deduplication information update log in its own node.

9. The storage system according to claim 1, wherein

when a chunk is newly stored in the storage device, information that an entry to the deduplication information has been added is stored in the deduplication information update log.

10. The storage system according to claim 1, wherein

when a content read request from the superior device is received, a content connecting only a chunk stored in its own nodes or a content connecting the chunk stored in its own node and a chunk stored in another node, are transmitted to the superior device.

11. The storage system according to claim 1, wherein

when a content deletion request is received from the superior device, information that a number of times of reference of a chunk constituting the content has been reduced is stored in the deduplication information update log, and the number of times of the deduplication information is reduced.

12. The storage system according to claim 11, wherein each node further has a container showing a storage information of the chunk in the storage device, and a deletion container list storing the information of a deletion target container;

and

information of a container where the number of times of reference of the deduplication information is zero is registered in to the deletion container list.

13. The storage system according to claim 12, wherein

when a container registered in the deletion container list is to be deleted in each node:

a content of a non-reflected deduplication information update log stored in another node is reflected in a deduplication information of its own node;

whether the number of times of reference of the deletion target container is zero or not is determined, and a container where the number is zero and a deduplication information corresponding to the container are deleted; and

the deletion is stored in the deduplication information update log.

14. The storage system according to claim 1, wherein

a timing for updating the deduplication information of its own node by the deduplication information update log of another node is one of the following:

every given period of time, or when a number of times of writing of contents to its own node or a write data capacity has reached a threshold value stored in advance.

15. A method for eliminating duplicated data of a storage system having a cluster configuration for deduplicating and storing a content from a superior device, wherein

the storage system has two or more nodes with a storage device for storing a content by dividing the same into one or more chunks having a fixed length or a variable length;

wherein each node includes;

a deduplication information including a storage information for storing the content in a given area within the storage device and managing the same, for managing a result of execution of the deduplication; and

a deduplication information update log for reflecting the result of deduplication in its own node to a deduplication information in another node;

when a deduplication information update log not yet reflected to the deduplication information of its own node exists in another node, updates the not-yet-reflected deduplication information update log to the deduplication information of its own node; and

based on the reflected deduplication information of its own node, performs deduplication by determining whether a duplicated chunk exists in a content to be stored in its own node.