DISTRIBUTED DEDUPLICATED STORAGE SYSTEM

• US 20120150826A1
• Filed: 12/13/2011
• Published: 06/14/2012
• Est. Priority Date: 12/14/2010
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Vijayan Retnamma, Manoj Kumar, Kottomtharayil, Rajiv, Attarde, Deepak Raghunath
• Granted Patent: US 9,020,900 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

A distributed, deduplicated storage system according to certain embodiments is arranged in a parallel configuration including multiple deduplication nodes. Deduplicated data is distributed across the deduplication nodes. The deduplication nodes can be networked together and communicate with one another according using a light-weight, customized communication scheme (e.g., a scheme based on FTP or HTTP). In some cases, deduplication management information including deduplication signatures and/or other metadata is stored separately from the deduplicated data in deduplication management nodes, improving performance and scalability.

Claims

1. A method of performing a storage operation in a distributed, deduplicated storage system, comprising:

receiving a request from a client computing device to restore a file comprising a plurality of data blocks, the request received at a first deduplication node of a plurality of deduplication nodes in communication with one another via a network, wherein the requested file is one of a plurality of deduplicated files that are distributed across the deduplication nodes;

in response to the request and using one or more processors, determining whether a copy of a particular data block of the plurality of data blocks in the requested file already exists at the first deduplication node;

if a copy of the particular data block of the plurality of data blocks already exists at the first deduplication node, accessing the particular data block from the first deduplication node; and

if a copy of the particular data block does not already exist at the first deduplication node, requesting the particular data block from another deduplication node of the plurality of deduplication nodes where a copy of the particular data block does exist.

2. The method of claim 1, wherein the other deduplication node transmits a copy of the particular data block in response to said requesting the particular data block, and wherein the method further comprises receiving the transmitted copy of the particular data block from the other deduplication node.

3. The method of claim 1, wherein the plurality of deduplication nodes communicate data blocks between one another without using network shares.

4. The method of claim 3, wherein the plurality of deduplication nodes communicate data blocks between one another without having a shared static mount path configuration.

5. The method of claim 1, wherein said requesting the particular data block is performed using a file-transfer protocol (FTP)-based service routine.

6. The method of claim 1, wherein said requesting the particular data block is performed using a hyper-text transfer protocol (HTTP)-based service routine.

7. The method of claim 1, wherein the method further comprises:

if the particular data block does not already exist at the first deduplication node, accessing a link stored at the first deduplication node and associated with the particular data block; and

based on the link, determining which of the other deduplication nodes a copy of the particular data block is stored on.

8. The method of claim 7, wherein the link includes information indicating which deduplication node the particular block is stored on and location information indicating where the particular data block is stored in the respective deduplication node.

9. The method of claim 7, further comprising, before receiving the request to restore the requested file:

performing a copy operation in which a file including a plurality of data blocks is copied in a deduplicated fashion to the plurality of deduplication nodes,

wherein the copied file comprises a plurality of data blocks, at least one of which is the particular data block,

further wherein the link corresponding to the particular data block was originally received by the first deduplication node during the copy operation and from one of at least one deduplication management node that is separate from the plurality of deduplication nodes and that stores deduplication management information.

10. The method of claim 9, wherein there are a plurality of deduplication management nodes performing the file copy operation comprises, for the particular data block in the copied file:

determining which of the deduplication management nodes to inquire of as to the presence of the particular data block on a deduplication node;

consulting the determined deduplication management node as to whether the particular data block is already stored on a deduplication node;

if the particular data block is already stored on a deduplication node, receiving a link to the particular data block from the determined deduplication management node, the link indicating which deduplication node the data block is already stored on;

storing a copy of the link at the first deduplication node; and

if the particular data block is not already stored on a deduplication node, storing a copy of the particular data block on the first deduplication node.

11. The method of claim 10, wherein said determining which of the deduplication management nodes to inquire of comprises performing a modulo operation on a hash of the particular data block of the copied file.

12. A distributed deduplicated storage system, comprising:

a plurality of deduplication nodes each comprising one or more processors and storage, the deduplication nodes in communication with one another via a network and data blocks corresponding to a plurality of deduplicated files distributed across the deduplication nodes;

a storage manager module configured to;

receive a request for a file comprised of a plurality of data blocks, the request received at a first deduplication node of the plurality of deduplication nodes;

in response to the request, determine whether a copy of a particular data block of the plurality of data blocks already exists at the first data deduplication node;

if a copy of the particular data block already exists at the first deduplication node, access the particular data block from the first deduplication node; and

if a copy of the particular data block does not already exist at the first deduplication node, request the data block from another deduplication node storing a copy of the data block.

13. The distributed deduplicated storage system of claim 12, wherein a data block server executing on the other deduplication node transmits a copy of the particular data block to the first deduplication node in response to the request for the data block.

14. The distributed deduplicated storage system of claim 12, wherein the plurality of deduplication nodes each comprise a respective data block server and wherein the deduplication nodes communicate the data blocks between one another using their respective data block servers and without using network shares.

15. The distributed deduplicated storage system of claim 12, wherein the plurality of deduplication nodes each comprise a respective data block server and wherein the deduplication nodes communicate data blocks between one another using their respective data block servers without having a shared static mount path configuration.

16. The distributed deduplicated storage system of claim 12, wherein the data block server is configured to perform the request using a file-transfer protocol (FTP)-based service routine.

17. The distributed deduplicated storage system of claim 12, wherein the data block server is configured to perform the request using a hyper-text transfer protocol (HTTP)-based service routine.