Distributed deduplicated storage system

• US 10,740,295 B2
• Filed: 01/19/2018
• Issued: 08/11/2020
• Est. Priority Date: 12/14/2010
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Vijayan, Manoj Kumar, Kottomtharayil, Rajiv, Attarde, Deepak Raghunath
• Primary Examiner(s): Torgrimson, Tyler J
• Application Number:
  US15/875,896
• Publication Number:
  US 20190026305A1
• Time in Patent Office: 935 Days
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

A distributed, deduplicated storage system according to certain embodiments is arranged in a parallel configuration including multiple deduplication nodes. Deduplicated data is distributed across the deduplication nodes. The deduplication nodes can be networked together and communicate with one another according using a light-weight, customized communication scheme (e.g., a scheme based on FTP or HTTP). In some cases, deduplication management information including deduplication signatures and/or other metadata is stored separately from the deduplicated data in deduplication management nodes, improving performance and scalability.

Claims

1. A distributed deduplicated storage system comprising:

a client device; and

one or more hardware processors configured with software instructions to perform operations of;

a data media agent, wherein the data media agent is configured to;

receive data from the client device,

split the data into at least first and second data blocks, and

transmit in parallel the first data block to a first deduplication database media agent and the second data block to a second deduplication database media agent;

the first deduplication database media agent having first deduplication management information associated therewith, wherein the first deduplication management information is stored separately from deduplicated data stored in secondary storage devices, the first deduplication database media agent configured to;

determine whether the first data block is stored in a secondary storage device by querying the first deduplication management information; and

the second deduplication database media agent having second deduplication management information associated therewith, wherein the second deduplication management information is stored separately from the deduplicated data stored in the secondary storage devices, the second deduplication database media agent configured to;

determine whether the second data block is stored in a secondary storage device by querying the second deduplication management information.

2. The distributed deduplicated storage system of claim 1, wherein the data transmitted to the data media agent is a file.

3. The distributed deduplicated storage system of claim 1, wherein the data transmitted to the data media agent includes two or more files.

4. The distributed deduplicated storage system of claim 1, wherein the first deduplication database media agent generates a first hash signature of the first data block, the second deduplication database media agent generates a second hash signature of the second data block.

5. The distributed deduplicated storage system of claim 4, wherein the first deduplication database media agent stores a copy of the second hash signature.

6. The distributed deduplicated storage system of claim 1, further including a plurality of data media agents, wherein the plurality of data media agents are in communication with one another and in communication with a plurality of deduplication database media agents.

7. The distributed deduplicated storage system of claim 6, wherein a first data media agent receives a request from the client device to restore a file comprising of one or more data blocks stored on two or more data media agents.

8. The distributed deduplicated storage system of claim 7, wherein the first data media agent transmits a first hash signature of the first data block to the second data media agent.

9. A distributed deduplicated storage system comprising:

a plurality of deduplication nodes each comprising one or more processors, one or more deduplication databases, and one or more storage devices, wherein the plurality of deduplication nodes are in communication with one another;

a first deduplication node of the plurality of deduplication nodes creates a first hash signature of a first data block of a plurality of data blocks associated with a file, wherein a copy of the first hash signature is stored in a first deduplication database, wherein the first deduplication database is stored separately from data stored on the one or more storage devices;

a first media agent that stores a copy of the first data block;

a second media agent that stores a copy of a second data block;

a second deduplication node of the plurality of deduplication nodes creates a second hash signature of the second data block associated with the file,

wherein the first deduplication node stores a copy of the second hash signature in a second deduplication database, wherein the second deduplication database is stored separately from data stored on the one or more storage devices; and

computer hardware configured to;

receive a request for the file comprised of the plurality of data blocks,

determine with the first deduplication node that the copy of the second data block was stored based at least in part on accessing the copy of the second hash signature, and

send a second request for the second data block.

10. The distributed deduplicated storage system of claim 9 wherein the second media agent is configured to respond to the second request using a file-transfer protocol (FTP)-based service routine.

11. The distributed deduplicated storage system of claim 9 wherein the second media agent is configured to perform the request using a hyper-text transfer protocol (HTTP)-based service routine.

12. The distributed deduplicated storage system of claim 9 wherein the second media agent transmits a copy of the second data block to the first media agent in response to the second request for the second data block.

13. The distributed deduplicated storage system of claim 9 wherein the first deduplication node further provides a path identifier associated with the second data block to the second media agent.

14. The distributed deduplicated storage system of claim 9 further comprising determining which of the plurality of deduplication nodes to send a request comprises performing a modulo operation on the second hash signature of the second data block.

15. The distributed deduplicated storage system of claim 9, the plurality of deduplication nodes are in communication with the first media agent and the second media agent.

16. The distributed deduplicated storage system of claim 9, wherein a first data media agent receives a request from a client device to restore a file comprising of one or more data blocks stored on two or more data media agents.

17. The distributed deduplicated storage system of claim 9, wherein the first data media agent transmits the first hash signature of the first data block to the second data media agent.

18. The distributed deduplicated storage system of claim 9, wherein the first media agent stores a copy of the second hash signature.