High availability distributed deduplicated storage system

• US 9,633,033 B2
• Filed: 01/10/2014
• Issued: 04/25/2017
• Est. Priority Date: 01/11/2013
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Vijayan, Manoj Kumar, Kochunni, Jaidev Oppath, Agrawal, Saurabh, Narulkar, Abhishek
• Primary Examiner(s): Torgrimson, Tyler
• Application Number:
  US14/152,509
• Publication Number:
  US 20140201170A1
• Time in Patent Office: 1,201 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A high availability distributed, deduplicated storage system according to certain embodiments is arranged to include multiple deduplication database media agents. The deduplication database media agents store signatures of data blocks stored in secondary storage. In addition, the deduplication database media agents are configured as failover deduplication database media agents in the event that one of the deduplication database media agents becomes unavailable.

Claims

1. A method of performing a storage operation in a distributed, deduplicated storage system, comprising:

receiving at a first secondary storage computing device of a plurality of secondary storage computing devices a request from a client computing device to backup a file comprising a plurality of data blocks and stored in primary storage,

wherein a first deduplication database computing device of a plurality of deduplication database computing devices communicatively coupled to the first secondary storage computing device is configured to store a first subset of signature blocks based at least in part on a data block distribution policy and is designated as a failover deduplication database computing device for a second deduplication database computing device of the plurality of deduplication database computing devices that is configured to store, based at least in part on the data block distribution policy, a second subset of the signature blocks that is different from and does not overlap with the first subset of the signature blocks,

wherein the plurality of deduplication database computing devices store the signature blocks corresponding to data blocks stored in secondary storage, wherein the data blocks stored in the secondary storage correspond to data blocks stored in primary storage, at least one signature block of the signature blocks comprising a signature of at least one data block of the plurality of data blocks, location information of the at least one data block in the secondary storage, and a reference count value indicative of a quantity of one or more references in the secondary storage to the at least one data block;

in response to the request and using one or more processors, calculating a signature of a particular data block of the plurality of data blocks using a signature function;

identifying the second deduplication database computing device as the deduplication database computing device assigned to store the signature of the particular data block;

determining that the second deduplication database computing device is unavailable; and

querying the first deduplication database computing device for the signature of the particular data block,

the method further comprising at least one of;

based at least in part on an indication from the first deduplication database computing device that the signature of the particular data block does not reside in the first deduplication database computing device, store the signature in a failover index, cause at least one storage device of the of the secondary storage to store a copy of the particular data block, and request the first deduplication database computing device to store the signature of the particular data block and a location of the copy of the particular data block, or

based at least in part on an indication from the first deduplication database computing device that the signature of the particular data block resides in the first deduplication database computing device, cause at least one storage device of the of the secondary storage to store a reference to a copy of the particular data block that is stored in the secondary storage.

2. The method of claim 1, wherein each deduplication database computing device is assigned to store a different group of unique signatures.

3. The method of claim 1, wherein the first deduplication database computing device is assigned to store a first set of signatures and the second deduplication database computing device is assigned to store a second set of signatures different from the first set of signatures.

4. The method of claim 1, wherein each deduplication database computing device of the plurality of deduplication database computing devices is identified as a failover deduplication database computing device to another one of the plurality of deduplication database computing devices.

5. The method of claim 1, further comprising identifying a third deduplication database computing device as the failover deduplication database computing device for the first deduplication database computing device and identifying the second deduplication database computing device as the failover deduplication database computing device for the third deduplication database computing device.

6. The method of claim 1, further comprising calculating a signature of each of the plurality of data blocks using the signature function.

7. A distributed deduplicated storage system, comprising:

a plurality of deduplication database computing devices configured to store signature blocks corresponding to a plurality of data blocks stored in one or more storage devices of secondary storage, at least one signature block of the signature blocks comprising a signature of at least one data block of the plurality of data blocks, location information of the at least one data block in the one or more storage devices, and a reference count value indicative of a quantity of one or more references in the secondary storage to the at least one data block,

wherein a first deduplication database computing device of the plurality of deduplication database computing devices is configured to store a first subset of the signature blocks based at least in part on a data block distribution policy and is designated as a failover deduplication database computing device for a second deduplication database computing device of the plurality of deduplication database computing devices that is configured to store, based at least in part on the data block distribution policy, a second subset of the signature blocks that is different from and does not overlap with the first subset of the signature blocks; and

a plurality of secondary storage computing devices communicatively coupled to the plurality of deduplication database computing devices, each of the plurality of secondary storage computing devices comprising one or more processors and storage, wherein at least one secondary storage computing device of the plurality of secondary storage computing devices further comprises a failover index and is configured to;

receive a request to backup a file comprising a plurality of data blocks and stored in primary storage,

calculate a signature of a particular data block of the plurality of data blocks using a signature function,

identify the second deduplication database computing device as the deduplication database computing device assigned to store the signature of the particular data block,

determine that the second deduplication database computing device is unavailable, and

query the first deduplication database computing device for the signature of the particular data block,

wherein the at least one secondary storage computing device is further configured to at least one of;

based at least in part on an indication from the first deduplication database computing device that the signature of the particular data block does not reside in the first deduplication database computing device, store the signature in the failover index, cause at least one storage device of the one or more storage devices of the secondary storage to store a copy of the particular data block, and request the first deduplication database computing device to store the signature of the particular data block and a location of the copy of the particular data block, or

based at least in part on an indication from the first deduplication database computing device that the signature of the particular data block resides in the first deduplication database computing device, cause at least one storage device of the one or more storage devices of the secondary storage to store a reference to a copy of the particular data block that is stored in the secondary storage.

8. The system of claim 7, wherein each deduplication database computing device is assigned to store unique signatures of different data blocks.

9. The system of claim 7, wherein the first deduplication database computing device is assigned to store a first set of signatures and the second deduplication database computing device is assigned to store a second set of signatures different from the first set of signatures.

10. The system of claim 7, wherein each deduplication database computing device of the plurality of deduplication database computing devices is identified as a failover deduplication database computing device to another one of the plurality of deduplication database computing devices.

11. The system of claim 7, further comprising a third deduplication database computing device identified as the failover deduplication database computing device for the first deduplication database computing device, wherein the second deduplication database computing device is identified as the failover deduplication database computing device for the third deduplication database computing device.

12. The system of claim 7, wherein the at least one of the plurality of secondary storage computing devices is further configured to calculate a signature of each of the plurality of data blocks using the signature function.