Dedicated client-side signature generator in a networked storage system

• US 10,387,269 B2
• Filed: 11/21/2017
• Issued: 08/20/2019
• Est. Priority Date: 06/13/2012
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Muller, Marcus S., Ngo, David
• Primary Examiner(s): Ortiz Ditren, Belix M
• Application Number:
  US15/820,152
• Publication Number:
  US 20180196720A1
• Time in Patent Office: 637 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1415 at system level
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/1847 specifically adapted to static storage, e.g. adapted to flash memory or SSD
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 : Synchronous replication
• CPC Class Codes: G06F 16/90 : Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 16/95 : Retrieval from the web
• CPC Class Codes: G06F 2201/80 : Database-specific techniques

Abstract

A storage system according to certain embodiments includes a client-side signature repository that includes information representative of a set of data blocks stored in primary storage. During storage operations of a client, the system can generate signatures corresponding to data blocks that are being stored in primary storage. The system can store the generated signatures in the client-side signature repository along with information regarding the location of the corresponding data block within primary storage. As additional instances of the data block are stored in primary storage, the system can store the location of the additional instances in the client-side signature repository.

Claims

1. A method of maintaining a signature repository in a data storage system, the method comprising:

generating, by a signature agent executing on one or more processors, signatures corresponding to a plurality of data units in a native format generated by one or more applications during a copy operation that copies the plurality of data units in the native format from a primary storage subsystem into a non-native backup format in a secondary storage subsystem, the primary storage subsystem that stores the plurality of data units in a native format is separate from the secondary storage subsystem that stores the copies of the plurality of data units in the non-native backup format;

maintaining a primary storage signature repository associated with the primary storage subsystem wherein the primary storage signature repository comprises signature blocks for one or more unique signatures associated with the plurality of data units stored in the native format in the primary storage subsystem;

maintaining a secondary storage signature repository associated with the secondary storage subsystem wherein the secondary storage signature repository comprises signature blocks for one or more unique signatures associated with the copies of the plurality of data units stored in the secondary storage subsystem in the non-native backup format;

wherein when a first data unit stored in a native format in a first location in the primary storage subsystem has the same unique signature as a second data unit stored in a native format in a second location in the primary storage subsystem, modifying the associated signature block in the primary storage signature repository to comprise;

a first data unit entry that identifies the first storage location of the first data unit in the native format in the primary storage subsystem; and

a second data unit entry that identifies the second storage location of the second data unit in the native format in the primary storage subsystem.

2. The method of claim 1, further comprising storing in a signature block a first client identifier that identifies a first client device associated with the first data unit, and a second client identifier that identifies a second client device associated with the second data unit.

3. The method of claim 2, wherein the first client device uses a first data store in the primary storage subsystem that is different than a second data store used by the second client device.

4. The method of claim 1, further comprising:

receiving a query including a plurality of signatures;

comparing the plurality of signatures included in the query with signature blocks in the primary storage signature repository to identify a first set of signatures received in the query that correspond to data units that reside in a primary data store of at least one client computing device of the plurality of client computing devices; and

for at least some of the signatures in the first set of signatures, accessing the corresponding data units from the primary data store of the at least one client computing device.

5. The method of claim 4, further comprising accessing the corresponding data units from the secondary storage subsystem for the signatures included in the query that are not included in the first set of signatures.

6. The method of claim 4, wherein the plurality of signatures included in the query correspond to a set of data units which represent a backed up version of a set of the primary data that is to be restored to the first primary data store.

7. The method of claim 1, further comprising:

in response to receipt of instructions to backup at least a subset of the plurality of data units in the primary storage subsystem, comparing a set of signatures corresponding to the subset of data units with entries in the primary storage signature repository;

based at least in part on the comparing, identifying a set of matching data units that match the subset of data units; and

accessing the set of matching data units for retrieval.

8. The method of claim 6, further comprising:

based at least in part on the comparing, identifying a first group of data units of the subset of data units that do not have a corresponding matching data unit; and

communicating the first group of data units to the secondary storage subsystem.

9. The method of claim 1, wherein the secondary storage subsystem comprises deduplicated data.

10. The method of claim 1, wherein the primary data store of at least one of a plurality of client computing devices comprises deduplicated data.

11. A storage system, comprising:

a signature repository agent executing on one or more processors,

a signature repository agent that executes on one or more processors the signature repository agent generates signatures corresponding to the plurality of data units in the a primary storage subsystem in a native format generated by one or more applications during a copy operation that copies the plurality of data units in the native format into a non-native backup format in a secondary storage subsystem, the primary storage subsystem that stores the plurality of data units in a native format is separate from the secondary storage subsystem that stores the copies of the plurality of data units in the non-native backup format;

a primary storage signature repository associated with the primary storage subsystem wherein the primary storage signature repository comprises signature blocks for one or more unique signatures associated with the plurality of data units in the native format in the primary storage subsystem;

a secondary storage signature repository associated with the secondary storage subsystem wherein the secondary storage signature repository comprises signature blocks for one or more unique signatures associated with the copies of the plurality of data units stored in the secondary storage subsystem in the non-native backup format;

wherein when a first data unit stored in a native format in a first location in the primary storage subsystem has the same unique signature as a second data unit stored in a native format in a second location in the primary storage subsystem, the signature repository agent modifies the associated signature block in the primary storage signature repository to comprise;

a first data unit entry that identifies the first storage location of the first data unit in the native format in the primary storage subsystem; and

a second data unit entry that identifies the second storage location of the second data unit in the native format in the primary storage subsystem.

12. The system of claim 11, wherein the associated signature block further comprises a first client identifier that identifies a first client device associated with the first data unit, and a second client identifier that identifies a second client device associated with the second data unit.

13. The system of claim 12, wherein the first client device uses a first data store in the primary storage subsystem that is different than a second data store used by the second client device.

14. The system of claim 11, wherein the signature repository agent is further configured to:

receive a query from the secondary storage subsystem, the query including a plurality of signatures;

compare the plurality of signatures with signature blocks in the signature repository to identify a first set of signatures from the plurality of signatures that correspond to data units of the plurality of data units that reside in a primary data store of at least one client computing device of the plurality of client computing devices; and

for at least some signatures in the first set of signatures, request the corresponding data units from the at least one client computing device.

15. The system of claim 14, wherein the signature repository agent to accesses data units from the secondary storage subsystem for the signatures that are not included in the first set of signatures.

16. The system of claim 14, wherein the plurality of signatures included in the query correspond to a set of data units which represent a backed up version of a set of the primary data that is to be restored to the first primary data store.

17. The system of claim 11, wherein in response to receipt of instructions to backup at least a subset of the plurality of data units in the primary storage subsystem, the signature repository agent compares a set of signatures corresponding to the subset of data units with entries in the primary storage signature repository and based at least in part on the comparing, identifies a set of matching data units that match the subset of data units and accesses the set of matching data units for retrieval.

18. The system of claim 17, wherein based at least in part on the comparing, the signature repository agent identifies a first group of data units of the subset of data units that do not have a corresponding matching data unit, and communicates the first group of data units to the secondary storage subsystem.

19. The method of claim 11, wherein the secondary storage subsystem comprises deduplicated data.

20. The method of claim 11, wherein the primary data store of at least one of a plurality of client computing devices comprises deduplicated data.