Intelligent data sourcing in a networked storage system

• US 9,218,376 B2
• Filed: 06/12/2013
• Issued: 12/22/2015
• Est. Priority Date: 06/13/2012
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Muller, Marcus S., Ngo, David
• Primary Examiner(s): PHAM, KHANH B
• Application Number:
  US13/916,467
• Publication Number:
  US 20130339390A1
• Time in Patent Office: 923 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1415 at system level
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/1847 specifically adapted to static storage, e.g. adapted to flash memory or SSD
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 : Synchronous replication
• CPC Class Codes: G06F 16/90 : Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 16/95 : Retrieval from the web
• CPC Class Codes: G06F 2201/80 : Database-specific techniques

Abstract

A storage system according to certain embodiments includes a repository of client-side data block signature information representative of a set of data blocks stored in a primary storage subsystem. In some cases, the system sources data blocks for secondary copy and restore operations from the primary storage subsystem instead of the secondary storage subsystem. Where multiple primary storage components (e.g., multiple client computing devices) contain copies of a data blocks involved in a secondary copy or restore operation, the system can decide which client to source the data block from based on sourcing criteria.

Claims

1. A method of sourcing data from storage associated with a pool of computing devices during a data storage operation associated with one of the computing devices in the pool, the method comprising:

obtaining signatures corresponding to data units that form a data set associated with a data storage operation, the data set corresponding to a version of one or more files of primary data of a first computing device in a pool of a plurality of computing devices, each respective computing device in the pool storing primary data generated by one or more software applications executing on the respective computing device, the primary data stored in at least one storage device associated with the respective computing device,

wherein the storage devices of the computing devices in the pool store a plurality of data units of primary data including at least the data set stored in the at least one storage device of the first computing device,

wherein each file of a plurality of files of primary data stored in the storage devices comprises at least one data unit of the plurality of data units,

wherein at least a first data unit of the data set forms at least a portion of a first file of primary data stored in the at least one storage device of the first computing device and a second data unit of the plurality of data units matches the first data unit and forms at least a portion of a second file of primary data stored in the at least one storage device of a second computing device of the plurality of computing devices, and

wherein the first file and the second file are generated by the one or more software applications executing on the first computing device and the second computing device, respectively;

populating, by one or more processors, a shared signature repository that includes;

signatures corresponding to at least each data unit of the plurality of data units, wherein a first signature corresponds to the first data unit and the second data unit; and

for each signature included in the signature repository, an indication as to one or more of the computing devices whose at least one storage device includes an independently generated data unit that corresponds to the signature, wherein each independently generated data unit forms at least a portion of a distinct file residing on the respective storage device, and wherein the shared signature repository includes at least a first indication that indicates a first location of the first data unit in the at least one storage device of the first computing device and a second location of the second data unit in the at least one storage device of the second computing device;

comparing the obtained signatures, including a signature of the first data unit, with the signature repository to identify one or more matching data units, including the second data unit, stored in the respective at least one storage device of the computing devices in the pool, wherein each of the one or more matching data units forms at least a portion of a read/write file residing in the respective storage device and is stored in a native format of the respective software application that generated the respective matching data unit;

consulting, by one or more processors, a priority policy; and

based on the priority policy, and for at least the first data unit in the data set, determining to access the second data unit rather than the first data unit for the data storage operation.

2. The method of claim 1, wherein the priority policy includes an indication as to a relative priority of one or more of the computing devices with respect to one or more others of the computing devices.

3. The method of claim 1, wherein the data storage operation comprises a backup copy operation in which a copy of the data set is stored in secondary storage that is separate from the pool of computing devices and from each at least one storage device of the respective computing devices.

4. The method of claim 1, wherein the data storage operation comprises a restore operation in which the data set is restored to the at least one storage device associated with the first client computing device.

5. The method of claim 4, wherein at least some of the data units in the data set are sourced from secondary storage which is separate from the pool of computing devices and is separate from each at least one storage device of the respective computing devices.

6. The method of claim 1, wherein said consulting is performed by a data sourcing module executing on one or more processors of a computing device that is separate from the pool, and wherein the signature repository is separate from each of the at least one storage devices.

7. The method of claim 1, wherein at least 10 percent of the data units are sourced from the at least one storage device of computing devices in the pool other than the first computing device.

8. The method of claim 1, wherein at least 25 percent of the data units are sourced from the at least one storage device of computing devices in the pool other than the first computing device.

9. The method of claim 1, wherein at least 50 percent of the data units are sourced from the at least one storage device of computing devices in the pool other than the first computing device.

10. The method of claim 1, wherein all of the data units are sourced from the at least one storage device of one or more computing devices in the pool other than the first computing device.

11. A storage system for sourcing data from storage associated with a pool of computing devices during a data storage operation associated with one of the computing devices in the pool, the storage system comprising:

a global signature repository including;

signatures corresponding to at least each data unit of a plurality of data units of primary data, the plurality of data units stored in storage devices associated with a plurality of computing devices in a pool,

wherein each file of a plurality of files of primary data stored in the storage devices comprises at least one data unit of the plurality of data units,

wherein at least a first data unit of the plurality of data units forms at least a portion of a first file of primary data stored in a first storage device associated with a first computing device of the plurality of computing devices and a second data unit of the plurality of data units matches the first data unit and forms at least a portion of a second file of primary data stored in a second storage device associated with a second computing device of the plurality of computing devices,

wherein the first file and the second file are generated by one or more software applications executing on the first computing device and the second computing device, respectively, and

wherein a first signature of the signatures corresponds to the first data unit and the second data unit; and

for each signature included in the signature repository, an indication as to one or more of the plurality of computing devices whose at least one storage device includes an independently generated data unit that corresponds to the signature, wherein each independently generated data unit forms at least a portion of a distinct file residing on the respective storage device, and wherein the global signature repository includes at least a first indication that indicates a first location of the first data unit in the first storage device and a second location of the second data unit in the second storage device; and

a repository agent executing in one or more processors and configured to;

obtain signatures corresponding to data units that form a data set associated with a data storage operation, the data set corresponding to a version of one or more files of primary data of the first computing device and including the first data unit;

compare the obtained signatures, including a signature of the first data unit, with the signature repository to identify one or more matching data units, including the second data unit, stored in the respective at least one storage device of the computing devices in the pool, wherein each of the one or more matching data units forms at least a portion of a read/write file residing in the respective storage device and is stored in a native format of the respective software application that generated the respective matching data unit;

consult a priority policy; and

based on the priority policy, and for at least the first data unit in the data set, determine to access the second data unit rather than the first data unit for the data storage operation.

12. The system of claim 11, wherein the priority policy includes an indication as to a relative priority of one or more of the computing devices with respect to one or more others of the computing devices.

13. The system of claim 11, wherein the data storage operation comprises a backup copy operation in which a copy of the data set is stored in secondary storage that is separate from the plurality of computing devices and from each at least one storage device of the respective computing devices.

14. The system of claim 11, wherein the data storage operation comprises a restore operation in which the data set is restored to the at least one storage device associated with the first client computing device.

15. The system of claim 14, wherein at least some of the data units in the data set are sourced from secondary storage which is separate from the plurality of computing devices and is separate from each at least one storage device of the respective computing devices.

16. The system of claim 11, wherein the repository agent executes on one or more processors of a computing device that is separate from the pool and wherein the signature repository is separate from the storage devices.

17. The system of claim 11, wherein at least 10 percent of the data units are sourced from the at least one storage device of computing devices in the pool other than the first computing device.

18. The system of claim 11, wherein at least 25 percent of the data units are sourced from the at least one storage device of computing devices in the pool other than the first computing device.

19. The system of claim 11, wherein at least 50 percent of the data units are sourced from the at least one storage device of computing devices in the pool other than the first computing device.

20. The system of claim 11, wherein all of the data units are sourced from the at least one storage device of one or more computing devices in the pool other than the first computing device.