System and method for performing multistream storage operations

• US 8,112,543 B2
• Filed: 06/24/2011
• Issued: 02/07/2012
• Est. Priority Date: 11/05/2004
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag, Kottomtharayil, Rajiv, Pawar, Rahul, Prahlad, Anand
• Primary Examiner(s): Hwang, Joon H
• Assistant Examiner(s): ISOM, JOHN W
• Application Number:
  US13/168,748
• Publication Number:
  US 20110314120A1
• Time in Patent Office: 228 Days
• Field of Search: 709/231, 709/239, 709/240, 709/241
• US Class Current: 709/231
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 12/00 Accessing, addressing or allocating within memory systems or architectures digital input from, or digital output to record carriers, e.g. to disk storage units, G06F3/06
• CPC Class Codes: G06F 12/02 Addressing or allocation; Relocation program address sequencing G06F9/00; arrangements for selecting an address in a digital store G11C8/00
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: G06F 3/0686 Libraries, e.g. tape libraries, jukebox
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems and methods for performing storage operations over multi-stream data paths are provided. Prior to performing a storage operation, it may be determined whether multi-streaming resources are available to perform a multi-stream storage operation. Availability of multi-streaming resources may be related to network pathways capable of supporting multi-stream storage operations, existing network load related to other storage operations being or to be performed, availability of components capable of supporting multi-stream storage operation, and other factors. If system resources to support multi-stream storage operations are not available, the system may optionally perform a traditional storage operation that does not incorporate multiple data streams.

Claims

1. A method for performing a dynamic multi-stream storage operation to transmit data from a sub-client within a data store to one or more storage devices, the method comprising:

subdividing at least one client storage volume into a plurality of sub-clients, wherein each sub-client comprises one or more folders;

receiving a request to transmit data from a first sub-client of the plurality of sub-clients to one or more storage devices;

determining whether two or more streaming communication paths are available to transmit the data based on pathway characteristics of the two or more streaming communication paths, wherein the pathway characteristics comprise bandwidth of network pathways, existing network load, and predicted network load;

identifying folders in the first sub-client based on a file type in the folders in the first sub-client;

weighting the folders in the first sub-client based on the amount of data in the folders of the first sub-client;

subdividing the folders in the first sub-client into two or more subsets based at least in part on the file type and the weighting;

transmitting the data from each of the two or more subsets of the first sub-client to the one or more storage devices via at least one of the two or more streaming communications paths based at least in part on the pathway characteristics and on the weighting; and

storing the transmitted data from each of the two or more subsets of the first sub-client in the one or more storage devices, wherein

the determining whether the two or more streaming communication paths are available further includes determining whether a transmission time threshold is exceeded by transmitting data over one streaming communication path.

2. The method of claim 1, wherein the determining whether the two or more streaming communication paths are available further comprises determining that storage network resources within the streaming communication paths required for performing multi-stream storage operations are available.

3. The method of claim 1, wherein the determining whether the two or more streaming communication paths are available further includes determining that performing a multi-stream storage operation over only one streaming communication path would exceed a load threshold.

4. The method of claim 1, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on a priority level of the folders.

5. The method of claim 1, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on an impact of scheduled data transmissions.

6. The method of claim 1, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on the number of items in the folders.

7. The method of claim 1, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on available storage devices within the streaming communication paths.

8. The method of claim 7 wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on a capacity of the available storage devices.

9. The method of claim 1, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on a transmission time of data associated with the first sub-client.

10. The method of claim 1, wherein the allocating each of the two or more subsets to a respective one of the two or more streaming communication paths is performed dynamically and is based at least in part on a number of the subsets of the first sub-client.

11. The method of claim 1, wherein the each folder of the sub-client is transmitted via a separate streaming communication path.

12. A method for performing a multi-stream storage operation in a data storage system, the method comprising:

subdividing at least one client storage volume into a plurality of sub-clients, wherein each sub-client comprises one or more folders;

receiving a request to transmit data from a first sub-client of the plurality of sub-clients to one or more storage devices;

determining whether a first streaming communication path is available to transmit the data based on a first pathway characteristic;

determining whether a second streaming communication path is available to transmit the data based on a second pathway characteristic;

identifying folders in the first sub-client based on a file type in the folders in the first sub-client;

weighting the folders in the first sub-client based on the amount of data in the folders in the first sub-client;

dynamically dividing the folders in the first sub-client into two or more subsets based at least in part on the file type, the weighting, and an impact of scheduled data transmissions;

associating a first subset of the two or more subsets to the first streaming communication path based at least in part on the characteristics of the streaming communication paths and on the weighting associated with the folders;

associating a second subset of the two or more subsets to the second streaming communication path based at least in part on the characteristics of the streaming communication paths and on the weighting associated with the folders; and

transmitting data from each of the two or more subsets of the first sub-client to one or more storage devices via at least one of the two or more streaming communications paths based at least in part on the characteristics of the streaming communication paths and on the weighting, wherein

the determining whether the second streaming communication path is available further includes determining whether a transmission time threshold is exceeded by transmitting data over the first streaming communication path.

13. The method of claim 12, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on a priority level of the folders.

14. The method of claim 12, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on available storage devices within the streaming communication paths.

15. The method of claim 14, wherein the subdividing the folders further comprises subdividing the folders in the first sub-client based at least in part on a capacity of the available storage devices.

16. A multi-stream storage operation device comprising:

a non-transitory computer readable medium storing

a data agent executing in one or more processors on a client computer, the executing comprising;

receiving a request for a storage operation from a storage manager;

subdividing at least one client storage volume into a plurality of sub-clients, wherein each sub-client comprises one or more folders;

identifying folders of a first sub-client of the plurality of sub-clients based on one or more file types in the folders of the first sub-client;

weighting the folders of the first sub-client based on the amount of data in the folders of the first sub-client;

subdividing the folders of the first sub-client into two or more subsets based at least in part on one of the file types in the folders and the weighting of the folders; and

transmitting data from each of the two or more subsets of the first sub-client to the one or more storage devices via at least one of the two or more streaming communications paths based on the pathway characteristics, wherein

the pathway characteristics include a determination whether a transmission time threshold is exceeded by transmitting data over at least one of the two or more streaming communication paths.

17. The device of claim 16, wherein the data agent subdivides the folders based at least in part on a bandwidth of the storage network.

18. The device of claim 16, wherein the data agent subdivides the folders based at least in part on an impact of scheduled data transmissions.

19. The device of claim 16, wherein the data agent subdivides the folders based at least in part on an availability of the storage network resources.

20. The device of claim 16, wherein the pathway characteristics further comprise characteristics related to the network pathways and an impact of scheduled data transmissions.