System and method for performing multistream storage operations

• US 7,975,061 B1
• Filed: 11/07/2005
• Issued: 07/05/2011
• Est. Priority Date: 11/05/2004
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Pawar, Rahul, Gokhale, Parag, Prahlad, Anand, Kottomtharayil, Rajiv
• Primary Examiner(s): Tang; Karen C
• Assistant Examiner(s): ISOM, JOHN W
• Application Number:
  US11/269,520
• Time in Patent Office: 2,066 Days
• Field of Search: 709/231, 709/239, 709/240, 709/241
• US Class Current: 709/231
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 12/00 Accessing, addressing or allocating within memory systems or architectures digital input from, or digital output to record carriers, e.g. to disk storage units, G06F3/06
• CPC Class Codes: G06F 12/02 Addressing or allocation; Relocation program address sequencing G06F9/00; arrangements for selecting an address in a digital store G11C8/00
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: G06F 3/0686 Libraries, e.g. tape libraries, jukebox
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems and methods for performing storage operations over multi-stream data paths are provided. Prior to performing a storage operation, it may be determined whether multi-streaming resources are available to perform a multi-stream storage operation. Availability of multi-streaming resources may be related to network pathways capable of supporting multi-stream storage operations, existing network load related to other storage operations being or to be performed, availability of components capable of supporting multi-stream storage operation, and other factors. If system resources to support multi-stream storage operations are not available, the system may optionally perform a traditional storage operation that does not incorporate multiple data streams.

Claims

1. A method for performing a dynamic multi-stream storage operation to transmit data from a sub-client within a data store to one or more storage devices, the method comprising:

subdividing at least one client storage volume into a plurality of sub-clients based on scheduled storage operations that occur on network pathways at overlapping times, wherein each sub-client comprises one or more folders in the storage volume;

receiving a request for a storage operation to transmit data from a first of the sub-clients to one or more storage devices, wherein the storage operation does not prohibit a multi-stream storage operation;

analyzing characteristics of two or more streaming communication paths, wherein the analyzed characteristics comprise bandwidth of network pathways, existing network load, and predicted network load and determining whether the two or more streaming communication paths are available to perform the storage operation based on the analyzed characteristics;

analyzing characteristics associated with the first sub-client by identifying folders based on file type in the folders, and by weighting the folders based on the amount of data in the folders;

dynamically subdividing the folders in the first sub-client into two or more subsets to perform the storage operation with the two or more streaming communication paths, based at least in part on the file type in the folders, the weighting associated with the folders, and an impact of the other scheduled storage operations;

allocating each of the two or more subsets to a respective one of the two or more streaming communications paths based at least in part on the analyzed characteristics of the two or more streaming communication paths and on the weighting associated with the folders;

performing the storage operation by transmitting the data from each of the two or more subsets of the first sub-client to the one or more storage devices via the respective streaming communications paths to which each of the subsets has been allocated; and

storing the transmitted data from each of the two or more subsets of the first sub-client on the one or more storage devices;

wherein at least one of the two or more streaming communication paths extends between two storage operation cells, wherein the two storage operation cells are part of a hierarchically organized system comprising a master storage manager and a group of storage operation cells, wherein each storage operation cell in the group of storage operation cells is hierarchically subordinate to the master storage manager and includes at least a storage manager for coordinating and controlling storage management operations within the storage operation cell, and wherein the hierarchical organization facilitates system security between storage operation cells;

conducting, by the master storage manager, an analysis regarding the availability and characteristics of additional network resources for supporting a multi-stream storage operation by communicating with the storage managers in the group of storage operation cells; and

communicating, by the master storage manager, security-related information between the storage operation cells, wherein the security-related information includes at least one of access protocol information, firewall information, a port to use, a password, and a certificate, and wherein the security-related information is required to perform storage operations between the storage operation cells,

wherein the determining whether the two or more streaming communication paths are available further includes determining whether a transmission time threshold is exceeded by performing a multi-stream storage operation only over one streaming communication path.

2. The method of claim 1 wherein the analyzing characteristics associated with the first sub-client data stored in the first sub client further includes analyzing a size of the data stored in the sub-client.

3. The method of claim 1 wherein the analyzing characteristics associated with the first sub-client further includes analyzing the first sub-client based on application type of data stored in the first sub-client.

4. The method of claim 1 wherein the analyzing characteristics associated with the first sub-client further stored in includes analyzing the data stored in the based on a user specified criteria.

5. The method of claim 1 wherein the determining whether the two or more streaming communication paths are available further includes determining that storage network resources within the streaming communication paths required for performing multi-stream storage operations are available.

6. The method of claim 1 wherein the determining whether the two or more streaming communication paths are available further includes determining that a load threshold would be exceeded by performing a multi-stream storage operation only over one streaming communication path.

7. The method of claim 1 wherein the determining whether the two or more streaming communication paths are available further includes determining the bandwidth for the network pathways within each of the streaming communication paths.

8. The method of claim 1 wherein the subdividing the folders in the first sub-client into two or more subsets further includes subdividing the folders based at least in part, on available storage network resources within the streaming communication paths.

9. The method of claim 1 wherein the subdividing the folders in the first sub-client into two or more subsets is based at least in part on an availability of the storage network resources within the streaming communication paths and on a size of the first sub-client.

10. The method of claim 1 wherein the subdividing the folders in the first sub-client into two or more subsets is based at least in part on a capacity of the storage network resources within the streaming communication paths.

11. The method of claim 1 wherein the subdividing the folders in the first sub-client into two or more subsets is based at least in part on a transmission time of data associated with the first sub-client.

12. The method of claim 1 wherein the subdividing the folders in the first sub-client into two or more subsets is based at least in part on a number and receiving capacity of the one or more storage devices that receive data from the streaming communication paths.

13. The method of claim 1 wherein the allocating each of the two or more subsets to a respective one of the two or more streaming communication paths is performed dynamically and is based at least in part on a number of the subsets of the first sub-client.

14. A method for performing a multi-stream storage operation in a data storage system, the method comprising:

subdividing at least one client storage volume into a plurality, of sub-clients based on scheduled storage operations that occur on network pathways at overlapping times, wherein each sub-client comprises one or more folders in the storage volume;

receiving a request for a storage operation to transmit data from a first of the sub-clients to one or more storage devices, wherein the storage operation does not prohibit a multi-stream storage operation;

analyzing a first characteristic of a first streaming communication path and determining whether the first streaming communication path is available to perform the storage operation based on the analyzed first characteristic;

analyzing a second characteristic of a second streaming communication path and determining whether the second streaming communication path is available to perform the storage operation based on the analyzed second characteristic;

analyzing characteristics associated with the first sub-client by identifying folders based on file type in the folders, and by weighting the folders based on the amount of data in the folders;

dynamically dividing the folders in the first sub-client into two or more subsets to perform the storage operation with the first streaming communication path and the second streaming communication path, based at least in part on the file type in the folders, the weighting associated with the folders, and an impact of the other scheduled storage operations;

associating a first subset of the two or more subsets to the first streaming communication path based at least in part on the characteristics of the streaming communication paths and on the weighting associated with the folders;

associating a second subset of the two or more subsets to the second streaming communication path based at least in part on the characteristics of the streaming communication paths and on the weighting associated with the folders; and

performing the storage operation by transmitting data from each of the two or more subsets of the first sub-client to one or more storage devices via the respective streaming communications path to which each of the subsets has been associated;

wherein at least one of the streaming communication paths extends between two storage operation cells in a hierarchically organized system comprising a master storage manager and a group of storage operation cells, wherein each storage operation cell in the group of storage operation cells is hierarchically subordinate to the master storage manager and includes at least a storage manager for coordinating and controlling storage management operations within the storage operation cell;

conducting, by the master storage manager, an analysis regarding the availability and characteristics of additional network resources for supporting a multi-stream storage operation by communicating with the storage managers in the group of storage operation cells; and

communicating, by the master storage manager, security-related information between the storage operation cells, wherein the security-related information is required to perform storage operations between the storage operation cells, and wherein the security-related information includes at least one of access protocol information, firewall information, a port to use, a password, and a certificate,

wherein the determining whether the second streaming communication path is available further includes determining whether a transmission time threshold is exceeded by performing a multi-stream storage operation only over the first streaming communication path.

15. A method for performing a multi-stream storage operation to transmit data from a sub-client within a data store to one or more storage devices, the method comprising:

subdividing at least one client storage volume into a plurality of sub-clients based on scheduled storage operations that occur on network pathways at overlapping times, wherein each sub-client comprises one or more folders in the storage volume;

analyzing characteristics associated with a first of the sub-clients by identifying folders based on file type in the folders, and by weighting the folders based on the amount of data in the folders;

dynamically subdividing the folders in the first sub-client into two or more subsets to perform the storage operation with two or more streaming communication paths, based at least in part on characteristics associated with the two or more streaming communication paths, including characteristics related to the network pathways, and at least the file type in the folders, the weighting associated with the folders, an impact of the other scheduled storage operations, and a determination whether a transmission time threshold is exceeded by performing a multi-stream storage operation only over one streaming communication path;

allocating each of the two or more subsets to a respective one of the two or more streaming communications paths based on characteristics associated with the two or more streaming communication paths, including characteristics related to the network pathways, and on the weighting associated with the folders; and

performing the storage operation by transmitting data from each of the two or more subsets of the first sub-client to the one or more storage devices via the respective one of the streaming communications paths to which each of the subsets has been allocated;

wherein at least one of the two or more streaming communication paths extends between two storage operation cells in a hierarchically organized system comprising a master storage manager and a group of storage operation cells, wherein each storage operation cell in the group of storage operation cells is hierarchically subordinate to the master storage manager and includes at least a storage manager for coordinating and controlling storage management operations within the storage operation cell;

conducting, by the master storage manage, an analysis regarding the availability and characteristics of additional network resources for supporting a multi-stream storage operation by communicating with the storage managers in the group of storage operation cells; and

communicating, by the master storage manager, security-related information between the storage operation cells, wherein the security-related information is required to perform storage operations between the storage operation cells, and wherein the security-related information includes at least one of access protocol information, firewall information, a port to use, a password, and a certificate.