Systems and methods of data storage management, such as dynamic data stream allocation

• US 8,832,706 B2
• Filed: 12/21/2007
• Issued: 09/09/2014
• Est. Priority Date: 12/22/2006
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag, Klose, Michael F.
• Primary Examiner(s): Lee, Adam
• Application Number:
  US11/963,581
• Publication Number:
  US 20080288947A1
• Time in Patent Office: 2,454 Days
• Field of Search: None
• US Class Current: 718/104
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A system and method for choosing a stream to transfer data is described. In some cases, the system reviews running data storage operations and chooses a data stream based on the review. In some cases, the system chooses a stream based on the load of data to be transferred.

Claims

1. A method of transferring data to one or more storage media, the method comprising:

identifying data to be transferred to the one or more storage media;

receiving data transfer information associated with two or more data streams in a process of transferring the identified data to the one or more storage media, wherein the data transfer information comprises an amount of data currently being transferred by each of the two or more data streams and a data transfer rate of each of the two or more data streams;

selecting from the two or more data streams a data stream based at least in part on the received data transfer information;

pre-allocating space on the one or more storage media associated with the selected data stream;

storing, at a first time, the identified data to be transferred in a first queue associated with the selected data stream;

reassigning, at a second time later than the first time, the identified data to be transferred from the first queue to a second queue associated with a data stream other than the selected data stream, wherein the reassigning compensates for changing amounts of data being transferred by each of the two or more data streams, and the data transfer rates of each of the two or more data streams; and

transferring the identified data to be transferred to the pre-allocated space.

2. The method of claim 1, wherein receiving data transfer information comprises:

receiving header information for the transferring data on the two or more data streams; and

determining the amount of data transferring based on the received header information.

3. The method of claim 1, wherein the selecting comprises selecting the data stream with a comparatively lesser amount of data being transferred.

4. The method of claim 1, wherein the selecting is based at least in part on information related to available space on the one or more storage media.

5. The method of claim 1, wherein the data transfer information comprises a predicted amount of identified data to be transferred by each of the two or more data streams.

6. The method of claim 1, further comprising:

choosing data paths for transferring the identified data.

7. The method of claim 6, wherein the choosing is performed by:

exchanging information with a monitoring system to determine load information.

8. The method of claim 1, further comprising:

identifying the one or more storage media based on a storage policy that defines how storage operations are performed.

9. The method of claim 1, further comprising:

receiving storage device information; and

pre-allocating data based at least in part on the received storage device information.

10. The method of claim 1, wherein the data to be transferred includes snapshot copies of data on a primary storage device.

11. The method of claim 1 wherein data transfer rates of the two or more data streams are different from each other.

12. A system, at least partially implemented in hardware, of dynamically allocating a data stream to send data to one or more storage devices, the system comprising:

a first storage subsystem, wherein the first storage subsystem transfers data to first storage media;

a second storage subsystem, wherein the second storage subsystem transfers data to second storage media; and

a dynamic allocation component, wherein the dynamic allocation component;

receives load information associated with the first storage subsystem and the second storage subsystem, wherein the load information comprises an amount of data currently being transferred by each of the first storage subsystem and the second storage subsystem and a data transfer rate of each of the first storage subsystem and the second storage subsystem;

selects a storage subsystem based at least in part on load information associated with the first storage subsystem and the second storage subsystem;

pre-allocates a space on the storage media associated with the selected storage subsystem; and

stores, at a first time, the data in a first queue;

reassigns, at a second time later than the first time, the data from the first queue to a second queue, wherein the reassignment is performed to compensate for changing amounts of data being transferred by each of the first storage subsystem and the second storage subsystem, and the data transfer rates of each of the first storage subsystem and the second storage subsystem; and

transfers the data to the pre-allocated space.

13. The system of claim 12, wherein the dynamic allocation component is contained within the first storage subsystem or the second storage subsystem.

14. The system of claim 12, wherein the dynamic allocation component is contained with a server that communicates with the storage subsystems.

15. The system of claim 12, wherein the dynamic allocation component receives load information from a monitoring component.

16. The system of claim 12, wherein the dynamic allocation component receives storage device information from the first storage subsystem and the second storage subsystem and allocates the data based at least in part on the received storage device information.

17. The system of claim 12, wherein the dynamic allocation component communicates with the storage subsystems via a network.

18. The system of claim 12, wherein the first storage media and the second storage media are the same storage media.

19. The system of claim 12, wherein the first storage subsystem and the second storage subsystem send data to the same storage media.

20. A system for transferring data to one or more storage media, comprising:

means for receiving the data to be transferred to the one or more storage media;

means, coupled to the means for receiving, for analyzing load information from two or more data streams in transferring the data to the one or more storage media, wherein the load information comprises an amount of data currently being transferred by each of the two or more data streams and a data transfer rate of each of the two or more data streams;

means, coupled to the means for analyzing, for selecting from the two or more data streams a data stream based at least in part on the received load information;

means for pre-allocating space on the one or more storage media associated with the selected data stream;

means for storing, at a first time, the data to be transferred in a first queue associated with the selected data stream;

means for reassigning, at a second time later than the first time, the data to be transferred from the first queue to a second queue associated with a data stream other than the selected data stream, wherein the reassigning compensates for changing amounts of data being transferred by each of the two or more data streams, and the data transfer rates of each of the two or more data streams; and

means for transferring the data to be transferred to the pre-allocated space.

21. The system of claim 20, wherein the means for selecting from the two or more data streams selects a data stream allocated less data than at least one other data stream.

22. A computer program product comprising a computer-readable, non-transitory medium having code stored thereon, the code, when executed, causing a processor to implement a method of transferring data to one or more storage media, the method comprising:

identifying data to be transferred to the one or more storage media;

receiving data transfer information associated with two or more data streams in a process of transferring the identified data to the one or more storage media, wherein the data transfer information comprises an amount of data currently being transferred by each of the two or more data streams and a data transfer rate of each of the two or more data streams;

selecting from the two or more data streams a data stream based at least in part on the received data transfer information;

pre-allocating space on the one or more storage media associated with the selected stream;

storing, at a first time, the identified data to be transferred in a first queue associated with the selected data stream;

reassigning, at a second time later than the first time, the identified data to be transferred from the first queue to a second queue associated with a data stream other than the selected data stream, wherein the reassigning compensates for changing amounts of data being transferred by each of the two or more data streams, and the data transfer rates of each of the two or more data streams; and

transferring the identified data to be transferred to the pre-allocated space.

23. The computer program product of claim 22, wherein receiving data transfer information comprises:

receiving header information for the transferring data on the two or more data streams; and

determining the amount of data being transferred based on the received header information.

24. The computer program product of claim 22, wherein the selecting comprises selecting the data stream with a comparatively lesser amount of data being transferred.

25. The computer program product of claim 22, wherein the selecting is based at least in part on information related to available space on the one or more storage media.