Modular systems and methods for managing data storage operations

• US 8,433,679 B2
• Filed: 06/09/2008
• Issued: 04/30/2013
• Est. Priority Date: 07/15/1999
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Crescenti, John, Kavuri, Srinivas, Oshinsky, David Alan, Prahlad, Anand
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): WITZENBURG, BRUCE A
• Application Number:
  US12/135,584
• Publication Number:
  US 20080244177A1
• Time in Patent Office: 1,786 Days
• Field of Search: 707/612
• US Class Current: 707/612
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: Y10S 707/99939 Privileged access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

The invention is a modular backup and retrieval system. The software modules making up the backup and retrieval system run independently, and can run either on the same computing devices or on different computing devices. The modular software system coordinates and performs backups of various computing devices communicating to the modules. At least one module on one of the computing devices acts as a system manager for a network backup regimen. A management component acts as a manager for the archival and restoration of the computing devices on the network. It manages and allocates library media usage, maintains backup scheduling and levels, and supervises or maintains the archives themselves through pruning or aging policies. A second software module acts as a manager for each particular library media.

Claims

1. A modular system for managing data storage operations in a network environment, the system comprising:

a plurality of storage devices, wherein a first storage device is a magnetic disk and a second storage device is a tape drive;

a plurality of first modules, each first module operating on a particular one of a plurality of first computing devices and configured to identify data from the particular first computing device to be associated with one or more storage operations, wherein the data comprises files that are organized into data bundles;

a plurality of second modules operating on a plurality of second computing devices and communicatively coupled to the plurality of first modules and to the plurality of storage devices via a network, the plurality of second modules being configured to receive the identified data from the plurality of first modules and to direct a physical transfer of the identified data to the plurality of storage devices; and

a manager module communicatively coupled to the plurality of first and second modules, the manager module being configured to initiate a first storage operation based on a schedule associated with one or more storage policies, wherein the manager module identifies at least one of the second modules to perform the storage of at least a first data bundle,

wherein the manager module is further configured to dynamically alter the schedule based on usage of the network and to select at least one first module from the plurality of first modules and at least one second module from the plurality of second modules to perform the first storage operation, said selection of the at least one second module being based on at least loads of the plurality of second modules,

wherein the at least one second module is configured to store the first data bundle by spanning the files in the first data bundle on at least the first storage device comprising the magnetic disk and the second storage device comprising the tape drive,

wherein the at least one second module creates and stores a first index, the first index is stored in association with the at least one second module, the first index maintaining at least a physical location of the files in the first data bundle, wherein the physical location in the first index details the physical addresses of the files spanning the first storage device comprising the magnetic disk, and the second storage device comprising the tape drive,

wherein the at least one second module is further configured to create and forward a second index to the manager module, wherein the second index is stored in association with the manager module, and comprises information that associates the first data bundle with the at least one second module, while retaining the physical addresses of the files in the first index such that the second index associated with the manager module comprises less information regarding the physical location of the files than the first index associated with the at least one second module,

wherein when retrieval of at least a portion of the first data bundle is requested from a client, the manager module uses the second index stored in association therewith, to identify that the at least one second module is associated with the first data bundle, and forwards the request to the at least one second module,

the manager module further configured to forward to the at least one second module, communications information by which the at least one second module can establish a communications link with the client,

the at least one second module configured to use the physical addresses in the second index stored in association with the second module, to retrieve the requested portion of first data bundle spanning the first storage device comprising the magnetic disk and the second storage device comprising the tape drive, and

the at least one second module further configured to provide the requested portion of the first data bundle to the client via the communications link indentified by the communications information provided by the manager module.

2. The modular system of claim 1, wherein the at least one first module is configured to filter types of data from the particular first computing device to include in the first storage operation.

3. The modular system of claim 1, wherein each of the plurality of first modules is application specific.

4. The modular system of claim 1, wherein the manager module is configured to initiate the first storage operation in response to a request from the at least one first module.

5. The modular system of claim 1, wherein the manager module is configured to initiate the first storage operation based on an aging policy.

6. The modular system of claim 1, wherein the manager module is configured to initiate the first storage operation based on a network load.

7. The modular system of claim 1, wherein the manager module is further configured to track configuration parameters of one or more applications running on the plurality of first computing devices.

8. The modular system of claim 1, wherein the at least one second module is further configured to store a first portion of the data associated with the first storage operation to a first storage device and a second portion of the data associated with the first storage operation to a second storage device.

9. The modular system of claim 1, wherein the at least one first module is further configured to manage a timing of the first storage operation.

10. The modular system of claim 1, wherein the at least one first module is further configured to prioritize the data to be stored according to the first storage operation.

11. A method for performing storage operations in a network environment, the method comprising:

receiving a request to perform a storage operation;

selecting with a manager module, a first media module from the plurality of media modules to perform the storage operation based on a schedule associated with one or more storage policies, wherein the storage operation stores files that are organized into data bundles;

selecting a first client module from a plurality of client modules to obtain from a first computing device data associated with the storage operation;

instructing the first media module to perform the storage of a first data bundle, wherein the first media module spans the files in the first data bundle on at least the first storage device comprising the magnetic disk, and the second storage device comprising the tape drive,

wherein the first media module creates and stores a first index recording a physical location of the files associated with the first data bundle, the first index stored in association with the first media module, wherein the physical location of the files in the first index details the physical addresses of the files spanning the at least the first storage device comprising the magnetic disk and the second storage device comprising the tape drive,

wherein the first media module further creates and forwards a second index to the manager module, the second index is stored in association with the manager module, the second index comprises metadata that associates the first data bundle with the first media module, while retaining the physical address of the files in the first index such that the second index associated with the manager module comprises less information regarding the physical location of the files than the first index associated with the first media module,

wherein when retrieval of at least a portion of the first data bundle is requested from a client, the manager module uses the first index to identify that the first media module is associated with the first data bundle, and forwards the request to the first media module,

forwarding with the manager module, communications information by which the first media module can establish a communications link with the client,

using the first media module to access the physical addresses in the second index to retrieve the requested portion of first data bundle spanning the at least two storage devices, and

providing with the first media module, the requested portion of the first data bundle to the client via the communications link indentified by the communications information provided by the manager module.

12. The method of claim 11, further comprising instructing a second media module of the plurality of media modules to retrieve the data from the first storage device.

13. The method of claim 12, wherein said retrieving comprises relocating the stored data from the first storage device to a second storage device of the plurality of storage devices, wherein the second storage device comprises a slower medium than the first storage device.

14. The method of claim 11, wherein the metadata further comprises information on at least one of origination information of the stored data and permissions associated with the stored data.

15. The method of claim 11, wherein the request to perform the storage operation is received from the first client module.

16. The method of claim 11, wherein the storage operation comprises an incremental backup operation.

17. The method of claim 16, wherein said selecting the first client module comprises sending to the first client module criteria for determining the data to include in the incremental backup operation.

18. The method of claim 11, wherein the request to perform the storage operation is based on a backup schedule.

19. The method of claim 11, wherein said instructing the first media module further comprises instructing the first media module to perform the physical transfer of a first portion of the data to the first storage device of the plurality of storage devices and to perform the physical transfer of a second portion of the data to a second storage device of the plurality of storage devices.