Data management and backup of distributed storage environment

• US 10,452,490 B2
• Filed: 10/20/2016
• Issued: 10/22/2019
• Est. Priority Date: 03/09/2016
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Pradhan, Manoj Kumar, Kumarasamy, Paramasivam, Zakharkin, Dmitriy Borisovich, Duraisamy, Arun Prabu
• Primary Examiner(s): Nguyen, Kim T
• Application Number:
  US15/299,120
• Publication Number:
  US 20170262346A1
• Time in Patent Office: 1,097 Days
• Field of Search: 709225
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: H04L 67/01 : Protocols
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/125 : involving control of end-device applications over a network
• CPC Class Codes: H04L 67/60 : Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70
• CPC Class Codes: H04L 69/40 : for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

In certain systems disclosed herein, a distributed data monitoring and management system is provided that can replicate a distributed storage environment. The distributed data monitoring and management system can intelligently and automatically configure data access nodes to form a structure that matches the distributed storage environment. By matching the structure of the distributed storage environment, the distributed structure of the data may be maintained, enabling the data to be backed up from and/or restored to the distributed storage environment and/or migrated to another distributed storage environment without altering the distribution of the data. Further, embodiments herein enable the transfer of data from a non-distributed environment to a distributed storage environment. Thus, in some cases, an entity can migrate data from a local storage structure to a network-based distributed storage structure.

Claims

1. A distributed system for monitoring data stored in a distributed computing environment, the distributed system comprising on or more processors:

a control agent implemented on a control computing system; and

a plurality of data access nodes implemented, respectively, on a plurality of computing systems,

wherein the control agent is programmed to;

access configuration information for a distributed storage environment from a configuration node of the distributed storage environment, wherein the distributed storage environment comprises a plurality of storage nodes;

determine a distributed file system type for the distributed storage environment based at least in part on the configuration information;

determine a total number of storage nodes within the distributed storage environment based at least in part on the configuration information;

make a determination on how to assign the plurality of data access nodes to back up the plurality of storage nodes by distributing the plurality of data access nodes among the plurality of storage nodes based at least in part on the distributed file system type and the total number of storage nodes;

assign the plurality of data access nodes to the plurality of storage nodes based on the determination; and

back up the plurality of storage nodes to the plurality of data access nodes.

2. The distributed system of claim 1, wherein the plurality of data access nodes comprise a plurality of data agents and the plurality of computing systems are part of a primary storage subsystem.

3. The distributed system of claim 1, wherein the plurality of data access nodes comprise a plurality of media agents and the plurality of computing systems are part of a secondary storage subsystem.

4. The distributed system of claim 1, wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the control agent is further programmed to re-distribute the plurality of data access nodes among the plurality of storage nodes.

5. The distributed system of claim 1, wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the control agent is further programmed to:

allocate a new data access node to the plurality of data access nodes; and

re-distribute the plurality of data access nodes with the new data access node among the plurality of storage nodes.

6. The distributed system of claim 1, wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the control agent is further programmed to reassign data access nodes from the plurality of data access nodes assigned to the storage node to one or more other storage nodes from the plurality of storage nodes that store at least a subset of data blocks stored on the storage node.

7. The distributed system of claim 1, wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the control agent is further programmed to reassign data access nodes from the plurality of data access nodes assigned to the storage node to a mirror node that mirrors data stored on the storage node.

8. The distributed system of claim 1, wherein the control agent is further programmed to distribute the plurality of data access nodes among the plurality of storage nodes based at least in part on size of files stored at the plurality of storage nodes.

9. The distributed system of claim 1, wherein the plurality of data access nodes comprises a plurality of heterogeneous data access nodes with differing backup capabilities.

10. The distributed system of claim 9, wherein the control agent is further programmed to distribute the plurality of data access nodes among the plurality of storage nodes by distributing the plurality of heterogeneous data access nodes based at least in part on the differing backup capabilities of the plurality of heterogeneous data access nodes.

11. A computer-implemented method of monitoring data stored in a distributed computing environment, the computer-implemented method comprising:

as implemented by a control agent comprising one or more hardware processors and configured with specific computer-executable instructions, accessing configuration information for a distributed storage environment from a configuration node of the distributed storage environment, wherein the distributed storage environment comprises a plurality of storage nodes;

determining a distributed file system type for the distributed storage environment based at least in part on the configuration information;

determining a total number of storage nodes within the distributed storage environment based at least in part on the configuration information;

determining how to assign a plurality of data access nodes to back up the plurality of storage nodes by distributing the plurality of data access nodes among the plurality of storage nodes based at least in part on the distributed file system type and the total number of storage nodes, wherein the plurality of data access nodes are implemented, respectively, on a plurality of computing systems;

assigning the plurality of data access nodes to the plurality of storage nodes based on the determining how to assign the plurality of data access nodes; and

backing up the plurality of storage nodes to the plurality of data access nodes.

12. The computer-implemented method of claim 11, wherein the plurality of data access nodes comprises a plurality of data agents and the plurality of computing systems are part of a primary storage subsystem.

13. The computer-implemented method of claim 11, wherein the plurality of data access nodes comprises a plurality of media agents and the plurality of computing systems are part of a secondary storage subsystem.

14. The computer-implemented method of claim 11, wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the computer-implemented method further comprises re-distributing the plurality of data access nodes among the plurality of storage nodes.

15. The computer-implemented method of claim 11, wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the computer-implemented method further comprises:

allocating a new data access node to the plurality of data access nodes; and

re-distributing the plurality of data access nodes with the new data access node among the plurality of storage nodes.

16. The computer-implemented method of claim 11, wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the computer-implemented method further comprises reassigning data access nodes from the plurality of data access nodes assigned to the storage node to one or more other storage nodes from the plurality of storage nodes that store at least a subset of data blocks stored on the storage node.

17. The computer-implemented method of claim 11, wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the computer-implemented method further comprises reassigning data access nodes from the plurality of data access nodes assigned to the storage node to a mirror node that mirrors data stored on the storage node.

18. The computer-implemented method of claim 11, further comprising distributing the plurality of data access nodes among the plurality of storage nodes based at least in part on size of files stored at the plurality of storage nodes.

19. The computer-implemented method of claim 11, wherein the plurality of data access nodes comprises a plurality of heterogeneous data access nodes with differing backup capabilities.

20. The computer-implemented method of claim 19, further comprising distributing the plurality of data access nodes among the plurality of storage nodes by distributing the plurality of heterogeneous data access nodes based at least in part on the differing backup capabilities of the plurality of heterogeneous data access nodes.