Multi-threaded scanning of distributed file systems

• US 10,389,810 B2
• Filed: 11/02/2016
• Issued: 08/20/2019
• Est. Priority Date: 11/02/2016
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Swallow, Douglas Cary, Pradhan, Manoj Kumar, Zakharkin, Dmitriy Borisovich
• Primary Examiner(s): Lazaro, David R
• Assistant Examiner(s): Shaw, Robert A
• Application Number:
  US15/341,849
• Publication Number:
  US 20180124174A1
• Time in Patent Office: 1,021 Days
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70

Abstract

Embodiments disclosed herein address the need to more efficiently backup a network-based storage environment that may be remote from a primary storage environment. For example, embodiments herein can provide a more efficient backup of a storage managed by a third-party entity. To improve the backup process, embodiments herein may optimize the number of scanning threads that are used to identify files that are to be backed up by, for example, analyzing the characteristics of the network and/or the network storage system to determine a number of scanning threads that will enable faster scanning of the network storage system while at the same time not overburden or be slowed down by a network between the network-based storage environment and the primary storage environment.

Claims

1. A computer-implemented method of performing multithreaded scanning of a network storage system, the computer-implemented method comprising:

as implemented by a data agent executing within a client computing device, wherein the client computing device is configured in an information management system;

determining network characteristics of a network used by the client computing device to communicate with a network storage system that is remote from the information management system and stores data accessible by the client computing device, wherein the data is associated with the data agent,

selecting an amount of scanning threads to initiate based at least in part on the network characteristics of the network, wherein the scanning threads are configured to scan a network storage repository of the network storage system to identify files to backup that are stored at the network storage repository,

triggering initiation of the amount of the scanning threads at the network storage system, and initiating scanning of the network storage repository using the scanning threads to identify the files to backup;

wherein to select the amount of scanning threads, the data agent balances an amount of time to complete the scanning of the network storage repository using the scanning threads with an amount of network resources to communicate results of the scanning.

2. The computer-implemented method of claim 1, further comprising:

receiving a scan result from the network storage system corresponding to a result of scanning the network storage repository; and

initiating backup of the network storage repository based at least in part on the scan result.

3. The computer-implemented method of claim 2, wherein the scan result identifies files to be backed up and a location of the files at the network storage repository.

4. The computer-implemented method of claim 2, further comprising:

receiving files to be backed up responsive to the initiating backup of the network storage repository; and

providing the files to a media agent for backup at a secondary storage system.

5. The computer-implemented method of claim 2, further comprising providing the scan result to a storage manager, wherein the initiating backup of the network storage repository occurs in response to a command from the storage manager.

6. The computer-implemented method of claim 1, further comprising:

sending a test packet to the network storage system over the network; and

receiving an acknowledgement packet from the network storage system.

7. The computer-implemented method of claim 6, wherein determining the network characteristics of the network is based at least in part on an amount of time between sending the test packet and receiving the acknowledgement packet.

8. The computer-implemented method of claim 6, wherein determining the network characteristics of the network is based at least in part on a number of times the test packet is sent before receiving the acknowledgement packet.

9. The computer-implemented method of claim 1, wherein determining the network characteristics of the network comprises performing a network test operation.

10. The computer-implemented method of claim 1, further comprising receiving a trigger to perform a backup process of the network storage system, wherein at least the initiating scanning of the network storage repository occurs in response to the trigger to perform the backup process.

11. The computer-implemented method of claim 1, wherein the selecting the amount of the scanning threads to initiate is further based at least in part on characteristics of the network storage system.

12. The computer-implemented method of claim 1, wherein scanning the network storage repository comprises traversing a file structure at the network storage repository.

13. A system for performing multi-threaded scanning of a network storage system, the system comprising:

an information management system implemented in computer hardware, wherein a client computing device in the information management system communicates over a network with a network storage system that comprises data files accessible by the client computing device, and wherein the client computing device executes a data agent that is associated with the data files;

wherein the data agent is configured to;

determine network characteristics of the network used by the client computing device to communicate with the network storage system;

select an amount of scanning threads to initiate based at least in part on the network characteristics of the network, wherein the scanning threads are configured to scan a network storage repository of the network storage system to identify data files to backup that are stored at the network storage repository,

trigger initiation of the amount of the scanning threads at the network storage system, and

initiate scanning of the network storage repository using the scanning threads to identify the files to backup; and

wherein to select the amount of scanning threads, the data agent is configured to balance an amount of time to complete the scanning of the network storage repository using the scanning threads with an amount of network resources to communicate results of the scanning.

14. The system of claim 13, wherein the information management system is further configured to:

receive a scan result from the network storage system corresponding to a result of scanning the network storage repository; and

initiate backup of the network storage repository based at least in part on the scan result.

15. The system of claim 14, wherein the information management system is further configured to:

receive files to be backed up responsive to the initiating backup of the network storage repository; and

provide the files to a media agent for backup at a secondary storage system.

16. The system of claim 14, wherein the information management system is further configured to provide the scan result to a storage manager, wherein the initiating backup of the network storage repository occurs in response to a command from the storage manager.

17. The system of claim 13, wherein the information management system is further configured to:

send a test packet to the network storage system over the network;

receive an acknowledgement packet from the network storage system; and

wherein the network characteristics are determined based at least in part on a result of sending the test packet and receiving the acknowledgement packet.

18. The system of claim 13, wherein the

selecting the amount of the scanning threads to initiate is further based at least in part on an organization of the network storage repository.

19. The system of claim 13, wherein the selecting the amount of the scanning threads to initiate is further based at least in part on characteristics of the network storage system.