Incrementally accumulating in-process performance data and hierarchical reporting thereof for a data stream in a secondary copy operation

• US 10,275,320 B2
• Filed: 06/26/2015
• Issued: 04/30/2019
• Est. Priority Date: 06/26/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Thanasekaran, Vijayan
• Primary Examiner(s): Harper, Eliyah S.
• Application Number:
  US14/752,653
• Publication Number:
  US 20160378614A1
• Time in Patent Office: 1,404 Days
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

Each data stream in a backup job is individually tracked by data agent(s) and media agent(s) in its path, generating performance data packets in-process and merging them into the processed data stream. The data stream thus incrementally accumulates performance data packets from any number of successive backup processes. The in-process tracking also captures hierarchical relationships among backup processes and in-process subtending tasks, so that the resulting performance report can depict parent and child operations. The hierarchical relationships are embedded into the performance data packets and may be analyzed by parsing the data stream. The media agent transfers the data packets belonging to the secondary copy to secondary storage. The media agent analyzes the performance data packets in the data stream and generates a performance report, which covers the data stream from source to destination, based on the accumulated information carried by the performance data packets. The media agent illustratively stores the performance report locally as a flat file.

Claims

1. A system for incrementally accumulating in-process measures of performance into individual data streams that carry data to be backed up into secondary copies, the system comprising:

a first computing device in communication with a first storage device comprising first data to be backed up to a secondary copy, wherein the first computing device comprises one or more processors and computer memory for executing a data agent;

a second computing device in communication with the first computing device, wherein the second computing device comprises one or more processors and computer memory for executing a media agent, and further wherein the second computing device is in communication with a second storage device for storing the secondary copy of the first data;

wherein the data agent, when executing on the first computing device, is configured to;

initiate a data stream originating at the first storage device and comprising the first data,

process the data stream using successive first backup processes for generating the secondary copy,

after each of the first backup processes used to process the data stream at the data agent;

(a) measure one or more performance characteristics of the respective first backup process including subtending tasks thereof,

(b) generate respective performance data packets that represent the measured one or more performance characteristics of the respective first backup process including the subtending tasks thereof, and

(c) accumulate the respective performance data packets into the data stream resulting from the respective first backup process, and

transmit the data stream, comprising the first data as processed by the data agent and further comprising the performance data packets accumulated by the data agent into the data stream, to the media agent; and

wherein the media agent, when executing on the second computing device, is configured to;

process the data stream as received from the data agent using successive second backup processes for generating the secondary copy,

after each of the second backup processes used to process the data stream at the media agent;

(i) measure one or more performance characteristics of the respective second backup process including subtending tasks thereof,

(ii) generate respective performance data packets that represent the measured one or more performance characteristics of the respective second backup process including the subtending tasks thereof, and

(iii) accumulate the respective performance data packets into the data stream resulting from the respective second backup process,

parse the data stream, comprising the first data as processed by the data agent and the media agent and further comprising the performance data packets accumulated into the data stream by the data agent and by the media agent,

store to the second storage device the secondary copy comprising the first data from the data stream as processed by the data agent and the media agent, and

generate a performance report for the data stream based on the performance data packets accumulated into the data stream by the data agent and by the media agent.

2. The system of claim 1 wherein the data agent is further configured to:

generate, in each of the first backup processes, indicators of hierarchical relationships among the respective first backup process and subtending tasks thereof, and

incorporate the indicators into the performance data packets generated by the data agent; and

wherein the performance report further depicts the hierarchical relationships among the depicted measures of performance, based on the indicators in the performance data packets.

3. The system of claim 1 wherein the media agent is further configured to:

generate, in each of the second backup processes, indicators of hierarchical relationships among the respective second backup process and subtending tasks thereof, and

incorporate the indicators into the performance data packets generated by the media agent; and

wherein the performance report further depicts the hierarchical relationships among the depicted measures of performance, based on the indicators in the performance data packets.

4. A method for accumulating performance metrics into a backup data stream in a storage management system, the method comprising:

performing a first backup process upon a data stream, by a data agent that executes on a client computing device, wherein the client computing device comprises one or more processors and computer-readable memory,

wherein the data stream comprises first data being backed up, and wherein the data stream originates at a first storage device that stores the first data, and

wherein performing the first backup process by the data agent comprises;

tracking in-process tasks of the first backup process and respective hierarchical relationships among the in-process tasks,

computing a first set of performance metrics of the tracked in-process tasks of the first backup process processing the data stream,

generating first performance data packets representing (a) the first set of performance metrics, and (b) associated hierarchical indicators representing the respective hierarchical relationships among the tracked in-process tasks, and

merging, by the first backup process into the data stream;

(i) the first performance data packets with (ii) data packets which result from the processing of the data stream by the first backup process;

transmitting the data stream, comprising the data packets which result from the processing of the data stream by the first backup process and further comprising the first performance data packets, by the first backup process, to a second backup process performed by the data agent; and

further merging into the data stream by the second backup process;

second performance data packets collectively representing at least a second set of performance metrics of the second backup process processing the data stream as received from the first backup process,

thereby resulting in the data stream comprising in-process performance data packets for the first backup process and also for the second backup process and further comprising data packets which result from the processing of the data stream by the first backup process and the second backup process.

5. The method of claim 4 wherein the performing of the first backup process by the data agent further comprises associating each performance metric in the first set with one or more hierarchical indicators based on the hierarchical relationships of the tracked in-process tasks of the first backup process.

6. The method of claim 4 further comprising:

transmitting, by the second backup process, the data stream comprising the data packets which result from the processing of the data stream by the first backup process and the second backup process, and further comprising the merged first performance data packets and second performance data packets, to a media agent component of the storage management system, wherein the media agent executes on a secondary storage computing device, which comprises one or more processors and computer-readable memory.

7. The method of claim 6 further comprising:

by the media agent, generating a secondary copy of the first data being backed up based on parsing the data packets which result from the processing of the data stream by the first backup process and the second backup process as received in the data stream from the data agent;

generating, by the media agent, a performance report for the data stream based on parsing the first performance data packets and the second performance data packets received in the data stream from the data agent,

wherein the performance report depicts at least the first set of performance metrics and the second set of performance metrics; and

storing the performance report, by the media agent, to a performance data log associated with the media agent.

8. The method of claim 6 wherein the performing of the first backup process by the data agent further comprises associating each performance metric in the first set with one or more hierarchical indicators based on the hierarchical relationships of the tracked in-process tasks of the first backup process;

and

further comprising generating, by the media agent, a performance report for the data stream based on parsing the first performance data packets and the second performance data packets received from the data agent,

wherein the performance report depicts at least the first set of performance metrics and hierarchical relationships therebetween.

9. The method of claim 8 further comprising:

storing the performance report, by the media agent, to a performance data log associated with the media agent.

10. The method of claim 8, wherein the performance report depicts the first set of performance metrics based on parsing the hierarchical indicators associated therewith, such that the performance report depicts parent-child relationships between in-process tasks performed by the data agent.

11. A method for accumulating performance metrics into a backup data stream in a storage management system, the method comprising:

receiving data stream from a data agent, by a media agent component of the storage management system,

wherein the media agent executes on a computing device, which comprises one or more processors and computer-readable memory, and

wherein the data stream comprises first data being backed up to a storage device in communication with the computing device, and further comprises first performance data packets representing one or more measures of performance of backup processes performed upon the data stream by the data agent;

performing, by the media agent, a first backup process upon the data stream,

wherein performing the first backup process comprises;

tracking in-process tasks of the first backup process and respective hierarchical relationships among the in-process tasks,

computing a first set of performance metrics of the tracked in-process tasks of the first backup process processing the data stream,

associating each performance metric in the first set with one or more hierarchical indicators based on the respective hierarchical relationships of the tracked in-process tasks of the first backup process,

generating second performance data packets representing (a) the first set of performance metrics, and (b) the associated one or more hierarchical indicators, and

merging, by the first backup process into the data stream;

(i) the second performance data packets with (ii) data packets which result from the processing of the data stream by the first backup process in the media agent,

thereby resulting in the data stream comprising the first performance data packets and the second performance data packets and further comprising data packets which result from the processing of the data stream by the first backup process;

generating, by the media agent, a secondary copy of the first data being backed up based on parsing the data packets which result from the processing of the data stream by the first backup process in the media agent; and

generating, by the media agent, a performance report for the data stream based at least in part on parsing the first performance data packets and the second performance data packets in the data stream.

12. The method of claim 11 further comprising:

storing the performance report, by the media agent, to a performance data log associated with the media agent.

13. The method of claim 11, wherein the performance report depicts the first set of performance metrics based at least in part on parsing the hierarchical indicators associated therewith.

14. The method of claim 11, wherein the performance report depicts the first set of performance metrics based at least in part on parsing the hierarchical indicators associated therewith, such that the performance report depicts parent-child relationships between in-process tasks performed by the first backup process of the media agent.

15. The method of claim 11 further comprising:

storing, by the media agent, a secondary copy of the first data to a second storage device which is in communication with the secondary storage computing device.

16. The method of method 15 further comprising:

accumulating, by the media agent, in addition to the first performance data packets and the second performance data packets, additional performance data packets generated by other backup processes performed by the media agent, wherein the additional performance data packets collectively comprise other performance metrics and associated hierarchical indicators relative to the other backup processes performed by the media agent; and

generating, by the media agent, the performance report for the data stream further based on parsing the additional performance data packets; and

wherein the secondary copy of the first data being backed up is further based on parsing data packets which result from the other backup processes performed by the media agent.