Monitoring, diagnosing, and repairing a management database in a data storage management system

• US 10,176,036 B2
• Filed: 10/25/2016
• Issued: 01/08/2019
• Est. Priority Date: 10/29/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Haridas, Ganesh, Narasimman, Sundar, Suriyanarayanan, Karthik, A S, Gunassekaran, Klose, Michael Frank
• Primary Examiner(s): McCarthy, Christopher S
• Application Number:
  US15/334,014
• Publication Number:
  US 20170123889A1
• Time in Patent Office: 805 Days
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

A lightweight always-on monitoring, collecting, diagnosing, and correcting utility operates in an enhanced storage manager that manages a data storage managements system. The always-on utility provides a comprehensive and pro-active approach, which is intended to reduce, if not altogether eliminate, the need for after-the-fact diagnostics. The always-on utility also enforces so-called best practices and other heuristics, which include pro-actively activating certain database settings that are not enabled by default; manipulating certain aspects of the database to improve performance; and reporting aspects that are outside best-practice parameters to the trouble report system so that system administrators and/or developers may intervene before a catastrophic failure occurs. In some cases, the best-practice parameters represent heuristics designed by the present inventors to improve the performance and general health of the management database.

Claims

1. A system comprising:

a computing device for executing a storage manager for managing a data storage management system,

wherein the computing device comprises one or more processors and non-transitory computer-readable memory for executing programming instructions from the computer-readable memory;

a database management system associated with the storage manager,

wherein the database management system comprises a database that stores information used by the storage manager to manage the data storage management system, including to manage data storage operations in the data storage management system, and

wherein the database management system comprises a plurality of native utilities for performing database management operations;

wherein the storage manager is programmed to;

monitor the database during active operation of the data storage management system,

diagnose a problem associated with the database, based on information about the database collected in the monitor operation, and

correct the diagnosed problem, at least in part, by instructing the database management system to perform a database management operation using one or more native utilities among the plurality of native utilities;

wherein the database is stored on a first mass storage device accessible by the storage manager; and

wherein the storage manager is further programmed to;

classify one or more indexes in the database as being busier, based on how often they are invoked, relative to other indexes in the database, and

move the one or more indexes classified as being busier from the first mass storage device to a second mass storage device, which is distinct from the first mass storage device and is also accessible by the storage manager.

2. The system of claim 1 wherein to correct the diagnosed problem, the storage manager is programmed to instruct the database management system to instantiate a number of additional temporary-database data structures in the database.

3. The system of claim 2 wherein the instantiated number of additional temporary-database data structures improve a performance metric of the database, wherein the performance metric caused the storage manager to diagnose the problem.

4. The system of claim 2 wherein the instantiated number of additional temporary-database data structures improve a performance metric of the database, wherein the performance metric measures responsiveness of the database.

5. The system of claim 2 wherein the instantiated number of additional temporary-database data structures improve a performance metric of the database, wherein the performance metric measures responsiveness of the database to one or more of:

read operations, write operations, and query processing.

6. The system of claim 1 wherein moving the one or more indexes classified as being busier to the second mass storage device from the first mass storage device improves a performance metric of the database.

7. The system of claim 1 wherein moving the one or more indexes classified as being busier to the second mass storage device from the first mass storage device improves a performance metric of the database, wherein the performance metric measures responsiveness of the database to one or more of:

read operations, write operations, and query processing.

8. The system of claim 1 wherein to monitor the database, the storage manager is programmed to instruct the database management system to check for data corruption in the database;

and

wherein to correct the diagnosed problem, the storage manager is programmed to;

(a) if the database management system reports that one or more database indexes are corrupted, instruct the database management system to perform at least one of;

re-generate the one or more database indexes and re-index the database, and

(b) if the database management system reports data corruption or data loss other than in the database indexes, initiate a database restore operation in the data storage management system.

9. The system of claim 1 wherein the storage manager is further programmed to:

request database health reports from the database management system,

identify database health issues by analyzing the database health reports, and

classify the identified database health issues based on importance of the respective database health issue to operation of the data storage management system.

10. The system of claim 1 wherein the database management system is a version of SQL server, and wherein the database is an SQL database.

11. The system of claim 1 wherein the database is a relational database that is organized according to a schema.

12. The system of claim 1 wherein the database management system executes on the computing device that hosts the storage manager.

13. The system of claim 1,

wherein the storage manager is further programmed to:

evaluate database resources that are locked in query execution during active operation of the data storage management system, and

generate a trouble report that identifies one or more database resources that remain locked longer than a pre-defined threshold.

14. The system of claim 13 wherein the storage manager is further programmed to transmit the trouble report to a trouble report system in communication with the storage manager.

15. The system of claim 13 wherein the one or more database resources that remain locked longer than the pre-defined threshold prevent further queries from executing, thereby causing the further queries to fail.

16. The system of claim 13 wherein the storage manager is further programmed to identify one or more queries to the management database that cause the one or more database resources to remain locked longer than the pre-defined threshold and to include the one or more queries in the trouble report.

17. A system comprising:

a computing device for executing a storage manager for managing a data storage management system,

wherein the computing device comprises one or more processors and non-transitory computer-readable memory for executing programming instructions from the computer-readable memory;

a database management system associated with the storage manager,

wherein the database management system comprises a database that stores information used by the storage manager to manage the data storage management system, including to manage data storage operations in the data storage management system, and

wherein the database management system comprises a plurality of native utilities for performing database management operations; and

wherein the storage manager is programmed to;

monitor the database during active operation of the data storage management system,

diagnose a problem associated with the database, based on information about the database collected in the monitor operation, and

correct the diagnosed problem, at least in part, by instructing the database management system to perform a database management operation using one or more native utilities among the plurality of native utilities; and

wherein to correct the diagnosed problem, the storage manager is programmed to;

determine, based on fragmentation information obtained from the database management system, whether a first table in the database exceeds a first fragmentation threshold,

if the first table in the database exceeds the first fragmentation threshold, instruct the database management system to re-index the first table, and

if the first table in the database does not exceed the first fragmentation threshold and exceeds a second fragmentation threshold, instruct the database management system to reorganize the first table.

18. The system of claim 17 wherein at least one of the re-indexing and the reorganizing of the first table by the database management system in response to instructions from the storage manager improves a performance metric of the database.

19. A system comprising:

a computing device for executing a storage manager for managing a data storage management system,

wherein the computing device comprises one or more processors and non-transitory computer-readable memory for executing programming instructions from the computer-readable memory;

a database management system associated with the storage manager,

wherein the database management system comprises a database that stores information used by the storage manager to manage the data storage management system, including to manage data storage operations in the data storage management system, and

wherein the database management system comprises a plurality of native utilities for performing database management operations;

wherein the storage manager is programmed to;

monitor the database during active operation of the data storage management system,

diagnose a problem associated with the database, based on information about the database collected in the monitor operation, and

correct the diagnosed problem, at least in part, by instructing the database management system to perform a database management operation using one or more native utilities among the plurality of native utilities;

wherein the database that stores information used by the storage manager is designated a management database; and

a report server comprising one or more processors and computer-readable memory for executing programming instructions from the computer-readable memory, and

wherein the report server further comprises a second database for storing historical information about the management database; and

wherein the storage manager is further programmed to;

extract, from the database management system, natively-generated statistics about the performance of the management database, wherein the natively-generated statistics are lost when the database management system resets the management database, and

transmit the extracted natively-generated statistics about database performance to the report server for storage to the second database, thereby overcoming loss of the natively-generated statistics on reset of the management database.