Apparatus, system and method for data collection, import and modeling

• US 9,411,865 B2
• Filed: 06/11/2013
• Issued: 08/09/2016
• Est. Priority Date: 08/07/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: McPhail, Iain
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Rayyan, Susan F
• Application Number:
  US13/914,719
• Publication Number:
  US 20140046952A1
• Time in Patent Office: 1,155 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A method of and system for data analysis, including interrogating a plurality of computer systems to generate for each computer system at least one dump file, each dump file including configuration and state information; extracting predetermined configuration and state information from the respective dump files; and storing the extracted configuration and state information in a database in a normalized format.

Claims

1. A method of data analysis for a backup system, the method comprising:

generating a dump file for each of a plurality of backup servers, each dump file comprising configuration and state information about each of the plurality of backup servers in a native format used by each of the plurality of backup servers on which data is stored, wherein the backup servers backup the data from a primary storage layer to a common media layer;

extracting a first predetermined configuration and state information from the respective dump files of the plurality of backup servers, the first predetermined configuration and state information being in different formats based on the dump file from which it was extracted;

translating the first predetermined configuration and state information from the format used by each of the plurality of backup servers into a normalized format, wherein the translated first configuration and state information comprises configuration and state information irrespective of which of the plurality of backup servers from which it was generated;

storing the translated first configuration and state information in a single database;

generating a dump file for each of a plurality of different computer systems of the primary storage layer, each dump file comprising configuration and state information about each of the plurality of computer systems in a format used by each of the plurality of computer systems on which the data is stored, wherein the plurality of computer systems include server computers, desktop computers, and laptop computers which are physically located across various sites and use hardware and software from different vendors;

extracting a second predetermined configuration and state information from the respective dump files of the plurality of different computer systems, the second predetermined configuration and state information being in different formats based on the dump file from which it was extracted;

translating the second predetermined configuration and state information from the format used by each of the plurality of computer systems into a normalized format, wherein the translated second configuration and state information comprises configuration and state information irrespective of which of the plurality of computer systems from which it was generated; and

storing the translated second configuration and state information in the single database;

determining what components are in the backup system, how the backup system works, how data is stored in the backup system, how efficiently data is stored in the backup system, a total capacity of the backup system, a remaining capacity of the backup system, and an operating cost of the backup system by analyzing the normalized first and second configuration and state information stored in the single database.

2. The method according to claim 1, wherein the format used by each of the plurality of computer systems is at least one of a vendor specific format and a product specific format.

3. The method according to claim 1, wherein generating the dump files for each of the plurality of backup servers and each of the plurality of different computer systems comprises:

querying each of the plurality of backup servers using an interface native to each of the plurality of backup servers; and

querying each of the plurality of different computer systems using an interface native to each of the plurality of different computer systems.

4. The method according to claim 3, wherein querying each of the plurality of computer systems using the interface native to each of the plurality of computer systems comprises:

running a predetermined series of commands, wherein the interface comprises a standard command line interface.

5. The method according to claim 1, wherein the configuration and state information relates to a plurality of objects stored on the plurality of backup servers and the plurality of different computer systems and comprises metadata about the plurality of objects.

6. The method according to claim 1, wherein the configuration and state information comprises metadata which describes a physical storage location of each object and includes customer data, site data, source data, node data, object data, and fragment data, wherein the fragment data describes a physical location of each the object.

7. A method of data analysis, the method comprising:

generating a dump file for each of a plurality of different computer systems, each dump file comprising product specific configuration and state information about each of the plurality of computer systems in a native format used by each of the plurality of computer systems on which data is stored;

extracting predetermined configuration and state information from the respective dump files of the plurality of different computer systems, the predetermined configuration and state information being in different native formats, wherein the predetermined configuration and state information comprises metadata which describes a physical storage location of each object and includes customer data, site data, source data, node data, object data, and fragment data, wherein the fragment data describes a physical location of each the object;

translating the predetermined configuration and state information from the native format used by each of the plurality of different computer systems into a normalized format, wherein the normalized configuration and state information comprises a format irrespective of which of the plurality of different computer systems from which it was generated; and

determining what components are in the backup system, how the backup system works, how data is stored in the backup system, how efficiently data is stored in the backup system, a total capacity of the backup system, a remaining capacity of the backup system, and an operating cost of the backup system by analyzing the normalized predetermined configuration and state information.

8. The method according to claim 7, wherein the native format used by each of the plurality of different computer systems is a vendor specific format.

9. The method according to claim 7, wherein generating the dump file for each of the plurality of different computer systems comprises:

querying each of the plurality of different computer systems using an interface native to each of the plurality of different computer systems.

10. The method according to claim 9, wherein querying each of the plurality of different computer systems using the interface native to each of the plurality of different computer systems comprises:

running a predetermined series of commands, wherein the interface comprises a standard command line interface.

11. The method according to claim 7, wherein the configuration and state information relates to a plurality of objects stored on the plurality of different computer systems and comprises metadata about the plurality of objects.

12. The method according to claim 7, wherein the plurality of different computer systems use at least two different proprietary data formats.

13. A method of data analysis, the method comprising:

generating a dump file for each of a plurality of different computer systems, each dump file comprising configuration and state information about each of the plurality of different computer systems in a native format used by each of the plurality of different computer systems on which data is stored;

extracting predetermined configuration and state information from the respective dump files of the plurality of different computer systems, the predetermined configuration and state information being in different native formats based on the dump file from which it was extracted;

translating the predetermined configuration and state information from the native format used by each of the plurality of different computer systems into a normalized format, wherein the translated configuration and state information comprises configuration and state information irrespective of which of the plurality of different computer systems from which it was generated; and

determining what components are in the backup system, how the backup system works, how data is stored in the backup system, how efficiently data is stored in the backup system, a total capacity of the backup system, a remaining capacity of the backup system, and an operating cost of the backup system by analyzing the normalized predetermined configuration and state information.

14. The method according to claim 13, wherein the native format used by each of the plurality of different computer systems is at least one of a vendor specific format and a product specific format.

15. The method according to claim 13, wherein generating the dump file for each of the plurality of different computer systems comprises:

querying each of the plurality of different computer systems using an interface native to each of the plurality of different computer systems.

16. The method according to claim 15, wherein querying each of the plurality of different computer systems using the interface native to each of the plurality of different computer systems comprises:

running a predetermined series of commands, wherein the interface comprises a standard command line interface.

17. The method according to claim 13, wherein the configuration and state information relates to a plurality of objects stored on the plurality of different computer systems and comprises metadata about the plurality of objects.

18. The method according to claim 13, wherein the configuration and state information comprises metadata which describes a physical storage location of each object and includes customer data, site data, source data, node data, object data, and fragment data, wherein the fragment data describes a physical location of each the object.

19. The method according to claim 13, wherein the plurality of different computer systems use at least two different proprietary data formats.