Systems and methods of unified reconstruction in storage systems

• US 8,930,496 B2
• Filed: 12/15/2006
• Issued: 01/06/2015
• Est. Priority Date: 12/19/2005
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Primary Examiner(s): Dharia, Rupal
• Assistant Examiner(s): Nguyen, Van Kim T
• Application Number:
  US11/639,855
• Publication Number:
  US 20070226535A1
• Time in Patent Office: 2,944 Days
• Field of Search: None
• US Class Current: 709/219
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Systems and methods for reconstructing unified data in an electronic storage network are provided which may include the identification and use of metadata stored centrally within the system. The metadata may be generated by a group of storage operation cells during storage operations within the network. The unified metadata is used to reconstruct data throughout the storage operation cells that may be missing, deleted or corrupt.

Claims

1. A system for maintaining metadata in an electronic storage network comprising:

a plurality of networks electronically linked to each other, the plurality of networks having a storage operation capability and which include client applications; and

primary data stored on at least a first storage device;

at least a first and second media agents comprising computer hardware having one or more computer processors, each of the first and second media agents configured to create at least secondary copies of first and second portions of the primary data on at least a second storage device,

wherein a first media agent creates a first set of metadata that is stored in a first metadata storage device, wherein the second media agent creates a second set of metadata that is stored in a second metadata storage device;

a centralized metadata storage device, the first and second media agents configured to copy the first and second sets of metadata to the centralized metadata storage device to create at least a copy of the first set of meta data and a copy of the second set of metadata on the centralized metadata storage device, wherein the centralized metadata storage device is different than the first and second metadata storage devices; and

at least a storage management component comprising computer hardware having one or more computer processors, the storage management component configured to identify the first media agent in association with the copy of the first set of metadata and the second media agent in association with the copy of the second set of metadata stored in the centralized metadata storage device, the at least one storage management component is further configured to;

determine whether the second data storage device that stores the secondary copies of the first and second portions of the primary data has a unique storage device identifier;

store the unique identifier of the second data storage device in association with the first and second sets of metadata stored on the centralized metadata storage device;

detect a loss of at least a portion of the first set of metadata on the first metadata storage device;

use information about the first media agent to locate the copy of the first set of metadata stored on the centralized metadata storage device;

reconstruct the loss of the portion of the first set of metadata on the first metadata storage device using the copy of the first set of metadata stored on the centralized metadata storage device based on the association of the copy of the first set of metadata with the first media agent; and

use the unique storage device identifier to locate the second storage device having the secondary copy of the first portion of the primary data.

2. The system of claim 1 wherein the storage management component identifies, collects, and stores metadata relating to system operation.

3. The system of claim 1 wherein the storage management component detects an interruption of metadata transfer from a first system component to a second system component.

4. The system of claim 3 wherein the storage management component compares the metadata in the first system component with metadata in the second system component to determine differences, if any.

5. The system of claim 4 wherein the storage management component determines whether metadata from the first system component can be transferred to the second system such that the first and second system components have substantially the same metadata if differences are determined.

6. The system of claim 5 wherein the storage management component coordinates transfer metadata from the first system component to the second system component if the first system component has metadata necessary to make the metadata in the second system component substantially the same as the metadata in the first system component.

7. The system of claim 5 wherein the storage management component searches the storage network for the metadata missing from the second system component if the storage management component determines that the missing metadata is not available from the first system component.

8. The system of claim 5 wherein the storage management component searches other components within a storage operation cell.

9. The system of claim 5 wherein the storage management component searches other storage operation cells.

10. The system of claim 5 wherein the storage management component searches other storage management components in other storage operation cells.

11. The system of claim 7 wherein the storage management component identifies portions of the missing metadata in different locations.

12. The system of claim 11 wherein the storage management component obtains copies the identified missing portions from the different locations to the storage device.

13. The system of claim 12 wherein the storage management component reconstructs the missing metadata from the identified missing portions.

14. The system of claim 13 wherein the storage management component reconstructs the missing metadata by integrating the identified missing portions into a metadata arrangement similar to or the same as an original arrangement of the missing metadata.

15. The system of claim 14 wherein the identified missing portions into a metadata are arranged based temporal criteria.

16. The system of claim 12 wherein the storage management component analyzes identified missing portions to determine whether the missing metadata may be reconstructed.

17. The system of claim 16 wherein the storage management component performs additional network searches if the identified missing portions of metadata are insufficient to reconstruct the missing metadata.

18. The system of claim 14 wherein the storage management component reconstructs the missing metadata on the storage devices and copies the reconstructed missing metadata from the storage device to the second system component.

19. The system of claim 7 wherein the missing metadata is not available from the first system component because the missing metadata is corrupted, has been deleted or is unavailable due to hardware or software failure.

20. A method for maintaining and restoring metadata in an electronic storage network which includes a client computer networked to at least another client computer or a server, comprising:

managing a storage operation network including client applications, running on the client computer, with a storage management component, resident on at least one of the client computer, the other client computer, or the server;

copying with at least first and second media agents comprising computer hardware, first and second portions of primary data stored on at least a first storage device to at least a second storage device to create at least secondary copies of the first and second portions of the primary data on the second storage device, wherein a first media agent creates a first set of metadata that is stored in a first metadata storage device, and wherein the second media agent creates a second set of metadata that is stored in a second metadata storage device;

copying the first and second sets of metadata to a centralized metadata storage device to create a copy of the first set of metadata and a copy of the second set of metadata on the centralized metadata storage device, wherein the centralized metadata storage device is different than the first and second metadata storage devices; and

associating the first media agent with the copy of the first set of metadata stored in the centralized storage device, and associating the second media agent with the copy of the second set of metadata stored in the centralized metadata storage device;

determining whether the second data storage device that stores the secondary copies of the first and second portions of the primary data includes a unique storage device identifier;

storing the unique storage device identifier of the second data storage device in association with the first and second sets of metadata stored on the centralized metadata storage device;

detect a loss of at least a portion of the first set of metadata on the first metadata storage device;

use information about the first media agent to locate the copy of the first set of metadata stored on the centralized metadata storage device;

reconstructing the loss of the portion of the first set of metadata on the first metadata storage device using the copy of the first set of metadata stored on the centralized metadata storage device based on the association of the copy of the first set of metadata with the first media agent; and

using the unique storage identifier to locate the second storage device having the secondary copy of the first portion of the primary data.

21. The method of claim 20 further comprising identifying, collecting, and storing metadata relating to system operation.

22. The method of claim 20 further comprising detecting an interruption of metadata transfer from a first system component to a second system component.

23. The method of claim 22 further comprising compares the metadata in the first system component with metadata in the second system component to determine differences, if any.

24. The method of claim 23 further comprising determining whether metadata from the first system component can be transferred to the second system such that the first and second system components have substantially the same metadata if differences are determined.

25. The method of claim 24 further comprising coordinating transfer of metadata from the first system component to the second system component if the first system component has metadata necessary to make the metadata in the second system component substantially the same as the metadata in the first system component.

26. The method of claim 24 further comprising searching the storage network for the metadata missing from the second system component if the storage management component determines that the missing metadata is not available from the first system component.

27. The method of claim 26 wherein the searching further comprises searching other components within a storage operation cell.

28. The method of claim 26 wherein the searching further comprises searching other storage operation cells.

29. The method of claim 26 wherein the searching further comprises searching other storage management components in other storage operation cells.

30. The method of claim 26 further comprising identifying portions of the missing metadata in different locations.

31. The method of claim 30 further comprising obtaining copies of the identified missing portions from the different locations to the storage device.

32. The method of claim 31 further comprising reconstructing the missing metadata from the identified missing portions.

33. The method of claim 32 wherein the reconstructing further comprises integrating the identified missing portions into a metadata arrangement similar to or the same as an original arrangement of the missing metadata.

34. The method of claim 33 further comprising arranging the identified missing portions into a metadata on a temporal basis.

35. The method of claim 31 further comprising analyzing identified missing portions to determine whether the missing metadata may be reconstructed.

36. The method of claim 35 further comprising performing additional network searches if the identified missing portions of metadata are insufficient to reconstruct the missing metadata.

37. The method of claim 33 further comprising:

reconstructing the missing metadata on the storage devices; and

copying the reconstructed missing metadata from the storage device to the second system component.

38. The method of claim 26 wherein the missing metadata is not available from the first network component because the missing metadata is corrupted, has been deleted or is unavailable due to hardware or software failure.