Reducing file system data unavailability window by adapting hierarchical recovery framework

• US 8,924,773 B1
• Filed: 06/29/2012
• Issued: 12/30/2014
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Veeraswamy, Sairam
• Primary Examiner(s): MCCARTHY, CHRISTOPHER S
• Application Number:
  US13/538,126
• Time in Patent Office: 914 Days
• US Class Current: 714/6.1
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

A storage processor isolates and classifies file system faults and inconsistencies to produce a fault tree for inline error recovery. The storage processor isolates the metadata objects or objects that have internal inconsistencies, the storage processor creates a fault tree to map the file system; or portion of the file system with the errors or inconsistencies and the metadata objects dependent upon those objects. Once the fault having been mapped, determine the complexity of recovering the branch of the fault tree with the error. If the error is not too complex, performing recovery of the metadata objects online otherwise taking the system offline for file system recovery.

Claims

1. A method to reduce file system data unavailability upon detecting a fault in the file system, the file system including a hierarchy of objects, the hierarchy of objects stored on slices of a storage device group, the method comprising:

creating a fault tree representative of the hierarchy of objects;

identifying within the fault tree a fault branch, the fault branch identified by an object of the hierarchy of objects producing the fault in the file system;

denying access by the file system to objects of the fault branch of the fault tree;

repairing objects in the hierarchy of objects identified by the fault branch of the fault tree; and

permitting access by the file system to objects in the hierarchy of objects identified by the fault branch of the fault tree.

2. The method of claim 1, wherein repairing objects of the fault branch of the fault tree includes:

creating a recovery plan to bring a first set of objects of the hierarchy of objects identified by the fault branch of the fault tree into consistent reference to a second set of objects of the hierarchy of objects identified by of the fault branch of the fault tree;

setting a numeric value indicating a complexity level of the recovery plan;

upon the numeric value indicating the complexity level of the recovery plan being less than or equal to a predefined complexity level;

recovering valid values of the objects in the hierarchy of objects identified by fault branch of the fault tree;

allowing access by the file system to the objects of the hierarchy of objects identified by the fault branch of the fault tree;

upon the numeric value indicating the complexity level of the recovery plan being greater than the predefined complexity level;

saving the fault tree and the recovery plan to a save area; and

marking the file system for offline recovery.

3. The method of claim 2, wherein recovering the valid values of the objects includes:

identifying by reference to the object of the hierarchy of objects producing the fault, a slice of the slices of the storage device group on which the object is stored producing a hard error; and

replacing the slice with a replacement slice of the slices of the storage device group.

4. The method of claim 2, wherein recovering the valid values of the objects of the hierarchy of objects identified by the fault tree includes:

locating objects of the hierarchy of objects identified by the fault branch of the fault tree with inconsistent references to other objects of the hierarchy of objects identified by the fault branch of the fault tree and setting the inconsistent references to valid values based on a set of known good objects of the hierarchy of objects identified by the fault branch of the fault tree.

5. The method of claim 2, wherein the marking the file system for offline recovery includes:

taking the file system offline;

determining the numeric value indicating the complexity level of the recovery plan saved in the save area;

upon the numeric value indicating the complexity level of the recovery plan being less than or equal to a second predefined complexity level;

recovering the valid values of objects in the fault tree;

upon the numeric value indicating the complexity level of the recovery plan being greater than the second predefined complexity level;

performing a full file system recovery; and

bringing the file system online.

6. The method of claim 2, wherein the numeric value indicating the complexity level of the recovery plan is based on a number of objects required to be set to a consistent state.

7. The method of claim 2, wherein detecting the fault in the file system includes at least one of:

receiving the fault in the I/O path of the object of the hierarchy of objects producing the fault, and

performing as a background process consistency checking of the hierarchy of objects.

8. The method of claim 1, wherein the file system includes an inode;

and

wherein creating the fault tree includes mapping the inode of the file system to a fault node of the fault branch of the fault tree.

9. A storage system that reduces file system data unavailability, the system comprising:

a storage device group providing persistent storage of file system data,

the file system data including a hierarchy of objects, the hierarchy of objects stored onto slices of the storage device group;

a controller circuit constructed and configured to;

detect a fault;

create a fault tree representative of the hierarchy of objects;

identify within the fault tree a fault branch, the fault branch identified by an object of the hierarchy of objects producing the fault in the file system;

deny access by the file system to objects of the hierarchy of objects identified by the fault branch of the fault tree;

repair objects in the hierarchy of objects identified by the fault branch of the fault tree; and

permit access by the file system to objects of the hierarchy of objects identified by the fault branch of the fault tree.

10. The system of claim 9, wherein the controller circuit configured to repair objects in the fault branch is further configured to:

create a recovery plan to bring a first set of objects of the hierarchy of objects identified by the fault branch of the fault tree into consistent reference to a second set of objects of the hierarchy of objects identified by the fault branch of the fault tree;

setting a numeric value that indicates a complexity level of the recovery plan;

upon the numeric value that indicates the complexity level of the recovery plan being within a predefined complexity level;

recover valid values of objects in the fault tree;

allow access to branch of the fault tree by the file system;

upon the numeric value indicating the complexity level of the recovery plan being outside the predefined complexity level;

save the fault tree and the recovery plan to a save area; and

mark the file system for offline recovery.

11. The system of claim 10, wherein the controller circuit configured to recover valid values of objects of the hierarchy of objects identified by the fault tree is further configured to:

identify by reference to the object of the hierarchy of objects producing the fault, a slice of the slices of the storage device group on which the object is stored producing a hard error; and

replace the slice with a replacement slice of the slices of the storage device group.

12. The system of claim 10, wherein the controller circuit configured to recover valid values of objects of the hierarchy of objects identified by the fault tree is further configured to:

locate objects of the hierarchy of objects identified by the fault branch of the fault tree with inconsistent references to other objects of the hierarchy of objects identified by the fault branch in the fault tree and setting the inconsistent references to valid values based on a set of known good objects of the hierarchy of objects identified by the fault branch of the fault tree.

13. The system of claim 10, wherein the controller circuit configured to mark the file system for offline recovery is further configured to:

take the file system offline;

determine the numeric value indicating the complexity level of the recovery plan saved in the save area;

upon the numeric value that indicates the complexity level of the recovery plan being within a second predefined complexity level;

recover the valid values of objects of the hierarchy of objects identified by the fault tree;

upon the numeric value that indicates the complexity level of the recovery plan being outside the second predefined complexity level;

perform a full file system recovery; and

bring the file system online.

14. The system of claim 10, wherein the numeric value indicating the complexity level of the recovery plan is based on a number of objects of the hierarchy of objects identified by the fault branch of the fault tree required to be set to a consistent state.

15. The system of claim 10, wherein the controller circuit configured to detect a fault in the file system is further configured to:

receive the fault in the I/O path of the object of the hierarchy of objects producing the fault, and

perform as a background process consistency checking of the hierarchy of objects.

16. A non-transitory computer readable storage medium with a computer program stored thereon, the computer program executed by a processor to perform the method of reducing file system data unavailability upon detecting a fault in the file system, the file system including a hierarchy of objects, the hierarchy of objects stored on slices of a storage device group, the method comprising:

creating a fault tree representative of the hierarchy of objects;

identifying within the fault tree a fault branch, the fault branch identified by an object of the hierarchy of objects producing the fault in the file system;

denying access by the file system to objects of the fault branch of the fault tree;

repairing objects in the hierarchy of objects identified by the fault branch of the fault tree; and

permitting access by the file system of objects in the hierarchy of objects identified by the fault branch of the fault tree.

17. The non-transitory computer readable storage medium of claim 16, wherein repairing objects of the fault branch of the fault tree includes:

creating a recovery plan to bring a first set of objects of the hierarchy of objects identified by the fault branch of the fault tree into consistent reference to a second set of objects of the hierarchy of objects identified by of the fault branch of the fault tree;

setting a numeric value indicating a complexity level of the recovery plan;

upon the numeric value indicating the complexity level of the recovery plan being less than or equal to a predefined complexity level;

recovering valid values of the objects in the hierarchy of objects identified by fault branch of the fault tree;

accessing the objects of the hierarchy of objects identified by the fault branch of the fault tree by the file system;

upon the numeric value indicating the complexity level of the recovery plan being greater than the predefined complexity level;

saving the fault tree and the recovery plan to a save area; and

marking the file system for offline recovery.

18. The non-transitory computer readable storage medium of claim 17, wherein recovering the valid values of the objects includes:

identifying by reference to the object of the hierarchy of objects producing the fault, a slice of the slices of the storage device group on which the object is stored producing a hard error; and

replacing the slice with a replacement slice of the slices of the storage device group.

19. The non-transitory computer readable storage medium of claim 17, wherein recovering the valid values of the objects of the hierarchy of objects identified by the fault tree includes:

locating objects of the hierarchy of objects identified by the fault branch of the fault tree with inconsistent references to other objects of the hierarchy of objects identified by the fault branch of the fault tree and setting the inconsistent references to valid values based on a set of known good objects of the hierarchy of objects identified by the fault branch of the fault tree.

20. The non-transitory computer readable storage medium of claim 17, wherein marking the file system for offline recovery includes:

taking the file system offline;

determining the numeric value indicating the complexity level of the recovery plan saved in the save area;

upon the numeric value indicating the complexity level of the recovery plan being less than or equal to a second predefined complexity level;

recovering the valid values of objects in the fault tree;

upon the numeric value indicating the complexity level of the recovery plan being greater than the second predefined complexity level;

performing a full file system recovery; and

bringing the file system online.

21. The non-transitory computer readable storage medium of claim 17, wherein the numeric value indicating the complexity level of the recovery plan is based on a number of objects of the hierarchy of objects identified by the fault branch of the fault tree required to be set to a consistent state.

22. A method to increase file system data availability upon detecting a fault in a file system including a hierarchy of objects, the hierarchy of objects being stored on slices of a storage device group, the method comprising:

creating a fault tree representative of the hierarchy of objects;

identifying within the fault tree a fault branch, the fault branch being identified by an object of the hierarchy of objects producing the fault in the file system;

denying access by the file system to objects of the fault branch of the fault tree;

repairing objects of the fault branch of the fault tree; and

permitting access by the file system to the objects of the fault branch of the fault tree;

wherein repairing objects of the fault branch of the fault tree includes;

creating a recovery plan to bring a first set of objects of the fault branch of the fault tree into consistent reference to a second set of objects of the fault branch of the fault tree;

setting a numeric value indicating a complexity level of the recovery plan;

upon the numeric value indicating the complexity level of the recovery plan being less than or equal to a predefined complexity level;

recovering valid values of the objects of the fault branch of the fault tree; and

allowing access by the file system to the objects of the fault branch of the fault tree; and

upon the numeric value indicating the complexity level of the recovery plan being greater than the predefined complexity level;

saving the fault tree and the recovery plan to a save area; and

marking the file system for offline recovery.