Self healing file system

• US 7,694,191 B1
• Filed: 06/30/2007
• Issued: 04/06/2010
• Est. Priority Date: 06/30/2007
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Clark, Morgan A., Mullick, Sachin, Bono, Jean-Pierre, Veeraswamy, Sairam, Vahalia, Uresh K., Hayden, John M.
• Primary Examiner(s): Maskulinski; Michael C
• Application Number:
  US11/772,149
• Time in Patent Office: 1,011 Days
• Field of Search: None
• US Class Current: 714/48
• CPC Class Codes: G06F 11/004 Error avoidance G06F11/07 and subgroups take precedence
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 2211/104 Metadata, i.e. metadata associated with RAID systems with parity

Abstract

A self healing file system is designed for proactive detection and containment of faults, errors, and corruptions, in order to enable in place (online) and non-intrusive recovery. For proactive fault detection, the file system maintains certain per-block metadata of each file system block. The per-block metadata includes a redundancy check, and for file system data blocks, an inode identifier, and an offset of the file system data block in the file including the file system data block. The redundancy check is used to detect and mark bad file system blocks. The inode identifier and offset is used for validating connectivity of the file system blocks to the inodes, and for tracing bad blocks to files that contain the bad blocks.

Claims

1. A file server comprising:

data storage; and

at least one data processor coupled to the data storage for accessing the data storage;

wherein said at least one data processor is programmed for maintaining a file system in the data storage, the file system including file system blocks, the file system blocks including inodes of metadata of files in the file system, and data blocks of data of the files in the file system, and

wherein said at least one data processor is programmed for maintaining per-block metadata in the file system for each of the file system blocks, the per-block metadata including a redundancy check for detecting error in said each of the file system blocks, and for each of the file system data blocks, an inode identifier for identifying an associated one of the inodes of metadata of files in the file system, and an offset of said each of the file system data blocks in the file of the associated one of the inodes; and

wherein said at least one data processor is programmed for using the per-block metadata in the file system for detecting error in at least one of the file system blocks as indicated by the redundancy check for said at least one of the file system blocks, and for detecting error in at least one of said files as indicated by the inode identifier and offset in the per-block metadata for a file system data block of said at least one of the files.

2. The file server as claimed in claim 1, wherein said at least one data processor is programmed for using the per-block metadata in the file system for detecting error in at least one of said file system blocks by computing a cyclic redundancy check over the file system block and comparing the cyclic redundancy check to the redundancy check in the per-block metadata for said at least one file system block.

3. The file server as claimed in claim 1, wherein said at least one data processor is further programmed for responding to detection of an error in said at least one of said file system blocks by setting an indication of error in a block number field containing a block number of said at least one of said file system blocks.

4. The file server as claimed in claim 1, wherein said at least one data processor is further programmed for responding to detection of an error in said at least one of said file system blocks by tracing the error to an offset in a file in the file system as indicated by the offset and the inode identifier of the per-block metadata of said at least one of the file system blocks.

5. The file server as claimed in claim 1, wherein said at least one data processor is further programmed for detecting error when a free file system block is being allocated to a file in the file system upon checking the per-block metadata for the free file system block and finding that per-block metadata for the free file system block indicates that the free file system block already is assigned to an inode.

6. The file server as claimed in claim 1, wherein said at least one data processor is further programmed for detecting error when an allocated file system block is being de-allocated from a file in the file system upon checking the per-block metadata for the allocated file system block and finding that per-block metadata for the allocated file system block indicates that the allocated file system block already is not associated with an inode.

7. The file server as claimed in claim 1, wherein said at least one data processor is further programmed for performing file system recovery upon reading an indirect block from storage and finding that the redundancy check in the per-block metadata for the indirect block indicates error in the indirect block as read from storage.

8. The file server as claimed in claim 1, wherein the file server has a file system cache memory, and said at least one data processor is further programmed for resetting said at least one data processor upon reading an indirect block from the file system cache memory and finding that the redundancy check in the per-block metadata for the indirect block indicates error in the indirect block as read from the file system cache memory.

9. The file server as claimed in claim 1, wherein the file system includes an inode of a production file, and inodes of snapshot copies of the production file, and the inode of the production file is linked to the inodes of the snapshot copies to form a version chain having double links, each double link including a forward pointer and a reverse pointer, and said at least one data processor is programmed for using the double links in the version chain for detecting and correcting a single link error in the version chain.

10. The file server as claimed in claim 1, wherein the per-block metadata for each of the file system blocks includes an internal checksum, and said at least one processor is further programmed for responding to an error indicated by the internal checksum of the per-block metadata by attempting to rebuild the per-block metadata from inode and block linkages of the file system.

11. The file server as claimed in claim 1, wherein said at least one data processor is further programmed to repair the file system by awarding a certain file system block of the file system to an inode of the file system if:

(1) the inode claims the certain file system block; and

(2) the redundancy check in the per-block metadata of the inode is valid; and

(3) the certain file system block is marked allocated in a cylinder group bitmap of the file system and the cylinder group bitmap has a valid redundancy check; and

(4) the per-block metadata for the certain file system block says that the inode owns the certain file system block and an internal redundancy check for the per-block metadata of the certain file system block indicates that the per-block metadata for the certain file system block is valid, or else the internal redundancy check for the per-block metadata of the certain file system block indicates that the per-block metadata for the certain file system block is invalid; and

(5) no other inode with a valid redundancy check claims the certain file system block.

12. A method of operating a file server, said method comprising the steps of:

maintaining a file system in data storage of the file server, the file system including file system blocks, the file system blocks including inodes of metadata of files in the file system, and data blocks of data of the files in the file system; and

maintaining per-block metadata in the file system for each of the file system blocks, the per-block metadata including a redundancy check for detecting error in said each of the file system blocks, and for each of the file system data blocks, an inode identifier for identifying an associated one of the inodes of metadata of files in the file system, and an offset of said each of the file system data blocks in the file of the associated one of the inodes; and

using the per-block metadata in the file system for detecting error in at least one of the file system blocks as indicated by the redundancy check for said at least one of the file system blocks, and for detecting error in at least one of the files as indicated by the inode identifier and offset in the per-block metadata for a file system data block of said at least one of the files.

13. The method as claimed in claim 12, which further includes responding to detection of an error in said at least one of said file system blocks by setting an indication of error in a block number field containing a block number of said at least one of said file system blocks.

14. The method as claimed in claim 12, which further includes responding to detection of an error in said at least one of said file system blocks by tracing the error to an offset in a file in the file system as indicated by the offset and the inode identifier of the per-block metadata of said at least one of the file system blocks.

15. The method as claimed in claim 12, which further includes detecting error when a free file system block is being allocated to a file in the file system upon checking the per-block metadata for the free file system block and finding that per-block metadata for the free file system block indicates that the free file system block already is assigned to an inode.

16. The method as claimed in claim 12, which further includes detecting error when an allocated file system block is being de-allocated from a file in the file system upon checking the per-block metadata for the allocated file system block and finding that per-block metadata for the allocated file system block indicates that the allocated file system block already is not associated with an inode.

17. The method as claimed in claim 12, which further includes performing file system recovery upon reading an indirect block from storage and finding that the redundancy check in the per-block metadata for the indirect block indicates error in the indirect block as read from storage.

18. A method of operating a file server, said method comprising the steps of:

maintaining a file system in data storage of the file server, the file system including file system blocks, the file system blocks including inodes of metadata of files in the file system, and data blocks of data of the files in the file system;

maintaining per-block metadata in the file system for each of the file system blocks, the per-block metadata including a redundancy check for detecting error in said each of the file system blocks, and for each of the file system data blocks, an inode identifier for identifying an associated one of the inodes of metadata of files in the file system, and an offset of said each of the file system data blocks in the file of the associated one of the inodes; and

using the per-block metadata in the file system during a check of the file system by using the redundancy checks for validating the file system blocks for absence of error and by using the inode identifiers and offsets for validating connectivity of the file system data blocks to the inodes.

19. The method as claimed in claim 18, wherein the check of the file system includes a first phase of checking files of the file system, and a second phase of directory validation,

wherein the checking of each file in the first phase begins with a single thread checking the file system block hierarchy of the file until the single thread finds an indirect block of at least double indirection, and upon finding an indirect block of at least double indirection, the indirect block of at least double indirection is placed on a queue for checking of the file system block hierarchy of the file by another thread beginning with the indirect block of at least double indirection.

20. The method as claimed in claim 18, wherein the check of the file system includes a first phase of checking files of the file system, and a second phase of directory validation,

wherein the second phase of directory validation is performed by threads servicing a queue of small directories, a queue of medium size directories, and queue of large directories, in which only one large directory is under check at any given time, and medium directories are checked so long as the sum of the sizes of the medium directories being checked, plus the size of any large directory being checked, does not exceed a certain limit, and otherwise an available thread checks any small directory in the queue of small directories.