Data processing system with mechanism for restoring file systems based on transaction logs

• US 6,732,124 B1
• Filed: 02/09/2000
• Issued: 05/04/2004
• Est. Priority Date: 03/30/1999
• Status: Expired due to Fees

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Yamaguchi, Satoru, Taniwaki, Sadayoshi, Koseki, Michihiko, Sumi, Masashi, Yokoyama, Mamoru, Hamanaka, Seishiro
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US09/501,568
• Time in Patent Office: 1,546 Days
• Field of Search: 707/200-205, 707/7-9, 707/1, 707/100-104, 714/15-20, 714/7, 711/113-115, 711/119-120, 711/145-147, 711/100, 711/161-162, 711/142-143, 711/152, 709/211, 709/215-216, 709/223-225, 709/229, 710/200, 705/404
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 2212/466 Metadata, control data
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A data processing system having an efficient logging mechanism which stores log records for repairing a file system when its consistency is lost. When there is a transaction attempting to update metadata stored in metadata volumes, a metadata loading unit reads the requested metadata objects out of the volumes and loads them to a metadata cache. At that time, a metadata manager updates its internal database to record from which metadata volume each metadata object has been fetched. Each time the transaction updates a metadata object in the cache, a log collection unit collects a copy of the updated metadata object, together with a volume ID which indicates its home metadata volume. The collected data is temporarily stored in a log buffer, and finally saved into a log volume by a log writing unit.

Claims

1. A data processing system with a logging mechanism which stores log records for repairing an inconsistent file system, the system comprising:

a primary storage subsystem;

a secondary storage subsystem;

a plurality of metadata volumes, created in said secondary storage subsystem, which store a plurality of metadata objects describing files;

a log volume, created in said secondary storage subsystem, which stores log records describing updates made to the metadata objects;

a metadata cache, created in said primary storage subsystem, which temporarily stores the metadata objects;

metadata loading means, responsive to a transaction attempting to update metadata objects, for loading the requested metadata objects from said metadata volumes to said metadata cache;

metadata management means for holding metadata volume location identifiers associated with the metadata objects loaded to said metadata cache, the metadata volume location identifiers indicating in which of said metadata volumes the metadata objects were stored;

log collection means for collecting log records which indicate what updates were made to the metadata objects in said metadata cache, the log records containing the metadata volume location identifiers corresponding to the updated metadata objects;

a log buffer which stores the log record collected by said log collection means; and

log writing means for transferring the log records from said log buffer to said log volume.

2. A data processing system with a logging mechanism which stores log records for repairing an inconsistent file system, the system comprising:

a primary storage subsystem;

a secondary storage subsystem;

a metadata volume, created in said secondary storage subsystem, which store a plurality of metadata objects describing files;

a log volume, created in said secondary storage subsystem, which stores log records describing updates made to the metadata objects;

a metadata cache, created in said primary storage subsystem, which temporarily stores the metadata objects;

metadata loading means, responsive to a transaction attempting to update metadata objects, for loading the requested metadata objects from said metadata volume to said metadata cache;

log collection means for collecting log records which indicate what updates were made to the metadata objects in said metadata cache;

a log buffer which stores the log record collected by said log collection means;

log writing means for transporting the log records from said log buffer to said log volume at regular intervals, using storage space of said log volume as a circular buffer;

metadata writing means for writing back the updated metadata objects from said metadata cache to said metadata volume;

valid log range monitoring means for observing the writing back operation performed by said metadata writing means so as to check which updated metadata object has been written back to said metadata volume, and based on the observation, identifying a particular set of log records stored in said log volume as valid log records, the particular set of log records corresponding to such updated metadata objects that have not yet been written back to said metadata volume; and

file system restoration means, responsive to a file system restoration request, for correcting inconsistencies in the metadata objects stored in said metadata volume by using the valid log records identified by said valid log range monitoring means.

3. A data processing system with a logging mechanism which stores log records for repairing an inconsistent file system, the system comprising:

a primary storage subsystem;

a secondary storage subsystem;

a metadata volume, created in said secondary storage subsystem, which store a plurality of metadata objects describing files;

a metadata cache, created in said primary storage subsystem, which temporarily stores the metadata objects;

metadata loading means, responsive to a transaction attempting to update metadata objects, for loading the requested metadata objects from said metadata volume to said metadata cache; and

log collection means for examining what type of transaction is in progress, to determine whether the current transaction is likely to make a plurality of updates to a single metadata object, and if the current transaction is likely to do so, collecting only the final state of the updated metadata object.

4. A computer-readable medium storing a computer program for file system management which collects transmission log records for repairing an inconsistent file system, the computer program being designed to run on a computer having primary and secondary storage subsystems in order to cause the computer to function as:

a plurality of metadata volumes, created in the secondary storage subsystem, which store a plurality of metadata objects describing files;

a log volume, created in the secondary storage subsystem, which stores log records describing updates made to the metadata objects;

a metadata cache, created in the primary storage subsystem, which temporarily stores the metadata objects;

metadata loading means, responsive to each transaction attempting to updated metadata objects, for loading the requested metadata objects from said metadata volume to said metadata cache;

metadata management means for holding metadata volume location identifiers associated with the metadata objects loading to said metadata cache, the metadata volume location identifiers indicating in which of said metadata volumes the metadata objects were stored;

log collection means for collecting log records which indicate what updates were made to the metadata objects in said metadata cache, the log records containing the metadata volume location identifiers corresponding to the updated metadata objects;

a log buffer which stores the log record collected by said log collection means; and

log writing means for transporting the log records from said log buffer to said log volume.

5. A computer-readable medium storing a computer program for file system management which collects transmission log records for repairing an inconsistent file system, the computer program being designed to run on a computer having primary and secondary storage subsystems in order to cause the computer to function as:

a metadata volume, created in the secondary storage subsystem, which stores a plurality of metadata objects describing files;

a log volume, created in the secondary storage subsystem, which stores log records describing updates made to the metadata objects;

a metadata cache, created in the primary storage subsystem, which temporarily stores the metadata objects;

metadata loading means, responsive to each transaction attempting to update metadata objects, for loading the requested metadata objects from said metadata volume to said metadata cache;

log collection means for collecting log records which indicate what updates were made to the metadata objects in said metadata cache;

a log buffer which stores the log record collected by said log collection means;

log writing means for transporting the log records from said log buffer to said log volume at regular intervals, using storage space of said log volume as a circular buffer;

metadata writing means for writing back updated metadata objects from said metadata cache to said metadata volume;

valid log range monitoring means for observing the writing back operation performed by said metadata writing means so as to check which updated metadata object has been written back to said metadata volume, and based on the observation, identifying a particular set of log records stored in said log volume as valid log records, the particular set of log records corresponding to such updated metadata objects that have not yet been written back to said metadata volume; and

file system restoration means, responsive to a file system restoration request, for correcting inconsistencies in the metadata objects stored in said metadata volume by using the valid log records identified by said valid log range monitoring means.

6. A computer-readable medium storing a computer program for file system management which collects transmission log records for repairing an inconsistent file system, the computer program being designed to run on a computer having primary and secondary storage subsystems in order to cause the computer to function as:

a metadata volume, created in the secondary storage subsystem, which store a plurality of metadata objects describing files;

a metadata cache, created in the primary storage subsystem, which temporarily stores the metadata objects;

metadata loading means, responsive to each transaction attempting to update metadata objects, for loading the requested metadata objects from said metadata volume to said metadata cache; and

log collection means for examining what type of transaction is in progress, to determine whether the current transaction is likely to make a plurality of updates to a single metadata object, and if the current transaction is likely to do so, collecting only the final state of the updated metadata object.

7. A computer-readable medium storing a computer program for file system management which collects transmission log records for repairing an inconsistent file system, the computer program being designed to run on a computer having primary and secondary storage subsystems in order to cause the computer to function as:

allocation management data memory means for holding allocation management data for use in controlling allocation of metadata objects;

metadata allocation means, responsive to a first transaction requesting allocation of a free metadata object, for searching the allocation management data to find and allocate a free metadata object to the first transaction, and updating the allocation management data so as to indicate that the allocated metadata object is in use;

metadata deallocation means, responsive to a second transaction requesting deallocation of a specific metadata object being allocated, for updating the allocation management data so as to indicate that the specified metadata object is free; and

log collection means for collecting log records describing which part of the allocation management data was modified by the metadata allocation means and metadata deallocation means;

wherein each collected log record describes only a single resource allocated or deallocated to/from a single transaction.

8. A data processing system with a logging mechanism which stores log records for repairing an inconsistent file system, the system comprising:

allocation management data memory means for holding allocation management data for use in controlling allocation of metadata objects;

metadata allocation means, responsive to a first transaction requesting allocation of a free metadata object, for searching the allocation management data to find and allocate a free metadata object to the first transaction, and updating the allocation management data so as to indicate that the allocated metadata object is in use;

metadata deallocation means, responsive to a second transaction requesting deallocation of a specific metadata object being allocated, for updating the allocation management data so as to indicate that the specified metadata object is free; and

log collection means for collecting log records describing which part of the allocation management data was modified by the metadata allocation means and metadata deallocation means;

wherein each collected log record describes only a single resource allocated or deallocated to/from a single transaction.

9. The data processing system according to claim 8, wherein said single resource comprises a single data bit.