Shutdown recovery

• US 7,552,148 B2
• Filed: 02/28/2006
• Issued: 06/23/2009
• Est. Priority Date: 02/28/2006
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Suriyanarayanan, Guhan, Bjorner, Nikolaj S., Teodosiu, Dan, Liu, Huisheng
• Primary Examiner(s): Ali; Mohammad
• Assistant Examiner(s): Smith; Brannon W
• Application Number:
  US11/363,828
• Publication Number:
  US 20070220328A1
• Time in Patent Office: 1,211 Days
• Field of Search: 707/202, 714/19, 714/24
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata

Abstract

Aspects of the subject matter described herein relate to shutdown recovery for resource replication systems. In aspects, a mechanism is described in which a machine having replicated data thereon can recover from a dirty shutdown. First, the machine determines whether a dirty shutdown has occurred. If so, the machine automatically performs shutdown recovery by causing resource metadata stored by the machine to be consistent with resource data stored by the machine. This may involve fixing the resource metadata for updates to the resource data that were not flushed to disk or may involve deleting the resource metadata and restoring it from another machine replicating the data.

Claims

1. A computer-readable storage medium having stored computer-executable instructions which when executed by a computer perform a method for recovering from a dirty shutdown, the method comprising:

subsequent to a dirty shutdown of a member computer participating in a replica group, determining that a dirty shutdown has occurred on the member computer by detecting that a flag stored on the member computer is set, wherein the flag is set when a synchronization service running on the member computer begins a synchronization and the flag is cleared when the synchronization service finishes the synchronization such that the dirty shutdown is detected when the dirty shutdown occurs after the synchronization service begins and before the synchronization service finishes the synchronization, wherein the synchronization service performs the synchronization by receiving an update to a replicated file from an upstream partner of the replica group and applying the update to the replicated file stored on the member computer and updating a metadata entry corresponding to the update in a database of the member computer, wherein the metadata entry corresponding to the update includes a sequence number for the update;

upon detecting that the flag is set indicating that a dirty shutdown has occurred, determining that the synchronization service updated the metadata entry corresponding to the update but did not update the replicated file by detecting that the sequence number, included with the metadata entry is greater than a sequence number stored by a file system change monitor of the member computer, wherein the stored sequence number corresponds to a last journal entry accessed by the file system change monitor prior to the dirty shutdown such that the metadata entry indicates that the replicated file has been updated on the member computer even though the update was not applied to the replicated file;

upon detecting that the synchronization service updated the metadata entry without updating the corresponding replicated file, determining whether the sequence number stored by the file system change monitor is valid indicating that all file changes made prior to the stored sequence number have not been lost, wherein each time a change is made to a replicated file on the member computer, a journal entry is added to a journal such that the file system change monitor reads the journal entries and updates the metadata stored in the database to reflect each change, and wherein the file system change monitor stores the sequence number of the last journal entry that the file system change monitor accessed such that a loss of a file change is determined by performing the following steps;

accessing the stored sequence number and stored timestamp that correspond to the last journal entry read by the file system change monitor prior to the dirty shutdown;

accessing a current journal entry from the journal, the current journal entry having a sequence number that matches the stored sequence number; and

comparing a timestamp of the current journal entry with the stored timestamp and upon determining that the timestamp of the current journal entry matches the stored timestamp, determining that no file system changes were lost; and

upon determining that the synchronization service updated metadata without updating the corresponding replicated files and upon determining that no file system changes were lost, automatically performing a shutdown recovery comprising causing resource metadata stored by the member to be consistent with resource data stored by the member by performing the following steps for each metadata entry having a sequence number greater than the stored sequence number;

removing from a version vector of the member computer a corresponding version of the metadata entry in the version vector; and

determining whether a replicated file corresponding to the metadata entry is stored on the member computer such that;

upon detecting that the member computer stores a version of the replicated file, a fence value is assigned to the replicated file such that during a subsequent synchronization, the replicated file on the member computer will be updated with a version of the replicated file from an upstream partner having a higher fence value; and

upon detecting that the member computer does not store a version of the replicated file, the metadata entry is marked for deletion.

2. The computer-readable storage medium of claim 1, wherein the dirty shutdown comprises a synchronization service dirty shutdown that occurs when a synchronization service terminates abnormally, wherein the synchronization service is arranged to synchronize resources with other members participating in the replica group.

3. The computer-readable storage medium of claim 1, wherein the dirty shutdown comprises a machine dirty shutdown that occurs when a machine hosting the member crashes or loses power while a synchronization service is executing, wherein the synchronization service is arranged to synchronize resources with other members participating in the replica group.

4. The computer-readable storage medium of claim 1, wherein the dirty shutdown comprises a volume dirty shutdown in which a volume upon which the member stores resources loses power, becomes disconnected, or is forced to dismount while a synchronization service is executing, wherein the synchronization service is arranged to synchronize resources with other members participating in the replica group.

5. The computer-readable storage medium of claim 1, wherein the replica group comprises a set of resources that are replicated on members participating in the replica group.

6. A method comprising steps executed by a processor and a memory operably coupled to the processor, wherein the memory stores program instructions executable to perform the method, for recovering from a dirty shutdown, the method comprising:

subsequent to a dirty shutdown of a member computer participating in a replica group, determining that a dirty shutdown has occurred on the member computer by detecting that a flag stored on the member computer is set, wherein the flag is set when a synchronization service running on the member computer begins a synchronization and the flag is cleared when the synchronization service finishes the synchronization such that the dirty shutdown is detected when the dirty shutdown occurs after the synchronization service begins and before the synchronization service finishes the synchronization, wherein the synchronization service performs the synchronization by receiving an update to a replicated file from an upstream partner of the replica group and applying the update to the replicated file stored on the member computer and updating a metadata entry corresponding to the update in a database of the member computer, wherein the metadata entry corresponding to the update includes a sequence number for the update;

upon detecting that the flag is set indicating that a dirty shutdown has occurred, determining that the synchronization service updated the metadata entry corresponding to the update but did not update the replicated file by detecting that the sequence number, included with the metadata entry is greater than a sequence number stored by a file system change monitor of the member computer, wherein the stored sequence number corresponds to a last journal entry accessed by the file system change monitor prior to the dirty shutdown such that the metadata entry indicates that the replicated file has been updated on the member computer even though the update was not applied to the replicated file;

upon detecting that the synchronization service updated the metadata entry without updating the corresponding replicated file, determining whether the sequence number stored by the file system change monitor is valid indicating that all file changes made prior to the stored sequence number have not been lost, wherein each time a change is made to a replicated file on the member computer, a journal entry is added to a journal such that the file system change monitor reads the journal entries and updates the metadata stored in the database to reflect each change, and wherein the file system change monitor stores the sequence number of the last journal entry that the file system change monitor accessed such that a loss of a file change is determinable by performing the following steps;

accessing the stored sequence number and stored timestamp that correspond to the last journal entry read by the file system change monitor prior to the dirty shutdown;

accessing a current journal entry from the journal, the current journal entry having a sequence number that matches the stored sequence number; and

comparing a timestamp of the current journal entry with the stored timestamp and upon determining that the timestamp of the current journal entry matches the stored timestamp, determining that no file system changes were lost; and

upon determining that the synchronization service updated metadata without updating the corresponding replicated files and upon determining that no file system changes were lost, automatically performing a shutdown recovery comprising causing resource metadata stored by the member to be consistent with resource data stored by the member by performing the following steps for each metadata entry having a sequence number greater than the stored sequence number;

removing from a version vector of the member computer a corresponding version of the metadata entry in the version vector; and

determining whether a replicated file corresponding to the metadata entry is stored on the member computer such that;

upon detecting that the member computer stores a version of the replicated file, a fence value is assigned to the replicated file such that during a subsequent synchronization, the replicated file on the member computer will be updated with a version of the replicated file from an upstream partner having a higher fence value; and

upon detecting that the member computer does not store a version of the replicated file, the metadata entry is marked for deletion.

7. In a computing environment, an apparatus, comprising:

a journal arranged to store indications of updates to files in a file system;

a file system change monitor arranged to read the indications of updates to files in the file system and to update records of a metadata store based thereon, wherein the file system change monitor is further arranged to store a sequence number of the last journal entry that the file system change monitor has accessed to update the metadata store;

a synchronization service arranged to update the metadata store based at least in part on information other than the indications of updates stored by the journal; and

a shutdown recovery component arranged to perform the following method;

subsequent to a dirty shutdown of a member computer participating in a replica group, determining that a dirty shutdown has occurred on the member computer by detecting that a flag stored on the member computer is set, wherein the flag is set when the synchronization service running on the member computer begins a synchronization and the flag is cleared when the synchronization service finishes the synchronization such that the dirty shutdown is detected when the dirty shutdown occurs after the synchronization service begins and before the synchronization service finishes the synchronization, wherein the synchronization service performs the synchronization by receiving an update to a replicated file from an upstream partner of the replica group and applying the update to the replicated file stored on the member computer and updating a metadata entry corresponding to the update in a database of the member computer, wherein the metadata entry corresponding to the update includes a sequence number for the update;

upon detecting that the flag is set indicating that a dirty shutdown has occurred, determining that the synchronization service updated the metadata entry corresponding to the update but did not update the replicated file by detecting that the sequence number included with the metadata entry is greater than the sequence number stored by the file system change monitor, such that the metadata entry indicates that the replicated file has been updated in the file system even though the update was not applied to the replicated file;

upon detecting that the synchronization service updated the metadata entry without updating the corresponding replicated file, determining whether the sequence number stored by the file system change monitor is valid indicating that all file changes made prior to the stored sequence number have not been lost wherein a loss of a file change is determinable by performing the following steps;

accessing the stored sequence number and stored timestamp that correspond to the last journal entry read by the file system change monitor prior to the dirty shutdown;

accessing a current journal entry from the journal, the current journal entry having a sequence number that matches the stored sequence number; and

comparing a timestamp of the current journal entry with the stored timestamp and upon determining that the timestamp of the current journal entry matches the stored timestamp, determining that no file system changes were lost; and

upon determining that the synchronization service updated metadata without updating the corresponding replicated files and upon determining that no file system changes were lost, automatically performing a shutdown recovery comprising causing resource metadata stored by the member to be consistent with resource data stored by the member by performing the following steps for each metadata entry having a sequence number greater than the stored sequence number;

removing from a version vector of the member computer a corresponding version of the metadata entry in the version vector; and

determining whether a replicated file corresponding to the metadata entry is stored on the member computer such that;

upon detecting that the member computer stores a version of the replicated file, a fence value is assigned to the replicated file such that during a subsequent synchronization, the replicated file on the member computer will be updated with a version of the replicated file from an upstream partner having a higher fence value; and

upon detecting that the member computer does not store a version of the replicated file, the metadata entry is marked for deletion.

8. The apparatus of claim 7, wherein the synchronization service is arranged to communicate with other members participating in the replica group and to update the files in the file system and the metadata store based on updates received from the other members.