Server failure recovery

• US 9,170,892 B2
• Filed: 05/26/2011
• Issued: 10/27/2015
• Est. Priority Date: 04/19/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nightingale, Edmund B., Elson, Jeremy E.
• Primary Examiner(s): Srivastava, Vivek
• Assistant Examiner(s): BUKHARI, SIBTE H
• Application Number:
  US13/116,270
• Publication Number:
  US 20110258488A1
• Time in Patent Office: 1,615 Days
• Field of Search: 709/201, 709/2.21, 709220-221, 714/15, 714/2, 714/4.12
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component

Abstract

A metadata server configured to maintain storage assignment mappings in non-persistent storage is described herein. The tract storage assignment mappings associate servers with storage assignments, the storage assignments representing the data stored on the servers. Responsive to a failure, the metadata server receives the storage assignments from the servers and rebuilds the storage assignment mappings from the storage assignments. The metadata server is also configured to enable clients to operate during a recovery process for a failed server by providing the storage assignment mappings to the clients during the recovery process. Also during the recovery process, the replacement server for the failed server conditionally overwrites stored data with other data received from other servers as part of the recovery process. The replacement server conditionally overwrites based on version information associated with the data and version information associated with the other data, the version information being associated with one or more versions of the storage assignment mappings.

Claims

1. A metadata server comprising:

a processor; and

a recovery module configured to be executed by the processor for implementation of a recovery process associated with a failure of a server of a system, the recovery module to;

provide a new recovery plan to one or more replacement servers, the new recovery plan proposing the one or more replacement servers for replacement of the failed server;

update previous storage assignment mappings associated with a previous recovery plan based on the new recovery plan such that new storage assignment mappings specify the one or more replacement servers as being associated with storage assignments of the failed server;

transmit the new storage assignment mappings to one or more clients of the system during the recovery process to enable the one or more clients to operate during the recovery process;

manage, during the recovery process, writing of a first instance of the data by servers of the system to the one or more replacement servers of the system, the first instance of the data being stored by the servers and the failed server and the first instance of the data being associated with a first version identifier;

manage, during the recovery process, writing of a second instance of at least a portion of the data by the one or more clients to the one or more replacement servers of the system, the second instance of at least the portion of the data being associated with a second version identifier indicating that the second instance of at least the portion of the data is newer than corresponding data in the first instance of the data and

store, in the one or more replacement servers and based at least in part the second version identifier, the second instance of at least the portion of the data instead of the corresponding data in the first instance of the data.

2. The metadata server of claim 1, wherein the recovery module is further configured to, during the recovery process:

determine that the writing of the second instance of at least the portion of the data from the one or more clients to the one or more replacement services occurs before the writing of the corresponding data in the first instance of the data; and

ensure that the corresponding data in the first instance of the data does not overwrite the portion of data in the second instance of the data.

3. The metadata server of claim 1, wherein the new storage assignment mappings are transmitted responsive to requests from the one or more clients.

4. The metadata server of claim 1, wherein the recovery module is further configured to periodically receive heartbeat messages from servers of the system and determine the occurrence of a server failure by noting a threshold number of missing heartbeat messages.

5. One or more computer storage devices comprising computer-executable instructions stored on the computer storage devices that are configured to program a tract server of a system to perform operations including:

receiving first data from one or more other tract servers of the system as part of a recovery process, the first data being associated with a tract identifier and first version information corresponding to tract storage assignment mappings known to the one or more other tract servers, the tract storage assignment mappings associating storage assignments with tract servers of the system;

determining that second data associated with the tract identifier is stored by the tract server, the second data being associated with second version information corresponding to the tract storage assignment mappings, the second version information being different than the first version information;

determining whether the first version information or the second version information corresponds to an older version of the tract storage assignments mappings;

in response to determining that the first version information corresponds to the older version of the tract storage assignment mappings, either storing the first data in addition to the second data or discarding the first data;

in response to determining that the second version information corresponds to the older version of the tract storage assignment mappings, overwriting the second data with the first data; and

providing the second data or the first data to a client of the system in response to a read request from the client that is received during the recovery process specifying the tract identifier.

6. The one or more computer storage devices of claim 5, wherein storage of the tract server maintains tracts of data, tract identifiers of those tracts of data, and version information associated with versions of the tract storage assignment mappings used by clients of the system at times the clients wrote the tracts of data to the tract server.

7. The one or more computer storage devices of claim 5, wherein the operations further include:

comparing tract storage assignment mappings version information received in requests from clients of the system with tract storage assignment mappings version information used by the tract server; and

in response to determining that an individual client is using the older version of the tract storage assignment mappings, notifying the individual client that a newer version of the tract storage assignment mappings is available.

8. The one or more computer storage devices of claim 5, wherein the operations further include:

receiving a write request;

storing data included in the write request; and

transmitting the write request to the one or more other tract servers based at least in part on the write request or based at least in part on a newer version of the tract storage assignment mappings.

9. The one or more computer storage devices of claim 5, wherein the operations further include, prior to the receiving the first data, agreeing to a recovery plan proposed by a metadata server.

10. One or more computer storage devices comprising computer-executable instructions stored on the computer storage devices that are configured to program a tract server of a system to perform operations including:

receiving first data from one or more other tract servers of the system as part of a recovery process, the first data being associated with a tract identifier and first version information corresponding to tract storage assignment mappings known to the one or more other tract servers, the tract storage assignment mappings associating storage assignments with tract servers of the system;

determining that second data associated with the tract identifier is stored by the tract server, the second data being associated with second version information corresponding to the tract storage assignment mappings, the second version information being different than the first version information;

determining that the second version information corresponds to an older version of the tract storage assignment mappings compared to the first version information;

in response to determining that the second version corresponds to the older version of the tract storage assignment mappings, overwriting the second data with the first data;

receiving tract storage assignment mappings version information in a request from a client of the system;

comparing the tract storage assignment mappings version information received in the request from the client of the system with the first version information corresponding to the tract storage assignment mappings used by the tract server; and

in response to determining that the tract storage assignment mappings version information received in the request from the client includes the older version of the tract storage assignment mappings, notifying the client that an updated version of the tract storage assignment mappings is available.

11. The one or more computer storage devices of claim 10, wherein storage of the tract server maintains tracts of data, tract identifiers of those tracts of data, and version information associated with versions of the tract storage assignment mappings used by clients of the system at times the clients wrote the tracts of data to the tract server.

12. The one or more computer storage devices of claim 10, wherein the operations further include:

receiving a write request;

storing data included in the write request; and

transmitting the write request to the one or more other tract servers based at least in part on the write request or based at least in part on a newer version of the tract storage assignment mappings.

13. The one or more computer storage devices of claim 10, wherein the operations further include, prior to the receiving the first data, agreeing to a recovery plan proposed by a metadata server.