Server Failure Recovery

• US 20110258488A1
• Filed: 05/26/2011
• Published: 10/20/2011
• Est. Priority Date: 04/19/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Nightingale, Edmund B., Elson, Jeremy E.
• Granted Patent: US 9,170,892 B2
• Time in Patent Office: Days
• US Class Current: 714/15
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component

Abstract

A metadata server configured to maintain storage assignment mappings in non-persistent storage is described herein. The tract storage assignment mappings associate servers with storage assignments, the storage assignments representing the data stored on the servers. Responsive to a failure, the metadata server receives the storage assignments from the servers and rebuilds the storage assignment mappings from the storage assignments. The metadata server is also configured to enable clients to operate during a recovery process for a failed server by providing the storage assignment mappings to the clients during the recovery process. Also during the recovery process, the replacement server for the failed server conditionally overwrites stored data with other data received from other servers as part of the recovery process. The replacement server conditionally overwrites based on version information associated with the data and version information associated with the other data, the version information being associated with one or more versions of the storage assignment mappings

Claims

1. A method comprising:

maintaining, by a metadata server of a system, storage assignment mappings for the system in non-persistent storage, the storage assignment mappings associating servers of the system with storage assignments, the storage assignments representing the data of the system stored on the associated servers; and

responsive to a metadata server failure, recovering, by the metadata server, the storage assignment mappings by receiving the storage assignments from the servers and rebuilding the storage assignment mappings from the received storage assignments.

2. The method of claim 1, further comprising providing the storage assignment mappings to the servers during a system initialization phase to enable the servers to persistently store storage assignments that are associated with those servers in the storage assignment mappings.

3. The method of claim 1, further comprising requesting, by the metadata server, the storage assignments from the servers upon initializing after a failure or a powering down.

4. The method of claim 1, wherein the metadata server comprises a primary metadata server and a backup metadata server, and the primary metadata server and backup metadata server switch roles during and after a recovery from a failure of the primary metadata server.

5. The method of claim 4, wherein clients of the system are configured to request the storage assignment mappings from the backup metadata server when the primary metadata server does not respond to a previous request.

6. The method of claim 1, further comprising determining the occurrence of a server failure during recovery from the metadata server failure, and, in response, waiting to recover from the server failure until completion of the recovery from the metadata server failure.

7. A metadata server comprising:

a processor; and

a recovery module configured to be executed by the processor to provide storage assignment mappings to one or more clients of a system during a recovery process to enable the one or more clients to operate during the recovery process, the recovery process being associated with a failure of a server of the system.

8. The metadata server of claim 7, wherein the recovery module is further configured to update the storage assignment mappings based on a recovery plan, the updated storage assignment mappings specifying replacement servers for the failed server.

9. The metadata server of claim 7, wherein the recovery module is further configured to manage the recovery process, including managing writing of data by groups of servers of the system to replacement servers of the system, the data being stored by the groups of servers and the failed server.

10. The metadata server of claim 9, wherein the recovery process is completed when the groups of servers complete the writing of the data to the replacement servers.

11. The metadata server of claim 7, wherein the storage assignment mappings are provided responsive to requests from the one or more clients.

12. The metadata server of claim 7, wherein the recovery module is further configured to periodically receive heartbeat messages from servers of the system and determine the occurrence of a server failure by noting a threshold number of missing heartbeat messages.

13. One or more computer storage media comprising computer-executable instructions stored on the computer-readable storage media that are configured to program a tract server of a system to perform operations including:

receiving first data associated with a tract identifier from one or more other tract servers of the system, the other tract servers providing the first data as part of a recovery process;

determining that second data associated with the tract identifier is stored by the tract server; and

conditionally overwriting the second data with the first data based on first version information associated with the first data and second version information associated with the second data, the first and second version information being associated with one or more versions of tract storage assignment mappings of the system.

14. The one or more computer storage media of claim 13, wherein the tract server conditionally overwrites the second data if the first and second version information indicates that the first data is associated with a newer version of the tract storage assignment mappings than the second data.

15. The one or more computer storage media of claim 14, wherein the operations further include, in response to determining that the second data is associated with an older version of the tract storage assignment mappings than the first data, either storing the second data in addition to the first data or discarding the second data.

16. The one or more computer storage media of claim 15, wherein the operations further include providing the second data to a client of the system in response to a read request from the client that is received during the recovery process specifying the tract identifier.

17. The one or more computer storage media of claim 13, wherein storage of the tract server maintains tracts of data, tract identifiers of those tracts of data, and version information associated with versions of the tract storage assignment mappings used by clients of the system at times the clients wrote the tracts of data to the tract server.

18. The one or more computer storage media of claim 13, wherein the operations further include:

comparing tract storage assignment mappings version information received in a request from a client of the system with tract storage assignment mappings version information of a version of the tract storage assignment mappings used by the tract server; and

in response to determining that the client is using an older version of the tract storage assignment mappings, notifying the client that a newer version of the tract storage assignment mappings is available.

19. The one or more computer storage media of claim 13, wherein the operations further include:

receiving a write request;

storing data included in the write request; and

transmitting the write request to one or more other tract servers based on the write request or on the tract storage assignment mappings.

20. The one or more computer storage media of claim 13, wherein the operations further include, prior to receiving the first data, agreeing to a recovery plan proposed by the metadata server and receiving the tract storage assignment mappings.