Storage system having error detection and recovery

• US 6,523,130 B1
• Filed: 03/11/1999
• Issued: 02/18/2003
• Est. Priority Date: 03/11/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: DeMello, Marco A., Hickman, Robert Chris
• Primary Examiner(s): BADERMAN, SCOTT T
• Application Number:
  US09/266,450
• Time in Patent Office: 1,440 Days
• Field of Search: 714/4-6, 714/7, 714/11, 714/13, 714/17, 714/42, 714/43, 709/215, 709/223, 709/224, 707/202, 707/204
• US Class Current: 714/4.3
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2003/0697 device management, e.g. handlers, drivers, I/O schedulers
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems

Abstract

A massively scalable architecture for providing a self-monitoring and self-correcting storage system that is capable of handling hundreds of millions of users and tens of billions of files. The system includes one or more clusters storing data elements that are received from a plurality of clients. Each cluster comprises a plurality of storage servers. The storage system facilitates the addition of new storage servers, and the fast recovery of failed storage servers, by logging system transactions in multiple journals of different lengths. When a storage server fails, a cluster backup determines the time of failure and replays one of the journals in order to bring the failed storage server up to date.

Claims

1. A storage system comprising:

a plurality of clusters storing data elements from a plurality of clients, wherein each cluster comprises a plurality of storage servers;

a storage monitor communicatively coupled to the clusters for detecting when one of the storage servers fails;

a storage manager communicatively coupled to the storage monitor, wherein the storage monitor informs the storage manager when one of the storage servers fails; and

a partition master communicatively coupled to the plurality of clusters to assign each client a storage partition within one of the clusters, wherein the storage manager commands one of the storage servers to operate as the partition master in the event that the partition master fails.

2. A storage system comprising:

a plurality of clusters storing data elements from a plurality of clients, wherein each cluster comprises a plurality of storage servers;

a storage monitor communicatively coupled to the clusters for detecting when one of the storage servers fails;

a storage manager communicatively coupled to the storage monitor, wherein the storage monitor informs the storage manager when one of the storage servers fails; and

a write master included in each cluster to receive the data elements from the clients and to direct the storage servers to store the received data elements, wherein the storage manager commands one of the storage servers to operate as the write master in the event that the write master fails.

3. A storage system comprising:

a plurality of clusters storing data elements from a plurality of clients, wherein each cluster comprises a plurality of storage servers, and wherein the data elements are replicated on each of the plurality of storage servers in a given cluster;

a storage monitor communicatively coupled to the clusters for detecting when one of the storage servers fails;

a storage manager communicatively coupled to the storage monitor, wherein the storage monitor informs the storage manager when one of the storage servers fails; and

wherein each cluster includes a cluster backup that records requests to store the data elements stored by the storage servers of the respective cluster.

4. The storage system of claim 3, wherein the cluster backup records the requests in one or more journals.

5. The storage system of claim 4, wherein each journal contains requests spanning over different lengths of time.

6. The storage system of claim 4, wherein when a failed server is detected the storage manager commands the cluster backup to replay at least one of the journals to restore the failed server.

7. The storage system of claim 4, wherein the cluster backup replays at least one of the journals in order to add a storage server to the cluster.

8. A computing method comprising:

receiving client requests to store data elements in a storage system having a plurality of storage clusters, wherein each storage cluster has a plurality of storage servers;

storing the data elements in each of the storage servers of one of the storage clusters, wherein storing the data elements includes assigning each client a storage partition within one of the clusters;

monitoring the storage servers to detect when a storage server fails; and

promoting one of the storage servers to perform services of the failed storage server, wherein promoting one of the storage servers includes promoting one of the storage servers to assign each client a storage partition within one of the clusters.

9. A computing method comprising:

receiving client requests to store data elements in a storage system having a plurality of storage clusters, wherein each storage cluster has a plurality of storage servers, and wherein the data elements are replicated on each of the plurality of storage servers in a given storage cluster;

storing the data elements in each of the storage servers of one of the storage clusters;

monitoring the storage servers to detect when a storage server fails; and

wherein receiving a client request includes recording the request in at least one journal.

10. The method of claim 9, wherein recording the request includes recording the request in a plurality of journals that each span a different length of time.

11. The method of claim 9, wherein monitoring the storage servers includes the step of replaying at least one of the journals to restore the failed server.

12. A computer-readable medium having computer-executable instructions for storing information in a storage system having error detection and recovery comprising:

receiving client requests to store data elements in a storage system having a plurality of storage clusters, wherein each storage cluster has a plurality of storage servers;

storing the data elements in each of the storage servers of one of the storage clusters, wherein storing the data elements includes assigning each client a storage partition within one of the clusters;

monitoring the storage servers to detect when a storage server fails; and

promoting one of the storage servers to perform services of the failed storage server, wherein promoting one of the storage servers includes promoting one of the storage servers to assign each client a storage partition within one of the clusters.

13. A computer-readable medium having computer-executable instructions for storing information in a storage system having error detection and recovery comprising:

receiving client requests to store data elements in a storage system having a plurality of storage clusters, wherein each storage cluster has a plurality of storage servers, and wherein the data elements are replicated on each of the plurality of storage servers in a given storage cluster;

storing the data elements in each of the storage servers of one of the storage clusters;

monitoring the storage servers to detect when a storage server fails; and

wherein receiving a client request includes recording the request in at least one journal.