System and method for coordinating cluster state information

• US 7,953,924 B1
• Filed: 01/22/2010
• Issued: 05/31/2011
• Est. Priority Date: 03/03/2003
• Status: Expired due to Term

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: sen Sarma, Joydeep, Larson, Richard O., Rowe, Alan L.
• Primary Examiner(s): LANE, JOHN A
• Application Number:
  US12/692,030
• Time in Patent Office: 494 Days
• Field of Search: 711/111, 711/114, 711/152, 711/163, 709/216, 709/219
• US Class Current: 711/111
• CPC Class Codes: G06F 11/1441 Resetting or repowering
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 2003/0697 device management, e.g. handlers, drivers, I/O schedulers
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0637 Permissions
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A method for managing a plurality of servers is disclosed. Each server of the plurality of servers has access to data stored by other servers. The data is stored to one or more data storage devices. Coordinating information is written for the plurality of servers to a master mailbox record. The coordinating information includes data that each server uses to recover after a failure by a server. The master mailbox record is stored on a selected storage device at a location known to the plurality of servers, and the selected storage device is designated as a lock storage device. A plurality of lock storage devices is chosen so that in the event of failure of a server of the plurality of servers, at least one lock storage device will be available to the remaining servers.

Claims

1. A method for managing a plurality of computer servers, comprising:

configuring the plurality of servers to have access to data stored by each server of the plurality of servers;

storing the data on one or more data storage devices accessible by the plurality of servers;

writing coordinating information for the plurality of servers to a master mailbox record, where the coordinating information includes recovery data that the each server uses to recover after a failure by a failed server of the plurality of servers, and where the master mailbox record is stored on a selected storage device at a location known to the plurality of servers, the selected storage device designated as a selected lock storage device; and

choosing a plurality of lock storage devices so that upon of failure of the failed server, at least one lock storage device is available to the remaining servers of the plurality of servers.

2. The method as in claim 1, further comprising:

maintaining more than one copy of the master mailbox record on each lock storage device.

3. The method as in claim 1, further comprising:

using a slow write procedure to write the master mailbox record, where a slow write first writes a first copy of the master mailbox record; and

responsive to the first copy being recognized as successful, writing a second copy of the master mailbox record.

4. The method as in claim 1, further comprising:

determining if all writes of coordinating information succeeded; and

removing a lock storage device for which a write did not succeed from the plurality of lock storage devices.

5. The method as in claim 1, further comprising:

if after removal of the lock storage device for which the write failed leaves one or more remaining lock storage devices, writing the coordinating information a second time to the remaining lock storage devices, and if the writes of the coordinating information the second time are successful on the remaining lock storage devices, recognizing that the master mailbox record is correct on all of the lock storage devices.

6. The method as in claim 1, further comprising:

permitting direct access to the master mailbox record by the servers without a need for file system semantics so that contents of the master mailbox record are available to a server which has not experienced a failure.

7. The method as in claim 1, further comprising:

making the coordinating information written by a first server available to another server at some later time in order to prevent partitioning problems such as a loss of data consistency, a partition in space problem, and a partition in time problem, restart the system with missing data.

8. The method as in claim 1, further comprising:

using a writeable persistent storage device as the storage device.

9. The method as in claim 1, further comprising:

using a memory as the storage device.

10. The method as in claim 1, further comprising:

using a NVRAM as the storage device.

22. The method of claim 7, further comprising:

the partition in space problem including a failure of a connectivity between servers and storage devices that allow different servers to have access to different sets of storage devices at the same time, permitting the different sets of storage devices to have different data written thereto.

23. The method of claim 7, further comprising:

the partition in time problem including a loss of connectivity to one set of storage devices as a result of a failure of the storage fabric such as a power failure and the system continues to operate with the remaining set of storage devices but not with an other set of storage devices such that any newly written data will be stored only on the remaining set of storage devices, and if then connectivity is lost to the remaining set of storage devices and if connectivity is then restored only to the other set of storage devices.

26. The method as in claim 1, further comprising:

using magnetic disks as the storage devices.

27. The method as in claim 1, further comprising:

using RAID arrays as the data storage devices.

11. An apparatus of a plurality of computer servers, comprising:

each server of the plurality of servers having access to data stored by each server of the plurality of servers, the data stored on one or more data storage devices accessible by the plurality of servers;

a master mailbox record containing coordinating information for the plurality of servers, where the coordinating information includes recovery data that the each server of the plurality of servers uses to recover after a failure by a failed server, and where the master mailbox record is stored on a selected storage device at a location known to the plurality of servers, the selected storage device is designated as a lock storage device; and

a set of lock storage devices chosen so that upon failure of the failed server of the plurality of servers, at least one lock storage device is available to remaining servers of the plurality of servers.

12. The apparatus as in claim 11, further comprising:

more than one copy of the master mailbox record maintained on each lock storage device.

13. The apparatus as in claim 11, further comprising:

a slow write procedure used to write the master mailbox record, where a slow write first writes a first copy of the master mailbox record, and in response to the first copy being recognized as successful, writing a second copy of the master mailbox record.

14. The apparatus as in claim 11, further comprising:

the apparatus configured to determine if all writes of coordinating information succeeded, and responsive to a write not succeeding, removing a lock storage device for which the write did not succeed from the set of lock storage devices.

15. The apparatus as in claim 14, further comprising:

if after removal of the lock storage device for which the write failed leaves one or more remaining lock storage devices, the apparatus configured to write the coordinating information a second time to the remaining lock storage devices, and if the writes of the coordinating information the second time are successful on the remaining lock storage devices, recognizing that the master mailbox record is correct on all of the remaining lock storage devices.

16. The apparatus as in claim 11, further comprising:

the plurality of servers configured to permit direct access to the master mailbox record by a server without a need for file system semantics so that contents of the master mailbox record are available to any server which has not experienced a failure.

17. The apparatus as in claim 11, further comprising:

the coordinating information written by a first server is available to another server at some later time in order to prevent partitioning problems such as a loss of data consistency, a partition in space problem, and a partition in time problem, that restarts the system with missing data.

18. The apparatus as in claim 11, further comprising:

magnetic disks used as the storage devices.

19. The apparatus as in claim 11, further comprising:

a RAID array used as the data storage device.

20. The apparatus as in claim 11, further comprising:

a NVRAM used as the storage device.

24. The apparatus as in claim 17, further comprising:

the partition in space problem includes a failure of a connectivity between servers and storage devices that allow different servers to have access to different sets of storage devices at the same time, permitting the different sets of storage devices to have different data written thereto.

25. The apparatus as in claim 17, further comprising:

the partition in time problem includes a loss of connectivity to one set of storage devices as a result of a failure of the storage fabric such as a power failure and the system continues to operate with the remaining set of storage devices but not with an other set of storage devices such that any newly written data will be stored only on the remaining set of storage devices, and if then connectivity is lost to the remaining set of storage devices and if connectivity is then restored only to the other set of storage devices.

28. The apparatus as in claim 11, further comprising:

a writeable persistent storage device used as the storage device.

29. The apparatus as in claim 11, further comprising:

a memory used as the storage device.

21. A computer readable storage media, comprising:

said computer readable storage media containing program instructions for execution on a processor to manage a plurality of computer servers, the program instructions for,

configuring the plurality of servers to have access to data stored by each server of the plurality of servers;

storing the data on one or more data storage devices accessible by the plurality of servers;

writing coordinating information for the plurality of servers to a master mailbox record, where the coordinating information includes recovery data that the each server uses to recover after a failure by a failed server of the plurality of servers, and where the master mailbox record is stored on a selected storage device at a location known to the plurality of servers, the selected storage device designated as a selected lock storage device; and

choosing a plurality of lock storage devices so that upon failure of the failed server , at least one lock storage device is available to remaining servers of the plurality of servers.