Using a data storage system for cluster I/O failure determination

• US 8,095,828 B1
• Filed: 08/31/2009
• Issued: 01/10/2012
• Est. Priority Date: 08/31/2009
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Westenberg, Guido, Kruck, Joshua
• Primary Examiner(s): McCarthy, Christopher
• Application Number:
  US12/551,260
• Time in Patent Office: 862 Days
• Field of Search: None
• US Class Current: 714/20
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1464 for networked environments

Abstract

Techniques are disclosed relating to storing a log of write operations made to a first storage device by one of a plurality of host computers running an instance of a distributed application. The log of write operations is stored at a second storage device. The plurality of host computers communicate status information to the second storage device over respective communication paths. Upon a failure to communicate status information between one of the host computers and the second storage device, the second storage device reads from a predetermined location in the first storage device to determine whether the host computer is still performing write operations. If the second storage device reads an expected signature value written by the host computer, the host computer is deemed to have written data, which indicates that the host computer is operational but that the write operations have not been recorded by the second storage device.

Claims

1. A method, comprising:

a first storage device storing a log of write operations to a second storage device, wherein the write operations are made by different instances of a distributed application executing on a plurality of host computer systems; and

the first storage device determining whether a failure to receive status information from a first of the plurality of host computer systems indicates a) that write operations from the first host computer system to the second storage device have ceased, or b) that write operations are being made by the first host computer system to the second storage device without being logged by the first storage device.

2. The method of claim 1, wherein the plurality of host computer systems is arranged in a cluster configuration, and wherein the first and second storage devices are implemented within a storage area network (SAN).

3. The method of claim 1, wherein information in the log of write operations is usable to reconstruct a previous state of the second storage device, and wherein receipt of the status information by the first storage device from the first host computer system indicates that the first host computer system and a first communication path between the first host computer system and the first storage device on which the status information is transmitted are currently operational.

4. The method of claim 1, further comprising:

in response to determining that the write operations from the first host computer system to the second storage device have ceased, storing additional, subsequent write operations in the log, wherein the additional write operations are made by instances of the distributed application executing on ones of the plurality of host computer systems other than the first host computer system.

5. The method of claim 1, wherein said determining includes:

after detecting the failure to receive status information, the first storage device performing a check whether the second storage device is storing an indication that the first host computer system is performing write operations to the second storage device.

6. The method of claim 5, further comprising:

in response to the first storage device determining that write operations are being made by the first host computer system to the second storage device without being logged by the first storage device, the first storage device disallowing reconstruction of any previous state of the second storage device generated after the first storage device fails to receive the status information, wherein the disallowing continues until a corrective action is taken.

7. The method of claim 5, wherein the stored indication is stored in a storage location of the second storage device that previously stored data written by one of the instances of the distributed application.

8. A non-transitory computer readable medium having program instructions stored thereon that, if executed by a first of a plurality of host computers implementing a distributed application, cause the first host computer to perform a method comprising:

receiving information indicative of a write operation from a first instance of the distributed application executing on the first host computer;

providing the information indicative of the write operation to a first storage device;

providing information to a second storage device, wherein the information provided to the second storage device is usable to recreate the write operation;

sending heartbeat information to the second storage device via a first communication path to indicate that the first host computer is operational; and

in response to detecting an error in providing the information to the second storage device via the first communication path, communicating to second storage device via a second communication path to indicate that the first host computer remains operational.

9. The non-transitory computer readable medium of claim 8, wherein the plurality of host computers are within a cluster configuration, and wherein the first and second storage devices are located within a storage area network (SAN).

10. The non-transitory computer readable medium of claim 8, wherein said communicating to the second storage device via the second communication path includes the first host computer storing a signature in a memory storage location of the first storage device, wherein the signature indicates to the second storage device that the first host computer has written data to the first storage device.

11. The non-transitory computer readable medium of claim 10, the method further comprising the first host computer determining whether the stored signature has been modified by the second storage device within a predetermined time period, wherein modification of the signature indicates that the second storage device is operational.

12. The non-transitory computer readable medium of claim 11, the method further comprising:

in response to determining that the stored signature has not been modified within the predetermined time period, the first host computer replacing the stored signature with a set of data that was previously overwritten by said storing of the signature, wherein the set of data was written by an instance of the distributed application.

13. The non-transitory computer readable medium of claim 8, wherein the received information is captured from an I/O stack of the first host computer, wherein the I/O stack is configured to buffer write operations made by the first instance of the distributed application.

14. The non-transitory computer readable medium of claim 8, the method further comprising the first host computer providing additional information to the second storage device, wherein the additional information is usable to recreate write operations made to a plurality of other storage devices, and wherein said communicating includes the first host computer storing a signature on a single one of the second storage device and the plurality of other storage devices, wherein the signature indicates that the first host computer has written data to the second storage device or one of the plurality of other storage devices.

15. The non-transitory computer readable medium of claim 8, the method further comprising the first host computer indicating that the heartbeat information will no longer be sent to the second storage device.

16. A non-transitory computer readable medium having program instructions stored thereon that, if executed by a first storage system, cause the first storage system to perform a method comprising:

a first storage system maintaining information indicative of write operations made by a plurality of host computer systems to a second storage system, wherein said maintaining includes;

receiving status information from at least a first of the plurality of host computer systems, wherein the status information indicates that the first host computer system and a first communication path between the first host computer system and the first storage system are operational; and

in response to the first storage system not receiving the status information from the first host computer system within a predetermined time period, determining a) whether write operations from the first host computer system to the second storage system have ceased, or b) whether write operations are being made by the first host computer system to the second storage system without being recorded by the first storage system.

17. The non-transitory computer readable medium of claim 16, wherein host computers in the plurality of host computer systems are part of a computer cluster configuration, and wherein the second storage system is arranged in a storage area network (SAN) configuration.

18. The non-transitory computer readable medium of claim 16, wherein said determining includes the first storage system entering a mode of operation in which the first storage system searches for a signature that has been stored within a guard block on a first of a plurality of logical storage units of the second storage system by the first host computer system, wherein the signature indicates that the first host computer system has written data to the second storage system.

19. The non-transitory computer readable medium of claim 18, wherein the plurality of logical storage units are associated with a plurality of physical storage units located within a common enclosure, and wherein the first storage system searches for the signature on a single one of the plurality of logical storage units.

20. The non-transitory computer readable medium of claim 16, wherein said determining includes determining whether the first host computer system or the first communication path has failed.