Method for improving recovery performance from hardware and software errors in a fault-tolerant computer system

• US 5,812,748 A
• Filed: 05/16/1995
• Issued: 09/22/1998
• Est. Priority Date: 06/23/1993
• Status: Expired due to Term

• Current Assignee: EMC Corporation (Dell Technologies Inc.)
• Original Assignee: Vinca Corp. (Dell Technologies Inc.)
• Inventors: Rollins, Richard N., Marsden, Wally, Ohran, Richard S., Ohran, Michael R.
• Primary Examiner(s): Bowler, Alyssa H.
• Assistant Examiner(s): Davis, Jr., Walter D.
• Application Number:
  US08/442,415
• Time in Patent Office: 1,225 Days
• Field of Search: 395/800, 395/181, 395/183.05, 395/182.08, 395/182.11, 395/182.13, 395/182.09, 395/182.02, 395/800.01
• US Class Current: 714/4.5
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/2087 with a common controller
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G11C 29/74 using duplex memories, i.e. using dual copies
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A method for providing rapid recovery from a network file server failure through the use of a backup computer system. The backup computer system runs a special mass storage access program that communicates with a mass storage emulator program on the network file server, making the disks (or other mass storage devices) on the backup computer system appear like they were disks on the file server computer. By mirroring data by writing to both the mass storage of the file server and through the mass storage emulator and mass storage access program to the disks on the backup computer, a copy of the data on the file server computer is made. Optionally, selected portions of the data read through the mass storage emulator program can be altered before being returned as the result of the read operation on the file server. In the event of failure of the file server computer, the backup computer can replace the file server, using the copy of the file server'"'"'s data stored on its disks. A single backup computer can support a plurality of file server computers. Unlike other redundant file server configurations, this method does not require the backup computer system to be running the file server operating system.

Claims

1. A method for rapid recovery from a network file server failure, operating on a computer configuration comprising:

a plurality of computer systems adapted for responding to file server requests, each of said plurality of computer systems comprising;

a computer executing a file server operating system, and a mass storage device connected to said computer;

an additional computer system, comprising;

an additional computer which can execute said file server operating system, and

an additional mass storage system comprising at least one mass storage device, connected to said additional computer;

and means for communicating between each of said plurality of computer system and said additional computer system,

the recovery method comprising;

running a mass storage access program on said additional computer, said mass storage access program receiving mirroring data from each computer of said plurality of computer systems over said communicating means and writing said mirroring data to said additional mass storage system;

and for each computer system in said plurality of computer systems;

installing a mass storage emulator on said computer system for use by said file server operating system, said mass storage emulator taking mass storage write requests from said file server operating system and sending mirroring data indicative of said write request to said additional computer system over said communicating means;

initiating mirroring of data by writing said data both to said mass storage device of said computer system and through said mass storage emulator and said mass storage access program to said additional mass storage system, where said mass storage access program and said mass storage emulator makes a portion of said additional mass storage system appear as if said portion of said additional mass storage device were an extra mass storage device connected to said computer of said computer system in the same manner as said mass storage device is connected to said computer of said computer system;

and then when a failure of any of said plurality of computer systems is detected, performing at least the following steps;

transferring responsibility for responding to file server requests previously responded to by said failed computer system to said additional computer system; and

continuing to mirror data from said plurality of computer systems that have not failed to said additional computer system so that said additional computer system both responds to file server requests and mirrors data.

2. A method for recovery from failure of a network server in a network configuration comprising a plurality of network servers and a backup network server, said plurality of network servers being interconnected to said backup network server by means for communicating between said plurality of network servers and said backup network server, each of said plurality of network servers and said backup network server being adapted to execute a file serving operating system in order to process file server requests and each of said plurality of network servers and said backup network server comprising at least one attached mass storage device, said method comprising the steps of:

initiating operation of said file serving operating system on said plurality of network servers so that each of said plurality of network servers becomes operative to process file server requests;

selecting at least one of a first and second system configuration, wherein the first system configuration permits said backup network server to operate solely as a backup to said plurality of network servers, whereby file server requests are not processed by said backup network server until a failure of one of said plurality of network servers occurs, and wherein said second system configuration permits said backup network server to process different file server requests from any of said plurality of network servers both prior to and after occurrence of a failure of one of said network servers; and

initiating data mirroring from said each of said plurality of network servers to said backup network server via said means for communicating so that when a data write request is received by any of said plurality of network servers, data contained in said write request is written both to the at least one mass storage device attached to the network server which received the write request and to the at least one mass storage device attached to said backup network server.

3. A method for recovery from failure of a network server as recited in claim 2 wherein said initiating data mirroring step is accomplished by performing at least the steps of:

executing on each of said plurality of network servers a mass storage emulator program means for emulating a mass storage device so that each of said plurality of network servers appears to have at least one extra mass storage device in addition to the at least one mass storage device attached to each of said plurality of network servers; and

executing on said backup network server at least one mass storage access program means for communicating with said mass storage emulation program on each of said plurality of network servers over said means for communicating and for writing data received over said means for communicating to the at least one mass storage device attached to said backup network server so that when data is written to said mass storage emulation program on one of said plurality of network servers, the data is transferred from said mass storage emulation program on said one of said plurality of network servers to said backup network server and written to the at least one mass storage device attached thereto.

4. A method for recovery from failure of a network server as recited in claim 3 wherein said mass storage access program is adapted to return data to said mass storage emulation program in response to a read command and said mass storage access program comprises a cache of data so that when said mass storage access program receives, from any of said plurality of network servers via said means for communication, a read command comprising a request for data, the requested data is retrieved from said cache if the requested data resides therein and from the at least one mass storage device attached to said backup network server if the requested data does not reside in said cache.

5. A method for recovery from failure of a network server as recited in claim 2 wherein said backup network server is configured to process different file server requests from any of said plurality of network servers and said method further comprises the step of initiating data mirroring from said backup network server to at least one of said plurality of network servers via said means for communicating so that when a data write request is received by said backup network server, data contained in said write request is written both to the at least one mass storage device attached to said backup network server and to the at least one mass storage device attached to said at least one of said plurality of network servers.

6. A method for recovery from failure of a network server as recited in claim 5 further comprising the steps of:

detecting failure of any one of said plurality of network servers or said backup network server; and

if any one of said plurality of network servers has failed then transferring responsibility for processing file server requests that were previously processed by said failed network server to said backup network server and exiting the method; and

if said backup network server has failed, then transferring responsibility for processing file server requests that were previously processed by said backup network server to said at least one of said plurality of network servers and exiting the method.

7. A method for recovery from failure of a network server as recited in claim 2 further comprising the steps of:

detecting failure of any of said plurality of network servers; and

transferring responsibility for processing file server requests that were previously processed by said failed network server to said backup network server so that said network configuration continues to operate and does not fail even though one of said plurality of network servers has failed.

8. A method for recovery from failure of a network server as recited in claim 2 wherein if a read request is received by one of said plurality of network servers, the read request is filled by said one of said plurality of network servers by performing at least the following steps:

initiating at least one read command to retrieve data both from the at least one mass storage device attached to said one of said plurality of network servers and from the at least one mass storage device attached to said backup network server;

waiting for the first response to said at least one read command from one of either the at least one mass storage device attached to said one of said plurality of network servers or the at least one mass storage device attached to said backup network server; and

returning the retrieved data as the result of the received read request.

9. A method for recovery from failure of a network server as recited in claim 8 wherein if the at least one read command specifies data is to be read from at least one special mass storage location, then when the backup network server receives the at least one read command, the backup network server returns alternative data in response to said at least one read command and not the data stored in said at least one special mass storage location.

10. A method for recovery from failure of a network server as recited in claim 2 wherein said plurality of network servers comprise two network servers that each execute the same file server operating system and that individually process each received file server request in parallel so as to execute the same instructions thereby providing a fault tolerant computer system.

11. A method for recovery from a file server failure in a network configuration comprising a plurality of network servers and a backup network server interconnected by means for communicating between said plurality of network servers and said backup network server, each of said plurality of network servers and said backup network server comprising a file serving operating system and at least one mass storage device connected thereto, said method comprising the steps of:

selecting a network configuration wherein said plurality of network servers and said backup network server become operative to process different file server requests so that a given file server request is processed either by said backup network server or by at least one of said plurality of network servers; and

initiating data mirroring from said plurality of network servers to said backup network server via said means for communicating so that when a data write request is received by one of said plurality of network servers, data contained in said write request is written both to the at least one mass storage device of said one of said plurality of network servers and to the at least one mass storage device of said backup network server by performing at least the following steps;

executing on each of said plurality of network servers a mass storage emulator program means for emulating a mass storage device so that each of said plurality of network servers appears to have at least one extra mass storage device in addition to the at least one mass storage device connected to each of said plurality of network servers; and

executing on said backup network server at least one mass storage access program means for communicating with said mass storage emulation program on each of said plurality of network servers over said means for communicating and for writing data received over said means for communicating to the at least one mass storage device connected to said backup network server so that when data is written to said mass storage emulation program on any of said plurality of network servers, the data is transferred from said mass storage emulation program on said plurality of network servers to said backup network server and written to the at least one mass storage device connected thereto.

12. A method for recovery from failure of a network server as recited in claim 11 wherein said at least one mass storage access program is further adapted to return data to said mass storage emulation program in response to a read command and said mass storage access program comprises a cache of data so that when said mass storage access program receives, from one of said plurality of network servers via said means for communication, a read command comprising a request for data, the requested data is retrieved from said cache if the requested data resides therein and from the at least one mass storage device attached to said backup network server if the requested data does not reside in said cache.

13. A method for recovery from failure of a network server as recited in claim 12 wherein the method further comprises the step of initiating data mirroring from said backup network server to at least one of said plurality of network servers via said means for communicating so that when a data write request is received by said backup network server, data contained in said write request is written both to the at least one mass storage device connected to said backup network server and to the at least one mass storage device connected to said at least one of said plurality of network servers.

14. A method for recovery from failure of a network server as recited in claim 13 further comprising the steps of:

detecting the failure of either one of said plurality of network servers or the backup network server; and

if one of said plurality of network servers has failed, then transferring responsibility for processing file server requests that were previously processed by said one of said plurality of network servers to said backup network server and exiting the method; and

if said backup network server has failed, then transferring responsibility for processing file server requests that were previously processed by said backup network server to said at least one of said plurality of network servers and exiting the method.

15. A method for recovery from failure of a network server as recited in claim 12 wherein said plurality of network servers comprise two network servers that each execute the same file server operating system and that individually process each received file server request in parallel so as to execute the same instructions thereby providing a fault tolerant computer system.

16. A method for recovery from failure of a network server as recited in claim 12 wherein said plurality of network servers comprise a first network server and a second network server interconnected by means for communicating between said first network server and said second network server, said second network server operating as a backup to said first network server so that if said first network server fails, said second network server assumes responsibility for processing the file server requests previously processed by said first network server, and wherein said backup network server is connected to said second network server by means for communicating between said second network server and said backup network server, said backup network server operating as a backup to said second network server so that said if said second network server fails, said backup network server assumes responsibility for processing file server requests previously processed by said second network server.

17. A method for recovery from failure of a network, server in a network configuration comprising a plurality of network servers and a backup network server, said plurality of network servers being interconnected to said backup network server by means for communicating between said plurality of network servers and said backup network server, each of said plurality of network servers and said backup network server being adapted to execute a file serving operating system in order to process file server requests and each of said plurality of network servers and said backup network server comprising at least one attached mass storage device, said method comprising the steps of:

initiating operation of said file serving operating system on said plurality of network servers so that each of said plurality of network servers becomes operative to process file server requests;

initiating data mirroring from said each of said plurality of network servers to said backup network server via said means for communicating so that when a data write request is received by any of said plurality of network servers, data contained in said write request is written both to the at least one mass storage device attached to the network server which received the write request and to the at least one mass storage device attached to said backup network server by performing at least the following steps;

executing on each of said plurality of network servers a mass storage emulator program means for emulating a mass storage device so that each of said plurality of network servers appears to have at least one extra mass storage device in addition to the at least one mass storage device connected to each of said plurality of network servers; and

executing on said backup network server at least one mass storage access program means for communicating with said mass storage emulation program on each of said plurality of network servers over said means for communicating and for writing data received over said means for communicating to the at least one mass storage device connected to said backup network server so that when data is written to said mass storage emulation program on any of said plurality of network servers, the data is transferred from said mass storage emulation program on said plurality of network servers to said backup network server and written to the at least one mass storage device connected thereto;

detecting failure of one of said plurality of network servers;

transferring responsibility for processing file server requests previously processed by said failed one of said plurality of network servers to said backup network server; and

continuing data mirroring from the non-failed of said plurality of network servers to said backup network server.

18. A system for recovery from failure of a network server comprising:

a plurality of network servers, each comprising at least one attached mass storage device and a file server operating system, each of said plurality of network servers being adapted to process file server requests from clients;

a backup network server comprising at least one attached mass storage device;

means for communicating between at least one of said plurality of network servers and said backup network server so that data can be exchanged over said means for communicating;

wherein said at least one of said plurality of network servers comprises means for processing file server write requests in such a way that data in file server write requests is written both to said at least one mass storage device attached to said at least one of said plurality of network servers and to said backup network server via said means for communicating;

wherein said backup network server comprises means for writing data received over said means for communicating from said at least one of said plurality of network servers to the at least one mass storage device attached to said backup network server in order to mirror said data;

wherein said at least one of said plurality of network servers and said backup network server comprise means for executing a different sequence of instructions when a file server write request is processed by said at least one of said plurality of network servers so that if said at least one of said plurality of network servers encounters a software error that causes a failure, said backup network server will not encounter the same software error; and

wherein said at least one of said plurality of network servers further comprises a mass storage emulation program so that said at least one of said plurality of network servers appears to have another mass storage device attached thereto in addition to said at least one attached mass storage device, and wherein said backup network server comprises a mass storage access program that is adapted to communicate with said mass storage emulation program via said means for communicating and that is adapted to write data received over said means for communicating to the at least one mass storage device attached to said backup network server.

19. A system for recovery from failure of a network server as recited in claim 18 wherein said backup network server is further adapted to process file server requests that are different from the file server requests processed by said plurality of network servers.

20. A system for recovery from failure of a network server as recited in claim 18 wherein said backup network server is adapted to automatically detect failure of one of said plurality of network servers and to initiate transfer of responsibility for processing file server requests that were previously processed by said failed network server to said backup network server.

21. A system for recovery from failure of a network server comprising:

a plurality of network servers, each comprising at least one attached mass storage device and a file server operating system, each of said plurality of network servers being adapted to process file server requests from clients;

a backup network server comprising at least one attached mass storage device;

means for communicating between at least one of said plurality of network servers and said backup network server so that data can be exchanged over said means for communicating;

wherein said at least one of said plurality of network servers is adapted to process file server write requests in such a way that data in file server write requests is written both to said at least one mass storage device attached to said at least one of said plurality of network servers and to said backup network server via said means for communicating;

wherein said backup network server is adapted to write data received over said means for communicating from said at least one of said plurality of network servers to the at least one mass storage device attached to said backup network server in order to mirror said data, and wherein said backup network is further adapted to execute different instructions when a file server request is processed;

wherein said at least one of said plurality of network servers and said backup network server execute a different sequence of instructions when a file server write request is processed by said at least one of said plurality of network servers so that if said at least one of said plurality of network servers encounters a software error that causes a failure, said backup network server will not encounter the same software error; and

wherein said at least one of said plurality of network servers further comprises a mass storage emulation program so that said at least one of said plurality of network servers appears to have another mass storage device attached thereto in addition to said at least one attached mass storage device, and wherein said backup network server comprises a mass storage access program that is adapted to communicate with said mass storage emulation program via said means for communicating and that is adapted to write data received over said means for communicating to the at least one mass storage device attached to said backup network server.

22. A system for recovery from failure of a network server as recited in claim 21 wherein said plurality of network servers and said backup network server comprise different hardware configurations.

23. A system for recovery from failure of a network server as recited in claim 21 wherein said plurality of network servers comprises two network servers interconnected by means for communicating between said two network servers, each of said two network servers being adapted to process the same file server requests by executing the same sequence of instructions so that the contents of the memory and the contents of the at least one mass storage device of each of said two network servers are identical, and wherein said backup network server is connected to each of said two network servers via said means for communicating between at least one of said plurality of network servers and said backup network server.

24. A system for recovery from failure of a network server as recited in claim 21 wherein said backup network server is further adapted to process file server requests that are different from the file server requests processed by said plurality of network servers.

25. A system for recovery from failure of a network server as recited in claim 21 wherein said backup network server is further adapted to automatically detect failure of one of said plurality of network servers and to assume responsibility for processing file server requests that were previously processed by said failed network server.

26. As an article of manufacture, a computer program product comprising a computer-readable medium having program code means encoded thereon for use by a plurality of network servers and a backup network server connected together in a network configuration wherein a means for communicating is connected between each said network server and said backup network server, and each of said plurality of network servers and said backup network server comprising an attached mass storage device, said computer program code means comprising:

a mass storage emulation program means for use by said plurality of network servers, said mass storage emulation program means emulating a mass storage device and providing communication with said backup network server over said means for communicating in order to read data from or write data to the mass storage device attached to said backup network server;

a mass storage access program means for use by said backup network server, said mass storage access program means storing data on, and reading data from, said mass storage device attached to said backup network server and receiving data from a plurality of said mass storage emulation program means loaded on said plurality of said network servers, and said mass storage access program means writing said received data to the mass storage device attached to said backup network server; and

wherein said mass storage emulation program means and said mass storage access program means are together operable in either of a first and second system configuration wherein the first system configuration permits said backup network server to operate solely as a backup to said plurality of network servers, whereby file server requests are not processed by said backup network server until a failure of one of said plurality of network servers occurs, and wherein said second system configuration permits said backup network server to process different file server requests from any of said plurality of network servers both prior to and after occurrence of a failure of one of said network servers.