Hardware and software failover services for a file server

• US 5,987,621 A
• Filed: 05/05/1997
• Issued: 11/16/1999
• Est. Priority Date: 04/25/1997
• Status: Expired due to Term

• Current Assignee: EMC Corporation (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Gupta, Uday, Ting, Dennis P. J., Forecast, John, Duso, Wayne W., Vahalia, Uresh K
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): Iqbal, Nadeem
• Application Number:
  US08/851,507
• Time in Patent Office: 925 Days
• Field of Search: 395/182.02, 395/181, 395/182.04, 395/182.05, 395/182.06, 395/182.13, 395/182.18, 395/183.07, 395/183.19, 395/728, 395/734, 395/600, 395/650 , 395/200.37, 395/200.57, 395/200.54, 371/40.15, 714/2, 714/3, 714/4, 714/6, 714/8, 714/11, 714/13, 714/21, 714/25, 714/41, 714/47, 714/770
• US Class Current: 714/4.11
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: H04N 21/232 Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/2387 Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/2396 characterized by admission policies admission control, resource allocation in open networks H04L12/5692; arrangements for network security using user profiles for access control H04L63/102; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/2404 Monitoring of server processing errors or hardware failure error or fault detection G06F11/07; monitoring in general G06F11/30

Abstract

A file server includes stream server computers linking a cached disk array storage subsystem to a data network, and at least two controller servers for receiving requests for file access from network clients. At any given time one of the controller servers is active and another is inactive in servicing client requests. The active controller server selects one of the stream servers to service each request. A controller server failover mechanism is provided for recovering from a failure of the active controller server, and a stream server failover mechanism is provided for recovering from a failure of a stream server. The inactive controller server becomes active when it fails to receive a signal periodically transmitted by the active controller server. The active controller server begins stream server failover when it fails to receive a signal periodically transmitted by each stream server. To resume automatically an interrupted task, the tasks are organized as a series of transactions, and each transaction includes operations which can be duplicated without substantial disruption. The active controller server commits results of each transaction to memory of the cached disk array. Before becoming active, the inactive controller recovers the committed state of the interrupted tasks from the cached disk array.

Claims

1. A file server for serving clients in a data network, said file server comprising:

a cached disk array storage subsystem; and

a plurality of server computers linked to the cached disk array storage subsystem for responding to requests for file access from the clients in the data network by transferring data between the cached disk array storage subsystem and the data network,

wherein the plurality of server computers are programmed for;

(a) executing tasks for responding to requests for file access from the clients in the data network; and

(b) detecting a failure of one of the server computers causing interruption of execution of one of the tasks, and upon detecting the failure, for causing another one of the server computers to resume automatically the interrupted one of the tasks;

wherein the interrupted one of the tasks includes a series of transactions, and said plurality of server computers are programmed to record state information for the interrupted one of the tasks during execution of the interrupted one of the tasks, and for resuming automatically the interrupted one of the tasks by recovering the recorded state information for the interrupted one of the tasks including state information from at least one transaction completed prior to the interruption of the interrupted one of the tasks, and resuming execution of a next one of the transactions in the interrupted one of the tasks using the recovered state information from said at least one of the transactions completed prior to the interruption of the interrupted one of the tasks.

2. The file server as claimed in claim 1, wherein each of the server computers is programmed for periodically sending a signal to the other server computers, and wherein the other server computers are programmed to resume a task of said each of the server computers in response to a failure of said each of the server computers to periodically send the signal.

3. The file server as claimed in claim 2, which includes a data link linking the server computers for periodically sending the signal over the data link from said each of the server computers to the other server computers.

4. The file server as claimed in claim 1, wherein the plurality of server computers are programmed for executing tasks by maintaining state information of the tasks in memory of the cached disk array storage subsystem, and wherein the server computers are programmed for resuming interrupted ones of the tasks by recovering the state information for the interrupted ones of the tasks from the memory of the cached disk array, and resuming execution of the interrupted ones of the tasks using the recovered state information for the interrupted ones of the tasks.

5. The file server as claimed in claim 4, wherein the plurality of server computers are programmed for maintaining state information of the tasks in the memory of the cached disk array by writing updates of the state information to a log file in the memory of the cached disk array, and for recovering the state information of the interrupted ones of the tasks by reading from the log file updates of the state information.

6. The file server as claimed in claim 4, wherein the tasks are subdivided into a series of transactions, each of the transactions includes a series of operations that can be repeated without causing substantial disruption during the resuming of execution of the interrupted ones of the tasks, and the server computers are programmed for maintaining state information of the tasks in memory of the cached disk array by committing results of each transaction to the memory of the cached disk array before committing results of a next transaction in the series of transactions.

7. A file server for serving clients in a data network, said file server comprising, in combination:

a cached disk array storage subsystem;

a plurality of stream server computers linking the cached disk array storage subsystem to the data network for transfer of data streams between the cached disk array storage subsystem and the data network; and

at least two controller servers for receiving requests for file access from clients in the data network, said controller servers being connected to the stream server computers for selecting one of the stream server computers to service each request by streaming data between the cached disk array storage subsystem and the data network;

wherein the controller servers are programmed so that at any given time one of the controller servers is active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests, and at any given time one of the controller servers is inactive in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests, and wherein the inactive one of the controller servers is programmed to respond automatically to a failure of the active one of the controller servers by resuming tasks of the active one of the controllers having been interrupted by the failure, and becoming active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests;

wherein an interrupted one of said tasks includes a series of transactions, and the controller servers are programmed to record state information for the interrupted one of the tasks during execution of the interrupted one of the tasks, and for resuming automatically the interrupted one of the tasks by recovering the recorded state information for the interrupted one of the tasks including state information from at least one transaction completed prior to the interruption of the interrupted one of the tasks, and resuming execution of a next one of the transactions in the interrupted one of the tasks using the recovered state information from said at least one of the transactions completed prior to the interruption of the interrupted one of the tasks.

8. The file server as claimed in claim 7, wherein:

the active one of the controller servers is programmed for periodically sending a signal to the inactive one of the controller servers, and for executing tasks by maintaining state information of the tasks in memory of the cached disk array storage subsystem, and

the inactive one of the controller servers is programmed for responding to a failure to receive the signal by recovering the state information of the interrupted tasks from the memory of the cached disk array storage system, and becoming active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests.

9. The file server as claimed in claim 8, wherein the memory of the cached disk array stores system state information indicating which of the controller servers is active and which of the controller servers is inactive, the controller servers are programmed to read the system state information to determine whether or not to become active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests, and wherein the inactive one of the controller servers is programmed to become active in response to a failure to receive the signal by changing the system state information so that the active one of the controller servers becomes inactive.

10. The file server as claimed in claim 9, wherein the active one of the controller servers is programmed to read the system state information before changing the state information of the tasks in the memory of the cached disk array storage subsystem, so that the active one of the controller servers will not change the state information of the tasks in the memory of the cached disk array a certain time after the inactive one of the controller servers becomes active.

11. The file server as claimed in claim 8, wherein the plurality of server computers are programmed for maintaining state information of the tasks in the memory of the cached disk array by writing updates of the state information to a log file in the memory of the cached disk array, and for recovering the state information of the interrupted ones of the tasks by reading from the log file updates of the state information.

12. The file server as claimed in claim 8, wherein the tasks are subdivided into a series of transactions, each of the transactions includes a series of operations that can be repeated without causing substantial disruption during the resuming of execution of the interrupted ones of the tasks, and the controller servers are programmed for maintaining state information of the tasks in memory of the cached disk array by committing results of each transaction to the memory of the cached disk array before committing results of a next transaction in the series of transactions.

13. The file server as claimed in claim 7, wherein the active one of the controller servers is programmed to perform a stream server failover task for detecting a failure of one of the stream server computers and in response selecting another one of the stream server computers to resume servicing of a client request interrupted by the failure of one of the stream server computers.

14. The file server as claimed in claim 13, wherein the controller servers and the stream server computers share a data link over which are periodically transmitted status signals including a status signal transmitted from the active one of the controller servers to the inactive one of the controller servers and a status signal transmitted from each of the stream server computers to the active one of the controller servers, and wherein the inactive one of the controller servers is programmed for detecting a failure of the active one of the controller servers upon failing to receive the status signal from the active one of the controller servers, and wherein the active one of the controller servers is programmed for detecting a failure of one of the stream server computers upon failing to receive the status signal from one of the stream server computers.

15. A file server for serving clients in a data network, said file server comprising, in combination:

a cached disk array storage subsystem;

a plurality of stream server computers linking the cached disk array storage subsystem to the data network for transfer of data streams between the cached disk array storage subsystem and the data network; and

at least one controller server for receiving a request for file access from a client in the data network, said controller server being connected to the stream server computers for selecting one of the stream server computers to service the request by streaming data between the cached disk array storage subsystem and the data network;

wherein the controller server is programmed to perform a stream server failover task for detecting a failure of one of the stream server computers and in response selecting another one of the stream server computers to resume servicing of a client request interrupted by the failure of one of the stream server computers;

wherein the stream servers are programmed to execute tasks to service client requests, and an interrupted one of the tasks includes a series of transactions, and said stream server computers are programmed to record state information for the interrupted one of the tasks during execution of the interrupted one of the tasks, and upon interruption of the interrupted one of the tasks, for resuming automatically the interrupted one of the tasks by recovering the recorded state information for the interrupted one of the tasks including state information from at least one transaction completed prior to the interruption of the interrupted one of the tasks, and resuming execution of a next one of the transactions in the interrupted one of the tasks by using the recovered state information from said at least one of the transactions completed prior to the interruption of the interrupted one of the tasks.

16. The file server as claimed in claim 15, wherein each of the stream server computers is programmed for periodically sending a signal to the controller server, and wherein the controller server is programmed to respond to a failure to receive the signal from said each of the stream server computers by checking whether any client requests are currently being serviced by said each of the stream server computers, and upon finding that a client request is currently being serviced by said each of the stream server computers, selecting another stream server computer to assume servicing of the client request currently being serviced by said each of the stream server computers.

17. The file server as claimed in claim 16, wherein the controller server and the stream server computers share a data link over which is periodically transmitted the signal from said each of the stream server computers to the controller server.

18. The file server as claimed in claim 15, wherein the client request includes a failover mode selection signal, and wherein the controller server is programmed for checking the failover mode selection signal to select one of a plurality of predetermined modes for resuming servicing of the client request found to have been interrupted by the failure of one of the stream server computers.

19. The file server as claimed in claim 18, wherein the modes include a transparent failover mode in which the controller server re-routes a data stream from the stream server computer assigned to assume servicing of the client request, and a client control failover mode in which the controller server informs the client of the stream server computer failure, and the client re-routes a data stream from the stream server computer selected to assume servicing of the client request.

20. The file server as claimed in claim 19, wherein the controller server is programmed to default to the client control failover mode when the client has selected the transparent failover mode and the client but not the controller server has the capability for re-routing a data stream from the stream server computer selected to resume servicing of the client request.

21. A file server for serving clients in a data network, said file server comprising, in combination:

a cached disk array storage subsystem; and

a plurality of stream server computers linking the cached disk array storage subsystem to the data network for transfer of data streams between the cached disk array storage subsystem and the data network, at least one of the stream server computers being programmed for responding to a request from a client for file access by streaming data between the cached disk array storage subsystem and the data network;

wherein the file server is programmed to perform stream server failover, in response to a failure of said one of the stream server computers, in which another one of the stream server computers resumes servicing of a client request interrupted by the failure of said one of the stream server computers, and

wherein the client request includes a failover mode selection signal, and wherein the file server is programmed for checking the failover mode selection signal to select one of a plurality of predetermined modes for resuming servicing of the client request found to have been interrupted by the failure of said one of the stream server computers.

22. The file server as claimed in claim 21, wherein the modes include a transparent failover mode in which the file server re-routes a data stream from the stream server computer resuming servicing of the client request, and a client control failover mode in which the file server informs the client of the stream server computer failure, and the client re-routes a data stream from the stream server computer resuming servicing of the client request.

23. The file server as claimed in claim 22, wherein the controller server is programmed to default to the client control failover mode when the client has selected the transparent failover mode and the client but not the controller server has the capability for re-routing a data stream from the stream server computer selected to resume servicing of the client request.

24. A method of operating a file server for serving clients in a data network, said file server having a cached disk array storage subsystem and a plurality of server computers linked to the cached disk array storage subsystem for transferring data between the cached disk array storage subsystem and the data network, said method comprising the steps of:

(a) said server computers executing tasks for responding to requests for file access from the clients in the data network; and

(b) detecting a failure of one of the server computers causing interruption of execution of one of the tasks, and upon detecting the failure, causing another one of the server computers to resume automatically the interrupted one of the tasks;

wherein the interrupted one of the tasks includes a series of transactions, execution of the interrupted one of the tasks includes recording state information from at least one transaction completed prior to the interruption of the interrupted one of the tasks, and resuming automatically the interrupted one of the tasks includes recovering the recorded state information from said at least one transaction completed prior to the interruption of the interrupted one of the tasks, and resuming execution of a next one of the transactions in the interrupted one of the tasks using the recovered state information from said at least one of the transactions completed prior to the interruption of the interrupted one of the tasks.

25. The method as claimed in claim 24, wherein each of the server computers periodically sends a signal to the other server computers, and wherein one of the other server computers resumes a task of said each of the server computers in response to a failure of said each of the server computers to periodically send the signal.

26. The method as claimed in claim 24, wherein the plurality of server computers execute tasks by maintaining state information of the tasks in memory of the cached disk array storage subsystem, and wherein the server computers resume interrupted ones of the tasks by recovering the state information for the interrupted ones of the tasks from the memory of the cached disk array, and resuming execution of the interrupted ones of the tasks using the recovered state information for the interrupted ones of the tasks.

27. The method as claimed in claim 26, wherein the plurality of server computers maintain state information of the tasks in the memory of the cached disk array by writing updates of the state information to a log file in the memory of the cached disk array, and recovering the state information of the interrupted ones of the tasks by reading from the log file updates of the state information.

28. The method as claimed in claim 26, wherein the tasks are subdivided into a series of transactions, each of the transactions includes a series of operations that can be repeated without causing substantial disruption during the resuming of execution of the interrupted ones of the tasks, and the controller servers maintain state information of the tasks in memory of the cached disk array by committing results of each transaction to the memory of the cached disk array before committing results of a next transaction in the series of transactions.

29. A method of operating a file server for serving clients in a data network, the file server having a cached disk array storage subsystem, a plurality of stream server computers linking the cached disk array storage subsystem to the data network for transfer of data streams between the cached disk array storage subsystem and the data network, and at least two controller servers for receiving a request for file access from a client in the data network, said method comprising the steps of:

(a) selecting one of the controller servers to be active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests, and selecting another one of the controller servers to be inactive in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests, and

(b) the inactive one of the controller servers responding automatically to a failure of the active one of the controller servers by resuming tasks of the active one of the controllers having been interrupted by the failure, and becoming active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests;

wherein an interrupted one of said tasks includes a series of transactions, and the active one of the controller servers records state information for the interrupted one of the tasks during execution of the interrupted one of the tasks, and the inactive one of the controller servers resumes automatically the interrupted one of the tasks by recovering the recorded state information for the interrupted one of the tasks including state information from at least one transaction completed prior to the interruption of the interrupted one of the tasks, and resuming execution of a next one of the transactions in the interrupted one of the tasks using the recovered state information from said at least one of the transactions completed prior to the interruption of the interrupted one of the tasks.

30. The method as claimed in claim 29, wherein:

the active one of the controller servers periodically sends a signal to the inactive one of the controller servers, and executes tasks by maintaining state information of the tasks in memory of the cached disk array storage subsystem, and

the inactive one of the controller servers responds to a failure to receive the signal by recovering the state information of the interrupted tasks from the memory of the cached disk array storage system, and becoming active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests.

31. The method as claimed in claim 30, wherein the memory of the cached disk array stores system state information indicating which of the controller servers is selected to be active and which of the controller servers is selected to be inactive, the controller servers read the system state information to determine whether or not to become active in executing tasks for responding to requests for file access from clients in the data network by selecting respective ones of the stream server computers to service the requests, and wherein the inactive one of the controller servers becomes active in response to a failure to receive the signal by changing the system state information so that the active one of the controller servers becomes inactive.

32. The method as claimed in claim 31, wherein the active one of the controller servers reads the system state information before changing the state information of the tasks in the memory of the cached disk array storage subsystem, so that the active one of the controller servers will not change the state information of the tasks in the memory of the cached disk array a certain time after the inactive one of the controller servers becomes active.

33. The method as claimed in claim 30, wherein the controller servers maintain state information of the tasks in the memory of the cached disk array by writing updates of the state information to a log file in the memory of the cached disk array, and recovering the state information of the interrupted ones of the tasks by reading from the log file updates of the state information.

34. The method as claimed in claim 30, wherein the tasks are subdivided into a series of transactions, each of the transactions includes a series of operations that can be repeated without causing substantial disruption during the resuming of execution of the interrupted ones of the tasks, and the controller servers are programmed for maintaining state information of the tasks in memory of the cached disk array by committing results of each transaction to the memory of the cached disk array before committing results of a next transaction in the series of transactions.

35. The method as claimed in claim 29, wherein the active one of the controller servers performs a stream server failover task for detecting a failure of one of the stream server computers and in response selecting another one of the stream server computers to resume servicing of a client request interrupted by the failure of one of the stream server computers.

36. The method as claimed in claim 35, wherein the controller servers and the stream server computers share a data link over which are periodically transmitted status signals including a status signal transmitted from the active one of the controller servers to the inactive one of the controller servers and a status signal transmitted from each of the stream server computers to the active one of the controller servers, and wherein the inactive one of the controller servers detects a failure of the active one of the controller servers upon failing to receive the status signal from the active one of the controller servers, and wherein the active one of the controller servers detects a failure of one of the stream server computers upon failing to receive the status signal from one of the stream server computers.

37. A method of operating a file server for serving clients in a data network, said file server having a cached disk array storage subsystem, a plurality of stream server computers linking the cached disk array storage subsystem to the data network for transfer of data streams between the cached disk array storage subsystem and the data network, and at least one controller server for receiving a request for file access from a client in the data network, said controller server being connected to the stream server computers for selecting one of the stream server computers to service the request by streaming data between the cached disk array storage subsystem and the data network, said method comprising the steps of:

the controller server performing a stream server failover task including the detection of a failure of one of the stream server computers, and in response selecting another one of the stream server computers to resume servicing of a client request interrupted by the failure of one of the stream server computers; and

said another one of the stream server computers resuming the servicing of the client request having been interrupted by the failure of one of the stream server computers;

wherein the servicing of the client request interrupted by the failure of one of the stream server computers includes execution of an interrupted task including a series of transactions, execution of the interrupted task includes recording state information from at least one transaction completed prior to the interruption of the interrupted task, and the resuming of servicing of the client request having been interrupted by the failure of one of the stream server computers includes recovering the recorded state information from said at least one transaction completed prior to the interruption of the interrupted task, and resuming execution of a next one of the transactions in the interrupted task using the recovered state information from said at least one of the transactions completed prior to the interruption of the interrupted task.

38. The method as claimed in claim 37, wherein each of the stream server computers periodically sends a signal to the controller server, and wherein the controller server responds to a failure to receive the signal from said each of the stream server computers by checking whether any client requests are currently being serviced by said each of the stream server computers, and upon finding that a client request is currently being serviced by said each of the stream server computers, selecting another stream server computer to assume servicing of the client request currently being serviced by said each of the stream server computers.

39. The method as claimed in claim 38, wherein the controller server and the stream server computers share a data link over which is periodically transmitted the signal from said each of the stream server computers to the controller server.

40. The method as claimed in claim 37, wherein the client request includes a failover mode selection signal, and wherein the controller server checks the failover mode selection signal to select one of a plurality of predetermined modes for resuming servicing of the client request found to have been interrupted by the failure of one of the stream server computers.

41. The method as claimed in claim 40, wherein the modes include a transparent failover mode in which the controller server re-routes a data stream from the stream server resuming servicing of the client request, and a client control failover mode in which the controller server informs the client of the stream server computer failure, and the client re-routes a data stream from the stream server computer resuming servicing of the client request.

42. The method as claimed in claim 41, wherein the controller server defaults to the client control failover mode when the client has selected the transparent failover mode and the client but not the controller server has the capability for re-routing a data stream from the stream server computer resuming servicing of the client request.

43. A method of operating a file server for serving clients in a data network, said file server having a cached disk array storage subsystem, and a plurality of stream server computers linking the cached disk array storage subsystem to the data network for transfer of data streams between the cached disk array storage subsystem and the data network, wherein said method comprises the steps of:

one of the stream server computers responding to a request from a client for file access by streaming data between the cached disk array storage subsystem and the data network; and

the file server performing stream server failover in response to a failure of said one of the stream server computers in which another one of the stream server computers resumes servicing of a client request interrupted by the failure of said one of the stream server computers;

wherein the client request includes a failover mode selection signal, and wherein the file server checks the failover mode selection signal to select one of a plurality of predetermined modes for resuming servicing of the client request found to have been interrupted by the failure of said one of the stream server computers.

44. The method as claimed in claim 43, wherein the modes include a transparent failover mode in which the file server re-routes a data stream from the stream server computer resuming servicing of the client request, and a client control failover mode in which the file server informs the client of the stream server computer failure, and the client re-routes a data stream from the stream server computer resuming servicing of the client request.

45. The method as claimed in claim 44, wherein the controller server defaults to the client control failover mode when the client has selected the transparent failover mode and the client but not the controller server has the capability for re-routing a data stream from the stream server computer resuming servicing of the client request.