Method and apparatus for file system disaster recovery

• US 6,144,999 A
• Filed: 05/29/1998
• Issued: 11/07/2000
• Est. Priority Date: 05/29/1998
• Status: Expired due to Fees

• Current Assignee: Sun Microsystems Incorporated (Oracle Corporation), Pacific Advanced Technology, Inc. (PAT Industries, Inc.)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Khalidi, Yousef A., Talluri, Madhusudhan, Dion, David, Swaroop, Anil
• Primary Examiner(s): Meky, Moustafa M.
• Application Number:
  US09/087,261
• Time in Patent Office: 893 Days
• Field of Search: 709/203, 709/212, 709/213, 709/217, 709/218, 709/219, 709/223, 709/224, 709/225, 709/242, 707/10, 707/200, 707/201, 707/202, 707/204, 714/15
• US Class Current: 709/219
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2074 Asynchronous techniques

Abstract

A file disaster recovery system that employs geographical replication of data from a local site to remote site in a such a manner that file requests from clients of the local site can be handled by a file server on the remote site following a failover from the local site to the remote site. Geographical data replication software running on a local server checkpoints to a log in local stable storage all information on file operations that change the file state of the local file system. According to a selected mode, the local geographical data replication software flushes information in the log pertaining to the file operations since the last flush to the remote site. At the remote site, compatible remote geographical data replication software running on a remote file server receives the flushed log and replicates in sequence order the file operations represented in the flushed log. The results of the operations are stored on remote stable storage. The local and remote servers can be clusters or single servers. There is no need for commonality, except for operating and file systems, between the local and remote sites. Because operations are replicated and not low level, formatted data, the local and remote stable file storage can be completely different.

Claims

1. A geographical data replication system comprising:

a local primary server configured to run a first high availability file system (FHAFS) and a local file system;

a local secondary server coupled to the local primary server configured to run the local file system and respond to FHAFS mini-transactions initiated by the local primary;

first dual-ported file storage coupled to the local servers with which the local servers interact via the local file system;

at least one client configured to issue local file system requests to the local primary;

the local primary being configured in accordance with the FHAFS to checkpoint to the local secondary mini-transactions conveying application state, enabling the local secondary, when the local primary fails, to take over operations of the local primary consistent with the checkpointed application state, the local secondary only being active when the local primary is inactive;

a local server that is whichever of the local primary and secondary is active configured to intercept the local file requests and to determine which of the local file requests is one of a first set of file requests that will alter file state of the local file system; and

a log file stored on the first dual-ported file storage in which the local server writes operations and data required to serve the first set of file requests, the local server being configured to periodically flush the log file to a remote site, enabling the remote site, when the local site fails, to serve the local file requests with little or no loss of the file state;

such that, when it is active, the local primary is configured to pass file checkpoints to the local secondary server in conjunction with the mini-transaction checkpoints issued by the FHAFS, enabling the local secondary, when the local primary fails, to complete uncompleted local file requests and to handle subsequent local file requests consistent with the file and application state.

2. The geographical data replication system of claim 1, wherein the local file system is UFS and the FHAFS is HA-PXFS.

3. The geographical data replication system claim 1, further comprising a membership monitor configured to detect failure of the local primary and to activate the local secondary when the local primary fails.

4. The geographical data replication system of claim 3, wherein, when activated by the membership monitor, the local secondary is configured to:

mount the local file system;

obtain pointers to local file system objects represented in the file checkpoints; and

complete file operations in progress at time of failure of the local primary.

5. The geographical data replication system of claim 1, wherein:

the local server is configured to enter a unique transaction id into the log file corresponding to each of the file system requests;

the file checkpoint sent by the local primary to the local secondary for a particular file system request comprises a respective particular transaction id;

upon receiving the file checkpoint, the local secondary is configured to allocate a state object associated with the particular transaction id and acknowledge the file checkpoint;

after sending the file checkpoint, the local primary is configured to update the dual-ported storage in accordance with the particular file system request;

the local primary is configured to enter a commit message into the log file when the particular file system request is successfully completed and an invalidate message otherwise;

the local primary returns result of the particular file system request to the requesting client; and

the client sends an asynchronous forget message to the local secondary.

6. The geographical data replication system of claim 1,

wherein the remote site comprises at least one remote server running a second file system and second stable file storage;

further comprising;

the remote server being configured to update state of the second file system in accordance with the flushed log file by performing the operations on the data represented in the flushed log file;

such that, whenever failover occurs from the local to the remote site, the remote server is able to serve the requests from the clients with little or no loss of file state.

7. The geographical data replication system of claim 6, wherein the second stable file storage is not required to be identical to the first dual-ported file storage.

8. The geographical data replication system of claim 6, wherein:

the at least one remote server comprises a remote primary server and a remote secondary server;

the second file system is a second high availability file system (SHAFS); and

the second stable file storage comprises second dual-ported file storage accessible to both the remote primary and secondary servers.

9. The geographical data replication system of claim 8, wherein the SHAFS is HA-PXFS.

10. The geographical data replication system of claim 6, further comprising:

a first identifier maintained by the first high availability file system for each of the files on the first dual-ported file storage, the first identifier being transmitted by the local server to the remote server for each of the flushed files;

a second identifier maintained by the second file system for each of the flushed files replicated on the remote server; and

a mapping table on the remote server that maintains a mapping between the first and second identifiers enabling the remote server to serve the requests from the clients.

11. The geographical data replication system of claim 10, wherein:

the first identifier is a persistent local identifier of the flushed file on the local site; and

the second identifier is a remote identifier of the flushed file on the remote site.

12. The geographical data replication system of claim 11, wherein:

the first identifier includes a persistent local directory identifier of a directory on the local site in which the flushed file is stored and a persistent local name identifier associated with a name on the local site identifying the flushed file; and

the second identifier includes a remote directory identifier of a directory on the remote site in which the flushed file is stored and a remote name identifier associated with a name on the remote site identifying the flushed file.

13. The geographical data replication system of claim 6, wherein the log comprises:

a plurality of records, each of the records including;

a body holding file information identifying, among other things, a respective file operation and its associated data, and

a file header enabling the geographical replication systems to decode the log, including the body.

14. The geographical data replication system of claim 13, wherein the local site includes a log anchor indicating:

which of the records in the log file were last flushed; and

location of next record to be written to the log file.

15. The geographical data replication system of claim 6, wherein the local and remote servers operate in respective modes that define the file state coherency between the local and the remote sites.

16. The geographical data replication system of claim 15, wherein the file state is one of 1-safe, 2-safe and very-safe.

17. The geographical data replication system of claim 16, wherein the local server comprises one of:

a cluster; and

a single server.

18. The geographical data replication system of claim 1, wherein the local site includes a cache that the local file system uses to satisfy the requests before applying the requests to the first dual-ported file storage;

such that the local server is configured to intercept only those file requests that are applied by the local file system to the first dual-ported file storage.

19. The geographical data replication system of claim 1, wherein the first set of file requests that will alter file state of the local file system include selected:

file data methods;

file attribute methods; and

directory methods.

20. The geographical data replication system of claim 19, wherein, when the FHAFS is HA-PXFS:

the file data methods include

page_-- out;

write_-- out;

sync;

uiowrite; and

fsync;

the file attribute methods include;

set_-- attributes;

set_-- secattr;

ioctl; and

the directory methods include;

create_-- fobj;

remove_-- fobj;

create_-- symlink;

rename_-- fobj;

link_-- fobj;

create_-- dir; and

remove_-- dir.

21. The geographical data replication system of claim 1, wherein the data comprises one of:

partitions; and

files.

22. A geographical data replication method for use in a network including:

a local primary server configured to run a first high availability file system (FHAFS) and a local file system, a local secondary server coupled to the local primary server configured to run the local file system and respond to FHAFS mini-transactions initiated by the local primary, first dual-ported file storage coupled to the local servers with which the local servers interact via the local file system, and at least one client configured to issue local file system requests to the local primary, the local primary being configured in accordance with the FHAFS to checkpoint to the local secondary mini-transactions conveying application state, enabling the local secondary, when the local primary fails, to take over operations of the local primary consistent with the checkpointed application state, the local secondary only being active when the local primary is inactive, the method comprising;

on whichever of the local primary and secondary is active;

intercepting the local file requests;

determining which of the local file requests is one of a first set of file requests that will alter file state of the local file system;

writing to a log file stored on the first dual-ported file storage operations and data required to serve the first set of file requests; and

periodically flushing the log file to a remote site;

on the remote site;

using the information in the log file, serving the local file requests with little or no loss of the file state when the local site fails; and

on the local primary, when it is active;

passing file checkpoints to the local secondary in conjunction with the mini-transaction checkpoints issued by the FHAFS, enabling the local secondary, when the local primary fails, to complete uncompleted local file requests and to handle subsequent local file requests consistent with the file and application state.

23. The geographical data replication method of claim 22, wherein the local file system is UFS and the FHAFS is HA-PXFS.

24. The geographical data replication method of claim 22, further comprising:

detecting failure of the local primary; and

activating the local secondary when the local primary fails.

25. The geographical data replication method of claim 24, further comprising:

following the activating step, on the local secondary;

mounting the local file system;

obtaining pointers to local file system objects represented in the file checkpoints; and

completing file operations in progress at time of failure of the local primary.

26. The geographical data replication method of claim 22, further comprising:

on the local primary, prior to the file checkpoint passing step, entering a unique transaction id into the log file corresponding to each of the file system requests, such that the file checkpoint passing step for a particular file system request includes passing a respective transaction id for that request;

on the local secondary, upon receiving the file checkpoint,

allocating a state object associated with the particular transaction id; and

acknowledging the file checkpoint;

on the local primary, after sending the file checkpoint,

updating the dual-ported storage in accordance with the particular file system request;

entering a commit message into the log file when the particular file system request is successfully completed and an invalidate message otherwise; and

returning result of the particular file system request to the requesting client; and

upon receiving the result of the particular file system request, the client sending an asynchronous forget message to the local secondary.

27. The geographical data replication system of claim 22,

wherein the remote site comprises at least one remote server running a second file system and second stable file storage, further comprising:

on the remote server;

updating state of the second file system in accordance with the flushed log file by performing the operations on the data represented in the flushed log file; and

following failover from the local to the remote site, serving the requests from the clients with little or no loss of file state.

28. The geographical data replication system of claim 27, wherein the second stable file storage is not required to be identical to the first dual-ported file storage.

29. The geographical data replication system of claim 27, wherein:

the at least one remote server comprises a remote primary server and a remote secondary server;

the second file system is a second high availability file system (SHAFS); and

the second stable file storage comprises second dual-ported file storage accessible to both the remote primary and secondary servers.

30. The geographical data replication system of claim 29, wherein the SHAFS is HA-PXFS.

31. The geographical data replication system of claim 29, further comprising:

on the local site;

maintaining in the first high availability file system a first identifier for each of the files on the first dual-ported file storage, and

transmitting the first identifier to the remote server for each of the flushed files; and

on the remote site;

maintaining in the second file system a second identifier for each of the flushed files replicated on the remote server;

maintaining a mapping table associating the first and second identifiers; and

using the mapping table, serving the requests from the clients when failover occurs to the remote site.

32. The geographical data replication system of claim 31, wherein:

the first identifier is a persistent local identifier of the flushed file on the local site; and

the second identifier is a remote identifier of the flushed file on the remote site.

33. The geographical data replication system of claim 32, wherein:

the first identifier includes a persistent local directory identifier of a directory on the local site in which the flushed file is stored and a persistent local name identifier associated with a name on the local site identifying the flushed file; and

the second identifier includes a remote directory identifier of a directory on the remote site in which the flushed file is stored and a remote name identifier associated with a name on the remote site identifying the flushed file.

34. The geographical data replication system of claim 29, wherein the log file comprises:

a plurality of records, each of the records including;

a body holding file information identifying, among other things, a respective file operation and its associated data, and

a file header enabling the log file, including the body, to be decoded.

35. The geographical data replication system of claim 34, wherein the local site includes a log anchor indicating:

which of the records in the log file were last flushed; and

location of next record to be written to the log file.

36. The geographical data replication system of claim 29, wherein the local and remote servers operate in respective modes that define the file state coherency between the local and the remote sites.

37. The geographical data replication system of claim 36, wherein the file state is one of 1-safe, 2-safe and very-safe.

38. The geographical data replication system of claim 22, further comprising:

the local file system satisfying the requests from a cache before applying the requests to the first dual-ported file storage; and

intercepting only those file requests that are applied by the local file system to the first dual-ported file storage.

39. The geographical data replication system of claim 24, wherein the first set of file requests that will alter file state of the local file system include selected:

file data methods;

file attribute methods; and

directory methods.

40. The geographical data replication system of claim 39, wherein, when the FHAFS is HA-PXFS:

the file data methods include

page_-- out;

write_-- out;

sync;

uiowrite; and

fsync;

the file attribute methods include;

set_-- attributes;

set_-- secattr;

ioctl; and

the directory methods include;

create_-- fobj;

remove_-- fobj;

create_-- symlink;

rename_-- fobj;

link_-- fobj;

create_-- dir; and

remove_-- dir.

41. The geographical data replication system of claim 24, wherein the data comprises one of;

partitions; and

files.

42. The geographical data replication system of claim 24, wherein the local server comprises one of:

a cluster; and

a single server.

43. A computer program product for providing geographical data replication in a computer network including:

a local primary server configured to run a first high availability file system (FHAFS) and a local file system, a local secondary server coupled to the local primary server configured to run the local file system and respond to FHAFS mini-transactions initiated by the local primary, first dual-ported file storage coupled to the local servers with which the local servers interact via the local file system, and at least one client configured to issue local file system requests to the local primary, the local primary being configured in accordance with the FHAFS to checkpoint to the local secondary mini-transactions conveying application state, enabling the local secondary, when the local primary fails, to take over operations of the local primary consistent with the checkpointed application state, the local secondary only being active when the local primary is inactive, the computer program product including a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising;

local geographical data replication software that configures a local server that is whichever of the local primary and secondary is active to;

intercept the local file requests;

determine which of the local file requests is one of a first set of file requests that will alter file state of the local file system;

write to a log file stored on the first dual-ported file storage operations and data required to serve the first set of file requests; and

periodically flush the log file to a remote server, enabling the remote server, when the local site fails, to serve the local file requests with little or no loss of the file state; and

when it is active, the local primary being configured to pass file checkpoints to the local secondary server in conjunction with the mini-transaction checkpoints issued by the FHAFS, enabling the local secondary, when the local primary fails, to complete uncompleted local file requests and to handle subsequent local file requests consistent with the file and application state.

44. The computer program product of claim 43, wherein:

when the local site includes a cache that the first file system uses to satisfy the requests before applying the requests to a first stable file storage, the local server is configured to intercept only those requests that are applied by the first file system to the first stable file storage.

45. The computer program product of claim 43, wherein:

the local server is configured to transmit to the remote server a first identifier maintained for each of the files on the first stable file storage for each of the flushed files;

further comprising;

remote geographical data replication software that configures the remote server to maintain a mapping table on the remote server that maps the first identifier to a second identifier maintained by a second file system running on the remote server for each of the flushed files replicated on the remote server, enabling the remote server to serve the requests from the clients.

46. The computer program product of claim 45, wherein:

the first identifier is a persistent local identifier of the flushed file on the local site; and

the second identifier is a remote identifier of the flushed file on the remote site.

47. The computer program product of claim 45, wherein:

the first identifier includes a persistent local directory identifier of a directory on the local site in which the flushed file is stored and a persistent local name identifier associated with a name on the local site identifying the flushed file; and

the second identifier includes a remote directory identifier of a directory on the remote site in which the flushed file is stored and a remote name identifier associated with a name on the remote site identifying the flushed file.

48. The computer program product of claim 43, wherein the first set of file requests that will alter state of the first file system include selected:

file data methods;

file attribute methods; and

directory methods.

49. The computer program product of claim 48, wherein, when the first and second file systems are PXFS:

the file data methods include

page_-- out;

write_-- out;

sync;

uiowrite; and

fsync;

the file attribute methods include;

set_-- attributes;

set_-- secattr;

ioctl; and

the directory methods include;

create_-- fobj;

remove_-- fobj;

create_-- symlink;

rename_-- fobj;

link_-- fobj;

create_-- dir; and

remove_-- dir.

50. The computer program product of claim 43, wherein the log file comprises:

a plurality of records, each of the records including;

a body holding file information identifying, among other things, a respective file operation and its associated data, and

a file header enabling the local and remote servers to decode the log, including the body.

51. The computer program product of claim 50, wherein the local geographical data replication software is configured to maintain a log anchor indicating:

which of the records in the log file were last flushed; and

location of next record to be written to the log file.

52. The computer program product of claim 51, wherein the local and remote servers operate in respective modes that define the file state coherency between the local and the remote sites.

53. The computer program product of claim 52, wherein the file state is one of 1-safe, 2-safe and very-safe.

54. A geographical data replication system comprising:

a local server that is whichever of a local primary server and secondary server is active configured to intercept local file requests directed to a local file system and to determine which of the local file requests is one of a first set of file requests that will alter file state of the local file system; and

a persistent log file accessible to the local primary and local secondary in which the local server writes operations and data required to serve the first set of file requests, the local server being configured to periodically flush the log file to a remote server, enabling the remote server, when the local server fails, to serve the local file requests with little or no loss of the file state;

such that, when the local primary is active, the local primary is configured to pass file checkpoints to the local secondary server in conjunction with mini-transaction checkpoints issued by a first high availability file system (FHAFS) running on the local primary, enabling the local secondary, when the local primary fails, to complete uncompleted local file requests and to handle subsequent local file requests consistent with the file and application state.

55. A geographical data replication method, comprising:

on a local server that is whichever of a local primary server and secondary server is active;

intercepting local file requests directed to a local file system;

determining which of the local file requests is one of a first set of file requests that will alter file state of the local file system;

writing to a persistent log file accessible to the local primary and local secondary operations and data required to serve the first set of file requests; and

periodically flushing the log file to a remote server;

on the remote server;

using information in the log file, serving the local file requests with little or no loss of the file state when the local server fails; and

on the local primary, when it is active;

passing file checkpoints to the local secondary in conjunction with the mini-transaction checkpoints issued by a first high availability file system (FHAFS) running on the local primary, enabling the local secondary, when the local primary fails, to complete uncompleted local file requests and to handle subsequent local file requests consistent with the file and application state.

56. A computer program product for providing geographical data replication, the computer program product including a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:

local geographical data replication software that configures a local server that is whichever of a local primary and secondary is active to;

intercept local file requests directed to a local file system running on the local server;

determine which of the local file requests is one of a first set of file requests that will alter file state of the local file system;

write to a persistent log file accessible to the local primary and local secondary operations and data required to serve the first set of file requests; and

periodically flush the log file to a remote server, enabling the remote server, when the local server fails, to serve the local file requests with little or no loss of the file state; and

when it is active;

the local primary being configured to pass file checkpoints to the local secondary server in conjunction with the mini-transaction checkpoints issued by a first high availability file system (FHAFS) running on the local primary, enabling the local secondary, when the local primary fails, to complete uncompleted local file requests and to handle subsequent local file requests consistent with the file and application state.