System and method for distributed file system I/O recovery

• US 7,664,991 B1
• Filed: 12/17/2002
• Issued: 02/16/2010
• Est. Priority Date: 12/17/2002
• Status: Expired due to Fees

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Operating Corporation (Gen Digital Inc.)
• Inventors: Shyam, Nagaraj, Abraham, Sara, Narasimhan, Balaji, Huang, Shie-rei, Gunda, Laxmikant
• Primary Examiner(s): Baderman; Scott
• Assistant Examiner(s): Contino; Paul F
• Application Number:
  US10/321,120
• Time in Patent Office: 2,618 Days
• Field of Search: 714/4, 714/1, 714/43, 714/44, 714/56
• US Class Current: 714/43
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1443 Transmit or communication errors

Abstract

Embodiments of a system and method for distributed file system I/O recovery in storage networks. Embodiments may detect loss of access to a server in the storage network and recover application I/O requests in real-time once access to the server is restored. Embodiments may detect server and/or network failures and store failed and new I/O requests. Recovery from the failure (e.g. network reconnect, server node reboot, or failover, if this is a clustered environment) may be detected and, after recovery is detected, any stored failed and new I/O requests may be sent to the server. In one embodiment, to detect recovery from the failure, a failed I/O request may be repeatedly re-issued until the I/O request succeeds. Embodiments may be implemented in a variety of storage environments, including environments where clients issue direct I/O over a storage network to storage and control I/O over a network to a server.

Claims

1. A system, comprising:

one or more storage devices;

a file server node coupled to the one or more storage devices via a storage network; and

an application server node comprising an application and configured to access the server node via a network, wherein the application server node is a client of the file server node, wherein the application is configured to access data on the one or more storage devices by sending I/O requests to the file server node over the network;

wherein the application server node is configured to;

detect loss of access to the file server node via the network, wherein said loss generates one or more failed I/O requests from the application to the file server node;

store failed I/O requests and I/O requests issued from the application to the file server node after detection of the loss of access;

store security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

detect recovery of access to the file server node via the network;

using the stored security credentials and parameters, reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

send the stored I/O requests to the file server node after detection of the recovery and after reopening the files.

2. The system as recited in claim 1, wherein, to detect recovery of access to the file server node via the network, the application server node is further configured to repeatedly send one of the failed I/O requests to the file server node until the failed I/O request succeeds.

3. The system as recited in claim 1, wherein the system is a direct access file system, wherein the application server node is coupled to at least one of the one or more storage devices via the storage network, wherein the application server node is further configured to directly access the data on the one or more storage devices via direct data I/O requests over the storage network, and wherein the I/O requests from the application to the file server node over the network are control I/O requests.

4. The system as recited in claim 3, wherein the application is an Oracle Parallel Server (OPS) application.

5. A system, comprising:

one or more storage devices;

a cluster of two or more file server nodes coupled to the one or more storage devices via a storage network; and

an application server node comprising an application and configured to access the cluster of file server nodes via a network, wherein the application server node is a client of the cluster, wherein the application is configured to access data on the one or more storage devices by sending I/O requests to one of the cluster of file server nodes over the network;

wherein the application server node is configured to;

detect failure of the one of the cluster of file server nodes, wherein said failure generates one or more failed I/O requests from the application to the one of the cluster of file server nodes;

store failed I/O requests and I/O requests issued from the application to the one of the cluster of file server nodes after detection of the failure;

store security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

detect failover to another one of the cluster of file server nodes;

using the stored security credentials and parameters reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

send the stored I/O requests to the other one of the cluster of file server nodes after detection of the failover and after reopening the files.

6. The system as recited in claim 5, wherein, to detect failover to another one of the cluster of file server nodes, the application server node is further configured to repeatedly send one of the failed I/O requests until the failed I/O request succeeds.

7. A system, comprising:

a processor; and

a memory comprising program instructions, wherein the program instructions are executable by the processor to;

detect loss of access to a file server node coupled to the system via a network, wherein said loss generates one or more failed I/O requests from an application of the system to the file server node, wherein the application is configured to access data on one or more storage devices coupled to the file server node via a storage network by sending I/O requests to the server node over the network;

store failed I/O requests and I/O requests issued from the application to the file server node after detection of the loss of access;

store security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

detect recovery of access to the file server node via the network;

using the stored security credentials and parameters reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

send the stored I/O requests to the file server node after detection of the recovery and after reopening the files.

8. The system as recited in claim 7, wherein, to detect recovery of access to the file server node via the network, the program instructions are further executable by the processor to repeatedly send one of the failed I/O requests to the file server node until the failed I/O request succeeds.

9. The system as recited in claim 7, wherein the system is an application server that also functions as a client node in a direct access file system, wherein the system is coupled to at least one of the one or more storage devices via the storage network, wherein the system is configured to directly access the data on the one or more storage devices via direct data I/O requests over the storage network, and wherein the I/O requests from the application to the file server node over the network are control I/O requests.

10. The system as recited in claim 9, wherein the application is an Oracle Parallel Server (OPS) application.

11. A system, comprising:

a processor; and

a memory comprising program instructions, wherein the program instructions are executable by the processor to;

detect failure of a file server node coupled to the system via a network, wherein said failure generates one or more failed I/O requests from an application of the system to the file server node, wherein the application is configured to access data on one or more storage devices coupled to the file server node via a storage network by sending I/O requests to the file server node over the network;

store failed I/O requests and I/O requests issued from the application to the file server node after detection of the loss of access;

store security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

detect failover to another file server node;

using the stored security credentials and parameters reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

send the stored I/O requests to the other file server node after detection of the failover and after reopening the files.

12. The system as recited in claim 11, wherein, to detect failover to another file server node, the program instructions are further executable by the processor to repeatedly send one of the failed I/O requests until the failed I/O request succeeds.

13. A system, comprising:

means for detecting loss of access from an application on an application server node to a file server node coupled to the application server node via a network, wherein the application server node is a client of the file server node, wherein the application is configured to access data on one or more storage devices coupled to the file server node via a storage network by sending I/O requests to the file server node over the network;

means for storing failed I/O requests and I/O requests issued from the application to the file server node after said detecting recovery of access to file the server node;

means for storing security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

means for detecting recovery of access to the file server node via the network; and

means for recovering said stored failed I/O requests and said stored I/O requests issued from the application to the file server node after said detecting recovery of access to the server node, wherein said recovering comprises using the stored security credentials and parameters to reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

means for sending the stored I/O requests to the file server node after the detection of the recovery and after reopening the files.

14. A method, comprising:

an application server node detecting loss of access to a file server node via a network, wherein the application server node is a client of the file server node, wherein said loss generates one or more failed I/O requests from an application on the application server node to the file server node;

wherein the application is configured to access data on one or more storage devices coupled to the file server node via a storage network by sending I/O requests to the file server node over the network;

the application server node storing failed I/O requests and I/O requests issued from the application to the file server node after detection of the loss of access;

the application server node storing security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

the application server node detecting recovery of access to the file server node via the network;

the application server node using the stored security credentials and parameters to reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

the application server node sending the stored I/O requests to the file server node after detection of the recovery and after reopening the files.

15. The method as recited in claim 14, wherein said detecting recovery of access to the file server node via the network comprises repeatedly sending one of the failed I/O requests to the file server node until the failed I/O request succeeds.

16. The method as recited in claim 14, wherein the application server node and the file server node are in a direct access file system, wherein the application server node is coupled to at least one of the one or more storage devices via the storage network, wherein the application server node is configured to directly access the data on the one or more storage devices via direct data I/O requests over the storage network, and wherein the I/O requests from the application to the file server node over the network are control I/O requests.

17. The method as recited in claim 16, wherein the application is an Oracle Parallel Server (OPS) application.

18. A method, comprising:

an application server node detecting failure of a file server node via a network, wherein the application server node is a client of the file server node, wherein said failure generates one or more failed I/O requests from an application on the application server node to the file server node;

wherein the application is configured to access data on one or more storage devices coupled to the file server node via a storage network by sending I/O requests to the file server node over the network;

the application server node storing failed I/O requests and I/O requests issued from the application to the file server node after detection of the loss of access;

the application server node storing security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

the application server node detecting failover to another file server node;

the application server node using the stored security credentials and parameters to reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

the application server node sending the stored I/O requests to the other file server node after detection of the failover and after reopening the files.

19. The method as recited in claim 18, wherein said detecting failover to another file server node comprises repeatedly sending one of the failed I/O requests to the file server node until the failed I/O request succeeds.

20. A computer-accessible storage medium comprising program instructions, wherein the program instructions are configured to implement:

an application server node detecting loss of access to a file server node via a network, wherein the application server node is a client of the file server node, wherein said loss generates one or more failed I/O requests from an application on the application server node to the file server node;

wherein the application is configured to access data on one or more storage devices coupled to the file server node via a storage network by sending I/O requests to the file server node over the network;

the application server node storing failed I/O requests and I/O requests issued from the application to the file server node after detection of the loss of access;

the application server node storing security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

the application server node detecting recovery of access to the file server node via the network;

the application server node using the stored security credentials and parameters to reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

the application server node sending the stored I/O requests to the file server node after detection of the recovery and after reopening the files.

21. The computer-accessible storage medium as recited in claim 20, wherein, in said detecting recovery of access to the file server node via the network, the program instructions are further configured to implement repeatedly sending one of the failed I/O requests to the file server node until the failed I/O request succeeds.

22. The computer-accessible storage medium as recited in claim 20, wherein the application server node and the file server node are in a direct access file system, wherein the application server node is coupled to at least one of the one or more storage devices via the storage network, wherein the application server node is configured to directly access the data on the one or more storage devices via direct data I/O requests over the storage network, and wherein the I/O requests from the application to the file server node over the network are control I/O requests.

23. The computer-accessible storage medium as recited in claim 22, wherein the application is an Oracle Parallel Server (OPS) application.

24. A computer-accessible storage medium comprising program instructions, wherein the program instructions are configured to implement:

an application server node detecting failure of a file server node via a network, wherein the application server node is a client of the file server node, wherein said failure generates one or more failed I/O requests from an application on the application server node to the file server node;

wherein the application is configured to access data on one or more storage devices coupled to the file server node via a storage network by sending I/O requests to the file server node over the network;

the application server node storing failed I/O requests and I/O requests issued from the application to the file server node after detection of the loss of access;

the application server node storing security credentials and parameters for the failed I/O requests and the I/O requests issued after detection of the loss of access;

the application server node detecting failover to another file server node;

the application server node using the stored security credentials and parameters to reopen files for the failed I/O requests and the I/O requests issued after detection of the loss of access; and

the application server node sending the stored I/O requests to the other file server node after detection of the failover and after reopening the files.

25. The computer-accessible storage medium as recited in claim 24, wherein, in said detecting failover to another file server node, the program instructions are further configured to implement repeatedly sending one of the failed I/O requests to the file server node until the failed I/O request succeeds.