Distributed file serving architecture system with metadata storage virtualization and data access at the data server connection speed

• US 20040225719A1
• Filed: 05/07/2003
• Published: 11/11/2004
• Est. Priority Date: 05/07/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kisley, Richard Victor, Knight, Philip Douglas
• Granted Patent: US 7,610,348 B2
• Time in Patent Office: Days
• US Class Current: 709/212
• CPC Class Codes: G06F 13/385 for adaptation of a particular data processing system to different peripheral devices
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Method, apparatus and program storage device that provides a distributed file serving architecture with metadata storage virtualization and data access at the data server connection speed is provided. A host issues a file access request including data target locations. The file access request including data target locations is processed. Remote direct memory access (RDMA) channel endpoint connection are issued in response to the processing of the file access request. An RDMA transfer of the file-block data associated with the file access request is made directly between a memory at the host and a data server.

Claims

1. A host, comprising:

a central processing unit for running an application thereon;

memory for storing data therein; and

a host bus adapter, coupled to the memory, the host bus adapter including an adapter processor configured for issuing an input/output request to a metadata server and for enabling direct memory access transfer of data associated with the input/output request directly between the memory and a data server.

2. The host of claim 1, wherein the adapter processor is further configured for enabling the direct memory access via an RDMA channel.

3. The host of claim 2, wherein the enabling of the direct memory access via an RDMA channel further comprises direct memory access from the host bus adapter to memory, wherein the memory comprises user-mode memory.

4. The host of claim 2, wherein the RDMA channel is initiated by the host.

5. The host of claim 2, wherein the RDMA channel is initiated by the data server.

6. The host of claim 1, wherein the adapter processor is further configured for receiving data block location information from the metadata server in response to issuing the input/output request.

7. The host of claim 1, wherein the adapter processor is further configured for issuing commands identifying memory locations at the host to the data server for enabling the direct memory access transfer of data associated with the input/output request between the memory and the data server.

8. The host of claim 1, wherein the adapter processor is further configured for notifying the metadata server of completion of the input/output request.

9. The host of claim 1, wherein the adapter processor is further configured with a client file access stack for controlling input/output requests, the client file access stack comprising:

file access protocol for receiving data requests from the application and issuing the input/output request;

a virtualization layer for providing virtual addressing and resolution; and

an RDMA transport layer for controlling connections with the host and authenticating memory access in support of the input/output request.

10. The host of claim 9, wherein the RDMA transport layer authenticates memory access by maintaining memory address mapping until an RDMA transport address of a data server is authenticated.

11. The host of claim 1, wherein the adapter processor is further configured with a client file access stack for controlling input/output requests, the client file access stack comprising:

file access protocol for receiving data requests from the application and issuing the input/output request;

an RDMA endpoint address adaptation layer for resolving a virtual address for the input/output request and opening an RDMA channel for the resolved virtual address;

an RDMA transport endpoint provider layer for maintaining memory mapping and authenticating memory access in support of the input/output request; and

an RDMA transport layer for controlling connections with the host.

12. The host of claim 1, wherein the input/output request comprises a file access request with data target locations.

13. The host of claim 1 further comprising an intelligent RDMA component for controlling the RDMA transfer of file-block data between the memory of the host and the storage device.

14. A metadata server, comprising:

memory for storing metadata associated with data on a data server; and

a processor configured for receiving an input/output request from a host and issuing to the data server input/output operation details based on the metadata for transferring file-block data directly between memory at the host and the data server.

15. The metadata server of claim 14, wherein the input/output request is a request to write data directly from a memory address at the host to the data server.

16. The metadata server of claim 14, wherein the input/output request is a request to read data referenced by input/output operation details from the data server directly into memory at the host.

17. The metadata server of claim 14, wherein the processor is configured to virtualize file-block data located at the data server.

18. The metadata server of claim 14, wherein the input/output request is a request for metadata, the processor returning requested metadata to a memory address at the host.

19. The metadata server of claim 14, wherein the processor returns the requested metadata via an RDMA channel between the host and the metadata server.

20. The metadata server of claim 14, wherein the processor is further configured for receiving a completion notice from the data server and issuing a file access response to the host indicating completion of a session associated with the input/output request.

21. The metadata server of claim 14, wherein the processor is further configured to invalidate cached data at the host when metadata associated with the cached data changes at the metadata server.

22. The metadata server of claim 14, wherein the processor is further configured to translates input/output request from a host from an RDMA file access language to a meta-file system language of the metadata server to allow the input/output request to be mapped to raw data blocks on the data server,.

23. The metadata server of claim 14 further comprising an intelligent RDMA component for controlling the RDMA transfer of file-block data between a memory address of the host and the data server.

24. A data server, comprising:

at least one storage device for storing file-block data thereon; and

a processor, operatively coupled to the storage device, the processor being configured for receiving from a metadata server input/output operation details associated with an input/output request of a host, for establishing a direct connection to a memory address at the host identified by the received input/output operation details, and for transferring file-block data referenced by the received input/output operation details directly between the memory address of the host and the at least one storage device.

25. The data server of claim 24, wherein the processor is further configured to provide a completion notice to the metadata server to signal completion of the file-data transfer between the memory address of the host and the storage device.

26. The data server of claim 24, wherein the processor further comprises a storage controller for presenting the at least one storage device as logical unit numbers and for satisfying the input/output request by processing file-block data between the memory address of the host and the at least one storage device.

27. The data server of claim 24, wherein the data server stack further comprises an intelligent RDMA component for controlling the RDMA transfer of file-block data between a memory address of the host and the at least one storage device.

28. The data server of claim 24, wherein the processor transfers the file-block data referenced by the received input/output operation details directly between the memory address of the host and the at least one storage device with minimum processing overhead by the operating system at the host.

29. The data server of claim 28, wherein the file-block data is transferred between the memory address at the host and the at least one storage device via an RDMA channel.

30. The data server of claim 29, wherein the RDMA channel is initiated by the host.

31. The data server of claim 29, wherein the RDMA channel is initiated by the data server.

32. A storage network, comprising:

a host including a first memory for use in processing an input/output request, the host being configured for issuing the input/output request;

a metadata server, coupled to the host, the metadata server including a second memory for storing metadata associated with file-block data, the metadata server further being configured for receiving the input/output request from the host and for providing input/output operation details for processing the input/output request; and

a data server, operatively coupled to the host and the metadata server, the data server including at least one storage device for storing the file-block data thereon, the data server being configured for receiving from the metadata server the input/output operation details for processing the input/output request, for establishing a direct connection to a memory address at the host identified by the input/output operation details from the metadata server, and for transferring file-block data referenced by the input/output operation details directly between the identified memory location at the host and the at least one storage device.

33. The storage network of claim 32, wherein input/output request passes between the host and the metadata server via an operations channel and the file-block data is transferred directly between the memory address of the host and the at least one storage device via an RDMA channel.

34. The storage network of claim 33, wherein the operations channel and the RDMA channel form an File-based RDMA Protocol (FRP) session.

35. The storage network of claim 32, wherein the data server is further configured to issue a completion notice to the metadata server upon completion of the transferring of the file-block data directly between the memory address of the host and the at least one storage device and to issue a file access response to the host indicating completion of a session associated with the input/output request.

36. The storage network of claim 32, wherein the metadata server further comprises an intelligent RDMA component for controlling the RDMA transfer of file-block data between the memory location at the host and the data server.

37. The storage network of claim 32, wherein the data server further comprises an intelligent RDMA component for controlling the RDMA transfer of file-block data between the memory location at the host and the data server.

38. A storage network, comprising:

a host being configured for issuing a file access request;

a metadata server, coupled to the host, the metadata server being configured for receiving the file access request from the host and for providing to the host data block information associated with the file access request; and

a data server, operatively coupled to the host, the data server including at least one storage device for storing file-block data thereon, the data server being configured for receiving commands identifying memory locations and for transferring file-block data directly between the identified memory location at the host and the at least one storage device.

39. The storage network of claim 38, wherein input/output request passes between the host and the metadata server via an operations channel and the file-block data is transferred directly between the memory address of the host and the at least one storage device via an RDMA channel.

40. The storage network of claim 39, wherein the operations channel and the RDMA channel form an File-based RDMA Protocol (FRP) session.

41. The storage network of claim 38, wherein the host further comprises an intelligent RDMA component for controlling the RDMA transfer of file-block data between a memory address of the host and the at least one storage device.

42. A program storage device readable by a computer, the medium tangibly embodying one or more programs of instructions executable by the computer to perform a method for making data access at data server connection speed in a distributed file serving architecture with metadata storage virtualization, the method comprising:

issuing an input/output request to a metadata server; and

transferring data associated with the input/output request directly between the memory and a data server.

43. The program storage device of claim 42, wherein the transferring is initiated by the host.

44. The program storage device of claim 42, wherein the transferring is initiated by the data server.

45. The program storage device of claim 42 further comprising receiving data block location information from the metadata server in response to issuing the input/output request.

46. The program storage device of claim 42, wherein the transferring further comprises issuing commands identifying memory locations at the host to the data server for enabling the direct memory access transfer of data associated with the input/output request between the memory and the data server.

47. The program storage device of claim 42 further comprising notifying the metadata server of completion of the input/output request.

48. The program storage device of claim 42 further comprising maintaining memory address mapping until an RDMA transport address of a data server is authenticated.

49. The program storage device of claim 42 further comprising controlling at the host the RDMA transfer of file-block data between the memory of the host and the storage device.

50. A program storage device readable by a computer, the medium tangibly embodying one or more programs of instructions executable by the computer to perform a method for enabling data access at data server connection speed in a distributed file serving architecture with metadata storage virtualization, the method comprising:

storing metadata associated with data on a data server; and

in response to an input/output request, issuing to the data server input/output operation details based on the metadata for transferring file-block data directly between memory at a host and the data server.

51. The program storage device of claim 50 further comprising returning metadata to a memory address at the host via an RDMA channel between the host and the metadata server.

52. The program storage device of claim 50 further comprising receiving a completion notice from the data server and issuing a file access response to the host indicating completion of a session associated with the input/output request.

53. The program storage device of claim 50 further comprising invalidating cached data at the host when metadata associated with the cached data changes at the metadata server.

54. The program storage device of claim 50 further comprising translating an input/output request from the host from an RDMA file access language to a meta-file system language of the metadata server to allow the input/output request to be mapped to raw data blocks on the data server

55. The program storage device of claim 50 further comprising controlling at the metadata server the RDMA transfer of file-block data between a memory address of the host and the data server.

56. A program storage device readable by a computer, the medium tangibly embodying one or more programs of instructions executable by the computer to perform a method for providing data access at data server connection speed in a distributed file serving architecture with metadata storage virtualization, the method comprising storing file-block data on a data server comprising at least one storage device;

receiving at a data server input/output operation details associated with an input/output request of a host from a metadata server;

establishing a direct connection from the data server to a memory address at the host identified by the received input/output operation details; and

transferring file-block data referenced by the received input/output operation details directly between the memory address of the host and the data server.

57. The program storage device of claim 56 further comprising providing a completion notice from the data server to the metadata server to signal completion of the file-data transfer between the memory address of the host and the storage device.

58. The program storage device of claim 56 further comprising presenting the at least one storage device as logical unit numbers and satisfying the input/output request by processing file-block data between the memory address of the host and the data server.

59. The program storage device of claim 56 further comprising controlling at the data server the RDMA transfer of file-block data via an RDMA channel between a memory address of the host and the data server.

60. The program storage device of claim 59, wherein the RDMA channel is initiated by the host.

61. The program storage device of claim 59, wherein the RDMA channel is initiated by the data server.

62. A program storage device readable by a computer, the medium tangibly embodying one or more programs of instructions executable by the computer to perform a method for providing a distributed file serving architecture with metadata storage virtualization and data access at data server connection speed, the method comprising:

issuing from a host a file access request including data target locations;

processing the file access request including data target locations;

issuing a remote direct memory access channel endpoint connection in response to the processing the file access request including data target locations; and

transferring file-block data associated with the file access request directly between a memory at the host and a data server.

63. The program storage device of claim 62, wherein the processing the file access request further comprises translating the data target locations to raw data blocks.

64. The program storage device of claim 63, wherein the translating the data target locations to raw data blocks is performed by the host.

65. The program storage device of claim 63 wherein the translating the data target locations to raw data blocks is performed by a metadata server.

66. The program storage device of claim 62, wherein the file access request comprises a request for the transfer of metadata, the method further comprising transferring the metadata from a metadata server to the memory at the host via an RDMA channel.

67. A host, comprising:

means for running an application thereon;

memory means for storing data therein; and

processing means configured for issuing an input/output request to a metadata server and for enabling direct memory access transfer of data associated with the input/output request directly between the memory and a data server.

68. A metadata server, comprising:

means for storing metadata associated with data on a data server; and

means for receiving an input/output request from a host and issuing to the data server input/output operation details based on the metadata for transferring file-block data directly between memory at the host and the data server.

69. A data server, comprising:

means for storing file-block data thereon; and

means, operatively coupled to the means for storing, for receiving from a metadata server input/output operation details associated with an input/output request of a host, for establishing a direct connection to a memory address at the host identified by the received input/output operation details, and for transferring file-block data referenced by the received input/output operation details directly between the memory address of the host and the at least one storage device.

70. A storage network, comprising:

hosting means including first memory means for use in processing an input/output request, the host being configured for issuing the input/output request;

metadata means, coupled to the hosting means, the metadata means including a second memory means for storing metadata associated with file-block data, the metadata means further being configured for receiving the input/output request from the hosting and for providing input/output operation details for processing the input/output request; and

a data means, operatively coupled to the hosting and the metadata means, the data means including at least one storage means for storing the file-block data thereon, the data means being configured for receiving from the metadata server the input/output operation details for processing the input/output request, for establishing a direct connection to a memory address at the hosting means identified by the input/output operation details from the metadata means, and for transferring file-block data referenced by the input/output operation details directly between the identified memory location at the hosting means and the data means.

71. A storage network, comprising:

hosting means for issuing a file access request;

metadata means, coupled to the hosting means, the metadata means receiving the file access request from the host and providing to the host data block information associated with the file access request; and

data means, operatively coupled to the hosting means, the data means including at least one storage means for storing file-block data thereon, the data means receiving commands identifying memory locations and transferring file-block data directly between the identified memory location at the hosting means and the data means.

72. A method for providing a distributed file serving architecture with metadata storage virtualization and data access at data server connection speed, comprising:

issuing from a host a file access request including data target locations;

processing the file access request including data target locations;

issuing a remote direct memory access channel endpoint connection in response to the processing the file access request including data target locations; and

transferring file-block data associated with the file access request directly between a memory at the host and a data server.