Method and apparatus for improving file access performance of distributed storage system

• US 8,086,634 B2
• Filed: 10/07/2008
• Issued: 12/27/2011
• Est. Priority Date: 10/07/2008
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Mimatsu, Yasuyuki
• Primary Examiner(s): Lee, Wilson
• Application Number:
  US12/246,520
• Publication Number:
  US 20100088335A1
• Time in Patent Office: 1,176 Days
• Field of Search: 707713-722, 707780-788
• US Class Current: 707/782
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/563 Data redirection of data network streams

Abstract

Embodiments of the invention provide methods and apparatus for improving the performance of file transfer to a client from a distributed storage system which provides single name space to clients. In one embodiment, a system for providing access to files in a distributed storage system comprises a plurality of storage nodes and at least one computer device connected via a network. Each storage node is configured, upon receiving a file access request for a file from one of the at least one computer device as a receiver storage node, to determine whether or not to inform the computer device making the file access request to redirect the file access request to an owner storage node of the file according to a preset policy. The preset policy defines conditions for whether to redirect the file access request based on at least one of file type or file size of the file.

Claims

1. A system for providing access to files in a distributed storage system, the system comprising:

a plurality of storage nodes and at least one computer device connected via a network;

wherein each storage node is configured, upon receiving a file access request for a file from one of the at least one computer device as a receiver storage node, to determine whether or not to inform the computer device making the file access request to redirect the file access request to an owner storage node of the file according to a preset policy;

wherein the preset policy defines conditions for whether to redirect the file access request based on at least one of file type or file size of the file; and

wherein the receiver storage node is configured, in not redirecting the file access request to the owner storage node, to read the file from the owner storage node and send the read file to the computer device making the file access request.

2. A system according to claim 1, wherein the receiver storage node is configured, in redirecting the file access request to the owner storage node, to send a prefetch request to the owner storage node specifying the file to be prefetched in cache.

3. A system according to claim 1, wherein the receiver storage node determines whether or not to inform the computer device making the file access request to redirect the file access request to the owner storage node by looking up a policy table which specifies the conditions for whether to redirect the file access request based on at least one of file type or file size of the file.

4. A system according to claim 3, wherein the conditions are based on both the file type and the file size of the file, and further based on at least one of the computer device requesting the file, an external path of the file, or at least one request parameter specified in the file access request.

5. A system according to claim 3, wherein the policy table is stored in each of the storage nodes.

6. A system according to claim 1, wherein the receiver storage node determines whether or not to inform the computer device making the file access request to redirect the file access request to the owner storage node by looking up a redirection flag for the file, the redirection flag specifying, for each file stored in the storage nodes, whether a file access request for the file associated therewith is to be redirected.

7. A system according to claim 6, wherein the redirection flag is set by performing the following at the time when a file is written into one of the storage nodes:

determining whether or not to redirect a file access request to an owner storage node of the file according to the preset policy based on at least one of the file type or the file size of the file; and

setting the redirection flag to redirect or not to redirect according to the determining step.

8. A system according to claim 1,

wherein the file of the file access request is divided into multiple portions and stored in a plurality of the storage nodes, and

wherein each storage node is configured, as the receiver storage node, to determine whether or not to inform the computer device making the file access request to redirect the file access request to owner storage nodes of the multiple portions of the file according to the preset policy.

9. A system according to claim 8, wherein the receiver storage node is configured, in redirecting the file access request to the owner storage nodes of the multiple portions of the file, to send a prefetch request to each owner storage node, other than the receiver storage node, specifying the portion of the file to be prefetched in cache.

10. A system according to claim 1, wherein each storage node is configured, upon receiving a migration request to migrate a file to a destination storage node, to extract an external path of the file to be migrated, read the file from an origination storage node, send an internal write request which includes the file and the external path of the file to the destination storage node, delete the migrated file from the origination storage node, and update a location record of the migrated file within the distributed storage system.

11. A system according to claim 1, wherein the preset policy defines conditions for whether to redirect the file access request based on at least one of the file type of the file, the file size of the file, the computer device requesting the file, an external path of the file, or at least one request parameter specified in the file access request.

20. A system according to claim 1, wherein the receiver storage node determines whether or not to inform the computer device making the file access request to redirect the file access request to the owner storage node by looking up a redirection flag for the file, the redirection flag specifying, for each file stored in the storage nodes, whether a file access request for the file associated therewith is to be redirected.

12. In a system having a distributed storage system and at least one computer device connected via a network, a method of providing access to files in a plurality of storage nodes in the distributed storage system, the method comprising:

upon receiving a file access request for a file from one of the at least one computer device, determining whether or not to inform the computer device making the file access request to redirect the file access request to an owner storage node of the file according to a preset policy;

wherein the preset policy defines conditions for whether to redirect the file access request based on at least one of file type or file size of the file; and

wherein determining whether or not to inform the computer device making the file access request to redirect the file access request to the owner storage node comprises looking up a redirection flag for the file, the redirection flag specifying, for each file stored in the storage nodes, whether a file access request for the file associated therewith is to be redirected.

13. A method according to claim 12, further comprising, if it is determined to redirect the file access request to the owner storage node, sending a prefetch request to the owner storage node specifying the file to be prefetched in cache.

14. A method according to claim 12, wherein determining whether or not to inform the computer device making the file access request to redirect the file access request to the owner storage node comprises looking up a policy table which specifies the conditions for whether to redirect the file access request based on at least one of file type or file size of the file.

15. A method according to claim 12, further comprising setting the redirection flag by performing the following at the time when a file is written into one of the storage nodes:

determining whether or not to redirect a file access request to an owner storage node of the file according to the preset policy based on at least one of the file type or the file size of the file; and

setting the redirection flag to redirect or not to redirect according to the determining step.

16. A method according to claim 12,

wherein the file of the file access request is divided into multiple portions and stored in a plurality of the storage nodes, and

wherein the determining step includes determining whether or not to inform the computer device making the file access request to redirect the file access request to owner storage nodes of the multiple portions of the file according to the preset policy.

17. A method according to claim 12, further comprising, upon receiving a migration request to migrate a file to a destination storage node:

extracting an external path of the file to be migrated;

reading the file from an origination storage node;

sending an internal write request which includes the file and the external path of the file to the destination storage node;

deleting the migrated file from the origination storage node; and

updating a location record of the migrated file within the distributed storage system.

18. A method according to claim 12, wherein the preset policy defines conditions for whether to redirect the file access request based on at least one of the file type of the file, the file size of the file, the computer device requesting the file, an external path of the file, or at least one request parameter specified in the file access request.

19. A system for providing access to files in a distributed storage system, the system comprising:

a plurality of storage nodes and at least one computer device connected via a network;

wherein each storage node is configured, upon receiving a file access request for a file from one of the at least one computer device as a receiver storage node, to determine whether or not to inform the computer device making the file access request to redirect the file access request to an owner storage node of the file according to a preset policy;

wherein the preset policy defines conditions for whether to redirect the file access request based on at least one of file type or file size of the file; and

wherein each storage node is configured, upon receiving a migration request to migrate a file to a destination storage node, to extract an external path of the file to be migrated, read the file from an origination storage node, send an internal write request which includes the file and the external path of the file to the destination storage node, delete the migrated file from the origination storage node, and update a location record of the migrated file within the distributed storage system.