Method and apparatus for data storage using striping

• US 20070271350A1
• Filed: 07/31/2007
• Published: 11/22/2007
• Est. Priority Date: 12/02/2003
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Spinnaker Networks, LLC (NetApp, Inc.)
• Inventors: Kazar, Michael, Sanzi, Richard
• Granted Patent: US 7,454,567 B2
• Time in Patent Office: Days
• US Class Current: 709/214
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

An apparatus for data storage includes a cluster of NFS servers. Each server has network ports for incoming file system requests and cluster traffic between servers. The apparatus includes a plurality of storage arrays in communication with the servers. The servers utilize a striped file system for storing data. A method for data storage. A method for establishing storage for a file. A method for removing a file from storage. A method for reading data in a file. A method for writing data in a file.

Claims

1. An apparatus for data storage comprising:

a cluster of NFS (network file system) servers, each server having network ports for incoming file system requests and cluster traffic between servers, each server has a network element and a disk element; and

a plurality of storage arrays in communication with the servers, the servers utilizing a striped file system for storing data for providing bandwidth to multiple disk elements, where the striped file system comprises a set of striped VFSes (virtual file systems) distributed among a number of disk elements of the cluster of servers, with one VFS of the set of striped VFSes per disk element, wherein a data file is striped among all the VFSes of the set of striped VFSes with different strips of the file'"'"'s data in different VFSes in the set of striped VFSes.

2. An apparatus as described in claim 1 wherein each disk element has a virtual file system with the virtual file system of each disk element together forming a striped VFS.

3. An apparatus as described in claim 2 wherein the different VFSes in the set of striped VFSes have a same vnode number.

4. An apparatus as described in claim 3 wherein a strip N of vnode B in the set of striped VFSes is stored on an I-th server where I=(B+N) and STRIPE_WIDTH and STRIPE_WIDTH is a number of strips in a striped VFS across all storage arrays holding the striped VFS.

5. An apparatus as described in claim 4 wherein one server of the cluster of servers is a meta-data server for one of the striped VFSes.

6. A method as described in claim 5 wherein all files of the one of the striped VFSes are represented at the meta-data server.

7. An apparatus as described in claim 6 wherein all disk elements for a virtual file system act as meta-data servers.

8. An apparatus as described in claim 7 wherein a file has attributes and each server for each file maintains a caching element that stores a last known version of the file attributes and ranges of modification time and change time values for assignment to write operation results.

9. An apparatus as described in claim 8 wherein each disk element which is not the meta-data server for a virtual file system is an input output secondary.

10. An apparatus as described in claim 9 wherein ranges of file modification times or file change times are reserved from the meta-data server by the input output secondary.

11. An apparatus as described in claim 10 wherein the modification and change times in the ranges obtained from the meta-data server are issued to operations already queued at the input output secondary.

12. An apparatus as described in claim 11 wherein modification and change times in the ranges obtained from the meta-data server are issued to operations received during a window of time after the ranges are reserved from the meta-data server by the input output secondary.

13. An apparatus as described in claim 12 wherein operations affecting all stripes of a file begin executions first at the meta-data server for a file, and then execute at all input output secondaries, such that operations at the input output secondaries wait only for already executing operations that have already finished their communication with the meta-data server.

14. An apparatus as described in claim 13 wherein operations follow one of at least two locking models, the first of which is to synchronize first with the meta-data server, then begin core execution by synchronizing with other operations executing at the input output secondary, and the second of which is to first synchronize at the meta-data server, and then to synchronize with operations at one or more input output secondaries that have begun core execution at the input output secondaries.

15. An apparatus as described in claim 14 wherein the cluster network is connected in a star topology.

16. An apparatus as described in claim 15 wherein the cluster network is a switched Ethernet.

17. A method for data storage comprising the steps of:

creating a file across a plurality of NFS (Network File System) servers, each server having a network element and a disk element;

writing data into the file as strips of the data in the servers, the strips together forming a stripe for providing bandwidth to multiple disk elements, where the striped file system comprises a set of striped VFSes (virtual file systems) distributed among a number of disk elements of the cluster of servers, with one VFS of the set of striped VFSes per disk element;

reading the strips of the data from the servers; and

deleting the strips from the servers.

18. A method as described in claim 17 including the step of storing in a caching element at each input output secondary for each active file at a meta-data server a last known version of attributes of the file which are good for a dallying period.

19. A method as described in claim 18 including the step of storing ranges of modification time and change time values in the caching element for assignment to write operations.

20. A method as described in claim 18 including the step of identifying a disk element for a virtual file system of an NFS server as a meta-data server and disk elements for the NFS servers which are not identified as the meta-data server as input output secondaries.

21. A method as described in claim 20 including the step of storing in a caching element at each input output secondary for each active file at a meta-data server a last known version of attributes of the file which are good for a dallying period.

22. A method as described in claim 21 including the step of storing ranges of modification time and change time values in the caching element for assignment to write operations.

23. A method as described in claim 22 including the step of making a status request by the caching element to the meta-data server to obtain a file'"'"'s current attributes.

24. A method as described in claim 23 wherein the making a status request step includes the step of obtaining modification time and change time ranges from the meta-data server.

25. A method as described in claim 24 including the step of queuing file read and file write requests at the input output secondary until the file read and file write requests are admitted by the cache element and complete execution.

26. A method as described in claim 25 including the step of tracking by the cache element of the file read and file write requests executing for the file and the ranges that are being read or written.

27. A method as described in claim 26 including the step of requesting the cache element move out of invalid node to read mode when a read operation must be executed.

28. A method as described in claim 27 including the step of checking a byte range affected by a file read request to ensure it does not overlap a byte range of any file write requests previously admitted and currently executing.

29. A method as described in claim 28 including the step of requesting, in response to a file write request that the cache element move into a write mode.

30. A method as described in claim 29 including the step of checking with the cache element the byte range affected by the file write request for overlap with any admitted and still executing file read or file write requests.

31. A method as described in claim 30 including the step, when executing a write request, of allocating a modification time and change time pair from the range of modification times and change times stored in the cache element.

32. A method as described in claim 31 including the step of checking the head of a queue of pending file read and file write requests to see if a head request can be admitted by the caching element after either a file read or file write request is completed.

33. A method as described in claim 32 including the steps of detecting by the cache element that a file length must be updated in response to a file write request, moving the cache element into exclusive mode;

and making a file write status call to the meta-data server to update length attributes of the file.

34. A method as described in claim 18 including the step of making a status request by a caching element to the meta-data server to obtain a file'"'"'s current attributes.

35. A method as described in claim 34 wherein the making a status request step includes the step of obtaining modification time and change time ranges from the meta-data server.

36. A method as described in claim 18 including the step of requesting a cache element move out of invalid node to read mode when a read operation must be executed.

37. A method as described in claim 18 including the step of requesting, in response to a file write request that a cache element move into a write mode.

38. A method as described in claim 18 including the steps of detecting by a cache element that a file length must be updated in response to a file write request, moving the cache element into exclusive mode;

and making a file write status call to a meta-data server to update length attributes of the file.

39. A method as described in claim 18 including the step of identifying a disk element for a virtual file system of an NFS (network file system) server as a meta-data server and disk elements for the NFS severs which are not identified as the meta-data server as input output secondaries.

40. A method as described in claim 17 wherein the writing step includes the step of writing the strips in different VFSes in the set of striped VFSes having a same vnode number.

41. A method as described in claim 40 wherein the writing step includes the step of storing a strip N of vnode B in the set of striped VFSes on an I-th server where I=(B+N) and STRIPE_WIDTH and STRIPE_WIDTH is a number of strips in a striped VFS across all storage arrays holding the striped VFS.

42. A method for reading data in a file comprising the steps of:

receiving an NFS (network file system) read request for data in the file at a network element;

determining by the network element which VFS stores at least one strip containing the data;

sending a file read request from the network element to at least one disk element of a plurality of servers storing a strip of the data;

obtaining current attributes associated with the file by each disk element;

reading the strips of the file from each disk element having the strips; and

generating a response in regard to the file read request.

43. A method for writing data in a file comprising the steps of:

receiving an NFS (network file system) write request for a file at a network element;

determining by the network element which VFS (virtual file system) is associated with the file;

sending a file write request from the network element to at least one disk element of a plurality of servers having a stripe of the VFS;

acquiring current attributes associated with the file; and

writing a predetermined number of bytes of the data into each VFS strip in succession until all of the data is written into the file.

44. An apparatus for data storage comprising:

a cluster of network-accessed file level servers, each server having network ports for incoming file system requests and cluster traffic between servers, each server has a network element and a disk element; and

a plurality of storage arrays in communication with the servers, the servers utilizing a striped file system for storing data, and where one disk element for a given file system acts as a meta-data server that maintains modification and change time attributes for each file, and where each server for each file maintains a caching element that stores a last known version of the file attributes and ranges of modification time and change time values for assignment to write operation results.

45. An apparatus as described in claim 44 wherein each disk element which is not the meta-data server is an input output secondary.

46. An apparatus as described in claim 45 wherein ranges of file modification times or file change times are reserved from the meta-data server by the input output secondary.

47. An apparatus as described in claim 46 wherein the modification and change times in the ranges obtained from the meta-data server are issued to operations already queued at the input output secondary.

48. An apparatus as described in claim 47 wherein modification and change times in the ranges obtained from the meta-data server are issued to operations received during a window of time after the ranges are reserved from the meta-data server by the input output secondary.

49. An apparatus as described in claim 48 wherein operations affecting all stripes of a file begin executions first at the meta-data server for a file, and then execute at all input output secondaries, such that operations at the input output secondaries wait only for already executing operations that have already finished their communication with the meta-data server.

50. An apparatus as described in claim 49 wherein operations follow one of at least two locking models, the first of which is to synchronize first with the meta-data server, then begin core execution by synchronizing with other operations executing at the input output secondary, and the second of which is to first synchronize at the meta-data server, and then to synchronize with operations at one or more input output secondaries that have begun core execution at the input output secondaries.

51. An apparatus as described in claim 50 wherein the cluster network is connected in a star topology.

52. An apparatus as described in claim 51 wherein the cluster network is a switched Ethernet.

53. An apparatus as described in claim 52 wherein the servers are NFS servers.

54. A method for data storage comprising the steps of:

receiving incoming file system requests at network ports of a cluster of network-accessed file level servers, and cluster traffic between servers at the ports, each server has a network element and a disk element; and

storing data utilizing a striped file system in a plurality of storage arrays in communication with the servers, where one disk element for a given file system acts as a meta-data server that maintains modification and change time attributes for each file, and where each server for each file maintains a caching element that stores a last known version of the file attributes and ranges of modification time and change time values for assignment to write operation results.

55. A method as described in claim 54 wherein each disk element which is not the meta-data server is an input output secondary, and wherein the storing step includes the step of reserving ranges of file modification times or file change times from the meta-data server by the input output secondary.

56. A method as described in claim 55 wherein the storing step includes the step of issuing the modification and change times in the ranges obtained from the meta-data server to operations already queued at the input output secondary.

57. A method as described in claim 56 wherein the issuing step includes issuing the modification and change times in the ranges obtained from the meta-data server to operations received during a window of time after the ranges are reserved from the meta-data server by the input output secondary.

58. A method as described in claim 57 including the step of executing operations affecting all stripes of a file beginning first at the meta-data server for a file, and then executing at all input output secondaries, such that operations at the input output secondaries wait only for already executing operations that have already finished their communication with the meta-data server.

59. A method as described in claim 58 including the step of executing operations following one of at least two locking models, the first of which is to synchronize first with the meta-data server, then begin core execution by synchronizing with other operations executing at the input output secondary, and the second of which is to first synchronize at the meta-data server, and then to synchronize with operations at one or more input output secondaries that have begun core execution at the input output secondaries.

60. A method as described in claim 59 wherein the servers are NFS servers.