Distributed file system and method

• US 8,041,735 B1
• Filed: 11/01/2002
• Issued: 10/18/2011
• Est. Priority Date: 11/01/2002
• Status: Active Grant

• Current Assignee: Hitachi Vantara, LLC (Hitachi, Ltd.)
• Original Assignee: Bluearc UK Ltd. (Hitachi, Ltd.)
• Inventors: Lacapra, Francesco, Willis, Trevor E., Cattaneo, Fiorenzo, Benham, Simon L., Aston, Christopher J.
• Primary Examiner(s): Rones; Charles
• Assistant Examiner(s): Chojnacki; Mellissa M
• Application Number:
  US10/286,153
• Time in Patent Office: 3,273 Days
• Field of Search: 707/783
• US Class Current: 707/783
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: Y10S 707/947 Human resources

Abstract

A distributed file system and method distributes file system objects across multiple self-contained volumes, where each volume is owned by a unique file system node. Logical links are used to reference a file system object between volumes. Each file system node includes a relocation directory in which is maintained hard links to locally-stored file system objects that are referenced from another file system node using logical links. Various file system operations that involve multiple volumes are performed without having to place a write lock on more than one volume at a time. Various caching schemes allow the various file system nodes to cache file system object data and metadata.

Claims

1. A method for using a plurality of cluster nodes, each of which has access to a plurality of logical storage volumes, to provide access by clients, in communication with the nodes, to data on the volumes, wherein each volume is restricted to be owned by a single node, wherein only the owner of a volume has permission to write to the volume, the method comprising:

storing, by a first node, a file system object in a first volume owned by the first node; and

creating, in the first node and in a second node, a logical link between the first volume and a second volume owned by the second node, said logical link referring to the file system object in the first volume, wherein the process of creating a logical link between the first volume and second volume includes creating a logical identifier in a directory of the second volume that symbolically identifies the file system object which is referenced by the logical link and creating in a relocation directory of the first volume a hard link to the file system object and creating in the first volume a unique association between the hard link and the logical identifier from the second volume with respect to the file system object, wherein the logical link is represented by the hard link, the logical identifier, and the unique association between the hard link and the logical identifier forms; and

wherein the hard link increases a reference count associated with the file system object in the first volume independent of the disposition of the logical identifier in the second volume such that the first volume is self-contained while allowing the file system object to be accessed symbolically from the second volume using the logical identifier without a physical cross-volume link;

a physical namespace invisible to the clients of the plurality of nodes, the physical namespace implemented through the plurality of volumes, each volume having an internal hierarchy that connects file system objects through hard links including the hard link in the relocation directory of the first volume; and

a logical namespace visible to the clients of the plurality of nodes that spans the entire file system across volumes and is made of file system objects connected via hard links and logical links such that the difference between hard links and logical links is hidden from the clients.

2. The method according to claim 1, further comprising:

storing the logical identifier in the relocation directory, in the first volume, that lists logical identifiers for all file system objects in the first volume that are logically referenced from volumes other than the first volume.

3. The method according to claim 1, wherein storage of the file system object and creation of the logical link are performed while placing a write-lock on no more than one volume at a time, so that multi-volume locking is avoided.

4. The method according to claim 3, wherein the first node assumes the role of write-lock manager for the first and second volumes in the course of storing the file system object and creating the logical link.

5. The method according to claim 2, wherein the relocation directory has a hierarchical structure.

6. The method according to claim 1, wherein creating the logical link includes assigning an identifier that is unique, at least in the first volume, as the logical identifier.

7. The method according to claim 6, wherein assigning a unique identifier includes assigning a number that is based on the count of previously assigned logical identifiers for file system objects in the first volume.

8. The method according to claim 1, further comprising:

permitting reads to a given storage volume to be made by any of the nodes.

9. The method according to claim 8, wherein a volume cache is associated with each of the volumes, and a node cache is associated with each of the nodes, the method further comprising:

handling a request, to read a sector in a selected volume, that has been received by an operative node that owns a volume distinct from the selected volume by causing the operative node to issue a first read request, for the sector, with a read-only lock to the node that is the owner of the selected volume and a second read request, for the sector, to the selected volume;

having the node that is the owner of the selected volume determine whether the sector is in the node cache associated with the node and

if not, granting the read-only lock and letting the selected volume satisfy the request;

if so, and if the sector is not write-locked, granting the read-only lock and sending the sector content from cache, of the node owning the selected volume, to the operative node and aborting the communication of the selected volume with respect to the request;

if so, and if the sector is write-locked, stalling affirmative action on the request until the read-only lock can be granted and then resuming processing as for when the sector is not write-locked.

10. The method according to claim 9, wherein granting the read-only lock includes recording the identity of the operative node and of the sector, so that caching states may be tracked.

11. The method according to claim 2, further comprising:

storing in the directory of the second volume a logical reference to the logical identifier stored in the relocation directory in the first volume.

12. The method according to claim 11, wherein the logical reference comprises:

the logical identifier; and

a numeric value that encodes a numeric string representing a name of the referenced file system object in the relocation directory of the first volume.

13. The method according to claim 11, further comprising:

restricting the logical reference to be the one and only logical reference to the logical identifier stored in the relocation directory in the first volume.

14. The method according to claim 4, wherein the first node assumes the role of write-lock manager for the first and second volumes in a semi-static fashion from the time the first node joins the plurality of cluster nodes until the first node leaves the plurality of cluster nodes.

15. The method according to claim 11, further comprising:

deleting the logical reference from the second volume; and

deleting the logical identifier from the relocation directory of the first volume.

16. The method according to claim 11, further comprising:

deleting the file system object from the first volume; and

deleting the logical reference from the second volume.

17. The method according to claim 11, further comprising:

determining that the logical reference references a non-existent file system object; and

deleting the logical reference from the second volume.

18. The method according to claim 17, wherein determining that the logical reference references a non-existent file system object comprises:

running a scavenger process in the second, the scavenger process examining the logical reference from time to time to determine whether the file system object it references exists.

19. The method according to claim 11, further comprising:

determining that the file system object in the first volume is no longer referenced by any logical references in the other volumes; and

deleting the file system object from the first volume.

20. The method according to claim 19, wherein determining that the file system object in the first volume is no longer referenced by any logical references in the other volumes comprises:

running a scavenger process in the first node, the scavenger process examining the relocation directory from time to time to determine whether the file system object is referenced by at least one logical reference in another volume.

21. The method according to claim 11, further comprising:

mapping a “

. . . ”

directory entry in the first volume to the directory containing the logical reference in the second volume.

22. The method according to claim 21, wherein mapping the “

. . . ”

directory entry in the first volume to the directory containing the logical reference in the second volume comprises;

storing in the relocation directory of the first volume a reference to the directory containing the logical reference in the second volume.

23. The method according to claim 22, wherein the reference to the directory containing the logical reference in the second volume comprises:

a volume identifier associated with the second volume; and

a node identifier associated with the directory containing the logical reference in the second volume.

24. The method according to claim 22, wherein the reference to the directory containing the logical reference in the second volume comprises:

a variable-length pathname to the directory containing the logical reference in the second volume.

25. The method according to claim 21, wherein mapping the “

. . . ”

directory entry in the first volume to the directory containing the logical reference in the second volume comprises;

storing in the first volume a separate file containing a reference to the directory containing the logical reference in the second volume.

26. The method according to claim 11, further comprising:

logically relocating the file system object from the first volume to a third volume owned by a third node by deleting the logical identifier from the relocation directory of the first volume;

storing a new logical identifier in the relocation directory of the first volume; and

storing in a directory of the third volume a new logical reference to the new logical identifier stored in the relocation directory in the first volume.

27. The method according to claim 26, further comprising:

deleting the logical reference from the directory of the second volume.

28. The method according to claim 11, further comprising:

referencing the file system object by a third volume by storing a new logical identifier in the relocation directory of the first volume; and

storing in a directory of the third volume a new logical reference to the new logical identifier stored in the relocation directory in the first volume.

29. The method according to claim 1, further comprising:

receiving by the second node a request to write to the file system object;

forwarding the request by the second node to the first node; and

writing to the file system object by the first node.

30. The method according to claim 1, further comprising:

receiving by the first node a request to write to the file system object; and

writing to the file system object by the first node.

31. The method according to claim 1, further comprising:

modifying the file system object by a third node by causing the third node to store a modified file system object in a third volume owned by the third node; and

creating a logical link between the second volume and the third volume, such logical link providing in the second volume a logical identifier for accessing the file system object in the third volume.

32. The method according to claim 1, wherein creating the logical link between the first volume and the second volume comprises:

sending by the first node to the second node a request to create the logical link; and

creating the logical link by the second node upon receiving the request.

33. The method according to claim 1, wherein storing the file system object in the first volume comprises:

receiving by the first node a request to create the file system object in the second node.

34. The method according to claim 1, wherein storing the file system object in the first volume comprises:

receiving by the second node a request to create the file system object in the second node; and

determining by the second node that the file system object should be created in the first node.

35. The method according to claim 1, wherein each volume is associated with a unique volume identifier, and wherein the method further comprises:

causing the second node to create a file handle for the file system object, the file handle including the volume identifier associated with the first volume.

36. The method according to claim 1, further comprising:

receiving a request to delete the file system object by a node other than the second node; and

directing the request to the second node; and

deleting the file system object by the second node.

37. The method according to claim 1, further comprising:

receiving a request to delete the file system object by the second node;

causing the logical link to be removed from the second node;

sending by the second node to the first node a request to delete the file system object; and

deleting the file system object by the first node.

38. The method according to claim 1, wherein the file system object comprises one of:

a file; and

a directory.

39. The method according to claim 26, wherein said logically relocating the file system object from the first volume to a third volume owned by a third node is performed automatically under predetermined conditions.

40. The method according to claim 11, further comprising:

relocating the file system object from the first volume to a third volume owned by a third node by

causing the file system object to be copied from the first volume to the third volume;

removing the logical identifier from the relocation directory of the first volume;

removing from the second volume the logical reference to the logical identifier stored in the relocation directory of the first volume;

creating a new logical link between the second volume and the third volume, such new logical link providing in the second volume a new logical identifier for accessing the file system object in the third volume; and

deleting the file system object from the first volume.

41. The method according to claim 40, further comprising:

storing the new logical identifier in a relocation directory of the third volume.

42. The method according to claim 41, further comprising:

storing in the directory of the second volume a new logical reference to the new logical identifier stored in the relocation directory in the third volume.

43. The method according to claim 40, wherein causing the file system object to be copied from the first volume to the third volume comprises:

copying the file system object from the first volume to the third volume using a lazy copy technique.

44. The method according to claim 11, further comprising:

relocating the file system object from the first volume to the second volume by causing the file system object to be copied from the first volume to the second volume;

removing the logical identifier from the relocation directory of the first volume; and

removing from the second volume the logical reference to the logical identifier stored in the relocation directory of the first volume.

45. The method according to claim 1, further comprising:

maintaining a copy of data relating to the file system object by at least one node other than the first node;

modifying the data relating to the file system object by the first node; and

causing the at least one other node to invalidate the data relating to the file system object.

46. The method according to claim 45, further comprising:

causing the at least one other node to reacquire the data relating to the file system object from the first volume.

47. The method according to claim 45, wherein causing the at least one other node to invalidate the copy of the portion of the file system object comprises:

maintaining by the first node a list of other nodes having a copy of data relating to the file system object; and

sending to each other node an indication that the data relating to the file system object was modified.

48. The method according to claim 45, wherein the data relating to the file system object comprises metadata, and wherein maintaining the copy of data relating to the file system object by at least one node other than the first node comprises:

maintaining, by the at least one other node, a metadata cache for storing the metadata, the metadata cache operating independently of any other cache used for storing a copy of file system object data.

49. The method according to claim 48, wherein maintaining the copy of data relating to the file system object by at least one node other than the first node further comprises:

requiring the at least one other node to obtain a read lock from the first node before reading metadata relating to the file system object from the first node.

50. The method according to claim 45, wherein the data relating to the file system object comprises file system object data and metadata, and wherein maintaining the copy of data relating to the file system object by at least one node other than the first node comprises:

maintaining a data cache for storing file system object data;

maintaining a metadata cache for storing metadata; and

mapping each datum in the data cache to corresponding metadata in the metadata cache.

51. The method according to claim 26, wherein the file system object is associated with a first name in the first volume, and wherein logically relocating the file system object from the first volume to the third volume further comprises:

associating the file system object with a second name in the third volume.

52. The method according to claim 1, further comprising:

designating one node as a coordinator for a multi-volume operation affecting the file system object, said coordinator being the owner of one of the volumes involved in the multi-volume operation.

53. The method according to claim 1, further comprising:

maintaining for each volume a local directory for temporarily storing references to file system objects and logical links that have been removed from a volume and replaced with a logical link to the file system object or with another logical link; and

using a local directory to recover from a failure during a multi-volume operation that affects the corresponding volume.