Scalable system for partitioning and accessing metadata over multiple servers

• US 8,433,735 B2
• Filed: 12/20/2010
• Issued: 04/30/2013
• Est. Priority Date: 01/20/2005
• Status: Expired due to Fees

• Current Assignee: F5 Networks, Inc. (F5 Incorporated)
• Original Assignee: F5 Networks, Inc. (F5 Incorporated)
• Inventors: Lacapra, Francesco
• Primary Examiner(s): Hu, Jensen
• Application Number:
  US12/972,825
• Publication Number:
  US 20110087696A1
• Time in Patent Office: 862 Days
• Field of Search: 707/999.002, 707/821, 707/828
• US Class Current: 707/828
• CPC Class Codes: H04L 61/4552 Lookup mechanisms between a plurality of directories; Synchronisation of directories, e.g. metadirectories

Abstract

In an aggregated file system, metadata is partitioned into multiple metadata volumes. On receipt of a file processing request, a file switch examines its mount entry cache to identify a target metadata volume that hosts the metadata of the requested file. The identification begins with mount entries at a root volume and continues recursively by examining a portion of the absolute pathname of the file until the target metadata volume is identified. Finally, the file switch forwards the request to a metadata server managing the target metadata volume. Since the identification process is carried out completely within the file switch, there is no need for multiple expensive network accesses to different metadata servers.

Claims

1. A method of handling a file processing request in an aggregated file system, comprising:

receiving a request with respect to a file, the request identifying an absolute pathname of the file in the aggregated file system;

identifying a target metadata volume that hosts metadata of the file among a plurality of metadata volumes in accordance with the request and one or more mount entries each associated with a respective one of the plurality of metadata volumes, including recursively;

identifying a mount entry when associated with a current metadata volume, wherein the mount entry identifies a relative pathname matching a respective portion of the absolute pathname, the identified relative pathname further comprising pathname sub-components of sizes ranging from one pathname component to a size of the identified relative pathname; and

selecting at least one of the pathname sub-components using the identified mount entry based upon a number indicated in a list entry pointed to by the identified mount entry, until no mount entry associated with a current metadata volume matches a respective portion of the absolute pathname, to set the current metadata volume; and

identifying the current metadata volume as the target metadata volume and sending the request to a metadata server hosting the identified target metadata volume in response to the request to a file switch for assembling the file.

2. The method of claim 1, wherein the recursively identifying includes producing a residual pathname by removing from the absolute pathname the relative pathname associated with each identified mount entry.

3. The method of claim 1, wherein the absolute pathname of the file is unique in the aggregated file system.

4. The method of claim 1, wherein at least a subset of the plurality of metadata volumes includes one or more mount entries, each mount entry identifying a respective distinct child metadata volume and an associated relative pathname.

5. The method of claim 4, wherein at least one metadata volume in the subset of the plurality of metadata volumes includes a reverse mapper, the reverse mapper identifying a parent metadata volume.

6. The method of claim 1, wherein:

the plurality of metadata volumes include at least one root metadata volume, one or more intermediate metadata volumes and one or more leaf metadata volumes, and wherein the root metadata volume is a metadata volume that has no parent metadata volume, the leaf metadata volume is a metadata volume that has no child metadata volume and the intermediate metadata volume is a metadata volume that has both at least one parent and at least one child metadata volumes;

the target metadata volume is one of the root metadata volumes when none of the child metadata volumes directly referenced by at least one of the root metadata volumes has an associated relative pathname matching a portion of the absolute pathname;

the target metadata volume is one of the intermediate metadata volumes when at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname and none of the child metadata volumes referenced by at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname; and

the target metadata volume is one of the leaf metadata volumes referenced by at least one of the root metadata volumes using one or more mount entries when at least one of the leaf metadata volumes has an associated relative pathname matching a portion of the absolute pathname.

7. The method of claim 1, further comprising:

forming an ordered pair comprising the identified target metadata volume and a residual pathname, if any, based upon said identifying of the target metadata volume.

8. The method of claim 1 further comprising:

accessing a plurality of mount entries, wherein each of the plurality of mount entries is associated with a respective one of the plurality of metadata volumes as a parent metadata volume and identifies a respective distinct child metadata volume and the relative pathname associated with the child metadata volume;

resetting the current metadata volume to the child metadata volume identified by the identified mount entry, until no mount entry associated with the current metadata volume is found that matches a respective portion of the absolute pathname, the target metadata volume comprising the last child metadata volume so identified; and

returning information identifying the target metadata volume, hosted metadata and its associated residual pathname in response to the request to a file switch for assembling the file in response to the request.

9. The method of claim 1, wherein the identified relative pathname has a size based upon a maximum number of pathname components of the relative pathname that can be handled by the identified mount entry.

10. An apparatus for handling a file processing request in an aggregated file system, comprising:

at least one processor; and

memory coupled to the at least one processor which is configured to execute program instructions stored in the memory comprising;

receiving a request with respect to a specified file, the request including an absolute pathname of the specified file in the aggregated file system;

identifying a target metadata volume that hosts metadata of the file among a plurality of metadata volumes in accordance with the request and one or more mount entries each associated with a respective one of the plurality of metadata volumes, including recursively;

identifying a mount entry when associated with a current metadata volume, wherein the mount entry identifies a relative pathname matching a respective portion of the absolute pathname, the identified relative pathname further comprising pathname sub-components of sizes ranging from one pathname component to a size of the identified relative pathname; and

selecting at least one of the pathname sub-components using the identified mount entry based upon a number indicated in a list entry pointed to by the identified mount entry, until no mount entry associated with a current metadata volume matches a respective portion of the absolute pathname, to set the current metadata volume; and

identifying the current metadata volume as the target metadata volume and sending the request to a metadata server hosting the identified target metadata volume in response to the request to a file switch for assembling the file.

11. The apparatus of claim 10, wherein the recursively identifying includes producing a residual pathname by removing from the absolute pathname the relative pathname associated with each identified mount entry.

12. The apparatus of claim 10, wherein the absolute pathname of the file is unique in the aggregated file system.

13. The apparatus of claim 10, wherein at least a subset of the plurality of metadata volumes includes one or more mount entries, each mount entry identifying a respective distinct child metadata volume and an associated relative pathname.

14. The apparatus of claim 13, wherein at least one metadata volume in the subset of the plurality of metadata volumes includes a reverse mapper, the reverse mapper identifying a parent metadata volume.

15. The apparatus of claim 10, wherein:

the plurality of metadata volumes include at least one root metadata volume, one or more intermediate metadata volumes and one or more leaf metadata volumes, and wherein the root metadata volume is a metadata volume that has no parent metadata volume, the leaf metadata volume is a metadata volume that has no child metadata volume and the intermediate metadata volume is a metadata volume that has both at least one parent and at least one child metadata volumes;

the target metadata volume is one of the root metadata volumes when none of the child metadata volumes directly referenced by at least one of the root metadata volumes has an associated relative pathname matching a portion of the absolute pathname;

the target metadata volume is one of the intermediate metadata volumes when at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname and none of the child metadata volumes referenced by at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname; and

the target metadata volume is one of the leaf metadata volumes referenced by at least one of the root metadata volumes using one or more mount entries when at least one of the leaf metadata volumes has an associated relative pathname matching a portion of the absolute pathname.

16. The apparatus of claim 10 wherein the at least one processor is further configured to execute programmed instructions stored in the memory further comprising forming an ordered pair comprising the identified target metadata volume and a residual pathname, if any, based upon said identifying of the target metadata volume.

17. The apparatus of claim 10 wherein the at least one processor is further configured to execute programmed instructions stored in the memory further comprising:

accessing a plurality of mount entries, wherein each of the plurality of mount entries is associated with a respective one of the plurality of metadata volumes as a parent metadata volume and identifies a respective distinct child metadata volume and the relative pathname associated with the child metadata volume;

resetting the current metadata volume to the child metadata volume identified by the identified mount entry, until no mount entry associated with the current metadata volume is found that matches a respective portion of the absolute pathname, the target metadata volume comprising the last child metadata volume so identified; and

returning information identifying the target metadata volume, hosted metadata and its associated residual pathname in response to the request to a file switch for assembling the file in response to the request.

18. The apparatus of claim 10, wherein the identified relative pathname has a size based upon a maximum number of pathname components of the relative pathname that can be handled by the identified mount entry.

19. A non-transitory computer readable medium having stored thereon instructions for handling a file processing request in an aggregated file system comprising machine executable code which when executed by at least one processor, causes the processor to perform steps comprising:

receiving a request with respect to a specified file, the request including an absolute pathname of the specified file in the aggregated file system;

identifying a target metadata volume that hosts metadata of the file among a plurality of metadata volumes in accordance with the request and one or more mount entries each associated with a respective one of the plurality of metadata volumes, including recursively;

identifying a mount entry when associated with a current metadata volume, wherein the mount entry identifies a relative pathname matching a respective portion of the absolute pathname, the identified relative pathname further comprising pathname sub-components of sizes ranging from one pathname component to a size of the identified relative pathname; and

selecting at least one of the pathname sub-components using the identified mount entry based upon a number indicated in a list entry pointed to by the identified mount entry, until no mount entry associated with a current metadata volume matches a respective portion of the absolute pathname, to set the current metadata volume; and

identifying the current metadata volume as the target metadata volume and sending the request to a metadata server hosting the identified target metadata volume in response to the request to a file switch for assembling the file.

20. The medium of claim 19, wherein the recursively identifying includes producing a residual pathname by removing from the absolute pathname the relative pathname associated with each identified mount entry.

21. The medium of claim 19, wherein the absolute pathname of the file is unique in the aggregated file system.

22. The medium of claim 19, wherein at least a subset of the plurality of metadata volumes includes one or more mount entries, each mount entry identifying a respective distinct child metadata volume and an associated relative pathname.

23. The medium of claim 22, wherein at least one metadata volume in the subset of the plurality of metadata volumes includes a reverse mapper, the reverse mapper identifying a parent metadata volume.

24. The medium of claim 19, wherein:

the plurality of metadata volumes include at least one root metadata volume, one or more intermediate metadata volumes and one or more leaf metadata volumes, and wherein the root metadata volume is a metadata volume that has no parent metadata volume, the leaf metadata volume is a metadata volume that has no child metadata volume and the intermediate metadata volume is a metadata volume that has both at least one parent and at least one child metadata volumes;

the target metadata volume is one of the root metadata volumes when none of the child metadata volumes directly referenced by at least one of the root metadata volumes has an associated relative pathname matching a portion of the absolute pathname;

the target metadata volume is one of the intermediate metadata volumes when at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname and none of the child metadata volumes referenced by at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname; and

the target metadata volume is one of the leaf metadata volumes referenced by at least one of the root metadata volumes using one or more mount entries when at least one of the leaf metadata volumes has an associated relative pathname matching a portion of the absolute pathname.

25. The medium of claim 19, further having stored thereon instructions that when executed by the processor causes the process to perform steps comprising forming an ordered pair comprising the identified target metadata volume and a residual pathname, if any, based upon said identifying of the target metadata volume.

26. The medium of claim 19 further having stored thereon instructions that when executed by the processor causes the process to perform steps comprising:

accessing a plurality of mount entries, wherein each of the plurality of mount entries is associated with a respective one of the plurality of metadata volumes as a parent metadata volume and identifies a respective distinct child metadata volume and the relative pathname associated with the child metadata volume;

resetting the current metadata volume to the child metadata volume identified by the identified mount entry, until no mount entry associated with the current metadata volume is found that matches a respective portion of the absolute pathname, the target metadata volume comprising the last child metadata volume so identified; and

returning information identifying the target metadata volume, hosted metadata and its associated residual pathname in response to the request to a file switch for assembling the file in response to the request.

27. The medium of claim 19, wherein the identified relative pathname has a size based upon a maximum number of pathname components of the relative pathname that can be handled by the identified mount entry.

28. An aggregated file system, comprising:

a plurality of file servers;

a plurality of metadata servers including a plurality of metadata volumes; and

a plurality of file switches, each file switch including;

at least one processor; and

memory coupled to the at least one processor which is configured to execute program instructions stored in the memory comprising;

receiving a request with respect to a specified file, the request including an absolute pathname of the specified file in the aggregated file system;

identifying a target metadata volume that hosts metadata of the file among the plurality of metadata volumes in accordance with the request and one or more mount entries each associated with a respective one of the plurality of metadata volumes, including recursively;

identifying a mount entry when associated with a current metadata volume, wherein the mount entry identifies a relative pathname matching a respective portion of the absolute pathname, the identified relative pathname further comprising pathname sub-components of sizes ranging from one pathname component to a size of the identified relative pathname; and

selecting at least one of the pathname sub-components using the identified mount entry based upon a number indicated in a list entry pointed to by the identified mount entry, until no mount entry associated with a current metadata volume matches a respective portion of the absolute pathname, to set the current metadata volume; and

identifying the current metadata volume as the target metadata volume and sending the request to a metadata server hosting the identified target metadata volume in response to the request to a file switch for assembling the file.

29. The system of claim 28, wherein the recursively identifying includes producing a residual pathname by removing from the absolute pathname the relative pathname associated with each identified mount entry.

30. The system of claim 28, wherein the absolute pathname of the file is unique in the aggregated file system.

31. The system of claim 28, wherein at least a subset of the plurality of metadata volumes includes one or more mount entries, each mount entry identifying a respective distinct child metadata volume and an associated relative pathname.

32. The system of claim 31, wherein at least one metadata volume in the subset of the plurality of metadata volumes includes a reverse mapper, the reverse mapper identifying a parent metadata volume.

33. The system of claim 28, wherein:

the plurality of metadata volumes include at least one root metadata volume, one or more intermediate metadata volumes and one or more leaf metadata volumes, and wherein the root metadata volume is a metadata volume that has no parent metadata volume, the leaf metadata volume is a metadata volume that has no child metadata volume and the intermediate metadata volume is a metadata volume that has both at least one parent and at least one child metadata volumes;

the target metadata volume is one of the root metadata volumes when none of the child metadata volumes directly referenced by at least one of the root metadata volumes has an associated relative pathname matching a portion of the absolute pathname;

the target metadata volume is one of the intermediate metadata volumes when at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname and none of the child metadata volumes referenced by at least one of the intermediate metadata volumes has an associated relative pathname matching a portion of the absolute pathname; and

the target metadata volume is one of the leaf metadata volumes referenced by at least one of the root metadata volumes using one or more mount entries when at least one of the leaf metadata volumes has an associated relative pathname matching a portion of the absolute pathname.

34. The system of claim 28 wherein the at least one processor is further configured to execute programmed instructions stored in the memory further comprising forming an ordered pair comprising the identified target metadata volume and a residual pathname, if any, based upon said identifying of the target metadata volume.

35. The system of claim 28 wherein the at least one processor is further configured to execute programmed instructions stored in the memory further comprising:

accessing a plurality of mount entries, wherein each of the plurality of mount entries is associated with a respective one of the plurality of metadata volumes as a parent metadata volume and identifies a respective distinct child metadata volume and the relative pathname associated with the child metadata volume;

resetting the current metadata volume to the child metadata volume identified by the identified mount entry, until no mount entry associated with the current metadata volume is found that matches a respective portion of the absolute pathname, the target metadata volume comprising the last child metadata volume so identified; and

returning information identifying the target metadata volume, hosted metadata and its associated residual pathname in response to the request to a file switch for assembling the file in response to the request.

36. The system of claim 28, wherein the identified relative pathname has a size based upon a maximum number of pathname components of the relative pathname that can be handled by the identified mount entry.