Directory aggregation for files distributed over a plurality of servers in a switched file system

• US 8,396,895 B2
• Filed: 11/21/2005
• Issued: 03/12/2013
• Est. Priority Date: 01/11/2001
• Status: Expired due to Fees

• Current Assignee: RPX Corporation
• Original Assignee: F5 Networks, Inc. (F5 Incorporated)
• Inventors: Miloushev, Vladimir, Nickolov, Peter
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Khakhar, Nirav K
• Application Number:
  US11/285,677
• Publication Number:
  US 20060080353A1
• Time in Patent Office: 2,668 Days
• Field of Search: 707/670, 707/795
• US Class Current: 707/795
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 61/45 Network directories; Name-to-address mapping
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A file switch, logically positioned between client computers and file servers in a computer network, distributes user files among multiple file servers using an aggregated directory mechanism. A hierarchical directory structure is created on the file servers and used to store metadata files, which store metadata for each user file to indicate where data files, containing portions of the user file, are stored. The file switch automatically spreads the data files and metadata files over a large number of distinct directories on multiple file servers, preventing large number of data files from being stored in a single directory on a single file server. In response to a directory enumeration request from a client computer, one or more directories of metadata files on one or more of the file servers are enumerated, instead of enumerating the data file that store the user file portions.

Claims

1. A file switch for use in a computer network having a group of file servers and a plurality of client computers, wherein the plurality of client computers are coupled to the group of file servers through the file switch, the file switch comprising:

at least one processing unit for executing computer programs;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

receiving, at the file switch, a user request for a directory listing of user files for a specified directory;

accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile from the directory of metafiles storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored, the particular file server selected at least based upon the size of the one or more strips;

constructing, at the file switch, a directory listing of the user files from the directory of metafiles; and

sending the directory listing from the file switch to a particular client computer of the plurality of client computers.

2. The file switch of claim 1, wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.

3. The file switch of claim 2, wherein the aggregated file size is stored in a data field of the directory listing of the metafiles.

4. The file switch of claim 1, wherein the computer programs of the file aggregation module include instructions for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.

5. The file switch of claim 1, wherein the specified directory has an associated file path;

and

the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

6. A method of managing a directory listing of files in a computer network, comprising:

coupling a plurality of client computers to a group of file servers through a file switch in a computer network;

receiving, at the file switch, a user request for a directory listing of user files for a specified directory;

accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored, the particular file server selected at least based upon the size of the one or more strips;

constructing, at the file switch, a directory listing of the user files from the directory of metafiles; and

sending the directory listing from the file switch to a particular client computer of the plurality of client computers.

7. A file switch for use in a computer network having a group of file servers and a plurality of client computers, wherein the plurality of client computers are coupled to the group of file servers through the file switch, the file switch comprising:

at least one processing unit for executing computer programs;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

receiving, at the file switch, a user request for a directory listing of user files for a specified directory;

accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein at least one of the plurality of file servers is selected at least based upon the size of the one or more strips;

aggregating, at the file switch, the accessed directories of metafiles to construct an aggregated directory listing of the user files; and

sending the aggregated directory listing from the file switch to a particular client computer of the plurality of client computers.

8. The file switch of claim 7, wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.

9. The file switch of claim 8, wherein the aggregated file size is stored in a data field of the respective directory listing of the metafiles.

10. The file switch of claim 7, the computer programs of the file aggregation module including instructions, activated when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, for updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.

11. The file switch of claim 7, wherein

the specified directory has an associated file path;

and

each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

12. A method of managing a directory listing of files in a computer network, comprising:

coupling a plurality of client computers to a group of file servers through a file switch in a computer network;

receiving, at the file switch, a user request for a directory listing of user files for a specified directory;

accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein at least one of the plurality of file servers is selected at least based upon the size of the one or more strips;

aggregating, at the file switch, the accessed directories of metafiles to construct an aggregated directory listing of the user files; and

sending the aggregated directory listing from the file switch to a particular client computer of the plurality of client computers.

13. A file switch for use in a computer network having a group of file servers and a plurality of client computers, wherein the plurality of client computers are coupled to the group of file servers through the file switch, the file switch comprising:

at least one processing unit for executing computer programs;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

determining, at the file switch, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;

for each stripe portion of the user file to be stored in a respective one of the determined set of servers, generating, at the file switch, a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server, wherein each stripe portion comprises one or more strips that have a respective size, at least one of the determined set of servers being selected at least based upon the size of the one or more strips; and

storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.

14. The file switch of claim 13, wherein the computer programs of the file aggregation module include instructions for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.

15. The file switch of claim 13, wherein the computer programs of the file aggregation module include instructions for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.

16. A method of managing a directory listing of files in a computer network, comprising:

coupling a group of client computers to a group of file servers through a file switch in a computer network;

determining, at the file switch, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;

for each stripe portion of the user file to be stored in a respective one of the determined set of servers, generating, at the file switch, a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server, wherein each stripe portion comprises one or more strips that have a respective size, at least one of the determined set of servers being selected at least based upon the size of the one or more strips; and

storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.