Directory aggregation for files distributed over a plurality of servers in a switched file system

• US 20040133606A1
• Filed: 01/02/2003
• Published: 07/08/2004
• Est. Priority Date: 01/02/2003
• Status: Abandoned Application

• Current Assignee: RPX Corporation
• Original Assignee: Z-Force Communications, Inc. (F5 Incorporated)
• Inventors: Nickolov, Peter, Miloushev, Vladimir
• Application Number:
  US10/336,833
• Publication Number:
  US 20040133606A1
• Time in Patent Office: Days
• US Class Current: 707/200
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 61/45 Network directories; Name-to-address mapping
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A file switch, logically positioned between client computers and file servers in a computer network, distributes user files among multiple file servers using an aggregated directory mechanism. A hierarchical directory structure is created on the file servers and used to store metadata files, which store metadata for each user file to indicate where data files, containing portions of the user file, are stored. The file switch automatically spreads the data files and metadata files over a large number of distinct directories on multiple file servers, preventing large number of data files from being stored in a single directory on a single file server. In response to a directory enumeration request from a client computer, one or more directories of metadata files on one or more of the file servers is enumerated, instead of enumerating the data file that store the user file portions.

Claims

1. A method of generating a directory listing of user files, where each user file of at least a plurality of the user files is divided into portions stored on a plurality of file servers, among a group of file servers in a computer network, comprising:

receiving a user request for a directory listing of user files for a specified directory;

accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored; and

constructing a directory listing of the user files from the directory of metafiles.

2. The method of claim 1, wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.

3. The method of claim 2, wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.

4. The method of claim 1, including:

whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, updating the directory of metafiles to indicate the changed aggregated file size of the user file.

5. The method of claim 1, wherein

the specified directory has an associated file path;

and

the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

6. A file switch for use in a computer network having a group of file servers and a plurality of client computers, the file switch comprising:

at least one processing unit for executing computer programs;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

receiving a user request for a directory listing of user files for a specified directory;

accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored;

constructing a directory listing of the user files from the directory of metafiles; and

sending the directory listing to a particular client computer of the plurality of client computers.

7. The file switch of claim 6, wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.

8. The file switch of claim 7, wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.

9. The file switch of claim 6, wherein the computer programs of the file aggregation module include instructions for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.

10. The file switch of claim 6, wherein the specified directory has an associated file path;

and

the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

11. A system for use in a computer network having a plurality of client computers, the system comprising:

a group of file servers;

a file switch, the file switch including;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

means for receiving a user request for a directory listing of user files for a specified directory;

means for accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored;

means for constructing a directory listing of the user files from the directory of metafiles; and

means for sending the directory listing to a particular client computer of the plurality of client computers.

12. The system of claim 11, wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.

13. The system of claim 11, wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.

14. The system of claim 11, including means for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.

15. The system of claim 11, wherein the specified directory has an associated file path;

and

the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

16. A system for use in a computer network having a plurality of client computers, the system comprising:

a group of file servers;

a file switch, the file switch including;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

at least one processing unit for executing computer programs;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

receiving a user request for a directory listing of user files for a specified directory;

accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored;

constructing a directory listing of the user files from the directory of metafiles; and

sending the directory listing to a particular client computer of the plurality of client computers.

17. The system of claim 16, wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.

18. The system of claim 17, wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.

19. The system of claim 16, wherein the computer programs of the file aggregation module include instructions for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.

20. The system of claim 16, wherein the specified directory has an associated file path;

and

the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

21. A method of generating a directory listing of user files, where each user file of at least a plurality of the user files is divided into portions stored on a plurality of file servers, among a group of file servers in a computer network, comprising:

receiving a user request for a directory listing of user files for a specified directory;

accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file; and

aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files.

22. The method of claim 21, wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.

23. The method of claim 22, wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.

24. The method of claim 21, including:

whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.

25. The method of claim 21, wherein

the specified directory has an associated file path;

and

each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

26. A file switch for use in a computer network having a group of file servers and a plurality of client computers, the file switch comprising:

at least one processing unit for executing computer programs;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

receiving a user request for a directory listing of user files for a specified directory;

accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file;

aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files; and

sending the aggregated directory listing to a particular client computer of the plurality of client computers.

27. The file switch of claim 26, wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.

28. The file switch of claim 27, wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.

29. The file switch of claim 26, the computer programs of the file aggregation module including instructions, activated when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, for updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.

30. The file switch of claim 26, wherein

the specified directory has an associated file path;

and

each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

31. A system for use in a computer network having a plurality of client computers, the system comprising:

a group of file servers;

a file switch, the file switch including;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

means for receiving a user request for a directory listing of user files for a specified directory;

means for accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file;

means for aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files; and

means for sending the aggregated directory listing to a particular client computer of the plurality of client computers.

32. The system of claim 31, wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.

33. The system of claim 32, wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.

34. The system of claim 31, including means for updating the corresponding directory of metafiles, when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, to indicate the changed aggregated file size of the user file.

35. The system of claim 31, wherein

the specified directory has an associated file path;

and

each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

36. A system for use in a computer network having a plurality of client computers, the system comprising:

a group of file servers;

a file switch, the file switch including;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

at least one processing unit for executing computer programs;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

receiving a user request for a directory listing of user files for a specified directory;

accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file;

aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files; and

sending the aggregated directory listing to a particular client computer of the plurality of client computers.

37. The system of claim 36, wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.

38. The system of claim 37, wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.

39. The system of claim 36, the computer programs of the file aggregation module including instructions, activated when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, for updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.

40. The system of claim 36, wherein

the specified directory has an associated file path;

and

each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.

41. A method for determining storage locations of a user file in a group of file servers in a computer network, comprising the steps of:

in accordance with a predefined set of aggregation rules, determining a set of file servers from the group of file servers for storing the user file;

for each portion of the user file to be stored in a respective one of the determined set of servers, generating a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and

storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.

42. The method of claim 41, wherein the file path for each portion of the user file is generated by generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.

43. The method of claim 41, including generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.

44. A file switch for use in a computer network having a group of file servers and a plurality of client computers, the file switch comprising:

at least one processing unit for executing computer programs;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

a file aggregation module including one or more computer programs, the computer programs including instructions for;

determining, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;

for each portion of the user file to be stored in a respective one of the determined set of servers, generating a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and

storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.

45. The file switch of claim 44, wherein the computer programs of the file aggregation module include instructions for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.

46. The file switch of claim 44, wherein the computer programs of the file aggregation module include instructions for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.

47. A system for use in a computer network having a plurality of client computers, the system comprising:

a group of file servers;

a file switch, the file switch including;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;

means for determining, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;

means for generating, for each portion of the user file to be stored in a respective one of the determined set of servers, a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and

means for storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.

48. The system of claim 47, including means for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.

49. The system of claim 47, including means for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.

50. A system for use in a computer network having a plurality of client computers, the system comprising:

a group of file servers;

a file switch, the file switch including;

at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file; and

a file aggregation module including one or more computer programs, the computer programs including instructions for;

determining, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;

for each portion of the user file to be stored in a respective one of the determined set of servers, generating a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and

storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.

51. The system of claim 50, wherein the computer programs of the file aggregation module include instructions for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.

52. The system of claim 50, wherein the computer programs of the file aggregation module include instructions for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.