Building a meta file system from file system cells

• US 6,775,679 B2
• Filed: 03/20/2001
• Issued: 08/10/2004
• Est. Priority Date: 03/20/2001
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Gupta, Uday K.
• Primary Examiner(s): AMSBURY, WAYNE P
• Application Number:
  US09/812,740
• Publication Number:
  US 20020138502A1
• Time in Patent Office: 1,239 Days
• Field of Search: 707/102, 711/1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

File system cells are linked together to form a meta file system that appears to a user or application program to be a single file system. Each file system cell may have a conventional file system format, and can be indistinguishable from a conventional file system except for information, such as directory entry attributes, indicating one or more links to other file system cells. These external links may include direct links that are hidden from the user or application program and define a hierarchy of the meta file system cells, and indirect links that appear in the user-visible file system. The meta file system substantially reduces crash recovery time because each file system cell functions as a consistency unit that can be repaired only if needed. The meta file system also permits the file system cells to be accessed concurrently by multiple processors in a file server.

Claims

1. A method comprising:

selecting a file subsystem in a first file system;

removing the selected file subsystem from the first file system, converting the selected file subsystem into a second file system, and inserting a link in the first file system from a directory entry for the selected file subsystem to a root directory of the second file system; and

presenting the combination of the first file system and the second file system to a user or application program so that combination of the first file system and the second file system appears to the user or application program as one file system.

2. The method as claimed in claim 1, wherein the converting of the selected file subsystem into the second file system includes adding file system metadata to the selected file subsystem.

3. The method as claimed in claim 1, which includes creating a hierarchy of multiple file systems in such a way that the hierarchy appears to a user or application program as a single file system.

4. The method as claimed in claim 1, which includes hiding the link from view of a user or application program.

5. The method as claimed in claim 1, which includes providing the directory entry for the selected file subsystem with an attribute indicating that the entry is for an external link.

6. The method as claimed in claim 1, wherein the link is inserted by inserting a link file in the first file system, the link file including information for locating the second file system.

7. The method as claimed in claim 1, wherein the first file system includes a link between a portion of the first file system outside the selected file subsystem to the inside of the selected file subsystem, and the method further includes providing the link with a target specification that does not change when the method of claim 1 is repeated upon the first and second file systems.

8. The method as claimed in claim 1, wherein the first file system includes a link between a portion of the first file system outside the selected file subsystem to the inside of the selected file subsystem, and the method further includes providing a user or application with a view of objects in the first and second file systems that does not change from before and after performing the method of claim 1, and providing the link with a target specification which includes a name of a path in the view of objects in the first and second file systems that does not change from before and after performing the method of claim 1.

9. The method as claimed in claim 1, wherein the method includes accessing each of the first and second file systems by checking whether said each of the first and second file systems has an attribute indicating that the first file system has at least one external link to another file system, and when said each of the first and second file systems does not have an attribute indicating that the first file system has at least one external link to another file system, invoking at least one file system manager routine that does not recognize external links, and when said each file system does have an attribute indicating that the first file system has at least one external link to another file system, invoking at least one file system manager routine that recognizes external links.

10. The method as claimed in claim 1, which includes recovering from a system crash by checking whether each of the first and second file systems has a file system attribute indicating whether or not said each of the first and second file systems needs to be checked for consistency after the system crash.

11. The method of claim 1, which further includes operating a first processor to access the first file system to check the first file system for consistency after a system crash and concurrently operating a second processor to access the second file system to check the second file system for consistency after the system crash.

12. The method of claim 1, which further includes operating a first processor to manage user or application program access to the first file system and concurrently operating a second processor to manage user or application program access to the second file system.

13. The method of claim 12, wherein the selecting of the file subsystem is based on loading of the processors in order to balance loading upon the processors.

14. A collection of multiple file systems contained in data storage, the multiple file systems being linked in a hierarchy by direct links in the file systems, the direct links being hidden from a user or application program in such a way that the hierarchy appears to the user or application program as a single file system.

15. The collection of multiple file systems as claimed in claim 14, wherein the direct links include hidden files in the file systems, respective directory entries for the hidden files are included in the file systems and the respective directory entries for the hidden files each include an attribute indicating that the entry is for an external link.

16. The collection of multiple file systems as claimed in claim 14, wherein each file system that includes at least one direct link to another file system in the hierarchy also includes file system meta data including an attribute indicating that said each file system includes at least one direct link to another file system in the hierarchy.

17. The collection of multiple file systems as claimed in claim 14, wherein each direct link from a first file system in the collection of multiple file systems to a second file system in the collection of multiple file systems includes a link file in the first file system, and the link file contains information for locating the second file system.

18. The collection of multiple file systems as claimed in claim 14, further including at least one indirect link from a first file system in the collection of multiple file systems to a second file system in the collection of multiple file systems, the indirect link being visible to the user or application program, and the indirect link including a target specification that is a name of a path in said single file system visible to the user or application program.

19. A method of operating a network file server to provide users in the network with concurrent access to objects distributed over a plurality of file systems in data storage of the network file server, the network file server having a plurality of data mover computers for responding to users requesting access to the data storage, the method comprising:

(a) configuring the data mover computers so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

(b) when each data mover computer receives a user request for access to an object in one of the file systems, said each data mover computer directs the request to the data mover computer that has exclusive management of access to said one of the file systems;

wherein the file systems are linked in a hierarchy and said each data mover computer accesses a hidden root directory in the hierarchy of file system in order to direct the request to the data mover computer that has exclusive management of access to a respective one of the plurality of file systems containing the object.

20. The method as claimed in claim 19, wherein the hidden root directory is contained in a file system that is replicated in each of the data mover computers.

21. A method of operating a network file server to provide users in the network with concurrent access to objects distributed over a plurality of file systems in data storage of the network file server, the network file server having a plurality of data mover computers for responding to users requesting access to the data storage, the method comprising:

(a) configuring the data mover computers so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

(b) when each data mover computer receives a user request for access to an object in one of the file systems, said each data mover computer directs the request to the data mover computer that has exclusive management of access to said one of the file systems;

wherein after a system crash, each data mover performs a consistency check upon the file systems in said each data mover'"'"'s respective subset of file systems.

22. A method of operating a network file server to provide users in the network with concurrent access to objects distributed over a plurality of file systems in data storage of the network file server, the network file server having a plurality of data mover computers for responding to users requesting access to the data storage, the method comprising:

(a) configuring the data mover computers so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

(b) when each data mover computer receives a user request for access to an object in one of the file systems, said each data mover computer directs the request to the data mover computer that has exclusive management of access to said one of the file systems;

which further includes load balancing upon the data movers by reconfiguring the file systems in order to move objects from file systems managed by more heavily loaded data movers to file systems managed by less heavily loaded data movers.

23. The method as claimed in claim 22, wherein each data mover computer monitors loading upon said each data mover to determine whether said each data mover computer is heavily loaded, and upon determining that said each data mover computer is heavily loaded, said each data mover computer polls other data mover computers to search for a less heavily loaded data mover computer.

24. The method as claimed in claim 22, wherein the objects are moved by shifting file subsystems.

25. The method as claimed in claim 24, wherein file subsystems are shifted by removing the a selected file subsystem from a first file system, converting the selected file subsystem into a second file system, and inserting a link in the first file system from a directory entry for the selected file subsystem to a root directory of the second file system.

26. A method of operating a network file server to provide users in the network with concurrent access to objects distributed over a plurality of file systems in data storage of the network file server, the network file server having a plurality of data mover computers for responding to users requesting access to the data storage, the method comprising:

(a) configuring the data mover computers so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

(b) when each data mover computer receives a user request for access to an object in one of the file systems, said each data mover computer directs the request to the data mover computer that has exclusive management of access to said one of the file systems;

which further includes load balancing upon the data movers by moving ownership of file systems from more heavily loaded data movers to less heavily loaded data movers.

27. A network file server comprising:

data storage; and

a plurality of data mover computers adapted for coupling to a data network and coupled to the data storage to provide users in the network with concurrent access to objects distributed over a plurality of file systems in the data storage of the network file server;

wherein the data mover computers are configured so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

each data mover computer is programmed to receive a user request for access to an object in one of the file systems, and directs the request to the data mover computer that has exclusive management of access to said one of the file systems; and

wherein the file systems are linked in a hierarchy and said each data mover computer is programmed to access a hidden root directory in the hierarchy of file systems in order to direct the request to the data mover computer that has exclusive management of access to a respective one of the plurality of file systems containing the object.

28. The network file server as claimed in claim 27, wherein the hidden root directory is contained in a file system that is replicated in each of the data mover computers.

29. The network file server as claimed in claim 28, wherein the file system containing the hidden root directory contains no user-visible objects.

30. A network file server comprising:

data storage; and

a plurality of data mover computers adapted for coupling to a data network and coupled to the data storage to provide users in the network with concurrent access to objects distributed over a plurality of file systems in the data storage of the network file server;

wherein the data mover computers are configured so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

each data mover computer is programmed to receive a user request for access to an object in one of the file systems, and directs the request to the data mover computer that has exclusive management of access to said one of the file systems containing the object;

wherein each data mover is programmed to recover from a system crash by performing a consistency check upon the file systems in said each data mover'"'"'s respective subset of file systems.

31. A network file server comprising:

data storage; and

a plurality of data mover computers adapted for coupling to a data network and coupled to the data storage to provide users in the network with concurrent access to objects distributed over a plurality of file systems in the data storage of the network file server;

wherein the data mover computers are configured so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

each data mover computer is programmed to receive a user request for access to an object in one of the file systems, and directs the request to the data mover computer that has exclusive management of access to said one of the file systems containing the object;

wherein the data movers are programmed for load balancing by reconfiguring the file systems in order to move objects from file systems managed by more heavily loaded data movers to file systems managed by less heavily loaded data movers.

32. The network file server as claimed in claim 31, wherein each data mover computer is programmed for monitoring loading upon said each data mover to determine whether said each data mover computer is heavily loaded, and upon determining that said each data mover computer is heavily loaded, said each data mover computer polls other data mover computers to search for a less heavily loaded data mover computer.

33. The network file server as claimed in claim 31, wherein each data mover computer is programmed to move objects by shifting file subsystems.

34. The network file server as claimed in claim 33, wherein each data mover computer is programmed to shift file subsystems by removing the selected file subsystem from a first file system, converting the selected file subsystem into a second file system, and inserting a link in the first file system from a directory entry for the selected file subsystem to a root directory of the second file system.

35. A network file server comprising:

data storage; and

a plurality of data mover computers adapted for coupling to a data network and coupled to the data storage to provide users in the network with concurrent access to objects distributed over a plurality of file systems in the data storage of the network file server;

wherein the data mover computers are configured so that each data mover computer has exclusive management of access to a respective subset of the file systems; and

each data mover computer is programmed to receive a user request for access to an object in one of the file systems, and directs the request to the data mover computer that has exclusive management of access to said one of the file systems containing the object; and

wherein the data movers are programmed for load balancing by changing ownership of file systems from more heavily loaded data movers to less heavily loaded data movers.

36. A program storage device containing a program executable by a network file server, the network file server being adapted for coupling to at least one client for access of the client to a user-visible file system in storage in the network file server, objects in the user-visible file system being contained in a plurality of file systems, said program including a manager of the user-visible file system for receiving file system access calls intended for a file system manager, and the manager of the user-visible file system being executable for acting as an intermediary between the client and said file system manager in order to follow hidden links between the plurality of file systems so that the plurality of file systems appear to the client as one file system.

37. A program storage device containing a program executable by a network file server, the network file server being adapted for coupling to at least one client for access of the client to a user-visible file system in storage in the network file server, objects in the user-visible file system being contained in a plurality of file systems, said program including a manager of the user-visible file system for receiving file system access calls intended for a file system manager, and the manager of the user-visible file system being executable for acting as an intermediary between the client and said file system manager in order to follow hidden links between the plurality of file systems, wherein the manager of the user-visible file system is executable for accessing each of said plurality of file systems by checking whether said each of said plurality of file systems has an attribute indicating that said each of said plurality of file systems has external links, and when said each of said plurality of file systems does not have an attribute indicating that said each of said plurality of file systems has external links, calling at least one routine of the manager of the user-visible file system that does not recognize external links, and when said each of said plurality of file systems does have an attribute indicating that said each of said plurality of file systems has external links, executing at least one routine of the manager of the user-visible file system that recognizes external links.

38. The program storage device as claimed in claim 37, wherein the manager of the user-visible file system is executable to recover from a system crash by checking whether said each of said plurality of file systems has a file system attribute indicating whether or not said each of said plurality of file systems needs to be checked for consistency after the system crash.

39. A method comprising:

selecting a file subsystem in a first file system; and

removing the selected file subsystem from the first file system, converting the selected file subsystem into a second file system, and inserting a link in the first file system from a directory entry for the selected file subsystem to a root directory of the second file system;

wherein the link is a hidden file including information for locating the second file system.

40. The method as claimed in claim 39, wherein the method includes accessing each file system by checking whether said each file system has an attribute indicating that the first file system has at least one external link to another file system, and when said each file system does not have an attribute indicating that the first file system has at least one external link to another file system, invoking at least one file system manager routine that does not recognize external links, and when said each file system does have an attribute indicating that the first file system has at least one external link to another file system, invoking at least one file system manager routine that recognizes external links.

41. A method comprising:

selecting a file subsystem in a first file system;

removing the selected file subsystem from the first file system, converting the selected file subsystem into a second file system, and inserting a link in the first file system from a directory entry for the selected file subsystem to a root directory of the second file system; and

recovering from a system crash by checking whether each file system has a file system attribute indicating whether or not said each file system needs to be checked for consistency after the system crash.

42. A method comprising:

selecting a file subsystem in a first file system;

removing the selected file subsystem from the first file system, converting the selected file subsystem into a second file system, and inserting a link in the first file system from a directory entry for the selected file subsystem to a root directory of the second file system; and

operating a first processor to access the first file system to check the first file system for consistency after a system crash and concurrently operating a second processor to access the second file system to check the second file system for consistency after the system crash.

43. A method comprising:

selecting a file subsystem in a first file system;

removing the selected file subsystem from the first file system, converting the selected file subsystem into a second file system, and inserting a link in the first file system from a directory entry for the selected file subsystem to a root directory of the second file system; and

operating a first processor to manage user or application program access to the first file system and concurrently operating a second processor to manage user or application program access to the second file system.

44. The method of claim 43, wherein the selecting of the file subsystem is based on loading of the processors in order to balance loading upon the processors.