Parallel virtual file system

• US 5,897,638 A
• Filed: 06/16/1997
• Issued: 04/27/1999
• Est. Priority Date: 06/16/1997
• Status: Expired due to Term

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Software Corporation
• Inventors: Stanfill, Craig, Lasser, Cliff, Lordi, Robert
• Primary Examiner(s): Amsbury, Wayne
• Assistant Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US08/876,734
• Time in Patent Office: 680 Days
• Field of Search: 707/1, 707/2, 707/3, 707/6, 707/7, 707/8, 707/9, 707/10, 707/100, 707/101, 707/102, 707/103, 707/104, 707/200, 707/201, 707/205, 707/907, 707/513, 707/514, 707/500, 707/900, 707/902, 395/200.32, 395/200.49, 395/309, 395/726, 395/670, 395/672, 395/800.11
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: Y10S 707/99938 Concurrency, e.g. lock management in shared database
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A parallel virtual file system for parallel processing systems including single-processor systems having multiple storage devices. The invention manages partitioned files as follows: (1) partitions of a partitioned file are stored in a set of isomorphic "data trees"; (2) an additional directory tree, called the "control tree", is used to build a model of the intended structure of the data trees; (3) the combination of a control tree and a collection of data trees is referred to as a "multifile system", with sets of files referred to as "multifile" and sets of directories referred to as "multi-directories;" data elements of a multifile or multidirectory are referred to as "data plies" and control elements of a multifile or multidirectory are referred to as "control plies"; (4) a set of multifile subroutines is provided for accessing and modifying the multifile system in such a way so as to preserve the isomorphic data structures for the partitioned data; (5) the multifile subroutines use a distributed computing environment which provides for remote procedure calls and, in one of the embodiments of the invention, for a distributed transaction processing protocol to ensure atomicity of structural changes to the multifile system; (6) interference of concurrent file system operations is prevented by creating a "transactional" lock for each file system.

Claims

1. A method for creating and using a parallel virtual file system for a parallel processing system, comprising the steps of:

(a) partitioning a data file to be processed in the parallel processing system;

(b) storing the partitions on storage devices of the parallel processing system as a set of isomorphic data trees comprising data files and directories;

(c) providing a control tree on a storage device of the parallel processing system for storing a model of the file structure of the data trees, the combination of the control tree and the set of data trees defining a multifile system, with sets of isomorphic data files within the multifile system defining multifiles and sets of isomorphic directories within the multifile system defining multi-directories;

(d) providing a set of multifile subroutines for accessing and modifying the multifile system, wherein the multifile subroutines;

(1) effect file operations on processing components of the parallel processing system using remote procedure calls; and

(2) prevent interference of concurrent file operations by creating a transactional lock for multifiles and multi-directories involved in each file operation;

(3) preserve the isomorphism of the data trees by maintaining the model of the file structure of the data trees within the control tree before and after each file operation.

2. The method of claim 1, further including the step of applying a distributed transaction processing protocol to each file operation performed by the multifile subroutines to ensure atomicity of structural changes to the multifile system.

3. The method of claim 2, wherein the distributed transaction processing protocol includes a two-phase commit protocol to assure that all changes made by the file operations can be made permanent or can be rolled back.

4. The method of claim 2, wherein the distributed transaction processing protocol includes the step of recording auxiliary information before each file operation such that, if a failure occurs during a file operation, partial changes effected by the failed file operation may be undone or the file operation may be forced to completion.

5. The method of claim 1, wherein each data file and directory of every multifile and multidirectory is identified by a Universal Resource Locator designator.

6. A method for creating and using a parallel virtual file system for a parallel processing system, comprising the steps of:

(a) partitioning a data file to be processed in the parallel processing system;

(b) storing the partitions on storage devices of the parallel processing system, the storage devices having a native file system, as a set of isomorphic data trees comprising native data files and native directories;

(c) providing a control tree, as a native file on a storage device of the parallel processing system, for storing a model of the native file structure of the data trees, the combination of the control tree and the set of data trees defining a multifile system, with sets of isomorphic data files within the multifile system defining multifiles and sets of isomorphic directories within the multifile system defining multi-directories;

(d) providing a set of multifile subroutines for accessing and modifying the multifile system, wherein the multifile subroutines;

(1) effect native file operations on processing components of the parallel processing system using remote procedure calls; and

(2) prevent interference of concurrent native file operations by creating a transactional lock for multifiles and multi-directories involved in each native file operation;

(3) preserve the isomorphism of the data trees by maintaining the model of the native file structure of the data trees within the control tree.

7. The method of claim 6, further including the step of applying a distributed transaction processing protocol to each native file operation performed by the multifile subroutines to ensure atomicity of structural changes to the multifile system.

8. The method of claim 7, wherein the distributed transaction processing protocol includes a two-phase commit protocol to assure that all changes made by the native file operations can be made permanent or can be rolled back.

9. The method of claim 7, wherein the distributed transaction processing protocol includes the step of recording auxiliary information before each native file operation such that, if a failure occurs during a native file operation, partial changes effected by the failed native file operation may be undone or the native file operation may be forced to completion.

10. The method of claim 6, wherein each data file and directory of every multifile and multidirectory is identified by a Universal Resource Locator designator.

11. A computer program, residing on a computer-readable medium, for creating and using a parallel virtual file system for a parallel processing system, comprising instructions for causing a computer to:

(a) partition a data file to be processed in the parallel processing system;

(b) store the partitions on storage devices of the parallel processing system as a set of isomorphic data trees comprising data files and directories;

(c) provide a control tree on a storage device of the parallel processing system for storing a model of the file structure of the data trees, the combination of the control tree and the set of data trees defining a multifile system, with sets of isomorphic data files within the multifile system defining multifiles and sets of isomorphic directories within the multifile system defining multi-directories;

(d) provide a set of multifile subroutines for accessing and modifying the multifile system, wherein the multifile subroutines;

(1) effect file operations on processing components of the parallel processing system using remote procedure calls; and

(2) prevent interference of concurrent file operations by creating a transactional lock for multifiles and multi-directories involved in each file operation;

(3) preserve the isomorphism of the data trees by maintaining the model of the file structure of the data trees within the control tree before and after each file operation.

12. The program of claim 11, further comprising instructions for causing a computer to apply a distributed transaction processing protocol to each file operation performed by the multifile subroutines to ensure atomicity of structural changes to the multifile system.

13. The program of claim 12, wherein the distributed transaction processing protocol includes a two-phase commit protocol to assure that all changes made by the file operations can be made permanent or can be rolled back.

14. The program of claim 12, wherein the distributed transaction processing protocol further comprising instructions for causing a computer to record auxiliary information before each file operation such that, if a failure occurs during a file operation, partial changes effected by the failed file operation may be undone or the file operation may be forced to completion.

15. The program of claim 11, wherein each data file and directory of every multifile and multidirectory is identified by a Universal Resource Locator designator.

16. A program stored on a computer readable medium for creating and using a parallel virtual file system for a parallel processing system, wherein the processing system executes the following steps:

(a) partitioning a data file to be processed in the parallel processing system;

(b) storing the partitions on storage devices of the parallel processing system, the storage devices having a native file system, as a set of isomorphic data trees comprising native data files and native directories;

(c) providing a control tree, as a native file on a storage device of the parallel processing system, for storing a model of the native file structure of the data trees, the combination of the control tree and the set of data trees defining a multifile system, with sets of isomorphic data files within the multifile system defining multifiles and sets of isomorphic directories within the multifile system defining multi-directories;

(d) providing a set of multifile subroutines for accessing and modifying the multifile system, wherein the multifile subroutines;

(1) effect native file operations on processing components of the parallel processing system using remote procedure calls; and

(2) prevent interference of concurrent native file operations by creating a transactional lock for multifiles and multi-directories involved in each native file operation;

(3) preserve the isomorphism of the data trees by maintaining the model of the native file structure of the data trees within the control tree.

17. The program of claim 16, further including the step of applying a distributed transaction processing protocol to each native file operation performed by the multifile subroutines to ensure atomicity of structural changes to the multiple system.

18. The program of claim 17, wherein the distributed transaction processing protocol includes a two-phase commit protocol to assure that all changes made by the native file operations can be made permanent or can be rolled back.

19. The program of claim 17, wherein the distributed transaction processing protocol further comprising instructions for causing a computer to record auxiliary information before each native file operation such that, if a failure occurs during a native file operation, partial changes effected by the failed native file operation may be undone or the native file operation may be forced to completion.

20. The program of claim 16, wherein each data file and directory of every multifile and multidirectory is identified by a Universal Resource Locator designator.