Distributed file system and method of operating a distributed file system

• US 20060101025A1
• Filed: 10/01/2002
• Published: 05/11/2006
• Est. Priority Date: 10/01/2001
• Status: Active Grant

• Current Assignee: ParTec Cluster Competence Center GmbH
• Original Assignee: ParTec Cluster Competence Center GmbH
• Inventors: Isaila, Florin, Tichy, Walter
• Granted Patent: US 7,809,774 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers

Abstract

The present invention provides a distributed file system comprising a plurality of compute nodes and a plurality of input/output (I/O) nodes connected by an interconnection network wherein the system is adapted to use a common data representation for both physical and logical partitions of a file stored in the system and wherein the partitions are linearly addressable. The invention also provides a method of operating a distributed file system comprising a plurality of input/output (I/O) nodes and a plurality of compute nodes, the method comprising the steps of partitioning a file into a plurality of subfiles distributed across a plurality of I/O nodes; logically partitioning a file by setting a view on it; computing mappings between a linear space of a file and a linear space of a subfile; computing the intersection between a view and and a subfile; and performing data operations.

Claims

1. A distributed file system comprising a plurality of compute nodes and a plurality of input/output (I/O) nodes connected by an interconnection network wherein the system is adapted to use a common data representation for both physical and logical partitions of a file stored in the system and wherein elements of the partitions are linearly addressable.

2. A distributed file system according to claim 1 wherein the system includes a metadata manager adapted to gather information about a file from the plurality of I/O nodes and maintain the I/O nodes in a consistent state.

3. A distributed file system according to claim 2 wherein each I/O node comprises an I/O server performing the tasks of writing data to and reading data from subfiles stored on its respective I/O node.

4. A distributed file system according to claim 2 wherein an I/O server is adapted to maintain metadata about each of the subfiles stored at said I/O node and to deliver it to the metadata manager upon request.

5. A distributed file system according to claim 2 wherein the compute nodes are adapted to contact the metadata manager for any file operation that involves system metadata.

6. A distributed file system according to claim 1 wherein each compute node is programmed to perform a plurality of file operations comprising:

computing mapping between a linear space of a file and a linear space of a subfile;

computing mappings between a linear space of a view and linear spaces of subfiles;

performing an intersection algorithm between a view and a sub file; and

performing a data operation.

7. A method of operating a distributed file system comprising a plurality of input/output (I/O) nodes and a plurality of compute nodes, the method comprising the steps of:

(i) physically partitioning a file into subfiles,

(ii) logically partitioning a file into views;

(iii) performing mapping functions between subfiles and views; and

(iv) performing data redistribution between partitions.

8. The method of claim 7 wherein the subfiles and the views are linearly addressable and are described by sets of nested families of line segments.

9. The method according to claim 7 wherein the method arbitrarily partitions data on available storage media.

10. The method according to claim 7 wherein the method arbitrarily partitions data among several compute nodes.

11. The method according to claim 7 wherein the redistribution can be done between:

(i) two physical partitions,

(ii) two logical partitions, and

(iii) a logical and a physical partition

12. The method of claim 9 wherein said method comprises performing a plurality of algorithms comprising:

(i) an algorithm to compute a mapping of a file offset from a linear space of a file on a linear space of a subfile;

(ii) an algorithm to compute a mapping from a linear space of a subfile to a file;

(iii) an algorithm to compute a mapping between a sub file and a view;