Distributed file system with automated file management achieved by decoupling data analysis and movement operations

• US 6,542,930 B1
• Filed: 03/08/2000
• Issued: 04/01/2003
• Est. Priority Date: 03/08/2000
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Auvenshine, John Jason
• Primary Examiner(s): Eng, David Y.
• Application Number:
  US09/520,585
• Time in Patent Office: 1,119 Days
• Field of Search: 709/224, 709/201, 709/220, 709/221, 709/222
• US Class Current: 709/224
• CPC Class Codes: G06F 16/10 File systems; File servers

Abstract

A computer-driven file management technique is applied to a distributed file system with multiple storage “aggregates” to improve file distribution, user access, server workload, and the like. For a first aggregate, a recognition module analyzes file contents of all aggregates, plans a set of physical data movement operations to configure data stored upon a first one of the aggregates according to certain prescribed “goals,” and buffers the planned physical data movement operations. The planned set of physical data movement operations constitute a “virtual” data move. The reviewing, planning, and buffering operations are repeated for all remaining aggregates in the distributed file system. However, each repeated reviewing operation considers the file contents of the aggregates as if all previous virtual data moves had actually been performed. Asynchronously with the storage analysis and reconfiguration planning, the system processes the action buffer. Namely, the system reads buffer contents according to a prescribed order, consults predetermined action “rules,” and then carries out the physical data moves in accordance with the action rules. The action rules may specify desirable hours to move data or avoid moving data, for example.

Claims

1. A computer-driven method of managing file storage in a distributed file system that includes multiple file servers each including one or more storage aggregates, the method comprising operations of:

for a first aggregate, reviewing storage characteristics of all aggregates, planning one or more physical data movement operations to configure data stored upon the first aggregate according to predetermined criteria, and buffering actions representing the planned physical data movement operations;

repeating the reviewing, planning, and buffering steps for remaining aggregates in the distributed file system, where each repeated reviewing operation considers the file contents of the aggregates as if data movement operations represented by previously buffered actions had already been performed; and

asynchronously processing the buffer, comprising initiating physical data movement operations represented by the buffered actions in accordance with predetermined action rules.

2. The method of claim 1, where the reviewing and planning operations are implemented utilizing an expert system.

3. The method of claim 1, the where the actions comprise machine-readable instructions compatible with the file servers.

4. The method of claim 1, where each action comprises a data structure including names of one or more filesets subject to the physical data movement operations represented by the action, a source location of the physical data movement operation and a destination location of the physical data movement operation.

5. The method of claim 1, where:

the buffering operation also includes establishing a priority level for each action; and

the operation of initiating the physical data movement operations is conducted according to an order specified by the actions'"'"' priority levels.

6. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method for managing file storage in a distributed file system that includes multiple file servers each including one or more storage aggregates, the method comprising operations of:

for a first aggregate, reviewing storage characteristics of all aggregates, planning one or more physical data movement operations to configure data stored upon the first aggregate according to predetermined criteria, and buffering actions representing the planned physical data movement operations;

repeating the reviewing, planning, and buffering steps for remaining aggregates in the distributed file system, where each repeated reviewing operation considers the file contents of the aggregates as if data movement operations represented by previously buffered actions had already been performed; and

asynchronously processing the buffer, comprising initiating physical data movement operations represented by the buffered actions in accordance with predetermined action rules.

7. The medium of claim 6, where the reviewing and planning operations are implemented utilizing an expert system.

8. The medium of claim 6, where each action comprises a data structure including names of one or more filesets subject to the physical data movement operations represented by the action, a source location of the physical data movement operation and a destination location of the physical data movement operation.

9. The medium of claim 6, where the actions comprise machine-readable instructions compatible with the file servers.

10. The medium of claim 6, where:

the buffering operation also includes establishing a priority level for each action; and

the operation of initiating the physical data movement operations is conducted according to an order specified by the actions'"'"' priority levels.

11. A logic circuit of multiple interconnected electrically conductive elements configured to perform operations for managing file storage in a distributed file system that includes multiple file servers each including one or more storage aggregates, the operations comprising:

for a first aggregate, reviewing storage characteristics of all aggregates, planning one or more physical data movement operations to configure data stored upon the first aggregate according to predetermined criteria, and buffering actions representing the planned physical data movement operations;

repeating the reviewing, planning, and buffering steps for remaining aggregates in the distributed file system, where each repeated reviewing operation considers the file contents of the aggregates as if data movement operations represented by previously buffered actions had already been performed; and

asynchronously processing the buffer, comprising initiating physical data movement operations represented by the buffered actions in accordance with predetermined action rules.

12. An apparatus for managing files stored in a storage cell including multiple file severs, each coupled to file storage including one or more storage aggregates, the apparatus comprising:

an action buffer;

a goals database specifying target characteristics for contents of the storage aggregates;

a rules database containing action rules regulating file management activities in the storage aggregates;

a recognition module, coupled to the goals database and the action buffer, programmed to perform operations including;

for a first aggregate, reviewing storage characteristics of all aggregates, planning one or more physical data movement operations to configure data stored upon the first aggregate according to the goals database, and storing actions representing the planned physical data movement operations into the action buffer;

repeating the reviewing, planning, and buffering steps for remaining aggregates in the distributed file system, where each repeated reviewing operation considers the file contents of the aggregates as if data movement operations represented by previously buffered actions had already been performed; and

an action module, coupled to the rules database and the action buffer, programmed to asynchronously process the action buffer, comprising initiating physical data movement operations represented by the buffered actions in accordance with predetermined action rules.

13. The apparatus of claim 12, where the reviewing and planning operations are implemented utilizing an expert system.

14. The apparatus of claim 12, where the actions comprise machine-readable instructions compatible with the file servers.

15. The apparatus of claim 12, where each action comprises a data structure including names of one or more filesets subject to the physical data movement operations represented by the action, a source location of the physical data movement operation and a destination location of the physical data movement operation.

16. The apparatus of claim 12, where:

the buffering operation also includes establishing a priority level for each action; and

the operation of initiating the physical data movement operations is conducted according to an order specified by the actions'"'"' priority levels.

17. The apparatus of claim 12, further comprising the multiple file servers and the aggregates.