Determining File Allocation Based on File Operations

• US 20130304774A1
• Filed: 05/10/2012
• Published: 11/14/2013
• Est. Priority Date: 05/10/2012
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Lee, Bu Sung, Tan, Yu Shyang, Ko, Kok Leong Ryan, Jagadpramana, Peter
• Granted Patent: US 9,152,640 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A storage system may store files on multiple nodes. One or more logs may indicate operations performed on the files stored in the storage system and may identify the nodes that requested the operations. A new allocation or file placement scheme may be determined to reduce network traffic.

Claims

1. A system, comprising:

a communication interface to receive a log from each node of a cloud storage system, each log indicating a file operation history for each file stored on the respective node, the file operation history indicating operations performed on the stored files and which nodes requested the operations; and

an optimization engine to determine a new allocation of the files across the nodes of the cloud storage system to reduce network traffic caused by the operations performed on the files.

2. The system of claim 1, further comprising a file allocation engine to reallocate the files in the cloud storage system according to the new allocation determined by the optimization engine.

3. The system of claim 1, wherein for a given file, a local operation is an operation performed on the file by a node storing the file and a network operation is an operation performed on the file by a node different from the node storing the file.

4. The system of claim 3, wherein the optimization engine is configured to determine the new allocation of the files by, for each file:

assigning a fixed weight to local operations;

assigning a variable weight to network operations based on distance traveled from the node requesting the operation to the node storing the file;

calculating a total file operation cost by summing the local operation weights and the network operation weights;

determining whether storing the file on another node in the cloud storage system will reduce the total file operation cost given the file operation history of the file; and

selecting the node that would provide the lowest total file operation cost.

5. The system of claim 1, wherein the file operation history indicates a length of time that each file operation lasted and the optimization engine is configured to consider the length of time of each operation when determining the new allocation.

6. The system of claim 1, further comprising a visualization engine to generate a visual representation of the new allocation of the files.

7. The system of claim 1, wherein sequential local operations are indicated as a single operation in the file operation history.

8. The system of claim 1, further comprising a mapping layer to maintain a mapping between the new allocation of the files and previous allocations of the files so as to provide a persistent file path to the files for nodes requesting operations for the files based on a previous allocation.

9. A method, comprising:

receiving, using a processor, multiple lists of file operations for files stored on nodes in a distributed file system;

assigning, using a processor, a cost to each file operation based on whether the file operation was a local operation or a network operation;

calculating, using a processor, a total operation cost for each file by adding together the individual file operation costs for the respective file; and

for each file, determining, using a processor, whether the respective total operation cost can be reduced by storing the file on another node in the distributed file system.

10. The method of claim 9, further comprising generating a file placement scheme that reduces an overall file operation cost by reducing one or more total operation costs of files stored in the distributed file system.

11. The method of claim 10, further comprising generating a graphical representation of the file placement scheme for presentation to a user.

12. The method of claim 11, further comprising:

receiving an instruction from the user to implement the file placement scheme; and

reorganizing the files stored in the distributed file system according to the file placement scheme.

13. The method of claim 10,

wherein the multiple lists of file operations relate to file operations that occurred during a first period of time, and

wherein generating a file placement scheme that reduces an overall file operation cost takes into account file operations that occurred during a second period of time that occurred prior to the first period of time.

14. The method of claim 9, further comprising:

at each node in the distributed filing system,

tracking local operations for files stored on the respective node and network operations for files stored on the respective node;

generating a list of the local and network operations; and

sending the list of the local and network operations to a server,

wherein the multiple lists of file operations comprise the list of the local and network operations generated by each node in the distributed file system.

15. A non-transitory machine-readable storage medium encoded with instructions executable by a processor, the machine-readable medium comprising:

instructions to read multiple node operation logs associated with multiple nodes of a cloud storage system, each node operation log indicating a file operation history for each file stored on the node;

instructions to calculate an operation cost for each file based on which nodes in a cloud network requested the operations, the cost being proportional to a distance traveled between the node storing the file and the node requesting the operation; and

instructions to determine a more efficient file allocation in the cloud storage system to reduce the operation cost for one or more files.