Determining file allocation based on file operations

• US 9,152,640 B2
• Filed: 05/10/2012
• Issued: 10/06/2015
• Est. Priority Date: 05/10/2012
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Tan, Yu Shyang, Ko, Kok Leong Ryan, Jagadpramana, Peter, Lee, Bu Sung
• Primary Examiner(s): Casanova, Jorge A
• Application Number:
  US13/468,823
• Publication Number:
  US 20130304774A1
• Time in Patent Office: 1,244 Days
• Field of Search: 707/827
• US Class Current: 1/1
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A storage system may store files on multiple nodes. One or more logs may indicate operations performed on the files stored in the storage system and may identify the nodes that requested the operations. A new allocation or file placement scheme may be determined to reduce network traffic.

Claims

1. A system, comprising:

a communication interface to receive logs from a plurality of nodes that store files in a cloud storage system, each of the logs indicating a file operation history for a respective file stored on a respective node of the plurality of nodes, the file operation history indicating operations performed on the respective file and which nodes requested the operations; and

an optimization engine including at least one processor to determine, based on the file operation histories indicated by the logs, a new allocation of the files across the nodes of the cloud storage system to reduce network traffic caused by operations performed on the files, wherein the determining of the new allocation of the files comprises, for each given file of the files;

assigning first weights to respective local operations in which the given file requested by a requesting node is stored at the requesting node;

assigning variable weights to respective network operations in which the given file requested by the requesting node is stored at another node;

aggregating the first weights and the variable weights to produce a total operation cost for operations on the given file; and

use the total operation cost to determine the new allocation of the files.

2. The system of claim 1, further comprising a file allocation engine to reallocate the files in the cloud storage system according to the new allocation determined by the optimization engine.

3. The system of claim 1, wherein sequential local operations are indicated as a single operation in the file operation history for the respective file.

4. The system of claim 1, further comprising a mapping layer to maintain a mapping between the new allocation of the files and previous allocations of the files so as to provide a persistent file path to the files for nodes requesting operations for the files based on a previous allocation.

6. The system of claim 1, wherein the file operation history for the respective file indicates a length of time that each operation performed on the respective file lasted, and the optimization engine is configured to consider the length of time of each operation when determining the new allocation.

7. The system of claim 1, further comprising a visualization engine to generate a visual representation of the new allocation of the files.

5. A system comprising:

a communication interface to receive logs from a plurality of nodes that store files in a cloud storage system, each of the logs indicating a file operation history for a respective file stored on a respective node of the plurality of nodes, the file operation history indicating operations performed on the respective file and which nodes requested the operations, wherein for a given file, a local operation is an operation performed on the given file by a node storing the file, and a network operation is an operation performed on the given file by a node different from the node storing the file; and

an optimization engine including at least one processor to determine a new allocation of the files across the nodes of the cloud storage system to reduce network traffic caused by operations performed on the files, wherein the optimization engine is configured to determine the new allocation of the files by, for each file;

assigning a fixed weight to each of local operations;

assigning a variable weight to each of network operations based on distance traveled from a node requesting an operation to a node storing the file;

calculating a total file operation cost by a cost function based on the fixed weights and the variable weights;

determining whether storing the file on another node in the cloud storage system will reduce the total file operation cost given the file operation history of the file; and

selecting a node that would provide a lower total file operation cost.

8. The system of claim 5, wherein the cost function calculates the total file operation cost by summing the fixed weights and the variable weights.

9. A method, comprising:

receiving, using a processor, multiple lists of file operations for files stored on nodes in a distributed file system;

assigning, using a processor, a cost to each file operation based on whether the file operation is a local operation or a network operation, wherein the local operation is an operation on a given file of the files in which the given file is stored on a node that requested the given file, and wherein the network operation is an operation on the given file in which the given file is stored on a node different from the node that requested the given file, and wherein the cost assigned to the local operation is a fixed cost, and the cost assigned to the network operation is a variable cost dependent on a distance between a node storing the given file and the node that requested the given file;

calculating, using a processor, a total operation cost for each respective file of the files by adding together the costs of local operations and the costs of network operations for the respective file; and

for each respective file of the files, determining, using a processor, whether the respective total operation cost can be reduced by storing the respective file on another node in the distributed file system.

10. The method of claim 9, further comprising generating a file placement scheme that reduces an overall file operation cost by reducing one or more total operation costs of files stored in the distributed file system.

11. The method of claim 10, further comprising generating a graphical representation of the file placement scheme for presentation to a user.

12. The method of claim 11, further comprising:

receiving an instruction from the user to implement the file placement scheme; and

reorganizing the files stored in the distributed file system according to the file placement scheme.

13. The method of claim 10,

wherein the multiple lists of file operations relate to file operations that occurred during a first period of time, and

wherein generating the file placement scheme that reduces the overall file operation cost takes into account file operations that occurred during a second period of time that occurred prior to the first period of time.

14. The method of claim 9, further comprising:

at each respective node in the distributed filing system,

tracking local operations for files stored on the respective node and network operations for files stored on the respective node;

generating a list of the local and network operations; and

sending the list of the local and network operations to a server,

wherein the multiple lists of file operations comprise the list of the local and network operations generated by each respective node in the distributed file system.

15. A non-transitory machine-readable storage medium encoded with instructions executable by at least one processor to:

read node operation logs associated with multiple nodes of a cloud storage system, each node operation log indicating a file operation history for each file stored on a node of the multiple nodes;

calculate an operation cost for a respective file of a plurality of files, wherein the calculating comprises;

aggregating costs of local file operations for the respective file and costs of network file operations for the respective file, wherein a local file operation is an operation requested by a node that stores the respective file, and a network file operation is an operation requested by a node different from the node that stores the respective file, and wherein each of the costs of local file operations is a fixed cost, and each of the costs of network file operations is a variable cost dependent on a distance between the node that stores the respective file and the node that requested the network file operation; and

determine a different file allocation in the cloud storage system to reduce an operation cost for at least one of the plurality of files.

16. The non-transitory machine-readable storage medium of claim 15, wherein the file operation history for each respective file further identifies nodes performing operations on the respective file.

17. The non-transitory machine-readable storage medium of claim 16, wherein the file operation history for each respective file further identifies time durations of corresponding operations on the respective file.