Reducing Latency and Cost in Resilient Cloud File Systems

• US 20130179490A1
• Filed: 01/11/2012
• Published: 07/11/2013
• Est. Priority Date: 01/11/2012
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: Alcatel-Lucent USA, Inc. (Nokia Corporation)
• Inventors: Nandagopal, Thyaga, Ma, Yadi, Naga, Krishna Puttaswamy
• Granted Patent: US 8,868,647 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Various exemplary embodiments relate to a method of storing a file block in a cloud system including a plurality of data centers. The method may include; receiving the file block from a client; generating a plurality of chunks from the file block, wherein each chunk is smaller than the file block and the file block may be reconstructed from a subset of the chunks; distributing each chunk to one of the plurality of data centers; and storing the file block in a cache. Various exemplary embodiments relate to a cloud system for storing files. The system may include a plurality of data centers including a primary data center. The primary data center may include: a cache configured to store at least one complete file block; a chunk storage configured to store a chunk for each of a plurality of file blocks; a file encoder; and a file decoder.

Claims

1. A method of storing a file in a cloud system including a plurality of data centers, the method comprising:

receiving, at a first data center, a file block from a client;

generating a plurality of chunks from the file block, wherein each chunk is smaller than the file and the file block may be reconstructed from a subset of the chunks;

distributing the plurality of chunks to at least two of the plurality of data centers;

wherein at least a first chunk and a second chunk of the plurality of chunks are distributed to different ones of the plurality of data centers; and

storing the file block in a cache at the first data center.

2. The method of claim 1, further comprising:

receiving a request, from a client, to read the file block;

determining whether the file block is stored in the cache;

if the file block is stored in the cache, sending the file block stored in the cache to the client.

3. The method of claim 2, further comprising:

if the file block is not stored in the cache;

requesting chunks from the plurality of data centers;

receiving at least a subset of the plurality of chunks from the plurality of data centers;

reconstructing the file block from the subset of chunks;

storing the file block in a cache at the first data center; and

sending the reconstructed file block to the client.

4. The method of claim 1, wherein the step of generating a plurality of chunks from the file comprises using an erasure code to generate the plurality of chunks.

5. The method of claim 4, wherein the erasure code is at least one of Reed-Solomon codes, MDS codes, and LDPC codes.

6. The method of claim 1, further comprising:

receiving a request to write to the file block;

writing to the file block in the cache;

closing the file;

generating a second plurality of chunks from the file block; and

distributing each chunk of the second plurality of chunks to a different one of the plurality of data centers.

7. The method of claim 6, wherein the second plurality of chunks includes only modified chunks.

8. The method of claim 1, further comprising:

comparing an actual storage and transfer cost for a current cache size with a

a hypothetical storage and transfer cost for a previous cache size; and

adjusting the cache size based on the lower storage and transfer cost.

9. The method of claim 1, further comprising:

determining that the cache is full; and

removing a file block from the cache.

10. The method of claim 1, wherein the plurality of chunks are generated according to a systematic erasure code wherein the file block is divided into a subset of un-encoded chunks and a subset of coded chunks.

11. The method of claim 1, wherein the number of chunks is at least two more than the number in a subset of chunks.

12. A cloud system for storing a file, the system comprising:

a plurality of data centers including a primary data center, the primary data center comprising;

a cache configured to store at least one complete file block;

a chunk storage configured to store a chunk for each of a plurality of file blocks;

a file encoder configured to generate a plurality of chunks from the file block, wherein each chunk is smaller than the file and the file block may be reconstructed from a subset of the chunks; and

a file decoder configured to reconstruct a complete file block from a subset of the chunks.

13. The cloud system of claim 12, wherein the cache is a hard disk.

14. The cloud system of claim 12, wherein the primary data center further comprises a client interface configured to receive complete file blocks from a client and send complete file blocks to a client.

15. The cloud system of claim 12, wherein the primary data center further comprises a cloud interface configured to distribute a chunk of a plurality of chunks to each of the plurality of data centers and configured to receive a subset of chunks from the plurality of data centers.

16. The cloud system of claim 12, wherein the file encoder is configured to use erasure coding to generate the plurality of chunks.

17. The cloud system of claim 12, wherein the number of chunks in the subset is at least two less than the number of the plurality of chunks generated by the file encoder.

19. The tangible and non-transitory machine-readable storage medium of claim 17, wherein the instructions for generating a plurality of chunks from the file block comprise instruction for using an erasure code to generate the plurality of chunks.

20. The tangible and non-transitory machine-readable storage medium of claim 17, further comprising:

instructions for receiving a request to write to the file block;

instructions for writing to the file block in the cache;

instructions for closing the file;

instructions for generating a second plurality of chunks from the file block;

instructions for distributing each chunk of the second plurality of chunks to a different one of the plurality of data centers.

21. The tangible and non-transitory machine-readable storage medium of claim 17, further comprising:

instructions for comparing an actual storage and transfer cost for a current cache size with a hypothetical storage and transfer cost for a previous cache size; and

instructions for adjusting the cache size based on the lower storage and transfer cost.

22. The tangible and non-transitory machine-readable storage medium of claim 17, further comprising:

instructions for receiving a request to read the file block from a client;

instructions for determining whether the file block is stored in the cache;

if the file block is not stored in the cache;

instructions for requesting chunks from the plurality of data centers;

instructions for receiving a subset of chunks from the plurality of data centers;

instructions for reconstructing the file block from the subset of chunks; and

instructions for storing the file block in the cache; and

instructions for sending the file block to the client.

18. A tangible and non-transitory machine-readable storage medium encoded with instructions thereon for execution by a data center, the tangible and non-transitory machine-readable storage medium comprising:

instructions for receiving, at a first data center, a file block of a file from a client;

instructions for generating a plurality of chunks from the file block, wherein each chunk is smaller than the file and the file block may be reconstructed from a subset of the chunks;

instructions for distributing the plurality of chunks to at least two of the plurality of data centers;

wherein at least a first chunk and a second chunk of the plurality of chunks are distributed to different ones of the plurality of data centers; and

instructions for storing the file block in a cache at the first data center.

23. An apparatus for storing a file, the apparatus comprising:

a cache configured to store at least one complete file block;

a chunk storage configured to store a chunk for each of a plurality of file blocks;

a file encoder configured to generate a plurality of chunks from the file block, wherein each chunk is smaller than the file and the file block may be reconstructed from a subset of the chunks; and

a file decoder configured to reconstruct a complete file block from a subset of the chunks.

24. The apparatus of claim 23, further comprising

a cloud interface configured to distribute the plurality of chunks to at least two of a plurality of data centers;

wherein at least a first chunk and a second chunk of the plurality of chunks are distributed to different ones of the plurality of data centers.

25. The apparatus of claim 24, further comprising a client interface configured to:

receive a request, from a client, to read the file block;

determine whether the file block is stored in the cache; and

send the file block stored in the cache to the client,

wherein if the file block is not stored in the cache, the cloud interface is further configured to;

request chunks from the plurality of data centers; and

receive at least a subset of the plurality of chunks from the plurality of data centers,

wherein the file decoder is configured to reconstruct the file block from the subset of chunks and store the file block in the cache.