ADAPTIVE DATA STRIPING AND REPLICATION ACROSS MULTIPLE STORAGE CLOUDS FOR HIGH AVAILABILITY AND PERFORMANCE

• US 20140172929A1
• Filed: 12/14/2012
• Published: 06/19/2014
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Sedayao, Jeffrey C., Chahal, Sudip S.
• Granted Patent: US 9,348,840 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Technologies for adaptively striping data across multiple storage clouds include receiving user constraints corresponding one or more cloud storage providers, receiving a file to be striped across the cloud storage providers, splitting the received file into file blocks, allocating each of the file blocks to a different one of the cloud storage providers as a function of the user constraints and operating conditions of each of the cloud storage providers, and sending each of the file blocks to the cloud storage provider to which each file block is allocated. In some embodiments, file blocks may be re-allocated from one cloud storage provider to another cloud storage provider as a function of changing user constraints or operating conditions. In addition, each of the file blocks may be retrieved from the cloud storage providers to re-assemble the file.

Claims

1. A cloud resource allocation server for adaptively striping data across a plurality of storage clouds, the cloud resource allocation server comprising:

a communication module to receive (i) user constraints corresponding one or more cloud storage providers and (ii) a file to be striped across the one or more cloud storage providers; and

a block allocation module to (i) split the received file into a plurality of file blocks and (ii) allocate each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers as a function of the user constraints and operating conditions of each of the one or more cloud storage providers, each of the plurality of file blocks being allocated to a different cloud storage provider,

wherein the communication module further to send each of the plurality of file blocks to the cloud storage provider to which each file block is allocated.

2. The cloud resource allocation server of claim 1, wherein the block allocation module further to store location metadata for each of the plurality file blocks, the location metadata for each file block identifying which of the one or more cloud storage providers the file block is allocated and stored.

3. The cloud resource allocation server of claim 2, wherein the block allocation module further to store re-assembly metadata for re-assembling the file from each of the plurality of file blocks.

4. The cloud resource allocation server of claim 2, wherein the block allocation module further to replicate each allocated file block to another cloud storage provider, the another cloud storage provider being different from the cloud storage provider to which each file block is allocated.

5. The cloud resource allocation server of claim 2, wherein the block allocation module further to replicate the location metadata for each of the plurality file blocks to another cloud resource allocation server.

6. The cloud resource allocation server of claim 2, further comprising:

a change detection module to monitor for changes to the user constraints and the operating conditions of each of the one or more cloud storage providers, and

wherein the block allocation module further to (i) determine whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to the change detection module detecting a change to one or more of the user constraints and the operating conditions of the one storage provider, (ii) re-allocate one or more of the plurality of file blocks from the one cloud storage provider to the another cloud storage provider in response to determining to re-allocate the one or more of the plurality of file blocks, and (iii) update the location metadata for each of the plurality file blocks re-allocated from the one cloud storage provider to the another cloud storage provider.

7. The cloud resource allocation server of claim 6, wherein the user constraints comprise an availability preference, a performance level preference, and a cost preference of which each of the one or more cloud storage providers must satisfy to be allocated one or more of the plurality of file blocks, and

wherein to allocate each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers as a function of the user constraints and the operating conditions of each of the one or more cloud storage providers comprises to allocate each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers in response to determining that the cloud storage provider satisfies the availability preference, the performance level preference, and the cost preference.

8. The cloud resource allocation server of claim 7, wherein to determine whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to detecting a change to one or more of the user constraints and the operating conditions of the one storage provider comprises to determine whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to determining that the operating conditions of the one storage provider no longer satisfy one or more of the availability preference, the performance level preference, and the cost preference.

9. A method for adaptively striping data across a plurality of storage clouds, the method comprising:

receiving, on a cloud resource allocation server, user constraints corresponding one or more cloud storage providers;

receiving, on the cloud resource allocation server, a file to be striped across the one or more cloud storage providers;

splitting, on the cloud resource allocation server, the received file into a plurality of file blocks;

allocating, on the cloud resource allocation server, each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers as a function of the user constraints and operating conditions of each of the one or more cloud storage providers, each of the plurality of file blocks being allocated to a different cloud storage provider; and

sending, on the cloud resource allocation server, each of the plurality of file blocks to the cloud storage provider to which each file block is allocated.

10. The method of claim 9, further comprising storing, on the cloud resource allocation server, location metadata for each of the plurality file blocks, the location metadata for each file block identifying which of the one or more cloud storage providers the file block is allocated and stored.

11. The method of claim 10, further comprising storing, on the cloud resource allocation server, re-assembly metadata for re-assembling the file from each of the plurality of file blocks.

12. The method of claim 10, further comprising:

replicating, on the cloud resource allocation server, each allocated file block to another cloud storage provider, the another cloud storage provider being different from the cloud storage provider to which each file block is allocated; and

replicating, on the cloud resource allocation server, the location metadata for each of the plurality file blocks on another cloud resource allocation server.

13. The method of claim 10, further comprising:

monitoring, on the cloud resource allocation server, for changes to the user constraints;

monitoring, on the cloud resource allocation server, for changes to the operating conditions of each of the one or more cloud storage providers;

determining, on the cloud resource allocation server, whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to detecting a change to one or more of the user constraints and the operating conditions of the one storage provider;

re-allocating, on the cloud resource allocation server, one or more of the plurality of file blocks from the one cloud storage provider to the another cloud storage provider in response to determining to re-allocate the one or more of the plurality of file blocks; and

updating, on the cloud resource allocation server, the location metadata for each of the plurality file blocks re-allocated from the one cloud storage provider to the another cloud storage provider.

14. The method of claim 13, wherein the user constraints comprise an availability preference, a performance level preference, and a cost preference of which each of the one or more cloud storage providers must satisfy to be allocated one or more of the plurality of file blocks, and

wherein allocating each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers as a function of the user constraints and the operating conditions of each of the one or more cloud storage providers comprises allocating each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers in response to determining that the cloud storage provider satisfies the availability preference, the performance level preference, and the cost preference.

15. The method of claim 14, wherein determining whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to detecting a change to one or more of the user constraints and the operating conditions of the one storage provider comprises determining whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to determining that the operating conditions of the one storage provider no longer satisfy one or more of the availability preference, the performance level preference, and the cost preference.

16. One or more machine readable media comprising a plurality of instructions stored thereon that in response to being executed result in a cloud resource allocation server:

receiving user constraints corresponding one or more cloud storage providers;

receiving a file to be striped across the one or more cloud storage providers;

splitting the received file into a plurality of file blocks;

allocating each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers as a function of the user constraints and operating conditions of each of the one or more cloud storage providers, each of the plurality of file blocks being allocated to a different cloud storage provider; and

sending each of the plurality of file blocks to the cloud storage provider to which each file block is allocated.

17. The one or more machine readable media of claim 16, wherein the plurality of instructions further result in the cloud resource allocation server storing location metadata for each of the plurality file blocks, the location metadata for each file block identifying which of the one or more cloud storage providers the file block is allocated and stored.

18. The one or more machine readable media of claim 17, wherein the plurality of instructions further result in the cloud resource allocation server storing re-assembly metadata for re-assembling the file from each of the plurality of file blocks.

19. The one or more machine readable media of claim 17, wherein the plurality of instructions further result in the cloud resource allocation server:

monitoring for changes to the user constraints;

monitoring for changes to the operating conditions of each of the one or more cloud storage providers;

determining whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to detecting a change to one or more of the user constraints and the operating conditions of the one storage provider;

re-allocating one or more of the plurality of file blocks from the one cloud storage provider to the another cloud storage provider in response to determining to re-allocate the one or more of the plurality of file blocks; and

updating the location metadata for each of the plurality file blocks re-allocated from the one cloud storage provider to the another cloud storage provider.

20. The one or more machine readable media of claim 19, wherein the user constraints comprise an availability preference, a performance level preference, and a cost preference of which each of the one or more cloud storage providers must satisfy to be allocated one or more of the plurality of file blocks;

wherein allocating each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers as a function of the user constraints and the operating conditions of each of the one or more cloud storage providers comprises allocating each of the plurality of file blocks to a cloud storage provider of the one or more cloud storage providers in response to determining that the cloud storage provider satisfies the availability preference, the performance level preference, and the cost preference; and

wherein determining whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to detecting a change to one or more of the user constraints and the operating conditions of the one storage provider comprises determining whether to re-allocate one or more of the plurality of file blocks allocated to one cloud storage provider to another cloud storage provider in response to determining that the operating conditions of the one storage provider no longer satisfy one or more of the availability preference, the performance level preference, and the cost preference.

21. A computing device for re-assembling a file from a plurality of file blocks striped across a plurality of storage clouds, the computing device comprising:

a block location determination module to (i) determine that a first cloud storage provider of a plurality of cloud storage providers is storing a first file block of a particular file to be re-assembled, and (ii) determine that a second cloud storage provider of a plurality of cloud storage providers is storing a second file block of the particular file to be re-assembled;

a file re-assembly module to (i) request the first file block from the first cloud storage provider, and (ii) request the second file block from the second cloud storage provider;

a communication module to (i) receive the first file block from the first cloud storage provider, and (ii) receive the second file block from the second cloud storage provider; and

the file re-assembly module further to re-assemble the particular file to be re-assembled from the received first and second file blocks.

22. The computing device of claim 21 wherein the file re-assembly module further to determine whether the second file block of the particular file to be re-assembled exists.

23. The computing device of claim 21, wherein the block location determination module further to:

determine whether location metadata corresponding to the first file block is stored in a local metadata storage of the computing device; and

determine whether location metadata corresponding to the second file block is stored in the local metadata storage of the computing device.

24. The computing device of claim 23, wherein the block location determination module further to:

request the location metadata corresponding to the first file block from a cloud resource allocation server in response to determining that the location metadata corresponding to the first file block is not stored in the local metadata storage of the computing device; and

request the location metadata corresponding to the second file block from the cloud resource allocation server in response to determining that the location metadata corresponding to the second file block is not stored in the local metadata storage of the computing device.

25. The computing device of claim 24, wherein to determine that a first cloud storage provider of a plurality of cloud storage providers is storing a first file block of a particular file to be re-assembled comprises to determine that a first cloud storage provider of a plurality of cloud storage providers is storing a first file block of a particular file to be re-assembled as a function of the location metadata corresponding to the first file block, and

wherein to determine that a second cloud storage provider of a plurality of cloud storage providers is storing the second file block comprises to determine that a second cloud storage provider of a plurality of cloud storage providers is storing the second file block as a function of the location metadata corresponding to the second file block.

26. The computing device of claim 21, wherein the file reassembly module further to determine whether re-assembly metadata corresponding to the particular file to be re-assembled is stored in the local metadata storage of the computing device.

27. The computing device of claim 26, wherein the file reassembly module further to request the re-assembly metadata corresponding to the particular file to be re-assembled from a cloud resource allocation server in response to determining that the re-assembly metadata corresponding to the particular file to be re-assembled is not stored in the local metadata storage of the computing device.

28. The computing device of claim 27, wherein to re-assemble the particular file to be re-assembled from the received first and second file blocks comprises to re-assemble the particular file to be re-assembled from the received first and second file blocks as a function of the re-assembly metadata.