COMPRESSION PIPELINE FOR STORING DATA IN A STORAGE CLOUD

• US 20120089579A1
• Filed: 10/08/2010
• Published: 04/12/2012
• Est. Priority Date: 10/08/2010
• Status: Abandoned Application

• Current Assignee: Cirtas Systems, Inc.
• Original Assignee: Cirtas Systems, Inc.
• Inventors: Samuels, Allen, Ranade, Sandeep
• Application Number:
  US12/901,385
• Publication Number:
  US 20120089579A1
• Time in Patent Office: Days
• US Class Current: 707/693
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097

Abstract

A cloud storage appliance separates a point-in-time copy of a storage system into payload data chunks and metadata data chunks. The cloud storage appliance identifies a plurality of payload data chunks that have not been saved to a storage cloud. The cloud storage appliance compresses the plurality of payload data chunks. The cloud storage appliance groups the plurality of compressed payload data chunks into one or more cloud files, wherein each of the one or more cloud files is formatted for storage on the storage cloud. The cloud storage appliance then sends the one or more cloud files to the storage cloud.

Claims

1. A method comprising:

separating a point-in-time copy of a storage system into payload data chunks and metadata data chunks;

identifying a plurality of payload data chunks that have not been saved to a storage cloud;

compressing the plurality of payload data chunks;

grouping the plurality of compressed payload data chunks into one or more cloud files, wherein each of the one or more cloud files is formatted for storage on the storage cloud; and

sending the one or more cloud files to the storage cloud.

2. The method of claim 1, wherein the grouping comprises:

generating an empty cloud file, the empty cloud file being a container for holding compressed data chunks;

adding one or more compressed data chunks to the empty cloud file;

determining whether a quantity of compressed data chunks in the cloud file meets a first threshold;

determining whether a size of the cloud file meets a second threshold; and

if the quantity of compressed data chunks in the cloud file does not meet the first threshold and the size of the cloud file does not meet the second threshold, adding one or more additional compressed data chunks to the cloud file.

3. The method of claim 1, wherein compressing the plurality of data chunks comprises, for each data chunk:

selecting one or more reference chunks to compress the data chunk against, wherein each reference chunk is a previously compressed data chunk that has been stored in the storage cloud;

fetching the selected one or more reference chunks from a local cache; and

replacing at least a portion of the data chunk with references to the selected one or more reference chunks.

4. The method of claim 3, wherein at most two reference chunks are selected to compress the data chunk against.

5. The method of claim 1, wherein each cloud file includes a plurality of compressed data chunks, a directory that identifies where in the cloud file each of the plurality of compressed data chunks is located, and a header that identifies where in the cloud file the directory is located.

6. The method of claim 1, further comprising:

after all of the payload data chunks are compressed and sent to the storage cloud, performing the following;

identifying a plurality of metadata data chunks that have not been saved to the storage cloud;

compressing the plurality of metadata data chunks;

grouping the plurality of compressed metadata data chunks into one or more cloud files; and

sending the one or more cloud files to the storage cloud.

7. The method of claim 1, wherein the metadata data chunks are divided into first layer metadata data chunks and second layer metadata data chunks, and wherein all of the first layer metadata data chunks are compressed and sent to the storage cloud before any of the second layer metadata data chunks are compressed.

8. A computer readable storage medium including instructions that, when executed by a processing device, cause the processing device to perform a method comprising:

separating a point-in-time copy of a storage system into payload data chunks and metadata data chunks;

identifying a plurality of payload data chunks that have not been saved to a storage cloud;

compressing the plurality of payload data chunks;

grouping the plurality of compressed payload data chunks into one or more cloud files, wherein each of the one or more cloud files is formatted for storage on the storage cloud; and

sending the one or more cloud files to the storage cloud.

9. The computer readable storage medium of claim 8, wherein the grouping comprises:

generating an empty cloud file, the empty cloud file being a container for holding compressed data chunks;

adding one or more compressed data chunks to the empty cloud file;

determining whether a quantity of compressed data chunks in the cloud file meets a first threshold;

determining whether a size of the cloud file meets a second threshold; and

if the quantity of compressed data chunks in the cloud file does not meet the first threshold and the size of the cloud file does not meet the second threshold, adding one or more additional compressed data chunks to the cloud file.

10. The computer readable storage medium of claim 8, wherein compressing the plurality of data chunks comprises, for each data chunk:

selecting one or more reference chunks to compress the data chunk against, wherein each reference chunk is a previously compressed data chunk that has been stored in the storage cloud;

fetching the selected one or more reference chunks from a local cache; and

replacing at least a portion of the data chunk with references to the selected one or more reference chunks.

11. The computer readable storage medium of claim 10, wherein at most two reference chunks are selected to compress the data chunk against.

12. The computer readable storage medium of claim 8, wherein each cloud file includes a plurality of compressed data chunks, a directory that identifies where in the cloud file each of the plurality of compressed data chunks is located, and a header that identifies where in the cloud file the directory is located.

13. The computer readable storage medium of claim 8, the method further comprising:

after all of the payload data chunks are compressed and sent to the storage cloud, performing the following;

identifying a plurality of metadata data chunks that have not been saved to the storage cloud;

compressing the plurality of metadata data chunks;

grouping the plurality of compressed metadata data chunks into one or more cloud files; and

sending the one or more cloud files to the storage cloud.

14. The computer readable storage medium of claim 8, wherein the metadata data chunks are divided into first layer metadata data chunks and second layer metadata data chunks, and wherein all of the first layer metadata data chunks are compressed and sent to the storage cloud before any of the second layer metadata data chunks are compressed.

15. A storage appliance comprising:

a memory to store instructions for a compression pipeline module; and

a processing device to execute the instructions, wherein the instructions cause the processing device to;

separate a point-in-time copy of a storage system into payload data chunks and metadata data chunks;

identify a plurality of payload data chunks that have not been saved to a storage cloud;

compress the plurality of payload data chunks;

group the plurality of compressed payload data chunks into one or more cloud files, wherein each of the one or more cloud files is formatted for storage on the storage cloud; and

send the one or more cloud files to the storage cloud.

16. The storage appliance of claim 15, wherein to group the plurality of compressed payload data chunks into the one or more cloud files, the processing device:

generates an empty cloud file, the empty cloud file being a container for holding compressed data chunks;

adds one or more compressed data chunks to the empty cloud file;

determines whether a quantity of compressed data chunks in the cloud file meets a first threshold;

determines whether a size of the cloud file meets a second threshold; and

if the quantity of compressed data chunks in the cloud file does not meet the first threshold and the size of the cloud file does not meet the second threshold, adds one or more additional compressed data chunks to the cloud file.

17. The storage appliance of claim 15, wherein compressing the plurality of data chunks comprises, for each data chunk:

selecting one or more reference chunks to compress the data chunk against, wherein each reference chunk is a previously compressed data chunk that has been stored in the storage cloud;

fetching the selected one or more reference chunks from a local cache; and

replacing at least a portion of the data chunk with references to the selected one or more reference chunks.

18. The storage appliance of claim 17, wherein at most two reference chunks are selected to compress the data chunk against.

19. The storage appliance of claim 15, wherein each cloud file includes a plurality of compressed data chunks, a directory that identifies where in the cloud file each of the plurality of compressed data chunks is located, and a header that identifies where in the cloud file the directory is located.

20. The storage appliance of claim 15, further comprising the instructions to cause the processing device to perform the following after all of the payload data chunks are compressed and sent to the storage cloud:

identify a plurality of metadata data chunks that have not been saved to the storage cloud;

compress the plurality of metadata data chunks;

group the plurality of compressed metadata data chunks into one or more cloud files; and

send the one or more cloud files to the storage cloud.

21. The storage appliance of claim 15, wherein the metadata data chunks are divided into first layer metadata data chunks and second layer metadata data chunks, and wherein all of the first layer metadata data chunks are compressed and sent to the storage cloud before any of the second layer metadata data chunks are compressed.