Higher efficiency storage replication using compression

• US 8,775,375 B2
• Filed: 10/21/2011
• Issued: 07/08/2014
• Est. Priority Date: 03/08/2011
• Status: Expired due to Fees

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Holt, Gregory Lee, Gerrard, Clay, Goetz, David Patrick, Barton, Michael
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US13/278,948
• Publication Number:
  US 20120233117A1
• Time in Patent Office: 991 Days
• Field of Search: 707/610, 707/626, 707/637, 707999202-999204, 709/203, 709/219
• US Class Current: 707/637
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/5651 : Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 69/04 : Protocols for data compression, e.g. ROHC

Abstract

An improved scalable object storage system includes methods and systems allowing multiple clusters to work together. In one embodiment, there is a multi-cluster synchronization system between two or more clusters. The multi-cluster synchronization system uses variable compression to optimize the transfer of information between the clusters. Compression is used not only to minimize the total number of bytes sent between the two clusters, but to dynamically vary the size of the objects sent across the wire to optimize for higher throughput after considering packet loss, TCP windows, and block sizes. This includes both the packaging of multiple small files together into one larger compressed file, saving on TCP and header overhead, but also the chunking of large files into multiple smaller files that are less likely to have difficulties due to intermittent network congestion or errors.

Claims

1. A multi-cluster synchronization system, comprising:

a first cluster including a first cluster-internal network, the first cluster further including a first structured information repository and a first object storage, wherein the first structured information repository contains metadata corresponding to stored information objects in the first object storage, and wherein the first structured information repository and the first object storage are coupled via the first cluster-internal network;

an intercluster network coupling the first cluster and a remote cluster; and

an intercluster repository synchronizer including a compression module that selectively compresses multiple objects from the first object storage into a single file and identifies one or more files to compress and transmit to the remote cluster in compressed form, wherein the compression module further includes a programmable threshold, and wherein the compression module selectively compresses an object from the first object storage if a ratio of a compressed size of the object to a non-compressed size of the object exceeds the programmable threshold.

2. The system of claim 1, wherein the compression module selectively compresses multiple objects from the first object storage into a single compressed file if the metadata in the first structured information repository corresponding to the stored information objects has not changed.

3. The system of claim 1, wherein the compression module selectively compresses objects from the first object storage to minimize the network overhead associated with transmitting the objects to the remote cluster.

4. The system of claim 1, wherein the compression module selectively compresses objects from the first object storage to minimize the disk activity associated with reading from or writing to a computer-readable medium.

5. The system of claim 1, further including a network evaluator, wherein the size of a series of transmitted compressed objects transmitted by the intercluster repository synchronizer is modified based on observed network conditions.

6. The system of claim 1, wherein the compression module partitions large data objects into a smaller size for transmission.

7. The system of claim 1, wherein an object of the multiple objects is not a partition of a larger object.

8. A method of synchronizing objects in an object storage system, the method comprising:

identifying a set of stored information objects at a first location to be transferred to a second location;

analyzing the set of stored information objects to determine a compression scheme;

selectively compressing the set of stored information objects according to the compression scheme, wherein the selectively compressing includes compressing a stored information object if a ratio of a compressed size of the object to a non-compressed size of the object exceeds a programmable threshold; and

transmitting the selectively compressed objects to the second location;

wherein transmitting the selectively compressed objects results in the duplication of the set of stored information objects at the second location.

9. The method of claim 8, wherein analyzing the set of stored information objects to determine a compression scheme further comprises consulting the metadata associated with a prior replication and identifying files with metadata that has not changed since the prior replication.

10. The method of claim 9, wherein selectively compressing the set of stored information objects according to the compression scheme further comprises compressing all files that have changed into a single compressed file.

11. The method of claim 9, wherein selectively compressing the set of stored information objects according to the compression scheme further comprises compressing all files that have not changed into a single compressed file.

12. The method of claim 8, wherein analyzing the set of stored information objects to determine a compression scheme further comprises identifying files larger than the programmable threshold.

13. The method of claim 12, wherein selectively compressing the set of stored information objects according to the compression scheme further comprises compressing a plurality of files smaller than the programmable threshold into a single compressed file until the single compressed file is larger than the threshold.

14. The method of claim 12, wherein selectively compressing the set of stored information objects according to the compression scheme further comprises dividing a file larger than the programmable threshold into a plurality of files smaller than the programmable threshold.

15. The method of claim 8, wherein analyzing the set of stored information objects to determine a compression scheme further comprises evaluating network transmission conditions between the first location and the second location.

16. The method of claim 15, wherein selectively compressing the set of stored information objects according to the compression scheme further comprises varying the size of a transmitted compressed object depending on the evaluated network transmission conditions.

17. The method of claim 8, wherein the transmitting the selectively compressed objects to the second location further comprises establishing a secure connection to the second location.

18. The method of claim 8, wherein transmitting the selectively compressed objects to the second location includes transmitting the selectively compressed objects using HTTP.