Synchronization and ordering of multiple accessess in a distributed system

• US 8,554,951 B2
• Filed: 04/19/2011
• Issued: 10/08/2013
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Barton, Michael, Reese, Will, Dickinson, John A., Payne, Jay B., Thier, Charles B., Holt, Gregory
• Primary Examiner(s): Meky, Moustafa M
• Application Number:
  US13/089,476
• Publication Number:
  US 20120233490A1
• Time in Patent Office: 903 Days
• Field of Search: 709/248, 709/249, 713/400, 713/503
• US Class Current: 709/248
• CPC Class Codes: G06F 1/12 Synchronisation of different clock signals provided by a plurality of clock generators
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0671 In-line storage system
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Several different embodiments of a massively scalable object storage system are described. The object storage system is particularly useful for storage in a cloud computing installation whereby shared servers provide resources, software, and data to computers and other devices on demand. In several embodiments, the object storage system includes a ring implementation used to associate object storage commands with particular physical servers such that certain guarantees of consistency, availability, and performance can be met. In other embodiments, the object storage system includes a synchronization protocol used to order operations across a distributed system. In a third set of embodiments, the object storage system includes a metadata management system. In a fourth set of embodiments, the object storage system uses a structured information synchronization system. Features from each set of embodiments can be used to improve the performance and scalability of a cloud computing object storage system.

Claims

1. A system for coordinating events in a distributed system, comprising:

a plurality of subsidiary nodes coupled to a network, each subsidiary node including at least one processor, a computer-readable medium, and a communications interface, wherein information in a first subsidiary node needs to be synchronized with the information in a second subsidiary node in response to a time-varying series of requests;

a first gateway, including a first processor, a first local clock, and a first communications interface;

a second gateway, including a second processor, a second local clock, and a second communications interface;

a timekeeping node coupled to the network, including a master clock; and

a synchronization rectifier coupled to the first and second subsidiary nodes;

wherein the timekeeping node is operationally coupled to the first and second gateways to reduce clock skew between the master clock, the first local clock and the second local clock below a configurable threshold;

wherein the first gateway uses the first processor to timestamp a first request received over the first communications interface according to the time of the first local clock with a granularity at least equal to the configurable threshold;

wherein the second gateway uses the second processor to timestamp a second request received over the second communications interface according to the time of the second local clock with a granularity at least equal to the configurable threshold;

wherein synchronization between the first subsidiary node and the second subsidiary node is controlled by the later-occurring request if the first request and the second request are separated by a time greater than the configurable threshold;

and wherein synchronization between the first subsidiary node and the second subsidiary node is controlled by the synchronization rectifier if the first request and the second request are separated by a time smaller than the configurable threshold.

2. The system of claim 1, wherein the system further comprises a replicator adapted to copy information between the first subsidiary node and the second subsidiary node at a third time, wherein the replicator is further adapted to copy information between the second subsidiary node and the first subsidiary node at a fourth time, wherein the synchronization rectifier controls synchronization by ordering the requests according to which of the third and fourth times comes first.

3. The system of claim 1, wherein the synchronization rectifier controls synchronization by returning an error and preventing either request from completing.

4. The system of claim 1, wherein the synchronization rectifier controls synchronization by using geolocation on the request and ordering the requests according to geographical proximity.

5. The system of claim 1, wherein the synchronization rectifier controls synchronization by using latency measurements between the first gateway and the first request, and the second gateway and the second request, and ordering the requests according to observed latency.

6. The system of claim 1, wherein the synchronization rectifier controls synchronization by ordering the requests according to information contained in HTTP headers.

7. The system of claim 1, wherein the synchronization rectifier controls synchronization by ordering the requests according to information contained in the body of the request.

8. The system of claim 1, wherein the configurable threshold is less than or equal to a millisecond.

9. The system of claim 1, wherein the configurable threshold is less than or equal to a microsecond.