Massively scalable object storage

• US 9,197,483 B2
• Filed: 10/29/2012
• Issued: 11/24/2015
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Barton, Michael, Dickinson, John A., Payne, Jay B., Thier, Charles B., Holt, Gregory, Reese, Will
• Primary Examiner(s): Meky, Moustafa M
• Application Number:
  US13/662,685
• Publication Number:
  US 20130061084A1
• Time in Patent Office: 1,121 Days
• Field of Search: 709200-203, 709217-227, 709/248
• US Class Current: 1/1
• CPC Class Codes: G06F 1/12 Synchronisation of different clock signals provided by a plurality of clock generators
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0671 In-line storage system
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Several different embodiments of a massively scalable object storage system are described. The object storage system is particularly useful for storage in a cloud computing installation whereby shared servers provide resources, software, and data to computers and other devices on demand. In several embodiments, the object storage system includes a ring implementation used to associate object storage commands with particular physical servers such that certain guarantees of consistency, availability, and performance can be met. In other embodiments, the object storage system includes a synchronization protocol used to order operations across a distributed system. In a third set of embodiments, the object storage system includes a metadata management system. In a fourth set of embodiments, the object storage system uses a structured information synchronization system. Features from each set of embodiments can be used to improve the performance and scalability of a cloud computing object storage system.

Claims

1. A method for coordinating events in a distributed system, comprising:

synchronizing a first local clock at a first gateway with the master clock within a system skew threshold, and synchronizing a second local clock at a second gateway with the master clock within the system skew threshold, wherein the master clock is synchronized to coordinated universal time with a master skew threshold;

receiving, at the first gateway, a first request to manipulate a non-volatile data storage, and marking the first request with the time of reception according to the first local clock, with a granularity at least equal to the system skew threshold;

receiving, at the second gateway, a second request to manipulate the non-volatile data storage, and marking the second request with the time of reception according to the second local clock, with a granularity at least equal to the system skew threshold;

evaluating the first request and the second request to determine if they are unambiguously ordered by determining if the first request and the second request are temporally ordered with a granularity greater than the system skew threshold;

determining that the first request and the second request are not unambiguously ordered; and

modifying, based on determining that the first and second requests are not unambiguously ordered, the non-volatile data storage as directed by a deterministic tiebreaker.

2. The method of claim 1, wherein the master skew threshold is less than or equal to a tenth of a second.

3. The method of claim 1, wherein the tiebreaker throws an error for manual intervention.

4. The method of claim 1, wherein the tiebreaker asserts that the first copy to replicate wins and modifies the non-volatile storage according to the request associated with the first copy.

5. The method of claim 1, wherein the tiebreaker prevents modification of the non-volatile data storage.

6. The method of claim 1, wherein the tiebreaker measures the latency between the first gateway and a first address where the first request originated, and further measures the latency between the second gateway and a second address where the second request originated, and modifies the non-volatile storage according to the request exhibiting lower latency.

7. The method of claim 1, wherein the tiebreaker measures the geographic distance between the first gateway and a first address where the first request originated, and further measures the geographic distance between the second gateway and a second address where the second request originated, and modifies the non-volatile storage according to the request originating from a geographically closer address.

8. The method of claim 1, wherein the tiebreaker evaluates whether a merged request can be created including both the changes associated with the first request and the changes associated with the second request.

9. The method of claim 8, wherein the tiebreaker modifies the non-volatile storage according to the merged request if the merged request can be created including both the changes associated with the first request and the changes associated with the second request.

10. The method of claim 1, further comprising:

receiving, at the first gateway, a third request to manipulate the non-volatile data storage, and marking the third request with the time of reception according to the first local clock, with a granularity at least equal to the system skew threshold;

receiving, at the second gateway, a fourth request to manipulate the non-volatile data storage, and marking the fourth request with the time of reception according to the second local clock, with a granularity at least equal to the system skew threshold;

evaluating the third request and the fourth request to determine if they are unambiguously ordered by determining if the third request and the fourth request are temporally ordered with a granularity greater than the system skew threshold;

determining that the third request and the fourth request are unambiguously ordered; and

modifying, based on determining that the first and second requests are unambiguously ordered, the non-volatile data storage as directed in the later request.

11. A non-transient computer readable medium containing executable instructions, which when executed on one or more processors:

synchronizes a first local clock with a second local clock within a system skew threshold;

receives a first request to manipulate a system resource and marks the first request with the time of reception according to the first local clock, with a granularity at least equal to the system skew threshold;

receives a second request to manipulate the system resource and marks the second request with the time of reception according to the second local clock, with a granularity at least equal to the system skew threshold;

evaluates the first request and the second request to determine if they are unambiguously ordered by determining if the first request and the second request are temporally ordered with a granularity greater than the system skew threshold;

determines that the first request and the second request are not unambiguously ordered;

executes, based on the determination that the first and second requests are not unambiguously ordered, tiebreaker instructions controlling the system resource; and

returns a success or error depending on the outcome of the tiebreaker instructions.

12. The non-transient computer readable medium of claim 11, wherein the tiebreaker instructions attempt to create a merged request based on the first and second requests.

13. The non-transient computer readable medium of claim 12, wherein the tiebreaker returns success if the system resource can be manipulated according to the merged request, and error otherwise.

14. The non-transient computer readable medium of claim 11, wherein the tiebreaker throws an error for manual intervention.

15. The non-transient computer readable medium of claim 11, wherein the executable instructions, when executed on the one or more processors:

receives a third request to manipulate a second system resource and marks the third request with the time of reception according to the first local clock, with a granularity at least equal to the system skew threshold;

receives a fourth request to manipulate the system resource and marks the fourth request with the time of reception according to the second local clock, with a granularity at least equal to the system skew threshold;

evaluates the third request and the fourth request to determine if they are unambiguously ordered by determining if the third request and the fourth request are temporally ordered with a granularity greater than the system skew threshold;

determines that the third and fourth requests are unambiguously ordered; and

manipulates, based on the determination that the third and fourth requests are unambiguously ordered, the system resource as directed in the later request.

16. The non-transient computer readable medium of claim 11, wherein the tiebreaker asserts that the first copy to replicate wins and modifies the non-volatile storage according to the request associated with the first copy.

17. The non-transient computer readable medium of claim 11, wherein the tiebreaker prevents modification of the non-volatile data storage.

18. The non-transient computer readable medium of claim 11, wherein the system skew threshold is less than or equal to a microsecond.

19. The non-transient computer readable medium of claim 11, wherein the tiebreaker measures the latency between a first gateway that receives the first request and a first address where the first request originated, and further measures the latency between a second gateway that receives the second request and a second address where the second request originated, and modifies the non-volatile storage according to the request exhibiting lower latency.

20. The method of claim 11, wherein the tiebreaker measures the geographic distance between a first gateway that receive the first request and a first address where the first request originated, and further measures the geographic distance between a second gateway that receives the second request and a second address where the second request originated, and modifies the non-volatile storage according to the request originating from a geographically closer address.