Massively scalable object storage system

• US 10,104,175 B2
• Filed: 12/28/2015
• Issued: 10/16/2018
• Est. Priority Date: 03/08/2011
• Status: Expired due to Fees

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Barton, Michael, Reese, Will, Dickinson, John A., Payne, Jay B, Thier, Charles B, Holt, Gregory
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US14/981,623
• Publication Number:
  US 20160197996A1
• Time in Patent Office: 1,023 Days
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1837 Management specially adapted to peer-to-peer storage networks topology management mechanisms of peer-to-peer networks H04L67/1042
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Several different embodiments of a massively scalable object storage system are described. The object storage system is particularly useful for storage in a cloud computing installation whereby shared servers provide resources, software, and data to computers and other devices on demand. In several embodiments, the object storage system includes a ring implementation used to associate object storage commands with particular physical servers such that certain guarantees of consistency, availability, and performance can be met. In other embodiments, the object storage system includes a synchronization protocol used to order operations across a distributed system. In a third set of embodiments, the object storage system includes a metadata management system. In a fourth set of embodiments, the object storage system uses a structured information synchronization system. Features from each set of embodiments can be used to improve the performance and scalability of a cloud computing object storage system.

Claims

1. A method for managing data items in a distributed storage pool, comprising:

providing a plurality of physical storage pools, each storage pool including a plurality of storage nodes coupled to a network, and each storage node further providing a non-transitory computer readable medium for data storage;

storing a first replica of a data item in a first physical storage pool of the plurality of physical storage pools;

storing a second replica of the data item in a second physical storage pool of the plurality of physical storage pools;

receiving a modification instruction for the data item;

in response to receiving the modification instruction for the data item;

modifying the first replica of the data item;

creating a first modification sentinel file based on the modification instruction; and

storing the first modification sentinel file in the first physical storage pool; and

in response to encountering the first modification sentinel file during a data item replication process of the first physical storage pool;

modifying in accordance with the first modification sentinel file the second replica of the data item stored in the second physical storage pool;

creating a second modification sentinel file;

storing the second modification sentinel file in the second physical storage pool;

waiting for a configurable time;

deleting the first modification sentinel file in the first physical storage pool; and

deleting the second modification sentinel file in the second physical storage pool,

wherein the configurable time is longer than a worst-case replication propagation time between the first physical storage pool and the second physical storage pool.

2. The method of claim 1, wherein the modification instruction is a delete instruction, wherein the selective modification of a replica of the data item is performed by removing each data item from its physical storage pool;

and

wherein the modification sentinel file is a file deletion marker.

3. The method of claim 1, wherein the modification instruction is a purge instruction,

wherein the selective modification of a replica of the data item is performed by deleting and marking as free the physical drive space associated with each replica of the data item;

and

wherein the modification sentinel file is a file purge marker.

4. The method of claim 1, wherein the modification instruction is a version instruction, wherein the selective modification of a replica of the data item is performed by saving a copy of the existing data item in a second location, creating a new copy of each data item;

and wherein the modification sentinel file is a version marker.

5. The method of claim 1, wherein the first replica is stored at a first location and the second replica is stored at a second location, and

wherein the modification instruction is a time-to-live (ttl) instruction,

wherein the modification of a replica of the data item is performed by checking a third location for an update if the time since the last update is greater than the time indicated in the ttl instruction, and copying the data item from the third location if the data item in the third location has been changed;

and

wherein the modification sentinel file is a ttl marker.

6. The method of claim 1, wherein the first modification sentinel file is a 0 byte file.

7. The method of claim 1, further comprising:

applying a hash function to a name of the first replica of the data item, wherein the first modification sentinel file includes a first output of the hash function.

8. A system for managing data items in a distributed storage pool, the system comprising:

a distributed storage system coupled to a network, the distributed storage system including a first storage pool and a second storage pool from a plurality of physical storage pools, the first and second storage pools each including at least one processor, a computer readable medium, and a communications interface;

an object service that stores a first replica of a data item in the first storage pool and receives a modification instruction for the data item, wherein in response to receiving the modification instruction, the object service modifies the first replica of the data item, creates a first modification sentinel file based on the modification instruction, and stores the first modification sentinel file in the first storage pool;

a replicator that stores a second replica of the data item in the second storage pool and encounters the first modification sentinel file during a data item replication process of the first storage pool, wherein in response to encountering the first modification sentinel file, the replicator modifies in accordance with the first modification sentinel file the second replica of the data item stored in the second storage pool, creates a second modification sentinel file, and stores the second modification sentinel file in the second storage pool; and

a timer triggering an action at a configurable time;

wherein the replicator includes a triggerable action that deletes the first modification sentinel file in the first storage pool and deletes the second modification sentinel file in the second storage pool, and

wherein the configurable time is based upon a measured worst-case replication time between the first and second storage pools.

9. The system of claim 8, wherein the modification instruction is a delete instruction, wherein the object service modifies the first replica of the data item and the replicator modifies the second replica of the data item by removing the respective data items from their respective storage pools;

and wherein the modification sentinel files are file deletion markers.

10. The system of claim 8, wherein the modification instruction is a purge instruction, wherein the object service modifies the first replica of the data item and the replicator modifies the second replica of the data item by deleting and marking as free the respective data items from their respective storage pools;

and wherein the modification sentinel files are file purge markers.

11. The system of claim 8, wherein the modification instruction is a version instruction, wherein the object service modifies the first replica of the data item and the replicator modifies the second replica of the data item by saving a copy of the existing data item in a second location in their respective storage pools;

and wherein the modification sentinel files are version markers.

12. The system of claim 8, wherein the first replica is stored at a first location and the second replica is stored at a second location, and wherein the modification instruction is a time-to-live (ttl) instruction, wherein the object service modifies the first replica of the data item and the replicator modifies the second replica of the data item by checking a third location for an update if the time since the last update is greater than the time indicated in the ttl instruction, and copying the data item from the third location if the data item in the third location has been changed;

and wherein the modification sentinel files are ttl markers.

13. The system of claim 8, wherein the configurable time is longer than a worst-case replication propagation time between the first storage pool and the second storage pool.

14. The system of claim 8, wherein the first modification sentinel file is a 0 byte file.

15. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions that when executed by one or more processors causes the one or more processors to perform a method comprising:

storing a first replica of a data item in a first physical storage pool;

storing a second replica of the data item in a second physical storage pool;

in response to receiving a modification instruction for the data item, modifying the first replica of the data item, creating a first modification sentinel file based on the modification instruction, and storing the first modification sentinel file in the first physical storage pool;

in response to encountering the first modification sentinel file during a data item replication process of the first physical storage pool, modifying in accordance with the first modification sentinel file the second replica of the data item stored in the second physical storage pool, creating a second modification sentinel file, and storing the second modification sentinel file in the second physical storage pool;

waiting for a configurable time; and

deleting the first modification sentinel file in the first physical storage pool; and

deleting the second modification sentinel file in the second physical storage pool,

wherein the configurable time is based upon a measurement of a replication delay between the first and second physical storage pools.

16. The machine-readable medium of claim 15, wherein the first modification sentinel file is a 0 byte file.