Systems and methods for scalable object storage
First Claim
Patent Images
1. A system for providing distributed storage of payload and metadata for data objects, the system comprising:
- a manifest tracking subsystem comprising a plurality of manifest server computers communicatively interconnected by a network, the manifest tracking subsystem storing version manifests for the objects, wherein an object stored in the system has version manifests stored at multiple manifest server computers, wherein a version manifest for the object encodes the metadata for the object including forward-references to chunks of the object, wherein the chunks comprise sequences of payload bytes for the object; and
a chunk payload subsystem comprising a plurality of chunk server computers communicatively interconnected by the network, the chunk payload subsystem storing the chunks and encoding back-references to version manifests that reference the chunks, wherein each chunk is stored at multiple chunk server computers,wherein the back-references maintained by the plurality of chunk server computers comprise speculative back-references and verified back-references,wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, andwherein a verified back-reference corresponding to the speculative back-reference is created after the manifest tracking subsystem has completed a transaction that included the chunk put operation.
4 Assignments
0 Petitions
Accused Products
Abstract
The present disclosure provides an advantageous systems and related methods for scalable object storage. One embodiment relates to a system that includes at least a manifest tracking subsystem and a chunk payload subsystem. The manifest tracking subsystem stores version manifests and encodes forward-references from the version manifests to chunks. The chunk payload subsystem stores the chunks and encodes back-references to version manifests that reference the chunk. Other embodiments, aspects and features are also disclosed.
-
Citations
31 Claims
-
1. A system for providing distributed storage of payload and metadata for data objects, the system comprising:
-
a manifest tracking subsystem comprising a plurality of manifest server computers communicatively interconnected by a network, the manifest tracking subsystem storing version manifests for the objects, wherein an object stored in the system has version manifests stored at multiple manifest server computers, wherein a version manifest for the object encodes the metadata for the object including forward-references to chunks of the object, wherein the chunks comprise sequences of payload bytes for the object; and a chunk payload subsystem comprising a plurality of chunk server computers communicatively interconnected by the network, the chunk payload subsystem storing the chunks and encoding back-references to version manifests that reference the chunks, wherein each chunk is stored at multiple chunk server computers, wherein the back-references maintained by the plurality of chunk server computers comprise speculative back-references and verified back-references, wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, and wherein a verified back-reference corresponding to the speculative back-reference is created after the manifest tracking subsystem has completed a transaction that included the chunk put operation. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)
-
-
25. A method for providing storage of data objects using a distributed storage system, the method comprising:
-
storing chunks of a version of an object in a chunk payload subsystem comprising a plurality of chunk server computers, wherein each chunk is stored at multiple chunk server computers; encoding forward-references to the chunks of the version of the object in a version manifest that is stored in a manifest tracking subsystem comprising a plurality of manifest server computers, wherein an object stored in the system has version manifests stored at multiple manifest server computers; and encoding back-references from the chunks of the version of the object to the version manifest that contains the forward-references to the chunks, wherein a consistent hash algorithm is used on the forward references to obtain location information for the chunks, wherein the back-references comprise speculative back-references and verified back-references, wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, and wherein a verified back-reference corresponding to the speculative back-reference is created after the manifest tracking subsystem has completed a transaction that included the chunk put operation.
-
-
26. A method for putting an object into a distributed storage system, the method comprising:
-
receiving from a client of the system a request to put an object into the system; obtaining a transaction identifier for the request from a manifest tracking subsystem; dividing a payload of the object into a plurality of chunks; putting the plurality of chunks into a chunk payload subsystem, wherein each chunk is replicated so as to be stored at multiple chunk server computers of the chunk payload subsystem; putting a manifest to a manifest tracking subsystem, wherein the manifest includes forward references from the object to the plurality of chunks, wherein the manifest is written to multiple manifest server computers of the manifest tracking subsystem, and wherein a first consistent hashing algorithm is used on the forward references to obtain location information for the plurality of chunks, wherein the manifest is put into the manifest tracking subsystem after the plurality of chunks are put into the chunk payload subsystem; and writing back-references from the plurality of chunks to the object, wherein the back-references are written in the chunk payload subsystem, wherein a second consistent hashing algorithm is used on the back-references to obtain location information for the object, wherein the back-references comprise speculative back-references and verified back-references, wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, and wherein a verified back-reference corresponding to the speculative back-reference is created after the manifest tracking subsystem has completed a transaction that included the chunk put operation. - View Dependent Claims (27, 28, 29, 30, 31)
-
Specification