Object storage system with local transaction logs, a distributed namespace, and optimized support for user directories
First Claim
1. A method for a storage server within an object storage system to process a put transaction for a new version of an object associated with a new version manifest chunk, wherein the storage server contains an old version of the object associated with an old version manifest chunk, the method comprising:
- validating the new version manifest chunk;
saving the new version manifest chunk to persistent storage;
comparing a unique version identifier for the new version manifest with a unique version identifier for the old version manifest stored in a local name index and updating a local name index for the object'"'"'s name if the unique version identifier for the new version manifest is more current than the unique version identifier for the old version manifest stored in the local name index;
creating an entry in the storage server'"'"'s local transaction log recording the existence of the new version manifest;
sending an acknowledgement message for the put transaction; and
asynchronously processing the entry to update one or more namespace manifests of the object storage system;
wherein each transaction log entry specifying creation or deletion of version manifests is mapped into records to be distributed to namespace manifests, wherein the mapped records comprise inclusive mapped records and iterative mapped records; and
wherein the inclusive mapped records each comprises type, enclosing directory name, the remaining portion of the fully qualified object name which is not part of the enclosing directory name, unique version identifier, and cryptographic hash identifying token of the version manifest, wherein the type comprises data indicating a version manifest exists or data indicating a version manifest does not exist.
4 Assignments
0 Petitions
Accused Products
Abstract
The present invention relates to object storage systems that support hierarchical directories within a namespace manifest stored as an object. The namespace manifest is stored as namespace manifest shards that are determined based on a partial key derived from the full name of the referenced version manifest in each directory entry. Each storage server maintains a local transaction log that keeps track of changes to the namespace manifest in response to put transactions. The namespace manifest can be updated after each put transaction is acknowledged, which minimizes the number of write operations required for a put transaction. Updates to the namespace manifest are batched, distributed, and processed concurrently and in parallel by the storage servers that store the corresponding shards. This reduces the latency that would otherwise exist in the object storage cluster as a result of a put transaction. Updates to namespace manifest shards optionally occur in a batch process using MapReduce techniques.
143 Citations
7 Claims
-
1. A method for a storage server within an object storage system to process a put transaction for a new version of an object associated with a new version manifest chunk, wherein the storage server contains an old version of the object associated with an old version manifest chunk, the method comprising:
-
validating the new version manifest chunk; saving the new version manifest chunk to persistent storage; comparing a unique version identifier for the new version manifest with a unique version identifier for the old version manifest stored in a local name index and updating a local name index for the object'"'"'s name if the unique version identifier for the new version manifest is more current than the unique version identifier for the old version manifest stored in the local name index; creating an entry in the storage server'"'"'s local transaction log recording the existence of the new version manifest; sending an acknowledgement message for the put transaction; and asynchronously processing the entry to update one or more namespace manifests of the object storage system; wherein each transaction log entry specifying creation or deletion of version manifests is mapped into records to be distributed to namespace manifests, wherein the mapped records comprise inclusive mapped records and iterative mapped records; and wherein the inclusive mapped records each comprises type, enclosing directory name, the remaining portion of the fully qualified object name which is not part of the enclosing directory name, unique version identifier, and cryptographic hash identifying token of the version manifest, wherein the type comprises data indicating a version manifest exists or data indicating a version manifest does not exist.
-
-
2. A method for a storage server within an object storage system to process a put transaction for a new version of an object associated with a new version manifest chunk, wherein the storage server contains an old version of the object associated with an old version manifest chunk, the method comprising:
-
validating the new version manifest chunk; saving the new version manifest chunk to persistent storage; comparing a unique version identifier for the new version manifest with a unique version identifier for the old version manifest stored in a local name index and updating a local name index for the object'"'"'s name if the unique version identifier for the new version manifest is more current than the unique version identifier for the old version manifest stored in the local name index; creating an entry in the storage server'"'"'s local transaction log recording the existence of the new version manifest; sending an acknowledgement message for the put transaction; and asynchronously processing the entry to update one or more namespace manifests of the object storage system; wherein each transaction log entry specifying creation or deletion of version manifests is mapped into records to be distributed to namespace manifests, wherein the mapped records comprise inclusive mapped records and iterative mapped records; and wherein each iterative mapped record comprises type, enclosing directory name, and the immediate sub-directory name relative to the enclosing directory.
-
-
3. A method for a storage server within an object storage system to process a put transaction for a new version of an object associated with a new version manifest chunk, wherein the storage server contains an old version of the object associated with an old version manifest chunk, the method comprising:
-
validating the new version manifest chunk; saving the new version manifest chunk to persistent storage; comparing a unique version identifier for the new version manifest with a unique version identifier for the old version manifest stored in a local name index and updating a local name index for the object'"'"'s name if the unique version identifier for the new version manifest is more current than the unique version identifier for the old version manifest stored in the local name index; creating an entry in the storage server'"'"'s local transaction log recording the existence of the new version manifest; sending an acknowledgement message for the put transaction; and asynchronously processing the entry to update one or more namespace manifests of the object storage system; wherein each transaction log entry specifying creation or deletion of version manifests is mapped into records to be distributed to namespace manifests, wherein the mapped records comprise inclusive mapped records and iterative mapped records; and wherein each mapped record is assigned to a specific namespace manifest by a configuration rule assigning subsets of the entire namespace to specific namespace manifests, and to a specific shard of that namespace manifest, by the cryptographic hash of the enclosing path name. - View Dependent Claims (4, 5, 6, 7)
-
Specification