UNIFIED LOCAL STORAGE SUPPORTING FILE AND CLOUD OBJECT ACCESS

• US 20130185258A1
• Filed: 03/08/2012
• Published: 07/18/2013
• Est. Priority Date: 01/13/2012
• Status: Active Grant

• Current Assignee: Nexenta by DDN, Inc.
• Original Assignee: Nexenta Systems Incorporated (DataDirect Networks Incorporated)
• Inventors: BESTLER, Caitlin, AIZMAN, Alexander
• Granted Patent: US 8,849,759 B2
• Time in Patent Office: Days
• US Class Current: 707/638
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems

Abstract

A method and system for providing unified local storage support for file and cloud access is disclosed. The method comprises writing a chunk on a storage server, and replicating the chunk to other selected storage servers when necessary. The method and system further comprise writing a version manifest on the storage server; replicating the version manifest to other selected storage servers when necessary. Object puts or appends are implemented by first chunking the object, determining if the chunks are new, transferring the chunks if required, followed by creation of a new version manifest referencing the chunks. Finally, the method and system include providing concurrent file-oriented read and write access consistent with the stored version manifests and chunks.

Claims

1. A system comprising one or more federated storage servers, wherein each storage server comprises:

a processor;

plurality of storage devices; and

a computer readable medium comprising programmable instructions for unified file and cloud storage services that, when executed by the processor, is configured to perform a method, the method comprising;

writing a chunk on a storage server;

replicating the chunk to other selected storage servers when necessary;

writing a version manifest on the storage server;

replicating the version manifest to other selected storage servers when necessary, wherein chunks and version manifests interact to provide the proper version of the object; and

providing concurrent file oriented read and write access consistent with the stored version manifests and chunks.

2. A method for providing unified local storage support for file and cloud access, the method comprising:

writing a chunk on a storage server;

replicating the chunk to other selected storage servers when necessary;

writing a version manifest on the storage server;

replicating the version manifest to other selected storage servers when necessary, wherein chunks and version manifests interact to provide the proper version of the object; and

providing concurrent file oriented read and write access consistent with the stored version manifests and chunks.

3. The method of claim 2, wherein a payload of an object is stored as a plurality of chunks, wherein each chunk is uniquely identified by a chunk identifier derived from the chunk payload, and has associated metadata specifying:

a logical length of the chunk payload;

any compression algorithm used to compress the chunk payload; and

a length of the payload after compression.

4. The method of claim 3, wherein the at least one storage server provides persistent storage for each chunk image wherein the chunk image

is located within a plurality of chunk images using its chunk identifier;

encodes object metadata; and

encodes references to persistent storage blocks containing the compressed payload, or zero or more references to other federated storage servers that have stored the payload for this chunk.

5. The method of claim 2, wherein a mapped service holds a plurality of version manifests for each stored object.

6. The method of claim 5, wherein a mapped service holds an object directory for each stored object, wherein each object directory holds one or more version manifests.

7. The method of claim 5, wherein each service mapping references a single mapped service and is associated with specific access methods.

8. The method of claim 7, wherein a service mapping is a cloud-oriented container.

9. The method of claim 6, wherein each object directory contains one or more version manifests that specify the user defined metadata and data for the object directory as of a specific version.

10. The method of claim 9, wherein the identifier for each version manifest is generated without requiring global synchronization over the federated storage servers.

11. The method of claim 10, wherein each version manifest includes a plurality of chunk references, wherein a chunk reference includes a chunk identifier, and a logical length of the chunk of the uncompressed payload.

12. The method of claim 11, wherein a chunk reference further includes the logical offset of the chunk within the object.

13. The method of claim 11, wherein a chunk reference alternately encodes the logical offset, logical length and an inline payload of the chunk.

14. The method of claim 10, wherein a tiebreaker extends a timestamp to form a unique version identifier.

15. The method of claim 2, wherein the pending manifest encodes all changes to the object while performing file-oriented write operations on the object.

16. The method of claim 15, wherein the pending manifest maintains a set of blocks that have not been collected into fingerprinted chunks.

17. The method of claim 16, wherein a file-oriented synchronization or close operation results in:

transforming a pending manifest into a transactional manifest; and

applying the transactional manifest as a put or append transaction to the object.

18. The method of claim 17, wherein a new chunk image is formed from a snapshot of logically consecutive blocks within the pending manifest.

19. A method of avoiding a single server bottleneck within a set of federated storage servers, the method comprising:

allowing any storage server within the federated storage servers to accept transactions to create a blob;

creating the blob within local storage first by a transaction-initiating storage server;

replicating the blob to other selected storage servers based upon a consistent hashing algorithm by the transaction-initiating storage server; and

completing the transaction with a client by the transaction-initiating storage server.

20. The method of claim 19, wherein replicating a blob comprises:

replicating only chunk images not previously seen and retained by a target storage server; and

replicating transactional manifests only when originated on a source storage server.

21. The method of claim 20, further comprising applying a set of transactional manifests for a single object by different storage servers in an arbitrary order.

22. The method of claim 21, wherein applying each transaction manifest comprises:

determining a version manifest to which the changes in the transaction manifest will be applied;

saving the result of applied changes to the version manifest as a persistent version manifest with a unique version identifier taken from the transaction manifest; and

re-applying all already received but later versioned transaction manifests for this object.

23. The method of claim 22, wherein the re-applying all transactions with later version identifiers further comprises:

extracting a transactional manifest for this version manifest;

applying this transactional manifest to the prior version;

saving the resulting version manifest with the same timestamp and tiebreaker, but with the next sequence number; and

repeating the above steps for each previously applied version manifest.

24. The method of claim 22, wherein supplemental data to support object name queries and reference tracking of chunk images is performed on each storage server in a same local file system transaction that processes each newly created or received blob.