SYSTEMS AND METHODS FOR SCALABLE OBJECT STORAGE

• US 20130226978A1
• Filed: 09/21/2012
• Published: 08/29/2013
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: Nexenta by DDN, Inc.
• Original Assignee: Caitlin Bestler
• Inventors: BESTLER, Caitlin, AIZMAN, Alexander
• Granted Patent: US 8,745,095 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures

Abstract

The present disclosure provides an advantageous systems and related methods for scalable object storage. One embodiment relates to a system that includes at least a manifest tracking subsystem and a chunk payload subsystem. The manifest tracking subsystem stores version manifests and encodes forward-references from the version manifests to chunks. The chunk payload subsystem stores the chunks and encodes back-references to version manifests that reference the chunk. Other embodiments, aspects and features are also disclosed.

Claims

1. A system for providing distributed storage of data objects, the system comprising:

a manifest tracking subsystem which encodes forward-references from version manifests to chunks; and

a chunk payload subsystem which stores the chunks and encodes back-references to version manifests that reference the chunk.

2. The system of claim 1, wherein each forward-reference references a chunk by an identity of the chunk without identifying any server on which the chunk is stored.

3. The system of claim 2, wherein each back-reference references an object by an identity of the object without identifying any server on which the object is stored.

4. The system of claim 3, wherein the back-references maintained by the chunk payload system comprise speculative back-references and verified back-references.

5. The system of claim 4, wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system.

9. The system of claim 5, wherein the speculative back-reference is created even when the chunk previously existed in the chunk payload subsystem.

10. The system of claim 5, wherein a verified back-reference corresponding to the speculative back-reference is created only after the manifest tracking subsystem has completed a transaction that included the chunk put operation.

11. The system of claim 5, wherein the speculative back-reference is deleted if the speculative back-reference is of a lower class-of-storage than a verified back-reference from the chunk to the object.

12. The system of claim 5, wherein, in order to verify the speculative back-reference, the chunk payload subsystem queries the manifest tracking subsystem to confirm that there is a corresponding forward-reference to the chunk from the object.

13. The system of claim 12, wherein upon confirmation of the corresponding forward-reference, a verified back-reference from the chunk to the object is created if none already exists, and a class-of-storage attribute for the verified back-reference is updated if the verified back-reference already exists,

further wherein the speculative back-reference is then deleted.

14. The system of claim 13, wherein the class-of-storage attribute is updated to be a greater of the class-of-storage attributes in the speculative and the verified back-references.

15. The system of claim 12, wherein upon confirmation of the corresponding forward-reference, a reference count for the chunk for the specific class-of-storage is incremented,

further wherein the speculative back-reference is then deleted.

16. The system of claim 15, wherein reference counts are adjusted in response to notifications from the manifest tracking subsystem that a class-of-storage for the chunk from a specific object has changed.

17. The system of claim 1 further comprising:

a client which initiates an object put transaction,

wherein the client obtains a transaction identifier for the object put transaction from the manifest tracking subsystem by sending a request for transaction identifier that at least includes an identity of the object to be put.

18. The system of claim 17, wherein the request for transaction identifier includes a class-of-storage identifier.

19. The system of claim 17, wherein the manifest tracking subsystem cryptographically signs the transaction identifier, and wherein the transaction identifier encodes at least the identity of the object to be put and a unique transaction identifier.

20. The system of claim 19, wherein the transaction identifier is unique even when there are multiple concurrent transactions updating the object to be put.

21. The system of claim 17, wherein the chunk payload subsystem rejects a request for a chunk put operation if the request does not supply a transaction identifier which was cryptographically signed by the manifest subsystem.

22. The system of claim 4, wherein the manifest tracking subsystem and the chunk payload subsystem collaborate to maintain a class-of-storage attribute for each chunk stored in the system.

23. The system of claim 22, wherein the manifest tracking subsystem issues a notification of a class-of-storage update when a class-of-storage required for a chunk by an object has been updated.

24. The system of claim 23, wherein the chunk payload subsystem applies the class-of-storage update to a verified back-reference.

25. The system of claim 23, wherein the chunk payload subsystem creates the verified back-reference, if necessary, by confirming a speculative back-reference between the chunk and the object.

26. The system of claim 1, wherein a chunk get operation is validated using a transaction identifier obtained from the manifest tracking subsystem and supplied to the chunk payload subsystem with the chunk get operation.

27. The system of claim 26, wherein the chunk payload subsystem rejects the chunk get operation if the chunk does not have a back-reference to an object identified in the transaction identifier.

6. The system of claim 6, wherein the speculative back-reference encodes a transaction identifier supplied with the chunk put operation.

7. The system of claim 6, wherein a put cancelling operation that identifies the chunk and the transaction identifier causes deletion of the speculative back-reference.

8. The system of claim 6, wherein the speculative back-reference further encodes a timestamp of the object put transaction.

28. A method for providing storage of data objects using a distributed storage system, the method comprising:

encoding forward-references from version manifests to chunks in a manifest tracking subsystem which stores the version manifests; and

encoding back-references to version manifests that reference the chunk.

29. A method for putting an object into a distributed storage system, the method comprising:

receiving from a client of the system a request to put an object into the system;

obtaining a transaction identifier for the request from a manifest tracking subsystem;

dividing a payload of the object into a plurality of chunks;

putting the plurality of chunks into a chunk payload subsystem; and

putting a manifest into the manifest tracking subsystem,

wherein the manifest is put into the manifest tracking subsystem after the plurality of chunks are put into the chunk payload subsystem.

30. The method of claim 29 further comprising:

writing forward references from the object to the plurality of chunks, wherein the forward references are written in the manifest tracking subsystem.

31. The method of claim 30 further comprising:

writing back-references from the plurality of chunks to the object, wherein the back-references are written in the chunk payload subsystem.

32. The method of claim 31 further comprising:

incrementing a reference count for a chunk when a back-reference is confirmed.

33. The method of claim 31 further comprising:

creating a verified back-reference when a back-reference is confirmed.

34. The method of claim 29, wherein the transaction identifier encodes an identifier for the object.

35. The method of claim 33, wherein the transaction identifier is cryptographically signed by the manifest tracking subsystem.

36. The method of claim 29, wherein the transaction identifier is unique even when there are multiple concurrent transactions updating the object to be put.