Parallel transparent restructuring of immutable content in a distributed object storage system

• US 10,110,676 B2
• Filed: 12/24/2015
• Issued: 10/23/2018
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Nexenta by DDN, Inc.
• Original Assignee: Nexenta Systems Incorporated (DataDirect Networks Incorporated)
• Inventors: Bestler, Caitlin, Aizman, Alexander, Novak, Robert E.
• Primary Examiner(s): Eskandarnia, Arvin
• Application Number:
  US14/998,320
• Publication Number:
  US 20160205190A1
• Time in Patent Office: 1,034 Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: H04L 12/1854 with non-centralised forwarding system, e.g. chaincast
• CPC Class Codes: H04L 12/1877 Measures taken prior to transmission
• CPC Class Codes: H04L 47/15 in relation to multipoint traffic arrangements for broadcast or multicast in data networks H04L12/18
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

The present disclosure provides an innovative technique for parallel transparent restructuring of immutable content in a distributed object storage system. The content restructuring technique disclosed herein may be implemented with parallel operations by multiple storage servers in a live system. The restructuring is transparent in that the original CHIT may still be used to retrieve and validate the original content, such that a client requesting the original content need not be aware that the original content has been restructured. Furthermore, the restructuring is performed while maintaining support for pre-existing immutable metadata. Other embodiments, aspects and features are also disclosed. The content remains immutable from the perspective of the end user, but still may be advantageously restructured to optimize storage efficiency.

Claims

1. A method of restructuring content stored at a storage server of a distributed object storage system, the method comprising:

determining that an original chunk containing original content is to be restructured into an alternate chunk containing alternate content deriving the alternate content from the original content;

writing the alternate chunk; and

writing an alternate-index key-value tuple (KVT) that establishes the alternate chunk as containing an alternate encoding of the original chunk.

2. The method of claim 1, wherein the alternate-index KVT comprises:

a searchable key that includes an original chunk identifying token (CHIT) which identifies the original content; and

a value that includes an alternate CHIT which identifies the alternate content.

3. The method of claim 1, wherein the original content is of a category from a group of categories consisting of a payload category, a Content Manifest category, and a Version Manifest category.

4. The method of claim 1, wherein the alternate content is of a category from a group of categories consisting of a payload category, a Content Manifest category, and a Version Manifest category.

5. The method of claim 1, wherein the alternate content, when represented by an alternate manifest, comprises payload chunks using different chunk boundaries;

the method comprising;

generating alternate chunks with new boundaries;

creating an alternate manifest KVT to an original manifest that represents the original content; and

issuing verified back references from the original manifest to the alternate chunks referenced using the alternate manifest.

6. The method of claim 1, wherein the alternate content comprises a compressed version of the original content.

7. The method of claim 1, wherein the alternate content comprises a slice of a set of erasure-encoded slices generated from the original content.

8. The method of claim 1, wherein the alternate content comprises changes relative to a content of a referenced chunk.

9. The method of claim 1, further comprising:

reducing a retention requirement for the original content after successfully writing the alternate chunk and the alternate-index KVT.

10. The method of claim 9, wherein the retention requirement for a given replica of a chunk on a specific storage server is reduced by marking an original KVT for the original chunk as releasable.