Storing data and metadata in respective virtual shards on sharded storage systems

• US 9,811,546 B1
• Filed: 06/30/2014
• Issued: 11/07/2017
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Bent, John M., Faibish, Sorin, Zhang, Zhenhua, Liu, Xuezhao, Zhang, Jingwang
• Primary Examiner(s): Yen, Syling
• Application Number:
  US14/319,301
• Time in Patent Office: 1,226 Days
• Field of Search: 707791-796, 707802, 707999101-999103, 711114, 709217
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/2228 Indexing structures

Abstract

Techniques are provided for storing data and metadata on sharded storage arrays. In one embodiment, data is processed in a sharded distributed data storage system that stores data in a plurality of shards on one or more storage nodes by providing a plurality of addressable virtual shards within each of the shards, wherein at least a first one of the addressable virtual shards stores the data, and wherein at least a second one of the addressable virtual shards stores the metadata related to the data; obtaining the data from a compute node; and providing the data and the metadata related to the data stored to the sharded distributed data storage system for storage in the respective first and second addressable virtual shards. The metadata related to the data is stored together at a portion of a corresponding stripe for the data in the second one of the addressable virtual shards. A third one of the addressable virtual shards optionally stores a checksum value related to the data.

Claims

1. A method for processing data in a sharded distributed data storage system, wherein the sharded distributed data storage system stores said data in a plurality of shards on one or more storage nodes, said method comprising:

providing a plurality of addressable virtual shards within each of said plurality of shards, wherein at least a first one of said plurality of addressable virtual shards stores said data and wherein at least a second different one of said plurality of addressable virtual shards separately stores metadata related to said data, wherein said data and said corresponding metadata related to said data are stored within said first and second addressable virtual shards, respectively, with a same object offset;

obtaining, at a first burst buffer appliance, said data for a given shard from at least a second burst buffer appliance connected to said first burst buffer appliance by an interconnect network; and

providing, by said first burst buffer appliance, said data for said given shard and said metadata related to said data for said given shard to said sharded distributed data storage system using a single write operation for storage in said respective first and second addressable virtual shards.

2. The method of claim 1, wherein said metadata related to said data is generated by an application that generates said data executing on said compute node.

3. The method of claim 1, further comprising the step of generating said metadata related to said data.

4. The method of claim 1, wherein said metadata related to said data is stored together at a portion of a corresponding stripe for said data in the second one of said addressable virtual shards.

5. The method of claim 1, wherein at least a third one of said plurality of addressable virtual shards stores a checksum value related to said data.

6. The method of claim 5, wherein said data comprises a data chunk and wherein said checksum value corresponds to said data chunk.

7. The method of claim 5, wherein said data comprises a data chunk and wherein said data chunk is further divided into a plurality of sub-chunks and wherein each of a plurality of said checksum values corresponds to one of said sub-chunks.

8. The method of claim 7, wherein each of said plurality of checksum values corresponding to one of said sub-chunks is stored together at a portion of a corresponding stripe of the third one of said addressable virtual shards.

9. The method of claim 1, wherein each of said plurality of addressable virtual shards within each of said plurality of shards is indexed by a unique binary value.

10. The method of claim 1, wherein said method is implemented by said first burst buffer appliance.

22. The method of claim 1, further comprising the step of a first one of said first and second burst buffer appliances reading said given shard using a single read operation and providing at least a portion of said given shard to a second one of said first and second burst buffer appliances.

11. An apparatus for processing data in a sharded distributed data storage system, wherein the sharded distributed data storage system stores said data on a plurality of shards on one or more storage nodes, said apparatus comprising:

a memory; and

at least one hardware device operatively coupled to the memory and configured to;

provide a plurality of addressable virtual shards within each of said plurality of shards, wherein at least a first one of said plurality of addressable virtual shards stores said data and wherein at least a second different one of said plurality of addressable virtual shards separately stores metadata related to said data, wherein said data and said corresponding metadata related to said data are stored within said first and second addressable virtual shards, respectively, with a same object offset;

obtain, at a first burst buffer appliance, said data for a given shard from at least a second burst buffer appliance connected to said first burst buffer appliance by an interconnect network; and

provide, by said first burst buffer appliance, said data for said given shard and said metadata related to said data for said given shard to said sharded distributed data storage system using a single write operation for storage in said respective first and second addressable virtual shards.

12. The apparatus of claim 11, wherein said metadata related to said data is generated by an application that generates said data executing on said compute node.

13. The apparatus of claim 11, wherein said at least one hardware device is further configured to generate said metadata related to said data.

14. The apparatus of claim 11, wherein said metadata related to said data is stored together at a portion of a corresponding stripe for said data in the second one of said addressable virtual shards.

15. The apparatus of claim 11, wherein at least a third one of said plurality of addressable virtual shards stores a checksum value related to said data.

16. The apparatus of claim 15, wherein said data comprises a data chunk and wherein said checksum value corresponds to said data chunk.

17. The apparatus of claim 15, wherein said data comprises a data chunk and wherein said data chunk is further divided into a plurality of sub-chunks and wherein each of a plurality of said checksum values corresponds to one of said sub-chunks.

18. The apparatus of claim 17, wherein each of said plurality of checksum values corresponding to one of said sub-chunks is stored together at a portion of a corresponding stripe of the third one of said addressable virtual shards.

19. The apparatus of claim 11, wherein each of said plurality of addressable virtual shards within each of said plurality of shards is indexed by a unique binary value.

20. The apparatus of claim 11, wherein said apparatus comprises said first burst buffer appliance.

23. The apparatus of claim 11, wherein a first one of said first and second burst buffer appliances reads said given shard using a single read operation and providing at least a portion of said given shard to a second one of said first and second burst buffer appliances.

21. An article of manufacture for processing data in a sharded distributed data storage system, wherein the sharded distributed data storage system stores said data on a plurality of shards on one or more storage nodes, said article of manufacture comprising a non-transitory machine readable recordable storage medium containing one or more programs which when executed implement the steps of:

providing a plurality of addressable virtual shards within each of said plurality of shards, wherein at least a first one of said plurality of addressable virtual shards stores said data and wherein at least a second different one of said plurality of addressable virtual shards separately stores metadata related to said data, wherein said data and said corresponding metadata related to said data are stored within said first and second addressable virtual shards, respectively, with a same object offset;

obtaining, at a first burst buffer appliance, said data for a given shard from at least a second burst buffer appliance connected to said first burst buffer appliance by an interconnect network; and

providing, by said first burst buffer appliance, said data for said given shard and said metadata related to said data for said given shard to said sharded distributed data storage system using a single write operation for storage in said respective first and second addressable virtual shards.