Selective caching of erasure coded fragments in a distributed storage system

• US 10,503,654 B2
• Filed: 09/01/2016
• Issued: 12/10/2019
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Raghunath, Arun, Mesnier, Michael P., Zou, Yi
• Primary Examiner(s): Farrokh, Hashem
• Application Number:
  US15/254,824
• Publication Number:
  US 20180060242A1
• Time in Patent Office: 1,195 Days
• Field of Search: 711133
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/0888 using selective caching, e.g. bypass
• CPC Class Codes: G06F 12/121 using replacement algorithms
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc
• CPC Class Codes: G06F 2212/1044 Space efficiency improvement
• CPC Class Codes: G06F 2212/403 Error protection encoding, e.g. using parity or ECC codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes

Abstract

Methods and apparatus related to framework and/or methodology for selective caching of Erasure Coded fragments in a distributed storage system are described. In one embodiment, a plurality of fragments of a data object are generated. Each of the plurality of fragments is Erasure Coded (EC) prior to storage at a storage node of a plurality of storage nodes. Each of the plurality of fragments is transmitted with a caching hint to indicate whether that fragment is to be cached at the storage node. Other embodiments are also disclosed and claimed.

Claims

1. An apparatus comprising:

logic to generate a plurality of fragments of a data object to be Erasure Coded (EC) prior to storage at a storage node of a plurality of storage nodes,

wherein each of the plurality of fragments is to be transmitted with a caching hint to indicate whether that fragment is to be cached at the storage node, wherein a determination of whether to cache all of the plurality of fragments is to be based at least in part on;

a source of the data object, a destination of the data object, and a size-range based policy.

2. The apparatus of claim 1, comprising logic to determine where to store an EC fragment in a storage hierarchy of the storage node based at least in part on the caching hint.

3. The apparatus of claim 1, comprising logic to pass an eviction hint from the storage node to one or more neighboring storage nodes.

4. The apparatus of claim 1, wherein the logic is to pass an eviction hint to all storage that store the plurality of fragments.

5. The apparatus of claim 1, wherein the storage node is to comprise the logic.

6. The apparatus of claim 1, comprising logic to extract the caching hint and to add the caching hint to a write operation to a memory coupled to the storage node.

7. The apparatus of claim 1, wherein the plurality of storage nodes is to comprise a near storage node and/or a far storage node.

8. The apparatus of claim 1, wherein the plurality of storage nodes is to communicate via a network.

9. The apparatus of claim 8, wherein the network is to comprise a wired and/or a wireless network.

10. The apparatus of claim 1, wherein each of the plurality of storage nodes is to comprise one or more of:

a hard disk drive, a solid state drive, and a hybrid drive.

11. The apparatus of claim 1, wherein each of the plurality of storage nodes is to comprise volatile memory and/or non-volatile memory.

12. The apparatus of claim 11, wherein the non-volatile memory is to comprise one or more of:

nanowire memory, Ferro-electric Transistor Random Access Memory (FeTRAM), Magnetoresistive Random Access Memory (MRAM), flash memory, Spin Torque Transfer Random Access Memory (STTRAM), Resistive Random Access Memory, byte addressable 3-Dimensional Cross Point Memory, PCM (Phase Change Memory), write-in-place non-volatile memory, and volatile memory backed by a power reserve to retain data during power failure or power disruption.

13. The apparatus of claim 1, further comprising one or more of:

at least one processor, having one or more processor cores, communicatively coupled to the storage node, a battery communicatively coupled to the apparatus, or a network interface communicatively coupled to the apparatus.

14. The apparatus of claim 1, wherein the size-range based policy is to indicate that all fragments of objects whose size falls in a size range are to be cached at the storage node.

15. The apparatus of claim 1, wherein the caching hint is to be generated by a storage middleware logic.

16. A method comprising:

generating a plurality of fragments of a data object to be Erasure Coded (EC) prior to storage at a storage node of a plurality of storage nodes,

wherein each of the plurality of fragments is transmitted with a caching hint to indicate whether that fragment is to be cached at the storage node, wherein a determination of whether to cache all of the plurality of fragments is based at least in part on;

a source of the data object, a destination of the data object, and a size-range based policy.

17. The method of claim 16, further comprising determining where to store an EC fragment in a storage hierarchy of the storage node based at least in part on the caching hint.

18. The method of claim 16, further comprising passing an eviction hint from the storage node to one or more neighboring storage nodes.

19. The method of claim 16, further comprising extracting the caching hint and to add the caching hint to a write operation to a memory coupled to the storage node.

20. The method of claim 16, further comprising the plurality of storage nodes communicating via a network.

21. The method of claim 16, further comprising generating the caching hint at a storage middleware logic.

22. One or more non-transitory computer-readable medium comprising one or more instructions that when executed on at least one processor configure the at least one processor to perform one or more operations to:

generate a plurality of fragments of a data object to be Erasure Coded (EC) prior to storage at a storage node of a plurality of storage nodes,

wherein each of the plurality of fragments is to be transmitted with a caching hint to indicate whether that fragment is to be cached at the storage node, wherein a determination of whether to cache all of the plurality of fragments is to be based at least in part on;

a source of the data object, a destination of the data object, and a size-range based policy.

23. The one or more non-transitory computer-readable medium of claim 22, further comprising one or more instructions that when executed on the processor configure the processor to perform one or more operations to determine where to store an EC fragment in a storage hierarchy of the storage node based at least in part on the caching hint.

24. The one or more non-transitory computer-readable medium of claim 22, further comprising one or more instructions that when executed on the processor configure the processor to perform one or more operations to pass an eviction hint from the storage node to one or more neighboring storage nodes.

25. The one or more non-transitory computer-readable medium of claim 22, further comprising one or more instructions that when executed on the processor configure the processor to perform one or more operations to generate the caching hint at a storage middleware logic.