ADAPTABLE DATA CACHING MECHANISM FOR IN-MEMORY CLUSTER COMPUTING

• US 20160170882A1
• Filed: 05/14/2015
• Published: 06/16/2016
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: CHOI, Inseok Stephen, KI, Yang Seok
• Granted Patent: US 10,176,097 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 12/0806 Multiuser, multiprocessor or multiprocessing cache systems
• CPC Class Codes: G06F 12/0842 for multiprocessing or multitasking
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/0871 Allocation or management of cache space
• CPC Class Codes: G06F 2212/1021 Hit rate improvement
• CPC Class Codes: G06F 2212/1024 Latency reduction
• CPC Class Codes: G06F 2212/283 Plural cache memories
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 2212/286 Mirrored cache memory
• CPC Class Codes: G06F 2212/601 Reconfiguration of cache memory
• CPC Class Codes: G06F 2212/608 Details relating to cache mapping
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

An in-memory cluster computing framework node is described. The node includes storage devices having various priorities. The node also includes a resource monitor to monitor the operation of the storage devices. The node also includes a resource scheduler. When the resource monitor indicates that a storage device is at or approaching saturation, the resource scheduler can migrate data from that storage device to another storage device of lower priority.

Claims

1. An in-memory cluster computing framework node (305), comprising:

a processor (315);

a first storage device (220, 225, 230, 235, 130) storing cached data, the first storage device (220, 225, 230, 235, 130) having a first priority (240, 245, 250, 255, 260);

a second storage device (220, 225, 230, 235, 130) having a second priority (240, 245, 250, 255, 260);

a resource monitor (205) operative to monitor the first storage device (220, 225, 230, 235, 130); and

a resource scheduler (210) operative to migrate the cached data from the first storage device (220, 225, 230, 235, 130) to the second storage device (220, 225, 230, 235, 130) if the resource monitor (205) indicates that the first storage device (220, 225, 230, 235, 130) is saturated.

2. An in-memory cluster computing framework node (305) according to claim 1, wherein the resource monitor (205) is operative to determine the capabilities of the first storage device (220, 225, 230, 235, 130).

3. An in-memory cluster computing framework node (305) according to claim 1, wherein the resource scheduler (210) is operative to select the first storage device (220, 225, 230, 235, 130) to initially cache the data based on information provided by an application that uses the data.

4. An in-memory cluster computing framework node (305) according to claim 1, wherein:

the first priority (240, 245, 250, 255, 260) is higher than the second priority (240, 245, 250, 255, 260); and

the resource scheduler (210) is operative to select the first storage device (220, 225, 230, 235, 130) to initially cache the data as a higher priority (240, 245, 250, 255, 260) device.

5. An in-memory cluster computing framework node (305) according to claim 4, wherein the resource scheduler is operative to select the second storage device (220, 225, 230, 235, 130) for future data caching if the resource monitor (205) indicates that the first storage device (220, 225, 230, 235, 130) is saturated.

6. An in-memory cluster computing framework node (305) according to claim 1, further comprising a replicator (310) to replicate the cached data on a third storage device (220, 225, 230, 235, 130).

7. An in-memory cluster computing framework node (305) according to claim 6, wherein the third storage device (220, 225, 230, 235, 130) is in a second in-memory cluster computing framework node (305).

8. An in-memory cluster computing framework node (305) according to claim 1, wherein the data includes a resilient distributed dataset (RDD) on the first storage device (220, 225, 230, 235, 130).

9. An in-memory cluster computing framework node (305) according to claim 1, wherein the resource scheduler (210) is operative to migrate all data from the first storage device (220, 225, 230, 235, 130) to the second storage device (220, 225, 230, 235, 130).

10. An in-memory cluster computing framework node (305) according to claim 1, wherein the resource scheduler (210) is operative to migrate an oldest data from the first storage device (220, 225, 230, 235, 130) to the second storage device (220, 225, 230, 235, 130).

11. A method for caching data in an in-memory cluster computing framework, comprising:

caching (510) a data on a first storage device (220, 225, 230, 235, 130) with a first priority (240, 245, 250, 255, 260) in a cluster node (305);

monitoring (520) the operation of the first storage device (220, 225, 230, 235, 130); and

if the first storage device (220, 225, 230, 235, 130) is saturated, migrating (530) the cached data to a second storage device (220, 225, 230, 235, 130) with a second priority (240, 245, 250, 255, 260)).

12. A method according to claim 11, wherein monitoring (520) the operation of the first storage device (220, 225, 230, 235, 130) includes determining (505) a capability of the first storage device (220, 225, 230, 235, 130).

13. A method according to claim 11, wherein caching (510) a data on a first storage device (220, 225, 230, 235, 130) with a first priority (240, 245, 250, 255, 260) in a cluster node (305) includes caching (510) the data on the first storage device (220, 225, 230, 235, 130) in the cluster node (305), the first storage device (220, 225, 230, 235, 130) selected by an application using the data.

14. A method according to claim 11, wherein caching (510) a data on a first storage device (220, 225, 230, 235, 130) with a first priority (240, 245, 250, 255, 260) in a cluster node (305) includes caching (510) the data on the first storage device (220, 225, 230, 235, 130) in the cluster node (305), the first storage device (220, 225, 230, 235, 130) having a higher priority (240, 245, 250, 255, 260) among a plurality of devices.

15. A method according to claim 14, further comprising, if the first storage device (220, 225, 230, 235, 130) is saturated, re-directing (535) future cache requests for the first storage device (220, 225, 230, 235, 130) in the cluster node (305) in the cluster node (305) to the second storage device (220, 225, 230, 235, 130).

16. A method according to claim 11, further comprising replicating (515) the cached data on a third storage device (220, 225, 230, 235, 130).

17. A method according to claim 16, wherein replicating (515) the cached data on a third storage device (220, 225, 230, 235, 130) includes replicating (515) the cached data on the third storage device (220, 225, 230, 235, 130) in a second cluster node (305).

18. A method according to claim 11, wherein caching (510) a data on a first storage device (220, 225, 230, 235, 130) with a first priority (240, 245, 250, 255, 260) in a cluster node (305) includes caching a resilient distributed dataset (RDD) on the first storage device (220, 225, 230, 235, 130).

19. A method according to claim 11, wherein migrating (530) the cached data to a second storage device (220, 225, 230, 235, 130) with a second priority (240, 245, 250, 255, 260) includes migrating (705) all data on the first storage device (220, 225, 230, 235, 130) to the second storage device (220, 225, 230, 235, 130).

20. A method according to claim 11, wherein migrating (530) the cached data to a second storage device (220, 225, 230, 235, 130) with a second priority (240, 245, 250, 255, 260) includes migrating (710) an oldest data on the first storage device (220, 225, 230, 235, 130) to the second storage device (220, 225, 230, 235, 130).