ADAPTIVE CACHING AND DYNAMIC DELAY SCHEDULING FOR IN-MEMORY DATA ANALYTICS

• US 20180232153A1
• Filed: 04/11/2018
• Published: 08/16/2018
• Est. Priority Date: 12/29/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chiu, Lawrence Y., Saxena, Mohit, Xu, Erci
• Granted Patent: US 10,678,481 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0655 Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

In various embodiments, computer-implemented techniques for improving function of in-memory processing systems via adaptively caching datasets include: identifying data stored in a distributed filesystem, the data including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application; identifying one or more partitions of the data corresponding to the data to be processed; and selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application. Corresponding systems and computer-program products are also disclosed.

Claims

1. A computer-implemented method for adaptively caching datasets in an in-memory processing system, the method comprising:

identifying data stored in a distributed filesystem, including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application;

identifying one or more partitions of the data, the one or more partitions corresponding to the data to be processed by the in-memory processing application; and

selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application.

2. The computer-implemented method of claim 1, comprising deserializing the one or more partitions.

3. The computer-implemented method of claim 1, wherein the identifying the data stored in the distributed filesystem;

and the identifying the one or more partitions are each based at least in part on analyzing semantics of the in-memory processing application.

4. The computer-implemented method of claim 3, wherein analyzing the semantics of the in-memory processing application comprises identifying one or more application calls to the data stored in the distributed filesystem.

5. The computer-implemented method of claim 4, comprising providing the semantics to an adaptive caching manager of the in-memory processing system.

6. The computer-implemented method of claim 1, comprising selectively transferring the one or more partitions from the memory of the in-memory processing application to the distributed filesystem based on analyzing semantics of the in-memory processing application.

7. The computer-implemented method of claim 6, comprising serializing the one or more partitions.

8. An in-memory processing system for adaptively caching datasets, comprising:

a processor; and

logic executable by the processor to cause the processor to;

identify data stored in a distributed filesystem, the data including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application;

identify one or more partitions of the data, the one or more partitions corresponding to the data to be processed by the in-memory processing application; and

selectively transfer the one or more partitions from the distributed filesystem to a memory of the in-memory processing application.

9. The in-memory processing system of claim 8, comprising logic executable by the processor to cause the processor to:

selectively deserialize the one or more partitions; and

selectively serialize the one or more partitions.

10. The in-memory processing system of claim 8, wherein each of:

identifying the data stored in the distributed filesystem; and

identifying the one or more partitions;

are based at least in part on analyzing semantics of the in-memory processing application.

11. The in-memory processing system of claim 10, wherein analyzing the semantics of the in-memory processing application comprises identifying one or more application calls to the data stored in the distributed filesystem.

12. The in-memory processing system of claim 11, comprising logic executable by the processor to cause the in-memory processing application to provide the semantics to an adaptive caching manager of the in-memory processing system.

13. The in-memory processing system of claim 8, comprising logic executable by the processor to cause the processor to selectively transfer the one or more partitions from the memory of the in-memory processing application to the distributed filesystem based on analyzing semantics of the in-memory processing application.

14. A computer program product for adaptively caching datasets in an in-memory processing system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, and the program instructions are executable by a processor, to cause the processor to perform a method comprising:

identifying data stored in a distributed filesystem, including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application;

identifying one or more partitions of the data, the one or more partitions corresponding to the data to be processed by the in-memory processing application; and

selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application.

15. The computer program product of claim 14, comprising program instructions executable by the processor, to cause the processor to:

selectively deserialize the one or more partitions.

16. The computer program product of claim 14, wherein the identifying the data stored in the distributed filesystem;

and the identifying the one or more partitions are each based at least in part on analyzing semantics of the in-memory processing application.

17. The computer program product of claim 16, wherein analyzing the semantics of the in-memory processing application comprises identifying one or more application calls to the data stored in the distributed filesystem.

18. The computer program product of claim 17, comprising program instructions executable by the processor, to cause the processor to:

provide the semantics to an adaptive caching manager of the in-memory processing system.

19. The computer program product of claim 14, comprising program instructions executable by the processor, to cause the processor to:

selectively transfer the one or more partitions from the memory of the in-memory processing application to the distributed filesystem based on analyzing semantics of the in-memory processing application.

20. The computer program product of claim 14, comprising program instructions executable by the processor, to cause the processor to:

selectively serialize the one or more partitions.