CACHE MANAGEMENT FOR MAP-REDUCE APPLICATIONS

• US 20160062900A1
• Filed: 08/18/2015
• Published: 03/03/2016
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Liu, Liang, Qu, Junmei, Zhuang, Wei, Zhu, ChaoQiang
• Granted Patent: US 10,078,594 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0875 with dedicated cache, e.g. instruction or stack
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

A computer manages a cache for a MapReduce application based on a distributed file system that includes one or more storage medium by receiving a map request and receiving parameters for processing the map request. The parameters include a total data size to be processed, a size of each data record, and a number of map requests executing simultaneously. The computer determines a cache size for processing the map request, wherein the cache size is determined based on the received parameters for processing the map request and a machine learning model for a map request cache size and reads, based on the determined cache size, data from the one or more storage medium of the distributed file system into the cache. The computer processes the map request and writes an intermediate result data of the map request processing into the cache, based on the determined cache size.

Claims

1. A method for managing a cache for a MapReduce application on a distributed file system, the method comprising:

receiving, by a computer, a map request for a MapReduce application on a distributed file system that includes one or more storage medium;

receiving, by the computer, parameters for processing the map request, the parameters including a total data size to be processed, a size of each data record, and a number of map requests executing simultaneously;

determining, by the computer, a cache size for processing the map request, wherein the cache size is determined based on the received parameters for processing the map request and a machine learning model for a map request cache size;

reading, by the computer, based on the determined cache size, data from the one or more storage medium of the distributed file system into the cache;

processing, by the computer, the map request; and

writing, by the computer, an intermediate result data of the map request processing into the cache, based on the determined cache size.

2. The method according to claim 1, further comprising:

responsive to the received map request processing requiring an iterative computation, performing one or more times;

reading, by the computer, the intermediate result data from the cache, based on the determined cache size;

processing, by the computer, the map request; and

writing, by the computer, a second intermediate result data of the map request processing into the cache, based on the determined cache size.

3. The method according to claim 2, further comprising:

responsive to completing the iterative computation, writing, by the computer, a final computation result data of the map request processing into the cache, based on the determined cache size, for subsequent processing.

4. The method according to claim 1, wherein the machine learning model for the map request cache size is a first multiple linear regression model of a relationship between a map request processing time and the cache size in which the first multiple linear regression model is established based on historical records of a map request processed by at least one computing node, wherein the historical records of the map request include:

a processing time of the map request,

a total data size to be processed,

a size of each data record, and

a number of map requests executing simultaneously on the computing node.

5. The method according to claim 4, further comprising:

determining, by the computer, a processing time of the map request; and

correcting, by the computer, a coefficient of the first multiple linear regression model based on the determined map request processing time and the received parameters for processing the map request.

6. The method according to claim 3, further comprising:

receiving, by the computer, a reduce request for a MapReduce application on a distributed file system;

receiving, by the computer, second parameters for processing the reduce request, the second parameters including a total data size to be processed, a size of each data record, and a number of reduce requests executing simultaneously;

determining, by the computer, a second cache size for processing the reduce request, wherein the second cache size is determined based on the received second parameters and a second machine learning model for a reduce request cache size; and

reading, by the computer, the final computation result data of the map request processing from the cache, based on the determined second cache size.

7. The method according to claim 6, wherein the second machine learning model for the reduce request cache size is a second multiple linear regression model of a relationship between a reduce request processing time and a cache size, in which the second multiple linear regression model is established based on historical records of a reduce request processed by at least one computing node, wherein the historical records of the reduce request include:

a processing time of the reduce request,

a total data size to be processed,

a size of each data record, and

a number of reduce requests executing simultaneously on the computing node.

8. The method according to claim 7, further comprising:

determining, by the computer, a processing time of the reduce request; and

correcting, by the computer, a coefficient of the second multiple linear regression model based on the determined reduce request processing time and the received second parameters for processing the reduce request.

9. A computer program product for managing a cache for a MapReduce application on a distributed file system, the computer program product comprising one or more computer readable storage medium and program instructions stored on at least one of the one or more computer readable storage medium, the program instructions comprising:

program instructions to receive, by a computer, a map request for a MapReduce application on a distributed file system that includes one or more storage medium;

program instructions to receive, by the computer, parameters for processing the map request, the parameters including a total data size to be processed, a size of each data record, and a number of map requests executing simultaneously;

program instructions to determine, by the computer, a cache size for processing the map request, wherein the cache size is determined based on the received parameters for processing the map request and a machine learning model for a map request cache size;

program instructions to read, by the computer, based on the determined cache size, data from the one or more storage medium of the distributed file system into the cache;

program instructions to process, by the computer, the map request; and

program instructions to write, by the computer, an intermediate result data of the map request processing into the cache, based on the determined cache size.

10. The computer program product according to claim 9, further comprising:

responsive to the received map request processing requiring an iterative computation, performing one or more times;

program instructions to read, by the computer, the intermediate result data from the cache, based on the determined cache size;

program instructions to process, by the computer, the map request; and

program instructions to write, by the computer, a second intermediate result data of the map request processing into the cache, based on the determined cache size.

11. The computer program product according to claim 10, further comprising:

responsive to completing the iterative computation, program instructions to write, by the computer, a final computation result data of the map request processing into the cache, based on the determined cache size, for subsequent processing.

12. The computer program product according to claim 9, wherein the machine learning model for the map request cache size is a first multiple linear regression model of a relationship between a map request processing time and the cache size in which the first multiple linear regression model is established based on historical records of a map request processed by at least one computing node, wherein the historical records of the map request include:

a processing time of the map request,

a total data size to be processed,

a size of each data record, and

a number of map requests executing simultaneously on the computing node.

13. The computer program product according to claim 12, further comprising:

program instructions to determine, by the computer, a processing time of the map request; and

program instructions to correct, by the computer, a coefficient of the first multiple linear regression model based on the determined map request processing time and the received parameters for processing the map request.

14. The computer program product according to claim 11, further comprising:

program instructions to receive, by the computer, a reduce request for a MapReduce application on a distributed file system;

program instructions to receive, by the computer, second parameters for processing the reduce request, the second parameters including a total data size to be processed, a size of each data record, and a number of reduce requests executing simultaneously;

program instructions to determine, by the computer, a second cache size for processing the reduce request, wherein the second cache size is determined based on the received second parameters and a second machine learning model for a reduce request cache size; and

program instructions to read, by the computer, the final computation result data of the map request processing from the cache, based on the determined second cache size.

15. The computer program product according to claim 14, wherein the second machine learning model for the reduce request cache size is a second multiple linear regression model of a relationship between a reduce request processing time and a cache size, in which the second multiple linear regression model is established based on historical records of a reduce request processed by at least one computing node, wherein the historical records of the reduce request include:

a processing time of the reduce request,

a total data size to be processed,

a size of each data record, and

a number of reduce requests executing simultaneously on the computing node.

16. The computer program product according to claim 15, further comprising:

program instructions to determine, by the computer, a processing time of the reduce request; and

program instructions to correct, by the computer, a coefficient of the second multiple linear regression model based on the determined reduce request processing time and the received second parameters for processing the reduce request.

17. A computer system for managing a cache for a MapReduce application on a distributed file system, the computer system comprising one or more processors, one or more computer readable memories, one or more computer readable tangible storage medium, and program instructions stored on at least one of the one or more storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, the program instructions comprising:

program instructions to receive, by a computer, a map request for a MapReduce application on a distributed file system that includes one or more storage medium;

program instructions to receive, by the computer, parameters for processing the map request, the parameters including a total data size to be processed, a size of each data record, and a number of map requests executing simultaneously;

program instructions to determine, by the computer, a cache size for processing the map request, wherein the cache size is determined based on the received parameters for processing the map request and a machine learning model for a map request cache size;

program instructions to read, by the computer, based on the determined cache size, data from the one or more storage medium of the distributed file system into the cache;

program instructions to process, by the computer, the map request; and

program instructions to write, by the computer, an intermediate result data of the map request processing into the cache, based on the determined cache size.

18. The computer system according to claim 17, further comprising:

responsive to the received map request processing requiring an iterative computation, performing one or more times;

program instructions to read, by the computer, the intermediate result data from the cache, based on the determined cache size;

program instructions to process, by the computer, the map request; and

program instructions to write, by the computer, a second intermediate result data of the map request processing into the cache, based on the determined cache size.

19. The computer system according to claim 18, further comprising:

responsive to completing the iterative computation, program instructions to write, by the computer, a final computation result data of the map request processing into the cache, based on the determined cache size, for subsequent processing.

20. The computer system according to claim 17, wherein the machine learning model for the map request cache size is a first multiple linear regression model of a relationship between a map request processing time and the cache size in which the first multiple linear regression model is established based on historical records of a map request processed by at least one computing node, wherein the historical records of the map request include:

a processing time of the map request,

a total data size to be processed,

a size of each data record, and

a number of map requests executing simultaneously on the computing node.

21. The computer system according to claim 20, further comprising:

program instructions to determine, by the computer, a processing time of the map request; and

program instructions to correct, by the computer, a coefficient of the first multiple linear regression model based on the determined map request processing time and the received parameters for processing the map request.

22. The computer system according to claim 19, further comprising:

program instructions to receive, by the computer, a reduce request for a MapReduce application on a distributed file system;

program instructions to receive, by the computer, second parameters for processing the reduce request, the second parameters including a total data size to be processed, a size of each data record, and a number of reduce requests executing simultaneously;

program instructions to determine, by the computer, a second cache size for processing the reduce request, wherein the second cache size is determined based on the received second parameters and a second machine learning model for a reduce request cache size; and

program instructions to read, by the computer, the final computation result data of the map request processing from the cache, based on the determined second cache size.

23. The computer system according to claim 22, wherein the second machine learning model for the reduce request cache size is a second multiple linear regression model of a relationship between a reduce request processing time and a cache size, in which the second multiple linear regression model is established based on historical records of a reduce request processed by at least one computing node, wherein the historical records of the reduce request include:

a processing time of the reduce request,

a total data size to be processed,

a size of each data record, and

a number of reduce requests executing simultaneously on the computing node.

24. The computer system according to claim 23, further comprising:

program instructions to determine, by the computer, a processing time of the reduce request; and

program instructions to correct, by the computer, a coefficient of the second multiple linear regression model based on the determined reduce request processing time and the received second parameters for processing the reduce request.