DATA ARRANGEMENT MANAGEMENT IN A DISTRIBUTED DATA CLUSTER ENVIRONMENT OF A SHARED POOL OF CONFIGURABLE COMPUTING RESOURCES

• US 20170371928A1
• Filed: 06/28/2016
• Published: 12/28/2017
• Est. Priority Date: 06/28/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chainani, Naresh K., Cho, James H.
• Granted Patent: US 10,387,415 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24549 Run-time optimisation
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Disclosed aspects relate to data arrangement management in a distributed data cluster environment of a shared pool of configurable computing resources. In the distributed data cluster environment, a set of data is monitored for a data redistribution candidate trigger. The data redistribution candidate trigger is detected with respect to the set of data. Based on the data redistribution candidate trigger, the set of data is analyzed with respect to a candidate data redistribution action. Using the candidate data redistribution action, a new data arrangement associated with the set of data is determined. Accordingly, the new data arrangement is established.

Claims

1. A computer-implemented method for data arrangement management in a distributed data cluster environment of a shared pool of configurable computing resources, the method comprising:

monitoring, in the distributed data cluster environment, a set of data for a data redistribution candidate trigger;

detecting, in the distributed data cluster environment, the data redistribution candidate trigger with respect to the set of data;

analyzing, based on the data redistribution candidate trigger, the set of data with respect to a candidate data redistribution action;

determining, using the candidate data redistribution action, a new data arrangement associated with the set of data; and

establishing, in the distributed data cluster environment, the new data arrangement.

2. The method of claim 1, wherein the set of data includes a set of diagnostic metadata for the distributed data cluster environment.

3. The method of claim 1, wherein detecting the data redistribution candidate trigger includes:

detecting a data skew of a data structure which exceeds a threshold data skew value.

4. The method of claim 1, wherein detecting the data redistribution candidate trigger includes:

detecting a data structure which indicates a workload pattern.

5. The method of claim 1, wherein detecting the data redistribution candidate trigger includes:

detecting a data structure which exceeds a data transmission frequency threshold.

6. The method of claim 3, wherein the candidate data redistribution action includes:

a new distribution key for the data structure to reduce the data skew of the data structure.

7. The method of claim 4, wherein the candidate data redistribution action includes:

a new distribution key for the data structure to change the workload pattern to reduce data movement during a query operation.

8. The method of claim 5, wherein the candidate data redistribution action includes:

deploying the data structure which exceeds the data transmission frequency threshold to at least a threshold number of partitions in the distributed data cluster environment.

9. The method of claim 1, wherein the new data arrangement positively impacts a query performance metric when running a query in the distributed data cluster environment.

10. The method of claim 4, wherein the workload pattern includes a join on a common column.

11. The method of claim 4, wherein the workload pattern includes a set of query accesses which use a join key column other than a current distribution key column.

12. The method of claim 1, wherein the candidate data redistribution action includes a new distribution key which is range-based.

13. The method of claim 1, wherein the candidate data redistribution action includes a new distribution key which is hash-based.

14. The method of claim 1, further comprising:

presenting, to a user in response to determining the new data arrangement, a selection with respect to whether to initiate establishment of the new data arrangement; and

receiving, from the user in advance of establishing the new data arrangement, the selection to initiate establishment of the new data arrangement.

15. The method of claim 1, wherein the monitoring, the detecting, the analyzing, the determining, and the establishing each occur in an automated fashion without user intervention.

16. The method of claim 15, further comprising:

repeating the data arrangement management in a continuous fashion to adapt to a workload change.

17. The method of claim 1, further comprising:

metering use of the data arrangement management; and

generating an invoice based on the metered use.

18. A system for data arrangement management in a distributed data cluster environment of a shared pool of configurable computing resources, the system comprising:

a memory having a set of computer readable computer instructions, and

a processor for executing the set of computer readable instructions, the set of computer readable instructions including;

monitoring, in the distributed data cluster environment, a set of data for a data redistribution candidate trigger;

detecting, in the distributed data cluster environment, the data redistribution candidate trigger with respect to the set of data;

analyzing, based on the data redistribution candidate trigger, the set of data with respect to a candidate data redistribution action;

determining, using the candidate data redistribution action, a new data arrangement associated with the set of data; and

establishing, in the distributed data cluster environment, the new data arrangement.

19. A computer program product for data arrangement management in a distributed data cluster environment of a shared pool of configurable computing resources, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:

monitoring, in the distributed data cluster environment, a set of data for a data redistribution candidate trigger;

detecting, in the distributed data cluster environment, the data redistribution candidate trigger with respect to the set of data;

analyzing, based on the data redistribution candidate trigger, the set of data with respect to a candidate data redistribution action;

determining, using the candidate data redistribution action, a new data arrangement associated with the set of data; and

establishing, in the distributed data cluster environment, the new data arrangement.

20. The computer program product of claim 19, wherein at least one of:

the program instructions are stored in the computer readable storage medium in a data processing system, and wherein the program instructions were downloaded over a network from a remote data processing system;

or

the program instructions are stored in the computer readable storage medium in a server data processing system, and wherein the program instructions are downloaded over a network to the remote data processing system for use in a second computer readable storage medium with the remote data processing system.