System and method of optimization of in-memory data grid placement

• US 9,405,589 B2
• Filed: 06/06/2011
• Issued: 08/02/2016
• Est. Priority Date: 06/06/2011
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Gaur, Nitin, Bhogal, Kulvir S., Johnson, Christopher D., Kaplinger, Todd E., Berg, Douglas C.
• Primary Examiner(s): Dudek, Jr., Edward
• Assistant Examiner(s): Verderamo, III, Ralph A
• Application Number:
  US13/154,294
• Publication Number:
  US 20120311295A1
• Time in Patent Office: 1,884 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0674 Disk device
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

A method, system, and program product for optimizing distribution and availability of data partitions is disclosed. Placement logic is run for data partition distribution which optimizes a configuration of a memory data grid. A performance impact of adding new members to the memory data grid is provided. Replication of data partitions in the memory data grid is minimized. Optimum data partition distribution in the memory data grid is determined.

Claims

1. A method for optimizing data partition distribution in a memory data grid which includes a plurality of data partitions and a plurality of containers which contain the data partitions, comprising:

running placement logic for optimizing configuration of the memory data grid;

providing performance impact of adding new members to the memory data grid for adding available space to the memory data grid;

determining a number of containers for the plurality of partitions based on network latency;

for a primary partition found in a container to be removed, converting a replica of the found primary partition to the found primary partition in a container other than the container to be removed, wherein a total number of containers is based on the determined optimum number of containers;

creating a replica partition of the converted replica partition in a different one of the containers from the converted replica partition;

for a replica partition found in the container to be removed, creating the found replica partition in one of the containers other than the container to be removed, wherein the created replica partition is not in a container that contains a primary partition corresponding to the created replica partition; and

removing the container to be removed;

determining an optimal data partition distribution in the memory data grid based on network topology;

creating policies for deciding whether synchronization of the optimal data partition distribution should be synchronous or asynchronous;

reducing invalidation traffic by placing the plurality of data partitions with the plurality of grid containers in the memory data grid; and

dynamically changing a data partition synchronization to an optimal data partition synchronization based on one of the policies.

2. The method of claim 1, wherein the method includes minimizing internal communication among members of the memory data grid.

3. The method of claim 1, wherein the optimum partition distribution in the memory data grid is determined by observing network speed and latency, frequency of membership changes detected by the memory data grid, and frequency of data partition access.

4. The method of claim 3, wherein the optimum data partition distribution in the memory data grid is determined by observing existing desired quality of service goals, and internal communication between members of the memory data grid.

5. The method of claim 1, wherein optimizing data partition distribution in the memory data grid includes analyzing data partition usage patterns and co-locating data partition data most frequently accessed.

6. The method of claim 1, wherein optimizing data partition distribution in the memory data grid includes placing replica data partitions in a plurality of different data centers.

7. The method of claim 1, wherein optimizing data partition distribution in the memory data grid includes data partition placement that reduces group service communications between members of the memory data grid.

8. A computer program product for optimizing data partition distribution in a memory data grid which includes a plurality of data partitions and a plurality of containers which contain the data partitions, the computer program product comprising a

non-transitory computer readable storage medium having computer readable code embodied therewith, the computer readable program code being operable to:

run placement logic for optimizing configuration of the memory data grid;

provide performance impact of adding new members to the memory data grid for adding available space to the memory data grid;

determine a number of containers for the plurality of data partitions based on network latency;

for a primary partition found in a container to be removed, convert a replica of the found primary partition to the found primary partition in a container other than the container to be removed, wherein a total number of containers is based on the determined optimum number of containers;

create a replica partition of the converted replica partition in a different one of the containers from the converted replica partition;

for a replica partition found in the container to be removed, create the found replica partition in one of the containers other than the container to be removed, wherein the created replica partition is not in a container that contains a primary partition corresponding to the created replica partition;

remove the container to be removed;

determine an optimal data partition distribution in the memory data grid based on network topology;

create policies for deciding whether synchronization of the optimal data partition distribution should be synchronous or asynchronous;

reduce invalidation traffic by placing the plurality of data partitions with the plurality of grid containers in the memory data grid; and

dynamically change a data partition synchronization to an optimal data partition synchronization based on one of the policies.

9. The computer program product of claim 8, including computer readable program code that minimizes internal communication among members of the memory data grid.

10. The computer program product of claim 8, wherein the optimal data partition distribution in the memory data grid is determined by observing network speed and latency, frequency of membership changes detected by the memory data grid, and frequency of data partition access.

11. The computer program product of claim 10, wherein the optimal data partition distribution in the memory data grid is determined by observing existing desired quality of service goals, and internal communication between members of the memory data grid.

12. The computer program product of claim 8, wherein the optimizing data partition distribution in the memory data grid includes moving one of the plurality of data partitions from one container to another container in the memory data grid.

13. The computer program product of claim 8, wherein the optimizing data partition distribution in the memory data grid includes analyzing data partition usage patterns and co-locating data partition data most frequently accessed.

14. The computer program product of claim 8, wherein optimizing data partition distribution in the memory data grid includes placing replica data partitions in a plurality of different data centers.

15. The computer program product of claim 8, wherein optimizing data partition distribution in the memory data grid includes data partition placement that reduces group service communications between members of the memory data grid.

16. A system comprising:

a computer including a computer processor for optimizing data partition distribution in a memory data grid which includes a plurality of data partitions and a plurality of containers which contain the data partitions; and

a data partition optimization engine for synchronizing the plurality of data partitions, the data partition optimization engine being operable to;

run placement logic for optimizing configuration of the memory data grid;

provide performance impact of adding new members to the memory data grid for adding available space to the memory data grid;

determine a number of containers for the plurality of data partitions based on network latency;

for a primary partition found in a container to be removed, convert a replica of the found primary partition to the found primary partition in a container other than the container to be removed, wherein a total number of containers is based on the determined optimum number of containers;

create a replica partition of the converted replica partition in a different one of the containers from the converted replica partition;

for a replica partition found in the container to be removed, create the found replica partition in one of the containers other than the container to be removed, wherein the created replica partition is not in a container that contains a primary partition corresponding to the created replica partition; and

remove the container to be removed;

determine an optimal data partition distribution in the memory data grid based on network topology;

create policies for deciding whether synchronization of the optimal data partition distribution should be synchronous or asynchronous;

reduce invalidation traffic by placing the plurality of data partitions with the plurality of grid containers in the memory data grid; and

dynamically change a data partition synchronization to an optimal data partition synchronization based on one of the policies.

17. The system of claim 16, wherein determining optimum partition distribution in the memory data grid includes a policy driven by a service level agreement with grid transaction response criteria.

18. The system of claim 16, wherein determining optimum data partition distribution in the memory data grid includes a data partition distribution algorithm which ensures that primary and replica data partitions are not in a same container.

19. The system of claim 16, wherein determining optimum data partition distribution in the memory data grid includes monitoring data partition placements and grid membership changes.

20. The system of claim 16, wherein determining optimum data partition distribution in the memory data grid includes monitoring transaction times with a performance management interface.

21. The system of claim 16, wherein determining optimum data partition distribution in the memory data grid includes analyzing usage patterns of the data partitions.

22. The system of claim 16, determining optimum data partition distribution in the memory data grid includes determining whether optimal replication of data partitions is synchronous or asynchronous.