Data duplication detection in an in memory data grid (IMDG)

• US 9,613,121 B2
• Filed: 03/10/2014
• Issued: 04/04/2017
• Est. Priority Date: 03/10/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Berg, Douglas, Gaur, Nitin, Johnson, Christopher D., Martin, Brian K.
• Primary Examiner(s): LIAO, JASON G
• Application Number:
  US14/202,070
• Publication Number:
  US 20150254266A1
• Time in Patent Office: 1,121 Days
• Field of Search: None
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

Embodiments of the invention provide a method, system and computer program product for data duplication detection in an in memory data grid (IMDG). A method for data duplication detection in an IMDG includes computing a hash value for each binary data value in a key value pair of a partition in an IMDG. The method also includes generating a map including an entry for each unique computed hash value and one or more keys corresponding to binary data values of respective key value pairs from which the hash value had been uniquely computed. Thereafter, only those hash values in the map with multiple keys associated therewith are identified and binary data corresponding to the multiple keys of the identified hash values are reported as potential duplicate data in the IMDG.

Claims

1. A data processing system configured for data duplication detection in an in memory data grid (IMDG), the system comprising:

a host computing system comprising one or more computers each with memory and at least one processor;

at least one server communicatively coupled to the host computing system over a computer communications network, the at least one server hosting in memory an IMDG;

an IMDG interface executing in the memory of the host computing system and providing access to the IMDG; and

,

a data duplication detection module executing in the memory of the host computing system, the module comprising program code enabled upon execution to compute a hash value for each binary data value in a key value pair of a partition in the IMDG, to generate a map including an entry for each unique computed hash value and one or more keys corresponding to binary data values of respective key value pairs from which the hash value had been uniquely computed, to identify only those hash values in the map with multiple keys associated therewith, and to report binary data corresponding to the multiple keys of the identified hash values as potential duplicate data in the IMDG.

2. The system of claim 1, wherein the program code of the module is further enabled to remove the multiple keys of the identified hash value from the partition of the IMDG.

3. The system of claim 1, further comprising an agent in the partition of the IMDG, wherein the computing and generating is performed by the agent wherein the identifying and reporting is performed by a client of the interface to the IMDG.

4. The system of claim 1, further comprising multiple different agents, each disposed in a correspondingly different partition of the IMDG wherein the computing and generating is performed separately and independently by the different agents in the respectively different partitions of the IMDG and wherein the identifying and reporting is performed by a client of the interface to the IMDG.

5. The system of claim 1, further comprising multiple different agents, each disposed in a correspondingly different partition of different IMDGs hosted by the at least one server, wherein the computing and generating is performed separately and independently by the different agents in the respectively different partitions of the different IMDGs and wherein the identifying and reporting is performed by a coordinating client of the different IMDGs.

6. A computer program product for data duplication detection in an in memory data grid (IMDG), the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to perform a method comprising:

computing a hash value for each binary data value in a key value pair of a partition in an IMDG;

generating a map including an entry for each unique computed hash value and one or more keys corresponding to binary data values of respective key value pairs from which the hash value had been uniquely computed;

identifying only those hash values in the map with multiple keys associated therewith; and

,

reporting binary data corresponding to the multiple keys of the identified hash values as potential duplicate data in the IMDG.

7. The computer program product of claim 6, further comprising removing the multiple keys of the identified hash value from the partition of the IMDG.

8. The computer program product of claim 6, wherein the computing and generating is performed by an agent in the partition of the IMDG and wherein the identifying and reporting is performed by a client of an interface to the IMDG.

9. The computer program product of claim 6, wherein the computing and generating is performed separately and independently by different agents in respectively different partitions of the IMDG and wherein the identifying and reporting is performed by a client of an interface to the IMDG.

10. The computer program product of claim 6, wherein the computing and generating is performed separately and independently by different agents in respectively different partitions of different IMDGs and wherein the identifying and reporting is performed by a coordinating client of the different IMDGs.