Fault tolerant listener registration in the presence of node crashes in a data grid

• US 9,652,339 B2
• Filed: 01/29/2015
• Issued: 05/16/2017
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Burns, William Rosenquist, Markus, Mircea
• Primary Examiner(s): GUYTON, PHILIP A
• Application Number:
  US14/609,329
• Publication Number:
  US 20160124817A1
• Time in Patent Office: 838 Days
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2043 where the redundant components share a common memory address space
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06Q 20/102 Bill distribution or payments
• CPC Class Codes: G06Q 20/4016 : involving fraud or risk level assessment in transaction processing
• CPC Class Codes: G06Q 40/02 : Banking, e.g. interest calculation or account maintenance credit or loans G06Q40/03

Abstract

A processing device to perform operations comprising receive, from a listener of a second node in a data grid system, a filter defined by search criteria of a search query. The operations can include determine, at the first node, that a third node in the data grid has crashed. The operations can further include iterate over backup data of the third node that is stored at a memory of the first node to determine the backup data that matches the filter. The operations can further include communicate, to the listener, the backup data that matches the filter.

Claims

1. A non-transitory computer readable storage medium having instructions that, when executed by a processing device of a first node, cause the processing device to:

receive, from a listener of a second node in a data grid system, a filter defined by search criteria of a search query;

determine, at the first node, that a third node in the data grid has crashed;

iterate, by the processing device of the first node, over backup data of the third node that is stored at a memory of the first node to determine the backup data that matches the filter; and

communicate, to the listener, the backup data that matches the filter and a duplicate data notification, wherein the duplicate data notification indicates that the backup data may comprise, at least in part, data duplicative to filtered data communicated to the listener from the third node.

2. The non-transitory computer readable storage medium of claim 1, the processing device further to communicate, to the listener, data from a memory coupled to the processing device that matches the filter.

3. The non-transitory computer readable storage medium of claim 1, wherein the processing device is to receive, determine, iterate, and communicate during an iteration period of the search query.

4. The non-transitory computer readable storage medium of claim 1, wherein the first node, the second node, and the third node are part of a node cluster in the data grid system.

5. The non-transitory computer readable storage medium of claim 1, the processing device further to remove, at the first node, filters received from the third node when the processing device determines that the third node has crashed.

6. The non-transitory computer readable storage medium of claim 1, wherein to determine that the third node has crashed the processing device is further to:

receive, from the third node, heartbeat messages at periodic intervals, wherein each heartbeat message indicates that the third node is active;

determine that a heartbeat message of the heartbeat messages has not been received at the first node within a threshold period of time;

communicate, to a fourth node in the data grid, a check node request, wherein the check node request requests that the fourth node verify that the third node has crashed;

receive, from the fourth node, a crash notification when the third node has not responded to the check node request within the threshold period of time; and

determine that the third node has crashed in view of the crash notification.

7. The non-transitory computer readable storage medium of claim 1, the processing device further to rebuild a queue of modifications for the third node when the third node has crashed.

8. The non-transitory computer readable storage medium of claim 7, the processing device further to communicate the rebuilt queue to the listener.

9. A system comprising:

a memory to store data; and

a processor coupled to the memory, the processor to;

receive, at a first node in a data grid system, a filter from a listener of a second node in the data grid system, the filter defined by search criteria of a search query;

maintain, at the first node, a queue indicating modifications to the memory of a third node in the data grid system;

determine, at the first node, that the third node in the data grid has crashed;

order the queue to indicate the modifications to the data in a memory of the third node that occurred during an iteration period; and

communicate, to the listener, the queue of modification to the data of the third node.

10. The system of claim 9, the processor further to communicate, to the listener, data from a memory coupled to the processing device that matches the filter.

11. The system of claim 9, wherein the processor is further to:

iterate over, at the first node, backup data of the third node that is stored at the memory to determine the backup data that matches the filter; and

communicate, to the listener, the backup data that matches the filter.

12. The system of claim 9, wherein the rebuilt queue of changes to the data of the third node occurs during the iteration period of the search query.

13. The system of claim 9, wherein the processor is further to record, into a queue, modifications to the memory of the first node during the iteration period.

14. The system of claim 13, wherein the processing device is further to communicate the queue of the first node to the listener.

15. The system of claim 9, wherein to determine that the third node has crashed the processing device is further to:

receive, from the third node, heartbeat messages at periodic intervals, wherein each heartbeat message indicates that the third node is active;

determine that a heartbeat message of the heartbeat messages has not been received at the first node within a threshold period of time;

communicate, to a fourth node in the data grid, a check node request, wherein the check node request requests that the fourth node verify that the third node has crashed;

receive from the fourth node, a crash notification when the third node has not responded to the check node request within the threshold period of time; and

determine that the third node has crashed in view of the crash notification.

16. A method comprising:

communicating, from a listener of a first node in a data grid, a data filter to a second node and a third node in a data grid system;

receiving, at the listener, data from a memory of the second node that matches the data filter;

receiving, at the listener, backup data from a memory of the third node that matches the data filter;

receiving, at the listener, a duplicate data notification from the third node indicating that the backup data may contain data duplicative of the data received by the listener from the second node; and

combining, by a processing device executing the listener, the data from the memory of the second node and the backup data from the memory of the third node, wherein the combining comprises adding, to the data from the memory of the second node, a portion of the backup data from the memory of the third node that does not contain the duplicative data.

17. The method of claim 16, wherein:

the backup data is data of the second node stored at the third node, and

the backup data is communicated by the third node when the second node has crashed.

18. The method of claim 16, further comprising removing a duplicative data point of the duplicative data using an idempotent function.

19. The method of claim 16, further comprising receiving, at the listener when the second node has crashed, a rebuilt queue of changes to data of the second node.