FAULT TOLERANT LISTENER REGISTRATION IN THE PRESENCE OF NODE CRASHES IN A DATA GRID

• US 20160124817A1
• Filed: 01/29/2015
• Published: 05/05/2016
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Markus, Mircea, Burns, William Rosenquist
• Granted Patent: US 9,652,339 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2043 where the redundant components share a common memory address space
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06Q 20/102 Bill distribution or payments
• CPC Class Codes: G06Q 20/4016 : involving fraud or risk level assessment in transaction processing
• CPC Class Codes: G06Q 40/02 : Banking, e.g. interest calculation or account maintenance credit or loans G06Q40/03

Abstract

A processing device to perform operations comprising receive, from a listener of a second node in a data grid system, a filter defined by search criteria of a search query. The operations can include determine, at the first node, that a third node in the data grid has crashed. The operations can further include iterate over backup data of the third node that is stored at a memory of the first node to determine the backup data that matches the filter. The operations can further include communicate, to the listener, the backup data that matches the filter.

Claims

1. A non-transitory computer readable storage medium having instructions that, when executed by a processing device of a first node, cause the processing device to perform operations comprising:

receive, from a listener of a second node in a data grid system, a filter defined by search criteria of a search query;

determine, at the first node, that a third node in the data grid has crashed;

iterate, by the processing device of the first node, over backup data of the third node that is stored at a memory of the first node to determine the backup data that matches the filter; and

communicate, to the listener, the backup data that matches the filter.

2. The non-transitory computer readable storage medium of claim 1, the operations further comprising communicate, to the listener, a duplicate data notification, wherein the duplicate data notification indicates that the communicated backup data may comprise, at least in part, data duplicative to filtered data communicated to the listener from the third node.

3. The non-transitory computer readable storage medium of claim 1, the operations further comprising communicate, to the listener, data from a memory coupled to the processing device that matches the filter.

4. The non-transitory computer readable storage medium of claim 1, wherein the receive, determine, iterate, and communicate operations occur during an iteration period of the search query.

5. The non-transitory computer readable storage medium of claim 1, wherein the first node, the second node, and the third node are part of a node cluster in the data grid system.

6. The non-transitory computer readable storage medium of claim 1, the operations further comprising remove, at the first node, filters received from the third node when the processing device determines that the third node has crashed.

7. The non-transitory computer readable storage medium of claim 1, wherein the operation of determining that the third node has crashed further comprises:

receive, from the third node, heartbeat messages at periodic intervals, wherein each heartbeat message indicates that the third node is active;

determine that a heartbeat message has not been received at the first node within a threshold period of time;

communicate, to a fourth node in the data grid, a check node request, wherein the check node request requests that the fourth node verify that the third node has crashed;

receive, from the fourth node, a crash notification when the third node has not responded to the check node request within a threshold period of time; and

determine that the third node has crashed in view of the crash notification.

8. The non-transitory computer readable storage medium of claim 1, the operations further comprising rebuild a queue of modifications for the third node when the third node has crashed.

9. The non-transitory computer readable storage medium of claim 8, the operations further comprising communicating the rebuilt queue to the listener.

10. A system comprising:

a memory to store data; and

a processor coupled to the memory, the processor to perform operations comprising;

receiving, from a listener of a second node in a data grid system, a filter defined by search criteria of a search query;

determining, at the first node, that a third node in the data grid has crashed;

rebuilding, at the first node, a queue of changes to data in a memory of the third node when the third node has crashed; and

communicating, to the listener, the rebuilt queue of changes to the data of the third node.

11. The system of claim 10, the operations further comprising communicate, to the listener, data from a memory coupled to the processing device that matches the filter.

12. The system of claim 10, the operations further comprising:

iterate over, at the first node, backup data of the third node that is stored at the memory to determine the backup data that matches the filter; and

communicate, to the listener, the backup data that matches the filter.

13. The system of claim 10, wherein the rebuilt queue of changes to the data of the third node occurs during an iteration period of the search query.

14. The system of claim 10, the operations further comprising record, into a queue, modifications to the memory of the first node during an iteration period.

15. The system of claim 14, the operations further comprising communicating the queue of the first node to the listener.

16. The system of claim 10, wherein the operation of determining that the third node has crashed further comprises:

receiving, from the third node, heartbeat messages at periodic intervals, wherein each heartbeat message indicates that the third node is active;

determining that a heartbeat message has not been received at the first node within a threshold period of time;

communicating, to a fourth node in the data grid, a check node request, wherein the check node request requests that the fourth node verify that the third node has crashed;

receiving, from the fourth node, a crash notification when the third node has not responded to the check node request within a threshold period of time; and

determining that the third node has crashed in view of the crash notification.

17. A method comprising:

communicating, from a listener of a first node in a data grid, a data filter to a second node and a third node in a data grid system;

receiving, at the listener, data from a memory of the second node that matches the data filter;

receiving, at the listener, backup data from a memory of the third node that matches the data filter;

receiving, at the listener, a duplicate data notification from the third node indicating that the backup data may contain data duplicative of the data received by the listener from the second node; and

combining, by a processing device executing the listener, the data from the memory of the second node and the backup data from the memory of the third node, wherein the combining comprises adding, to the data from the memory of the second node, a portion of the backup data from the memory of the third node that does not contain the duplicative data.

18. The method of claim 17, wherein:

the backup data is data of the second node stored at the third node, and

the backup data is communicated by the third node when the second node has crashed.

19. The method of claim 17, further comprising removing a duplicative data point of the duplicative data using an idempotent function.

20. The method of claim 17, further comprising receiving, at the listener when the second node has crashed, a rebuilt queue of changes to data of the second node.