TRANSITIONING BETWEEN HISTORICAL AND REAL TIME DATA STREAMS IN THE PROCESSING OF DATA CHANGE MESSAGES

• US 20090070355A1
• Filed: 09/11/2007
• Published: 03/12/2009
• Est. Priority Date: 09/11/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: LOVE, Robert Daniel, KIRK, Lawrence E., VARKHEDI, Anil Venkatesh, CADARETTE, Paul Michael, WEIGEL, Charles Joseph III, UPTON, Gregg Andrew
• Granted Patent: US 7,827,299 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

Provided are techniques for data stream switching. While accepting data change messages from a historical data stream, in response to determining that the end of the historical data stream is approaching, consumption from a real time data stream is resumed. In accepting data change messages from one of a historical data stream and a real time data stream, a data change message identifier (DCMID) for a data change message is obtained. Then, it is determined whether the data change message is from the real time data stream. In response to determining that the data change message is from the real time data stream, the DCMID is processed against a historical data structure to determine whether the related data change message has been seen in the historical data stream. In response to determining that the data change message is from the historical data stream, the DCMID is processed against a real time data structure to determine whether the related data change message has been seen in the real time data stream. It is determined whether a threshold for a number of DCMIDs that correspond to each other in the real time data stream and the historical data stream has been met. In response to determining that the threshold has been met, there is a switch from capturing data change messages in the historical data stream to capturing data change messages in the real time data stream.

Claims

1. A computer-implemented method, comprising:

while accepting data change messages from one of a historical data stream and a real time data stream,

obtaining a data change message identifier (DCMID) for a data change message;

determining whether the data change message is from the real time data stream;

in response to determining that the data change message is from the real time data stream, processing the DCMID against a historical data structure to determine whether the related data change message has been seen in the historical data stream;

in response to determining that the data change message is from the historical data stream, processing the DCMID against a real time data structure to determine whether the related data change message has been seen in the real time data stream;

determining whether a threshold for a number of DCMIDs that correspond to each other in the real time data stream and the historical data stream has been met; and

in response to determining that the threshold has been met, switching from capturing data change messages in the historical data stream to capturing data change messages in the real time data stream.

2. The method of claim 1, wherein processing the DCMID using the historical data structure further comprises:

determining whether the DCMID is in the historical data structure;

in response to determining that the DCMID is in the historical data structure, removing an entry including the DCMID from the historical data structure and updating one or more counts of a number of matches in the historical data structure; and

in response to determining that the DCMID is not in the historical data structure, adding an entry including the DCMID to an end of the real time data structure.

3. The method of claim 2, wherein processing the DCMID using the real time data structure further comprises:

in response to determining that the data change message is from a historical data stream,

determining whether the DCMID is in a real time data structure;

in response to determining that the DCMID is in the real time data structure, removing an entry including the DCMID from the real time data structure and updating one or more counts of a number of matches in the real time data structure; and

in response to determining that the DCMID is not in the real time data structure, adding an entry including the DCMID to an end of historical data structure.

4. The method of claim 1, wherein the threshold comprises a data source specific threshold.

5. The method of claim 1, wherein the DCMID comprises a data source specific DCMID with a data source specific size and composition.

6. The method of claim 1, wherein the real time data structure stores real time data change messages and further comprising:

processing the real time data change messages in the real time data structure when the switching is concluded.

7. The method of claim 1, further comprising:

performing clean up by freeing resources associated with the switching, closing the historical data stream, and signaling that the switching has completed.

8. A computer program product comprising a computer useable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

while accepting data change messages from one of a historical data stream and a real time data stream,

obtain a data change message identifier (DCMID) for a data change message;

determine whether the data change message is from a real time data stream;

in response to determining that the data change message is from the real time data stream, process the DCMID against a historical data structure to determine whether the related data change message has been seen in the historical data stream;

in response to determining that the data change message is from the historical data stream, process the DCMID against a real time data structure to determine whether the related data change message has been seen in the real time data stream;

determine whether a threshold for a number of DCMIDs that correspond to each other in the real time data stream and the historical data stream has been met; and

in response to determining that the threshold has been met, switch from capturing data change messages in the historical data stream to capturing data change messages in the real time data stream.

9. The computer program product of claim 8, wherein when processing the DCMID using the historical data structure, the computer readable program when executed on a computer causes the computer to:

determine whether the DCMID is in the historical data structure;

in response to determining that the DCMID is in the historical data structure, remove an entry including the DCMID from the historical data structure and updating one or more counts of a number of matches in the historical data structure; and

in response to determining that the DCMID is not in the historical data structure, add an entry including the DCMID to an end of the real time data structure.

10. The computer program product of claim 9, wherein when processing the DCMID using the real time data structure, the computer readable program when executed on a computer causes the computer to:

in response to determining that the data change message is from a historical data stream,

determine whether the DCMID is in a real time data structure;

in response to determining that the DCMID is in the real time data structure, remove an entry including the DCMID from the real time data structure and updating one or more counts of a number of matches in the real time data structure; and

in response to determining that the DCMID is not in the real time data structure, add an entry including the DCMID to an end of the historical data structure.

11. The computer program product of claim 8, wherein the threshold comprises a data source specific threshold.

12. The computer program product of claim 8, wherein the DCMID comprises a data source specific DCMID with a data source specific size and composition.

13. The computer program product of claim 8, wherein the real time data structure stores real time data change messages and wherein the computer readable program when executed on a computer causes the computer to:

process the real time data change messages in the real time data structure when the switching is concluded.

14. The computer program product of claim 8, wherein the computer readable program when executed on a computer causes the computer to:

perform clean up by freeing resources associated with the switching, closing the historical data stream, and signaling that the switching has completed.

15. A system, comprising:

logic capable of performing operations, the operations comprising;

while accepting data change messages from one of a historical data stream and a real time data stream,

obtaining a data change message identifier (DCMID) for a data change message;

determining whether the data change message is from a real time data stream;

in response to determining that the data change message is from the real time data stream, processing the DCMID against a historical data structure to determine whether the related data change message has been seen in the historical data stream;

in response to determining that the data change message is from the historical data stream, processing the DCMID against a real time data structure to determine whether the related data change message has been seen in the real time data stream;

determining whether a threshold for a number of DCMIDs that correspond to each other in the real time data stream and the historical data stream has been met; and

in response to determining that the threshold has been met, switching from capturing data change messages in the historical data stream to capturing data change messages in the real time data stream.

16. The system of claim 15, wherein operations for processing the DCMID using the historical data structure further comprise:

determining whether the DCMID is in the historical data structure;

in response to determining that the DCMID is in the historical data structure, removing an entry including the DCMID from the historical data structure and updating one or more counts of a number of matches in the historical data structure; and

in response to determining that the DCMID is not in the historical data structure, adding an entry including the DCMID to an end of the real time data structure.

17. The system of claim 16, wherein operations for processing the DCMID using the real time data structure further comprise:

in response to determining that the data change message is from a historical data stream,

determining whether the DCMID is in a real time data structure;

in response to determining that the DCMID is in the real time data structure, removing an entry including the DCMID from the real time data structure and updating one or more counts of a number of matches in the real time data structure; and

in response to determining that the DCMID is not in the real time data structure, adding an entry including the DCMID to an end of the historical data structure.

18. The system of claim 15, wherein the threshold comprises a data source specific threshold.

19. The system of claim 15, wherein the DCMID comprises a data source specific DCMID with a data source specific size and composition.

20. The system of claim 15, wherein the real time data structure stores real time data change messages and the operations further comprise:

processing the real time data change messages in the real time data structure when the switching is concluded.

21. The system of claim 15, wherein the operations further comprise:

performing clean up by freeing resources associated with the switching, closing the historical data stream, and signaling that the switching has completed.