Checkpointing in distributed streaming platform for real-time applications

• US 9,298,788 B1
• Filed: 06/26/2013
• Issued: 03/29/2016
• Est. Priority Date: 03/11/2013
• Status: Expired due to Fees

• Current Assignee: DataTorrent, Inc.
• Original Assignee: DataTorrent, Inc.
• Inventors: Narsude, Chetan, Kekre, Amol J., Hoang, Phu D., Weise, Thomas
• Primary Examiner(s): Vy, Hung T
• Application Number:
  US13/928,351
• Time in Patent Office: 1,007 Days
• Field of Search: 707/682, 707/674, 707/718, 707/713, 707/756, 717/140, 717/1, 714/15, 714/16, 714/1
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/55 Push-based network services

Abstract

Software for a distributed streaming platform receives a data stream for an application running on a distributed streaming platform over a networked cluster of servers. The software converts the data into a plurality of data tuples structured according to a schema. And the software repeatedly emits a specified number of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window and by a trailing control tuple associated with the same ordinal identifier. Then the software emits a checkpointing tuple following the trailing control tuple after a specified number of streaming windows. The checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance.

Claims

1. A method, comprising:

receiving a data stream for an application running on a distributed streaming platform over a networked cluster of servers;

converting the data into a plurality of data tuples structured according to a schema;

repeatedly emitting a specified number of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window and by a trailing control tuple associated with the same ordinal identifier, wherein the streaming window is an atomic sequence of tuples that is associated with a recovery policy; and

emitting a checkpointing tuple following the trailing control tuple after a specified number of streaming windows, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.

2. The method of claim 1, wherein the recovery policy determines at least in part the specified number of streaming windows.

3. The method of claim 1, wherein the recovery policy is one of at least once or exactly once.

4. The method of claim 1, wherein checkpointing includes serializing the instance to persistent storage.

5. The method of claim 4, wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo.

6. The method of claim 1, wherein the data stream is accessed through a web Application Programming Interface.

7. The method of claim 1, wherein the application uses one or more stream modes from the group of stream modes consisting of in-line, in-node, and in-rack.

8. The method of claim 7, wherein the GUI is dashboard for decision support.

9. The method of claim 1, wherein the data tuples are displayed in a graphical user interface (GUI) following removal of the control tuples.

10. One or more computer-readable media persistently storing one or more programs, wherein the one or more programs, when executed, instructs one or more processors to perform the following operations:

receive a data stream for an application running on a distributed streaming platform over a networked cluster of servers;

convert the data into a plurality of data tuples structured according to a schema;

repeatedly emit a specified number of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window and by a trailing control tuple associated with the same ordinal identifier, wherein the streaming window is an atomic sequence of tuples that is associated with a recovery policy; and

emit a checkpointing tuple following the trailing control tuple after a specified number of streaming windows, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed in real time or near real time rather than offline.

11. The computer-readable media of claim 10, wherein the recovery policy determines at least in part the specified number of streaming windows.

12. The computer-readable media of claim 10, wherein the recovery policy is one of at least once or exactly once.

13. The computer-readable media of claim 10, wherein checkpointing includes serializing the instance to persistent storage.

14. The computer-readable media of claim 10, wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo.

15. The computer-readable media of claim 10, wherein the data stream is accessed through a web Application Programming Interface.

16. The computer-readable media of claim 10, wherein the application uses one or more stream modes from the group of stream modes consisting of in-line, in-node, and in-rack.

17. The computer-readable media of claim 10, wherein the data tuples are displayed in a graphical user interface (GUI) following removal of the control tuples.

18. The computer-readable media of claim 10, wherein the GUI is dashboard for decision support.

19. A method, comprising:

receiving a data stream for an application running on a distributed streaming platform over a networked cluster of servers, wherein the application uses one or more stream modes from the group of stream modes consisting of in-line, in-node, and in-rack;

converting the data into a plurality of data tuples structured according to a schema;

repeatedly emitting a specified number of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window and by a trailing control tuple associated with the same ordinal identifier, wherein the streaming window is an atomic sequence of tuples that is associated with a recovery policy; and

emitting a checkpointing tuple following the trailing control tuple after a specified number of streaming windows, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.

20. The method of claim 19, wherein checkpointing includes serializing the instance to persistent storage.