Distributed stream processing in the Cloud
First Claim
1. A computing device in a network-based computing environment, comprising:
- at least one processor;
at least one memory connected to the at least one processor; and
a distributed stream processing system stored in the at least one memory and executed by the at least one processor, comprisinga streaming job manager that monitors execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, the streaming job manager receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs,wherein the plurality of vertices includes at least one stream extractor vertex configured to consume events of one or more event streams, including continually waiting for and performing computations on data received in the one or more event streams; and
the streaming job manager is configured todetect, based on the monitored execution information, a failed vertex of the plurality of vertices, andrestart the failed vertex.
2 Assignments
0 Petitions
Accused Products
Abstract
A low-latency cloud-scale computation environment includes a query language, optimization, scheduling, fault tolerance and fault recovery. An event model can be used to extend a declarative query language so that temporal analysis of event of an event stream can be performed. Extractors and outputters can be used to define and implement functions that extend the capabilities of the event-based query language. A script written in the extended query language can be translated into an optimal parallel continuous execution plan. Execution of the plan can be orchestrated by a streaming job manager which schedules vertices on available computing machines. The streaming job manager can monitor overall job execution. Fault tolerance can be provided by tracking execution progress and data dependencies in each vertex. In the event of a failure, another instance of the failed vertex can be scheduled. An optimal recovery point can be determined based on checkpoints and data dependencies.
-
Citations
20 Claims
-
1. A computing device in a network-based computing environment, comprising:
-
at least one processor; at least one memory connected to the at least one processor; and a distributed stream processing system stored in the at least one memory and executed by the at least one processor, comprising a streaming job manager that monitors execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, the streaming job manager receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs, wherein the plurality of vertices includes at least one stream extractor vertex configured to consume events of one or more event streams, including continually waiting for and performing computations on data received in the one or more event streams; and the streaming job manager is configured to detect, based on the monitored execution information, a failed vertex of the plurality of vertices, and restart the failed vertex. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method in a distributed stream processing system implemented in at least one computing device, comprising
monitoring execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, said monitoring including receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs, the plurality of vertices including a stream extractor vertex; -
consuming events of one or more event streams at the stream extractor vertex, said consuming including continually waiting for data received in the one or more event streams, and performing computations on the data; detecting, based on the monitored execution information, a failed vertex of the plurality of vertices, and restarting the failed vertex. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processing circuit, perform a method for distributed stream processing, the method comprising:
-
monitoring execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, said monitoring including receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs, the plurality of vertices including a stream extractor vertex; and consuming events of one or more event streams at the stream extractor vertex, said consuming including continually waiting for data received in the one or more event streams, and performing computations on the data; detecting, based on the monitored execution information, a failed vertex of the plurality of vertices, and restarting the failed vertex. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification