DISTRIBUTED STREAM PROCESSING IN THE CLOUD

• US 20170339202A1
• Filed: 04/07/2017
• Published: 11/23/2017
• Est. Priority Date: 07/01/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhou, Jingren, Qian, Zhengping, Zabback, Peter, Lin, Wei
• Granted Patent: US 10,225,302 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

A low-latency cloud-scale computation environment includes a query language, optimization, scheduling, fault tolerance and fault recovery. An event model can be used to extend a declarative query language so that temporal analysis of event of an event stream can be performed. Extractors and outputters can be used to define and implement functions that extend the capabilities of the event-based query language. A script written in the extended query language can be translated into an optimal parallel continuous execution plan. Execution of the plan can be orchestrated by a streaming job manager which schedules vertices on available computing machines. The streaming job manager can monitor overall job execution. Fault tolerance can be provided by tracking execution progress and data dependencies in each vertex. In the event of a failure, another instance of the failed vertex can be scheduled. An optimal recovery point can be determined based on checkpoints and data dependencies.

Claims

1. A system comprising:

at least one processor;

a memory connected to the at least one processor; and

a distributed stream processing system that supports distributed stream processing comprising

a streaming job manager that monitors execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, the streaming job manager receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs,

wherein the plurality of vertices includes at least one stream extractor vertex configured to consume events of one or more event streams, including continually waiting for and performing computations on data received in the one or more event streams.