×

Stream processing at scale

  • US 10,613,903 B1
  • Filed: 10/08/2018
  • Issued: 04/07/2020
  • Est. Priority Date: 03/13/2015
  • Status: Active Grant
First Claim
Patent Images

1. A stream processing system, comprising:

  • a cluster manager;

    a cluster including a plurality of cluster nodes, wherein each cluster node includes computing resources and wherein the cluster nodes are managed by the cluster manager;

    a service scheduler, wherein the service scheduler receives resource offers from the cluster manager representing computing resources available on one or more of the cluster nodes and determines resources to accept and computations to run on the accepted resources; and

    a stream processor, wherein the stream processor includes one or more jobs, wherein each job includes two or more containers, including a first container and a second container, the first container including a topology master and the second container including a stream manager and one or more stream processing system (SPS) instances, wherein each SPS instance executes, on one or more of the cluster nodes, one or more tasks from a group of tasks associated with spouts and bolts,wherein each spout transfers tuples to one or more bolts and wherein each bolt performs a computation on the transferred tuples,wherein each SPS instance includes a gateway thread and a task execution thread, wherein each gateway thread controls communication and data movement in and out of the respective SPS instance,wherein the task execution thread performs a function on data in a data stream received by the gateway thread to arrive at a solution and transfers the solution through the gateway thread to the stream manager,wherein each thread executes as a task on one or more of the cluster nodes,wherein the gateway thread and the task execution thread use a queue to buffer metrics data transferred from the task execution thread to the gateway thread, andwherein the queue is reviewed periodically to determine if a size of the queue should be increased or decreased.

View all claims
  • 4 Assignments
Timeline View
Assignment View
    ×
    ×