×

Stream processing with multiple connections between local and central modelers

  • US 10,579,937 B2
  • Filed: 05/12/2015
  • Issued: 03/03/2020
  • Est. Priority Date: 09/08/2014
  • Status: Active Grant
First Claim
Patent Images

1. A method performed by a distributed stream processing system comprising a plurality of computers to implement a stream processing pipeline having a plurality of stages, the method comprising:

  • implementing, by a plurality of first local modelers executing on one or more first computers of the stream processing system, a first stage of the stream processing pipeline, including;

    processing, by each first local modeler, events in an event stream received by the first local modeler according to a first transformation of the first stage of the stream processing pipeline using first machine learning model parameters of a first machine learning model,providing, by each first local modeler of one or more of the first local modelers to a first central modeler executing on one or more computers of the stream processing system that are separate from each of the one or more first computers, a respective first representative value computed from events received by the first local modeler,computing, by the first central modeler, updated machine learning model parameters of the first machine learning model using the respective first representative values computed by the one or more first local modelers, andproviding, by the first central modeler to one or more of the first local modelers, the computed updated machine learning model parameters of the first machine learning model, thereby causing the first local modelers to process events according to the first transformation of the first stage of the stream processing pipeline using the updated machine learning model parameters for the first machine learning model;

    implementing, by a plurality of second local modelers executing on one or more second computers of the stream processing system that are separate from each of the one or more first computers of the first local modelers, a second stage of the stream processing pipeline, including;

    processing, by each second local modeler, one or more processed events generated by one or more first local modelers received by the second local modeler according to a second transformation of the second stage of the stream processing pipeline using second machine learning model parameters of a second machine learning model,providing, by each second local modeler of one or more of the second local modelers to a second central modeler executing on one or more computers of the stream processing system that are separate from each of the one or more second computers, a respective second representative value computed from the processed events received by the second local modeler,computing, by the second central modeler, updated machine learning model parameters of a second machine learning model using the respective second representative values computed by each respective second local modeler from the processed events generated by the one or more second local modelers, andproviding, by the second central modeler to one or more of the second local modelers, the computed updated machine learning model parameters of the second machine learning model, thereby causing the second local modelers to process events according to the second transformation of the second stage of the stream processing pipeline using the updated machine learning model parameters for the second machine learning model.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×