×

RECOVERY STRATEGY FOR A STREAM PROCESSING SYSTEM

  • US 20170083396A1
  • Filed: 01/22/2016
  • Published: 03/23/2017
  • Est. Priority Date: 09/18/2015
  • Status: Active Grant
First Claim
Patent Images

1. A method of providing a fault tolerance strategy for a stream processing system, the method including:

  • defining a computing grid that consumes data from a message bus as numerous batches, wherein the message bus queues events from one or more near real-time (NRT) data streams in numerous partitions and each event logged in a given partition is assigned a unique event offset;

    before processing, in the computing grid, a first batch from a first partition, identifying a current event offset and an end event offset in the first partition and locking the first batch to include events logged between the current event offset and the end event offset;

    persisting the current event offset and the end event offset;

    detecting failed processing of the first batch and responding by reprocessing the first batch in the computing grid, wherein, between the processing and the reprocessing, the end event offset is subject to change caused by the message bus queuing new events in the first partition; and

    restricting the reprocessing of the first batch to events logged between the current event offset and the end event offset, thereby preventing inclusion of the new events in the first batch.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×