Removing sets of tuples in a streaming environment

• US 10,545,931 B2
• Filed: 06/11/2015
• Issued: 01/28/2020
• Est. Priority Date: 04/17/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barsness, Eric L., Branson, Michael J., Santosuosso, John M.
• Primary Examiner(s): Gofman, Alex
• Assistant Examiner(s): Mian, Umar
• Application Number:
  US14/736,428
• Publication Number:
  US 20160306828A1
• Time in Patent Office: 1,692 Days
• Field of Search: None
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/248 Presentation of query results

Abstract

A stream of tuples is received to be processed by a plurality of processing elements operating on one or more computer processors. Each processing element has one or more stream operators. A first stream operator adds a set number to a first tuple in the stream of tuples. The first tuple having the set number indicates that the first tuple is part of a set of tuples. Each tuple in the set of tuples contains the set number. It is determined to delete the set of tuples, and a set of stream operators deletes tuples containing the set number.

Claims

1. A method for processing a stream of tuples, the method comprising:

receiving a stream of tuples to be processed by a plurality of processing elements operating on one or more computer processors, each processing element having one or more stream operators;

adding, by a first stream operator, a first specific set number, from a plurality of set numbers, to a first tuple in the stream of tuples in response to the first tuple containing one or more attributes associated with an entity, wherein the first tuple further contains metadata in addition to the one or more attributes, wherein the first tuple having the set number indicates that the first tuple is part of a set of tuples, and wherein each tuple in the set of tuples contains the set number;

changing, by a second stream operator and for the first tuple, the first set number to a second set number;

receiving the specific set number from an application administrator via a user interface; and

deleting, by each of a plurality of stream operators and in response to the receiving the specific set number from an application administrator via a user interface, a plurality of tuples based on each of the plurality of tuples containing the set number, wherein the plurality of stream operators includes a third stream operator of a first processing element, the third stream operator configured to delete tuples containing the set number after normal processing of the tuples by the second stream operator and prior to communicating the tuples to a next operator, and wherein the plurality of stream operators includes a fourth stream operator of a second processing element, the fourth stream operator configured to delete tuples containing the set number prior to normal processing by the fourth stream operator, and wherein each of the plurality of stream operators maintains an overall count of tuples deleted and a count at a set level of tuples deleted.