Removing sets of tuples in a streaming environment

• US 10,579,603 B2
• Filed: 04/17/2015
• Issued: 03/03/2020
• Est. Priority Date: 04/17/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barsness, Eric L., Branson, Michael J., Santosuosso, John M.
• Primary Examiner(s): Gofman, Alex
• Assistant Examiner(s): Mian, Umar
• Application Number:
  US14/689,087
• Publication Number:
  US 20160306842A1
• Time in Patent Office: 1,782 Days
• Field of Search: None
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/248 Presentation of query results

Abstract

A stream of tuples is received to be processed by a plurality of processing elements operating on one or more computer processors. Each processing element has one or more stream operators. A first stream operator adds a set number to a first tuple in the stream of tuples. The first tuple having the set number indicates that the first tuple is part of a set of tuples. Each tuple in the set of tuples contains the set number. It is determined to delete the set of tuples, and a set of stream operators deletes tuples containing the set number.

Claims

1. A system for processing a stream of tuples, the system comprising:

a plurality of processing elements to receive a stream of tuples, each processing element having one or more stream operators;

two or more processors; and

a memory containing an application that, when executed, causes at least one of the two or more processors to perform a method comprising;

adding, by a first stream operator, a first set number, from a plurality of set numbers, to a first tuple in the stream of tuples in response to the first tuple containing one or more attributes associated with an entity, wherein the first tuple further contains metadata in addition to the one or more attributes, wherein the first tuple having the first set number indicates that the first tuple is part of a set of tuples, and wherein each tuple in the set of tuples contains the first set number;

changing, by a second stream operator and for the first tuple, the first set number to a second set number;

receiving a specific set number from an application administrator via a user interface; and

deleting, by each of a plurality of stream operators and in response to the receiving the specific set number from an application administrator via a user interface, a plurality of tuples based on each of the plurality of tuples containing the specific set number, wherein the plurality of stream operators includes a third stream operator of a first processing element, the third stream operator configured to delete tuples containing the specific set number after normal processing of the tuples by the third stream operator and prior to communicating the tuples to a next operator, and wherein the plurality of stream operators includes a fourth stream operator of a second processing element, the fourth stream operator configured to delete tuples containing the specific set number prior to normal processing of the tuples by the fourth stream operator, and wherein each of the plurality of stream operators maintains an overall count of tuples deleted and a count at a set level of tuples deleted.

2. The system of claim 1, wherein the first stream operator created the first tuple.

3. The system of claim 1, wherein the first set number is added in response to the first tuple containing a plurality of attributes.

4. A computer program product for processing a stream of tuples, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by a computing device to cause the device to perform a method comprising:

receiving a stream of tuples to be processed by a plurality of processing elements operating on one or more computer processors, each processing element having one or more stream operators;

adding, by a first stream operator, a first set number, from a plurality of set numbers, to a first tuple in the stream of tuples in response to the first tuple containing one or more attributes associated with an entity, wherein the first tuple further contains metadata in addition to the one or more attributes, wherein the first tuple having the first set number indicates that the first tuple is part of a set of tuples, and wherein each tuple in the set of tuples contains the first set number;

changing, by a second stream operator and for the first tuple, the first set number to a second set number;

receiving a specific set number from an application administrator via a user interface; and

deleting, by each of a plurality of stream operators and in response to the receiving the specific set number from an application administrator via a user interface, a plurality of tuples based on each of the plurality of tuples containing the specific set number, wherein the plurality of stream operators includes a third stream operator of a first processing element, the third stream operator configured to delete tuples containing the specific set number after normal processing of the tuples by the third stream operator and prior to communicating the tuples to a next operator, and wherein the plurality of stream operators includes a fourth stream operator of a second processing element, the fourth stream operator configured to delete tuples containing the specific set number prior to normal processing of the tuples by the fourth stream operator, and wherein each of the plurality of stream operators maintains an overall count of tuples deleted and a count at a set level of tuples deleted.

5. The computer program product of claim 4, wherein the first stream operator created the first tuple.

6. The computer program product of claim 4, wherein the first set number is added in response to the first tuple containing a plurality of attributes.

7. A system for processing a stream of tuples, the system comprising:

a plurality of compute nodes communicatively coupled to each other over one or more communications networks, each compute node comprising one or more processors;

a plurality of processing elements to receive a stream of tuples, each of the plurality of processing elements executing on one of the plurality of compute nodes, each processing element having one or more stream operators, wherein the stream of tuples are communicated between the one or more stream operators as defined in an operator graph; and

a memory containing an application that, when executed, causes at least one of the plurality of compute nodes to perform a method comprising;

adding, by a first stream operator, a first set number, from a plurality of set numbers, to a first tuple in the stream of tuples in response to the first tuple containing one or more attributes associated with an entity, wherein the first tuple further contains metadata in addition to the one or more attributes, wherein the first tuple having the first set number indicates that the first tuple is part of a set of tuples, and wherein each tuple in the set of tuples contains the first set number;

changing, by a second stream operator and for the first tuple, the first set number to a second set number;

receiving a specific set number from an application administrator via a user interface; and

deleting, by each of a plurality of stream operators and in response to the receiving the specific set number from an application administrator via a user interface, a plurality of tuples based on each of the plurality of tuples containing the specific set number, wherein the plurality of stream operators includes a third stream operator of a first processing element, the third stream operator configured to delete tuples containing the specific set number after normal processing of the tuples by the third stream operator and prior to communicating the tuples to a next operator, and wherein the plurality of stream operators includes a fourth stream operator of a second processing element, the fourth stream operator configured to delete tuples containing the specific set number prior to normal processing of the tuples by the fourth stream operator, and wherein each of the plurality of stream operators maintains an overall count of tuples deleted and a count at a set level of tuples deleted.