Key-break and record-loop processing in parallel data transformation

• US 8,266,181 B2
• Filed: 05/27/2010
• Issued: 09/11/2012
• Est. Priority Date: 05/27/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Pu, Xiaoyan, McClean, Geoffrey M.
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US12/789,256
• Publication Number:
  US 20110295867A1
• Time in Patent Office: 838 Days
• Field of Search: 707600-831
• US Class Current: 707/791
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 9/4494 data driven

Abstract

Embodiments of the invention provide a method and apparatus for providing additional functionality to a data processing program. This is achieved by various means, including preprocessing records in a data volume, designating certain records with a key-break, and creating an aggregation structure that user programs may use to store previously-processed records from the data volume.

Claims

1. A method of processing data in a pipelined processing system, comprising:

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting

a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group; and

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record,

wherein the step of determining whether the record is the last record in a group further comprises;

reading a first record, belonging to a first group, and a second record, belonging to a second group, from the plurality of records; and

determining whether the first group and the second group are identical, wherein the first record is the last record in the first group if the first group and the second group are different.

2. A computer-readable storage medium containing a program which, when executed, performs an operation, comprising:

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting

a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group; and

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record,

wherein the step of determining whether the record is the last record in a group further comprises;

reading a first record, belonging to a first group, and a second record, belonging to a second group, from the plurality of records; and

determining whether the first group and the second group are identical, wherein the first record is the last record in the first group if the first group and the second group are different.

3. A system, comprising:

a computer processor; and

a memory containing a program that, when executed on the computer processor, performs an operation for processing data in a pipelined processing system, comprising;

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group; and

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record,

wherein the step of determining whether the record is the last record in a group further comprises;

reading a first record, belonging to a first group, and a second record, belonging to a second group, from the plurality of records; and

determining whether the first group and the second group are identical, wherein the first record is the last record in the first group if the first group and the second group are different.