Key-break and record-loop processing in parallel data transformation

• US 8,793,280 B2
• Filed: 06/13/2012
• Issued: 07/29/2014
• Est. Priority Date: 05/27/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: McClean, Geoffrey M., Pu, Xiaoyan
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US13/495,626
• Publication Number:
  US 20120259875A1
• Time in Patent Office: 776 Days
• Field of Search: 707600-899
• US Class Current: 707/791
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 9/4494 data driven

Abstract

Embodiments of the invention provide a method and apparatus for providing additional functionality to a data processing program. This is achieved by various means, including preprocessing records in a data volume, designating certain records with a key-break, and creating an aggregation structure that user programs may use to store previously-processed records from the data volume.

Claims

1. A method of processing data in a pipelined processing system, comprising:

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group;

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record; and

adding, in the first stage of the pipelined processing system, a second record from the plurality of records to an aggregation structure.

2. The method of claim 1, wherein the step of performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record comprises:

retrieving the second record from the aggregation structure; and

performing a transformation operation on the first record based on both the first record and the second record, to create a transformed record.

3. The method of claim 1, wherein the second record is added to the aggregation structure only when one or more predetermined conditions are satisfied.

4. The method of claim 1, further comprising:

writing the transformed record to a target data volume.

5. The method of claim 4, further comprising:

upon determining the key-break flag is set for the first record, writing a target key-break flag for the transformed record to the target data volume.

6. A non-transitory computer-readable storage medium containing a program which, when executed, performs an operation, comprising:

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group;

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record; and

adding, in the first stage of the pipelined processing system, a second record from the plurality of records to an aggregation structure.

7. A non-transitory computer-readable storage medium of claim 6, wherein the step of performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record comprises:

retrieving the second record from the aggregation structure; and

performing a transformation operation on the first record based on both the first record and the second record, to create a transformed record.

8. A non-transitory computer-readable storage medium of claim 7, wherein the second record is added to the aggregation structure only when one or more predetermined conditions are satisfied.

9. A non-transitory computer-readable storage medium of claim 6, the operation further comprising:

writing the transformed record to a target data volume.

10. A non-transitory computer-readable storage medium of claim 9, further comprising:

upon determining the key-break flag is set for the first record, writing a target key-break flag for the transformed record to the target data volume.

11. A system, comprising:

a computer processor; and

a memory containing a program that, when executed on the computer processor, performs an operation for processing data in a pipelined processing system, comprising;

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group;

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record; and

adding, in the first stage of the pipelined processing system, a second record from the plurality of records to an aggregation structure.

12. The system of claim 11, wherein the step of performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record comprises:

retrieving the second record from the aggregation structure; and

performing a transformation operation on the first record based on both the first record and the second record, to create a transformed record.

13. The system of claim 11, wherein the second record is added to the aggregation structure only when one or more predetermined conditions are satisfied.

14. The system of claim 11, further comprising:

writing the transformed record to a target data volume.

15. The system of claim 14, further comprising:

upon determining the key-break flag is set for the first record, writing a target key-break flag for the transformed record to the target data volume.