KEY-BREAK AND RECORD-LOOP PROCESSING IN PARALLEL DATA TRANSFORMATION

• US 20110295867A1
• Filed: 05/27/2010
• Published: 12/01/2011
• Est. Priority Date: 05/27/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Pu, Xiaoyan, Mcclean, Geoffrey M.
• Granted Patent: US 8,266,181 B2
• Time in Patent Office: Days
• US Class Current: 707/756
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 9/4494 data driven

Abstract

Embodiments of the invention provide a method and apparatus for providing additional functionality to a data processing program. This is achieved by various means, including preprocessing records in a data volume, designating certain records with a key-break, and creating an aggregation structure that user programs may use to store previously-processed records from the data volume.

Claims

1. A method of processing data in a pipelined processing system, comprising:

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group; and

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record.

2. The method of claim 1, wherein the step of determining whether the record is the last record in a group further comprises:

reading a first record, belonging to a first group, and a second record, belonging to a second group, from the plurality of records; and

determining whether the first group and the second group are identical, wherein the first record is the last record in the first group if the first group and the second group are different.

3. The method of claim 1, further comprising:

reading, in the first stage of the pipelined processing system, a second record from the plurality of records; and

adding the second record to an aggregation structure.

4. The method of claim 3, wherein the step of performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record comprises:

retrieving the second record from the aggregation structure; and

performing a transformation operation on the first record based on both the first record and the second record, to create a transformed record.

5. The method of claim 3, wherein the second record is added to the aggregation structure only when one or more predetermined conditions are satisfied.

6. The method of claim 1, further comprising:

writing the transformed record to a target data volume.

7. The method of claim 6, further comprising:

upon determining the key-break flag is set for the first record, writing a target key-break flag for the transformed record to the target data volume.

8. A computer-readable storage medium containing a program which, when executed, performs an operation, comprising:

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and upon determining the record is the last record in the group,

setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group; and

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record.

9. The computer-readable storage medium of claim 8, wherein the step of determining whether the record is the last record in a group further comprises:

reading a first record, belonging to a first group, and a second record, belonging to a second group, from the plurality of records; and

determining whether the first group and the second group are identical, wherein the first record is the last record in the first group if the first group and the second group are different.

10. The computer-readable storage medium of claim 8, the operation further comprising:

reading, in the first stage of the pipelined processing system, a second record from the plurality of records; and

adding the second record to an aggregation structure.

11. The computer-readable storage medium of claim 8, wherein the step of performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record comprises:

retrieving the second record from the aggregation structure; and

performing a transformation operation on the first record based on both the first record and the second record, to create a transformed record.

12. The computer-readable storage medium of claim 11, wherein the second record is added to the aggregation structure only when one or more predetermined conditions are satisfied.

13. The computer-readable storage medium of claim 8, the operation further comprising:

writing the transformed record to a target data volume.

14. The computer-readable storage medium of claim 13, further comprising:

upon determining the key-break flag is set for the first record, writing a target key-break flag for the transformed record to the target data volume.

15. A system, comprising:

a computer processor; and

a memory containing a program that, when executed on the computer processor, performs an operation for processing data in a pipelined processing system, comprising;

preprocessing a plurality of records, comprising;

for each record of the plurality of records;

determining whether the record is a last record in a group, and

upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;

reading, in a first stage of the pipelined processing system, a first record from the plurality of records;

determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group; and

upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record.

16. The system of claim 15, wherein the step of determining whether the record is the last record in a group further comprises:

reading a first record, belonging to a first group, and a second record, belonging to a second group, from the plurality of records; and

determining whether the first group and the second group are identical, wherein the first record is the last record in the first group if the first group and the second group are different.

17. The system of claim 15, wherein the program, when executed, further performs the steps of:

reading, in the first stage of the pipelined processing system, a second record from the plurality of records; and

adding the second record to an aggregation structure.

18. The system of claim 17, wherein the step of performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record comprises:

retrieving the second record from the aggregation structure; and

performing a transformation operation on the first record based on both the first record and the second record, to create a transformed record.

19. The system of claim 17, wherein the second record is added to the aggregation structure only when one or more predetermined conditions are satisfied.

20. The system of claim 15, further comprising:

writing the transformed record to a target data volume.

21. The system of claim 20, further comprising:

upon determining the key-break flag is set for the first record, writing a target key-break flag for the transformed record to the target data volume.