Method and system for implementing parallel transformations of records

• US 9,104,662 B2
• Filed: 08/08/2008
• Issued: 08/11/2015
• Est. Priority Date: 08/08/2008
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Shi, Jia, Lee, Jonghyun, Hansen, Roger, Srihari, Vinay H
• Primary Examiner(s): Hwa, Shyue Jiunn
• Application Number:
  US12/188,521
• Publication Number:
  US 20100036861A1
• Time in Patent Office: 2,559 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor

Abstract

An improved approach is described for implementing transformations of data records in high concurrency environments. Each transformation is performed in parallel at the source when the data record is first generated. According to one approach for data integrity validation, record generators compute an integrity checksum for a newly generated record before copying into a data unit in shared memory. Subsequent generators may aggregate integrity checksums for data records into checksums for data units incrementally. This approach achieves end-to-end protection of data records against corruption using an efficient method of maintaining verifiable data integrity. In another approach, compression and encryption data transformations may be performed by themselves, or in combination with an integrity checksum transformation.

Claims

1. A computer implemented method for transforming data records, comprising:

using at least one processor that executes one or more processing entities and is configured or programmed for performing a process, the process comprising;

identifying a data unit;

generating a first record and a second record to place into the data unit;

generating multiple checksums in parallel for the first record and the second record, wherein after generating the first and second records but before respective placement of the first and second records in the data unit;

generating a first checksum from the first record, and a second checksum from the second record, in which the first and second checksums are generated in parallel for the first and second records to be stored within the data unit;

storing the first and second checksums in a holding data structure configured to hold a plurality of checksums;

persistently storing the first record and the second record in the data unit; and

combining the multiple checksums into a combined checksum, wherein the combined checksum corresponds to an aggregate checksum for the data unit that comprises at least the first checksum and the second checksum from the holding data structure, and wherein the first record and the second record are persistently stored in the data unit with the aggregate checksum for the data unit.

2. The computer implemented method of claim 1 in which the data records comprise database log records.

3. The computer implemented method of claim 1 in which the first and second checksums are additive or commutative.

4. The computer implemented method of claim 1 in which the first and second checksums provide data integrity checks.

5. The computer implemented method of claim 1 in which a data record undergoes compression or encryption.

6. The computer implemented method of claim 1 in which a data record undergoes at least two of encryption;

compression; and

checksum.

7. The computer implemented method of claim 1 in which, when the aggregate checksum does not have a pre-existing value, a previously stored checksum is copied to the aggregate checksum.

8. The computer implemented method of claim 1 in which a previously stored checksum is aggregated with a pre-existing checks=value of the aggregate checksum for the data unit.

9. The computer implemented method of claim 1 in which the aggregate checksum for the data unit is stored within a header for the data unit.

10. The computer implemented method of claim 1 in which the first and second checksums are aggregated.

11. The computer implemented method of claim 10 in which a later record generator aggregates the first and second checksums.

12. The computer implemented method of claim 10 in which the first checksum is generated by a first record generator, the second checksum is generated by a second record generator, and the first and second checksums are aggregated by another entity.

13. The computer implemented method of claim 1 in which the first and second checksums are retained in un-aggregated states.

14. The computer implemented method of claim 1, the process further comprising:

determining integrity of content of the data unit using at least the first checksum or the second checksum before storing the content of the data unit in a persistent storage.

15. The computer implemented method of claim 1, in which the generating the first record to place into the data unit comprises:

invoking a strand lock to reserve a first part of a strand in the data unit, to ensure exclusive use of the first part of the strand in the data unit for the first record; and

the generating the second record to place into the data unit comprises;

invoking the strand lock to reserve a second part of the strand or of another strand in the data unit, to ensure exclusive use of the second part of the strand or of the another strand in the data unit for the second record.

16. The computer implemented method of claim 1, in which storing the first and second checksums in the holding data structure comprises:

identifying a slot in the holding data structure for the first checksum or the second checksum;

determining whether the slot includes a previously stored checksum;

aggregating the previously stored checksum into the aggregated checksum for the data unit; and

storing the first checksum or the second checksum in the slot.

17. A system for transforming data records, comprising:

at least one processor that is at least to;

identify a data unit;

generate a first record and a second record to place into the data unit;

generating multiple checksums in parallel for the first record and the second record, wherein after the first and second records are generated but before respective placement of the first and second records in the data unit;

generate a first checksum from the first record, and a second checksum from the second record, in which the first and second checksums are generated in parallel for the first and second records to be stored within the data unit;

store the first and second checksums in a holding data structure configured to hold a plurality of checksums;

persistently store the first record and the second record in the data unit; and

combining the multiple checksums into a combined checksum, wherein the combined checksum corresponds to an aggregate checksum for the data unit that comprises at least the first checksum and the second checksum from the holding data structure, and wherein the first record and the second record are persistently stored in the data unit with the aggregate checksum for the data unit.

18. The system of claim 17 in which the data records comprise database log records.

19. The system of claim 17 in which the first and second checksums are additive or commutative.

20. The system of claim 17 in which the first and second checksums provide data integrity checks.

21. The system of claim 17 in which a data record undergoes compression or encryption.

22. The system of claim 17 in which a data record undergoes at least two of encryption, compression, and checksum.

23. The system of claim 17 in which, when the aggregate checksum does not have a pre-existing value, a previously stored checksum is copied to the aggregate checksum.

24. The system of claim 17 in which a previously stored checksum is aggregated with a pre-existing checksum value of the aggregate checksum for the data unit.

25. The system of claim 17 in which the aggregate checksum for the data unit is stored within a header for the data unit.

26. The system of claim 17 in which the first and second checksums are aggregated.

27. The system of claim 26 in which a later record generator aggregates the first and second checksums.

28. The system of claim 26 in which the first checksum is generated by a first record generator, the second checksum is generated by a second record generator, and the first and second checksums are aggregated by another entity.

29. The system of claim 17 in which the first and second checksums are retained in un-aggregated states.

30. A computer program product that includes a non-transitory computer readable storage medium, the non-transitory computer readable storage medium comprising a plurality of computer instructions which, when executed by at least one processor, cause the at least one processor to perform a method for transforming data records, the method comprising:

using the at least one processor that is configured or programmed to implement a process, the process comprising;

identifying a data unit;

generating a first record and a second record to place into the data unit;

generating multiple checksums in parallel for the first record and the second record, wherein after generating the first and second records but before respective placement of the first and second records in the data unit;

generating a first checksum from the first record, and a second checksum from the second record, in which the first and second checksums are generated in parallel for the first and second records to be stored within the data unit;

storing the first and second checksums in a holding data structure configured to hold a plurality of checksums;

persistently storing the first record and the second record in the data unit; and

combining the multiple checksums into a combined checksum, wherein the combined checksum corresponds to an aggregate checksum for the data unit that comprises at least the first checksum and the second checksum from the holding data structure, and wherein the first record and the second record are persistently stored in the data unit with the aggregate checksum for the data unit.

31. The computer program product of claim 30 in which the data records comprise database log records.

32. The computer program product of claim 30 in which the first and second checksums are additive or commutative.

33. The computer program product of claim 30 in which the first and second checksums provide data integrity checks.

34. The computer program product of claim 30 in which a data record undergoes compression or encryption.

35. The computer program product of claim 30 in which a data record undergoes at least two of encryption, compression, and checksum.

36. The computer program product of claim 30 in which, when the aggregate checksum does not have a pre-existing value, a previously stored checksum is copied to the aggregate checksum.

37. The computer program product of claim 30 in which a previously stored checksum is aggregated with a pre-existing checksum value of the aggregate checksum for the data unit.

38. The computer program product of claim 30 in which the aggregate checksum for the data unit is stored within a header for the data unit.

39. The computer program product of claim 30 in which the first and second checksums are aggregated.

40. The computer program product of claim 39 in which a later record generator aggregates the first and second checksums.

41. The computer program product of claim 39 in which the first checksum is generated by a first record generator, the second checksum is generated by a second record generator, and the first and second checksums are aggregated by another entity.

42. The computer program product of claim 30 in which the first and second checksums are retained in un-aggregated states.