Method and apparatus for record pivoting to accelerate processing of data fields

• US 9,633,097 B2
• Filed: 04/23/2015
• Issued: 04/25/2017
• Est. Priority Date: 10/23/2012
• Status: Active Grant

• Current Assignee: Ip Reservoir, LLC
• Original Assignee: Ip Reservoir, LLC
• Inventors: Tidwell, Terry, St. John, Alex, Sewell, Daniel
• Primary Examiner(s): Moorthy, Aravind
• Application Number:
  US14/694,622
• Publication Number:
  US 20150310087A1
• Time in Patent Office: 733 Days
• Field of Search: 713189, 710 52
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 21/71 to assure secure computing or processing of information

Abstract

Various methods and apparatuses are described for performing high speed translations of data. In an example embodiment, record layout detection can be performed for data. In another example embodiment, data pivoting prior to field-specific data processing can be performed.

Claims

1. An apparatus comprising:

a processor;

a memory comprising a first buffer, a second buffer, and a third buffer; and

a member of the group consisting of (1) a graphics processor unit (GPU), (2) a multi-core processor, (3) a cell processor, and (4) a reconfigurable logic device;

wherein the processor is configured to (1) receive a plurality of incoming records, the incoming records comprising a plurality of fields, (2) write the incoming records to the first buffer, (3) identify which of the fields of the incoming records are to be processed by the member, and (4) write the identified fields to the second buffer;

wherein the member is configured to (1) read the fields in the second buffer, (2) perform a data processing operation on the read fields to generate a plurality of processed fields, and (3) write the processed fields to the third buffer; and

wherein the processor is further configured to (1) read the processed fields in the third buffer, (2) read the fields for the incoming records in the first buffer, and (3) merge the processed fields with the fields from the incoming records to generate a plurality of outgoing records such that the outgoing records include the processed fields in place of their corresponding fields of the incoming records.

2. The apparatus of claim 1 wherein the processor is configured to execute a first software module and a second software module;

the first software module comprising a plurality of instructions configured to cause the processor to (1) receive the incoming records, (2) write the incoming records to the first buffer, (3) identify which of the fields of the incoming records are to be processed by the member, and (4) write the identified fields to the second buffer;

the second software module comprising a plurality of instructions configured to cause the processor to (1) read the processed fields in the third buffer, (2) read the fields for the incoming records in the first buffer, and (3) merge the processed fields with the fields from the incoming records to generate the outgoing records such that the outgoing records include the processed fields in place of their corresponding fields of the incoming records.

3. The apparatus of claim 1 wherein the memory further comprises:

an input ring buffer in which the incoming records are stored; and

an output ring buffer in which the outgoing records are stored;

wherein the processor is configured to (1) receive the incoming records from the input ring buffer and (2) write the outgoing records into the output ring buffer.

4. The apparatus of claim 1 wherein the member is configured to perform an encryption operation on the fields read from the second buffer.

5. The apparatus of claim 4 wherein the encryption operation is format preserving encryption (FPE).

6. The apparatus of claim 1 wherein the second buffer comprises a pool of ingress buffers.

7. The apparatus of claim 6 wherein different fields of the incoming records are written to different ingress buffers to support multi-field processing by the member.

8. The apparatus of claim 1 wherein the third buffer comprises a pool of egress buffers.

9. The apparatus of claim 1 wherein the second buffer comprises a pool of buffers that can be selectively allocated as ingress buffers or egress buffers.

10. The apparatus of claim 1 wherein the member comprises the GPU.

11. The apparatus of claim 10 wherein the GPU is configured to perform an encryption operation on the fields read from the second buffer.

12. The apparatus of claim 11 wherein the encryption operation is format preserving encryption (FPE).

13. The apparatus of claim 10 wherein the second buffer comprises a pool of ingress buffers.

14. The apparatus of claim 13 wherein different fields of the incoming records are written to different ingress buffers to support multi-field processing by the GPU.

15. The apparatus of claim 10 wherein the memory further comprises:

an input ring buffer in which the incoming records are stored; and

an output ring buffer in which the outgoing records are stored;

wherein the processor is configured to (1) receive the incoming records from the input ring buffer and (2) write the outgoing records into the output ring buffer.

16. The apparatus of claim 10 wherein the processor is configured to execute a first software module and a second software module;

the first software module comprising a plurality of instructions configured to cause the processor to (1) receive the incoming records, (2) write the incoming records to the first buffer, (3) identify which of the fields of the incoming records are to be processed by the GPU, and (4) write the identified fields to the second buffer;

the second software module comprising a plurality of instructions configured to cause the processor to (1) read the processed fields in the third buffer, (2) read the fields for the incoming records in the first buffer, and (3) merge the processed fields with the fields from the incoming records to generate the outgoing records such that the outgoing records include the processed fields in place of their corresponding fields of the incoming records.

17. The apparatus of claim 10 wherein the processor is further configured to receive the incoming records as streaming records embodied in a data stream, the data stream comprising data arranged sequentially in a plurality of the fields, wherein each of a plurality of the fields of the streaming records has an associated field type, and wherein the processor is further configured to pivot a plurality of the identified fields of the same field type across different streaming records for grouping in the second buffer.

18. The apparatus of claim 17 wherein the GPU is further configured to perform the data processing operation on the fields read from the second buffer as a batch.

19. The apparatus of claim 18 wherein the streaming records include a plurality of social security number fields;

wherein the processor is further configured to pivot the social security number fields from a plurality of different streaming records into the second buffer; and

wherein the GPU is further configured to (1) encrypt the grouped social security number fields in the second buffer and (2) write the encrypted social security number fields into the third buffer.

20. The apparatus of claim 1 wherein the member comprises the multi-core processor.

21. The apparatus of claim 20 wherein the processor is further configured to receive the incoming records as streaming records embodied in a data stream, the data stream comprising data arranged sequentially in a plurality of the fields, wherein each of a plurality of the fields of the streaming records has an associated field type, and wherein the processor is further configured to pivot a plurality of the identified fields of the same field type across different streaming records for grouping in the second buffer.

22. The apparatus of claim 1 wherein the member comprises the cell processor.

23. The apparatus of claim 22 wherein the processor is further configured to receive the incoming records as streaming records embodied in a data stream, the data stream comprising data arranged sequentially in a plurality of the fields, wherein each of a plurality of the fields of the streaming records has an associated field type, and wherein the processor is further configured to pivot a plurality of the identified fields of the same field type across different streaming records for grouping in the second buffer.

24. The apparatus of claim 1 wherein the member comprises the reconfigurable logic device.

25. The apparatus of claim 24 wherein the reconfigurable logic device is configured to perform an encryption operation on the fields read from the second buffer.

26. The apparatus of claim 25 wherein the encryption operation is format preserving encryption (FPE).

27. The apparatus of claim 24 wherein the second buffer comprises a pool of ingress buffers.

28. The apparatus of claim 27 wherein different fields of the incoming records are written to different ingress buffers to support multi-field processing by the reconfigurable logic device.

29. The apparatus of claim 24 wherein the memory further comprises:

an input ring buffer in which the incoming records are stored; and

an output ring buffer in which the outgoing records are stored;

wherein the processor is configured to (1) receive the incoming records from the input ring buffer and (2) write the outgoing records into the output ring buffer.

30. The apparatus of claim 24 wherein the processor is configured to execute a first software module and a second software module;

the first software module comprising a plurality of instructions configured to cause the processor to (1) receive the incoming records, (2) write the incoming records to the first buffer, (3) identify which of the fields of the incoming records are to be processed by the reconfigurable logic device, and (4) write the identified fields to the second buffer;

the second software module comprising a plurality of instructions configured to cause the processor to (1) read the processed fields in the third buffer, (2) read the fields for the incoming records in the first buffer, and (3) merge the processed fields with the fields from the incoming records to generate the outgoing records such that the outgoing records include the processed fields in place of their corresponding fields of the incoming records.

31. The apparatus of claim 24 wherein the processor is further configured to receive the incoming records as streaming records embodied in a data stream, the data stream comprising data arranged sequentially in a plurality of the fields, wherein each of a plurality of the fields of the streaming records has an associated field type, and wherein the processor is further configured to pivot a plurality of the identified fields of the same field type across different streaming records for grouping in the second buffer.

32. The apparatus of claim 31 wherein the reconfigurable logic device is further configured to perform the data processing operation on the fields read from the second buffer as a batch.

33. The apparatus of claim 32 wherein the streaming records include a plurality of social security number fields;

wherein the processor is further configured to pivot the social security number fields from a plurality of different streaming records into the second buffer; and

wherein the reconfigurable logic device is further configured to (1) encrypt the grouped social security number fields in the second buffer and (2) write the encrypted social security number fields into the third buffer.