High speed processing of financial information using FPGA devices

• US 10,467,692 B2
• Filed: 10/09/2013
• Issued: 11/05/2019
• Est. Priority Date: 06/19/2006
• Status: Active Grant

• Current Assignee: Exegy Incorporated
• Original Assignee: Ip Reservoir, LLC
• Inventors: Parsons, Scott, Taylor, David E., Schuehler, David Vincent, Franklin, Mark A., Chamberlain, Roger D.
• Primary Examiner(s): Borlinghaus, Jason
• Application Number:
  US14/049,591
• Publication Number:
  US 20140040109A1
• Time in Patent Office: 2,218 Days
• Field of Search: None
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06Q 40/06 Asset management; Financial planning or analysis

Abstract

A high speed apparatus and method for processing a plurality of financial market data messages are disclosed. With respect to an exemplary embodiment, a reconfigurable logic device is employed to generate a plurality of financial market data messages from a plurality of the data fields, each generated message having a specified message format.

Claims

1. An apparatus for processing financial market data, the apparatus comprising:

a ticker plant configured to receive a financial market data feed from an exchange, wherein the financial market data comprises a plurality of streaming messages within the feed that represent a plurality of offers to buy and sell a plurality of financial instruments, the messages comprising a plurality of data values corresponding to a plurality of data fields, the ticker plant comprising a reconfigurable logic device, a processor in cooperation with the reconfigurable logic device, and a memory;

wherein the processor is configured to normalize the financial market data and deliver the normalized financial market data to the reconfigurable logic device;

wherein the memory is configured to store a data dictionary, the data dictionary comprising a plurality of entries corresponding to a message format, the entries configured to describe (1) a plurality of message fields for the message format, (2) a plurality of positions for the message fields with respect to the message format, (3) a plurality of sizes for the message fields with respect to the message format, and (4) metadata for the message fields with respect to the message format; and

wherein the reconfigurable logic device comprises firmware logic, the firmware logic configured to (1) access the memory to read the data dictionary entries, and (2) process the normalized financial market data and arrange the data values from the normalized financial market data into a plurality of the message fields in accordance with the read data dictionary entries to thereby generate from the processed normalized financial market data a plurality of financial market data messages having the message format, the generated financial market data messages comprising (i) a plurality of the arranged data values in the message fields, wherein the message fields are positioned and sized within the generated financial market data messages in accordance with the positions and sizes from the read data dictionary entries, and (ii) metadata from the read data dictionary entries corresponding to the message fields of the generated financial market data messages; and

wherein the reconfigurable logic device and the memory are configured to support generation of financial market data messages of different message formats by changing the data dictionary entries to be accessed and read by the firmware logic and without altering the firmware logic.

2. The apparatus of claim 1 wherein the metadata in the data dictionary entries comprise a field identifier and a data type.

3. The apparatus of claim 1 wherein the memory is external to the reconfigurable logic device.

4. The apparatus of claim 1 wherein the reconfigurable logic device comprises a plurality of firmware application modules (FAMs) arranged as a processing pipeline, the FAMs including a message formatter FAM and at least one FAM that is upstream from the message formatter FAM;

wherein the at least one upstream FAM is configured to receive streaming normalized financial market data and perform a data processing operation on at least a portion of the streaming normalized financial market data;

wherein the message formatter FAM is configured to (1) receive normalized financial market data that was previously processed by the at least one upstream FAM, and (2) perform the access, process, and arrange operations with respect to the received normalized financial market data to thereby generate the financial market data messages having the message format;

wherein the at least one upstream FAM is configured to operate simultaneously with the message formatter FAM in a pipelined fashion such that the message formatter FAM is configured to operate on normalized financial market data that was previously processed by the at least one upstream FAM while the at least one upstream FAM operates on different normalized financial market data.

5. The apparatus of claim 4 wherein the at least one upstream FAM comprises at least one member of the group consisting of (1) a symbol ID mapping FAM, (2) a value cache update FAM, (3) a top 10 lists generator FAM, (4) a minute bar generator FAM, (5) an interest entitlement filter FAM, and (6) a programmatic calculation engine FAM.

6. The apparatus of claim 4 wherein the processing pipeline further comprises (1) a message parsing FAM upstream from the at least one upstream FAM, and (2) a message synchronization buffer downstream from the message parsing FAM;

wherein the message parsing FAM is configured to (1) receive a normalized financial market data message, (2) parse the received normalized financial market data message into a plurality of fields of financial market data, and (3) provide at least a plurality of the parsed financial market data fields to the message synchronization buffer for buffering therein;

wherein the at least one upstream FAM is further configured to perform the data processing operation on a select subset of the parsed financial market data; and

wherein the message formatter FAM is configured to generate a plurality of the financial market data messages having the message format based on the buffered data fields in the message synchronization buffer and the processed financial market data produced by the at least one upstream FAM.

7. The apparatus of claim 4 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the processing pipeline being resident on the FPGA, the processing pipeline configured to operate at hardware processing speeds.

8. The apparatus of claim 1 wherein the message format comprises a fixed record format.

9. The apparatus of claim 1 wherein the message format comprises a FIX message format.

10. The apparatus of claim 1 wherein the message format comprises a FAST message format.

11. The apparatus of claim 1 wherein the data dictionary further comprises a plurality of entries corresponding to a plurality of different message formats, and wherein the reconfigurable logic device is further configured to access the data dictionary to determine a message format for each financial market data message that is to be generated.

12. The apparatus of claim 11 wherein the reconfigurable logic device comprises a plurality of firmware application modules (FAMs) arranged as a message transformation pipeline, the message transformation pipeline comprising a message parsing FAM and a message formatter FAM that is downstream from the message formatting FAM;

wherein the message parsing FAM is configured to (1) receive a plurality of streaming normalized financial market data messages having a first message format, (2) access the memory to read a plurality of data dictionary entries corresponding to the first message format, and (3) parse the received normalized financial market data messages into constituent fields of financial market data in accordance with the read data dictionary entries corresponding to the first message format; and

wherein the message formatter FAM is configured to (1) receive a plurality of parsed financial market data fields, and (2) perform the access, process, and arrange operations with respect to the received parsed financial market data fields and a second message format from the data dictionary to generate a plurality of financial market data messages having the second message format; and

wherein the reconfigurable logic device and the memory are configured to support processing of different message formats by changing the data dictionary entries to be accessed and read by the message parsing FAM and the message formatter FAM and without altering the message parsing FAM or the message formatter FAM.

13. The apparatus of claim 12 wherein the pipeline is further configured to validate the parsed financial market data fields.

14. The apparatus of claim 12 wherein the pipeline further comprises a message query filter downstream from the message parsing FAM and upstream from the message formatter FAM, wherein the message query filter is configured to exclude predetermined types of financial market data messages from processing by the message formatter FAM.

15. The apparatus of claim 12 wherein the first message format comprises a FIX message format.

16. The apparatus of claim 15 wherein the second message format comprises a FAST message format.

17. The apparatus of claim 12 wherein the first message format comprises a FAST message format.

18. The apparatus of claim 17 wherein the second message format comprises a FIX message format.

19. The apparatus of claim 12 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the pipeline being resident on the FPGA.

20. The apparatus of claim 12 wherein the message parsing FAM and the message formatter FAM share access to the same data dictionary.

21. The apparatus of claim 3 wherein the memory comprises at least one selected from the group consisting of an SRAM device and an SDRAM device.

45. The apparatus of claim 1 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), and wherein the firmware logic is resident on the FPGA.

52. The apparatus of claim 1 wherein the processor is further configured to (1) execute an operating system that includes a user space for a user mode and a kernel space for a kernel mode, (2) receive the financial market data feed through a network protocol stack, (3) use shared memory that is mapped into the kernel space and the user space to store the normalized financial market data, and (4) facilitate DMA transfers of the normalized financial market data from the shared memory to the reconfigurable logic device;

and

wherein the reconfigurable logic device is further configured to (1) receive normalized financial market data from the shared memory via the DMA transfers, and (2) perform the processing and arranging operations on the received normalized financial market data.

53. The apparatus of claim 52 wherein the shared memory comprises first shared memory that is mapped into the kernel space and the user space and second shared memory that is mapped into the kernel space and the user space;

wherein a first driver that executes within the kernel space of the operating system while the operating system is in the kernel mode is configured to (1) maintain a kernel level interface into the network protocol stack and (2) copy the streaming financial market data from the network protocol stack into the first shared memory without a transition to the user mode; and

wherein user mode code that executes within the user space of the operating system is configured to (1) access financial market data from the first shared memory, (2) normalize the accessed financial market data, and (3) write the normalized financial market data to the second shared memory.

54. The apparatus of claim 53 wherein a second driver that executes within the kernel space of the operating system while the operating system is in the kernel mode is configured to facilitate the DMA transfers of the normalized financial market data from the second shared memory to the reconfigurable logic device.

55. The apparatus of claim 54 wherein the user mode code comprises a plurality of threads, each thread configured for execution to normalize a different group of the accessed financial market data independently of the other threads.

56. The apparatus of claim 52 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA).

57. The apparatus of claim 52 wherein the shared memory comprises a ring buffer.

58. The apparatus of claim 1 wherein the reconfigurable logic device and the memory are configured to support generation of financial market data messages of different message formats by changing the content of the data dictionary entries to be accessed and read by the firmware logic and without altering the firmware logic.

60. The apparatus of claim 1 wherein the processor is further configured to execute software to normalize the financial market data.

22. A method for processing financial market data, the method comprising:

a ticker plant receiving a financial market data feed from an exchange, wherein the financial market data comprises a plurality of streaming messages within the feed that represent a plurality of offers to buy and sell a plurality of financial instruments, the messages comprising a plurality of data values corresponding to a plurality of data fields, the ticker plant comprising a reconfigurable logic device, a processor in cooperation with the reconfigurable logic device, and a memory;

the processor normalizing the financial market data and delivering the normalized financial market data to the reconfigurable logic device;

firmware logic deployed on the reconfigurable logic device accessing the memory to read a plurality of data dictionary entries, the accessed memory comprising a data dictionary, the data dictionary comprising a plurality of entries corresponding to a message format, the entries configured to describe (1) a plurality of message fields for the message format, (2) a plurality of positions for the message fields with respect to the message format, (3) a plurality of sizes for the message fields with respect to the message format, and (4) metadata for the message fields with respect to the message format;

the firmware logic processing the normalized financial market data and arranging the data values from the normalized financial market data into a plurality of the message fields in accordance with the read data dictionary entries to thereby generate from the processed normalized financial market data a plurality of financial market data messages having the message format, the generated financial market data messages comprising (i) a plurality of the arranged data values in the message fields, wherein the message fields are positioned and sized within the generated financial market data messages in accordance with the positions and sizes from the read data dictionary entries, and (ii) metadata from the read data dictionary entries corresponding to the message fields of the generated financial market data messages;

changing the data dictionary entries to be accessed and read by the firmware logic, wherein the changing step is performed without altering the firmware logic; and

the firmware logic repeating the accessing, processing, and arranging steps with respect to new normalized financial market data and the changed data dictionary entries to generate a plurality of new financial market data messages having a different format.

23. The method of claim 22 wherein the metadata in the data dictionary entries comprise a field identifier and a data type.

24. The method of claim 22 wherein the memory is external to the reconfigurable logic device.

25. The method of claim 22 wherein the reconfigurable logic device comprises a plurality of firmware application modules (FAMs) arranged as a processing pipeline, the FAMs including a message formatter FAM and at least one FAM that is upstream from the message formatter FAM, the method further comprising:

the at least one upstream FAM (1) receiving streaming normalized financial market data, and (2) performing a data processing operation on at least a portion of the streaming normalized financial market data;

the message formatter FAM (1) receiving normalized financial market data that was previously processed by the at least one upstream FAM, and (2) performing the accessing, processing, and arranging steps with respect to the received normalized financial market data to thereby generate the financial market data messages having the message format; and

the at least one upstream FAM operating simultaneously with the message formatter FAM in a pipelined fashion such that the message formatter FAM operates on normalized financial market data that was previously processed by the at least one upstream FAM while the at least one upstream FAM operates on different normalized financial market data.

26. The method of claim 25 wherein the at least one upstream FAM comprises at least one member of the group consisting of (1) a symbol ID mapping FAM, (2) a value cache update FAM, (3) a top 10 lists generator FAM, (4) a minute bar generator FAM, (5) an interest entitlement filter FAM, and (6) a programmatic calculation engine FAM.

27. The method of claim 25 wherein the processing pipeline further comprises (1) a message parsing FAM upstream from the at least one upstream FAM, and (2) a message synchronization buffer downstream from the message parsing FAM, the method further comprising:

the message parsing FAM (1) receiving a normalized financial market data message, (2) parsing the received normalized financial market data message into a plurality of fields of financial market data, and (3) providing at least a plurality of the parsed financial market data fields to the message synchronization buffer for buffering therein;

the at least one upstream FAM performing the data processing operation on a select subset of the parsed financial market data; and

the message formatter FAM generating a plurality of the financial market data messages having the message format based on the buffered data fields in the message synchronization buffer and the processed financial market data produced by the at least one upstream FAM.

28. The method of claim 25 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the processing pipeline being resident on the FPGA, the processing pipeline operating at hardware processing speeds.

29. The method of claim 22 wherein the message format comprises a fixed record format.

30. The method of claim 22 wherein the message format comprises a FIX message format.

31. The method of claim 22 wherein the message format comprises a FAST message format.

32. The method of claim 22 wherein the data dictionary further comprises a plurality of entries corresponding to a plurality of different message formats, the method further comprising:

the reconfigurable logic accessing the data dictionary to determine a message format for each financial market data message that is to be generated.

33. The method of claim 32 wherein the reconfigurable logic device comprises a plurality of firmware application modules (FAMs) arranged as a message transformation pipeline, the message transformation pipeline comprising a message parsing FAM and a message formatter FAM that is downstream from the message formatting FAM, the method further comprising:

the message parsing FAM (1) receiving a plurality of streaming normalized financial market data messages having a first message format, (2) accessing the memory to read a plurality of data dictionary entries corresponding to the first message format, and (3) parsing the received normalized financial market data messages into constituent fields of financial market data in accordance with the read data dictionary entries corresponding to the first message format;

the message formatter FAM (1) receiving a plurality of parsed financial market data fields, and (2) performing the accessing, processing, and arranging steps with respect to the received parsed financial market data fields and a second message format from the data dictionary to generate a plurality of financial market data messages having the second message format; and

the reconfigurable logic device and the memory supporting processing of different message formats by changing the data dictionary entries to be accessed and read by the message parsing FAM and the message formatter FAM and without altering the message parsing FAM or the message formatter FAM.

34. The method of claim 33 further comprising:

the pipeline validating the parsed financial market data fields.

35. The method of claim 33 wherein the pipeline further comprises a message query filter downstream from the message parsing FAM and upstream from the message formatter FAM, the method further comprising:

the message query filter excluding predetermined types of financial market data messages from processing by the message formatter FAM.

36. The method of claim 33 wherein the first message format comprises a FIX message format.

37. The method of claim 36 wherein the second message format comprises a FAST message format.

38. The method of claim 33 wherein the first message format comprises a FAST message format.

39. The method of claim 38 wherein the second message format comprises a FIX message format.

40. The method of claim 33 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the pipeline being resident on the FPGA.

41. The method of claim 33 wherein the message parsing FAM and the message formatter FAM share access to the same data dictionary.

42. The method of claim 24 wherein the memory comprises at least one selected from the group consisting of an SRAM device and an SDRAM device.

43. The method of claim 22 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), and wherein the firmware logic is resident on the FPGA.

44. The method of claim 22 further comprising:

changing the data dictionary entries that are accessed and read by the firmware logic such that the changed data dictionary entries define a different message format; and

repeating the accessing, processing, and arranging steps using the same firmware logic to generate financial market data messages in accordance with the different message format.

46. The method of claim 22 further comprising:

the processor executing an operating system that includes a user space for a user mode and a kernel space for a kernel mode, wherein the executing step comprises;

the processor receiving the financial market data feed through a network protocol stack;

the processor using shared memory that is mapped into the kernel space and the user space to store the normalized financial market data; and

the processor facilitating DMA transfers of the normalized financial market data from the shared memory to the reconfigurable logic device; and

the reconfigurable logic device (1) receiving normalized financial market data from the shared memory via the DMA transfers, and (2) performing the processing and arranging steps on the received normalized financial market data.

47. The method of claim 46 wherein the shared memory comprises first shared memory that is mapped into the kernel space and the user space and second shared memory that is mapped into the kernel space and the user space, and wherein the executing step further comprises:

a first driver executing within the kernel space of the operating system while the operating system is in the kernel mode, wherein the first driver executing step comprises the first driver (1) maintaining a kernel level interface into the network protocol stack and (2) copying the streaming financial market data from the network protocol stack into the first shared memory without a transition to the user mode; and

user mode code executing within the user space of the operating system, wherein the user mode code executing step comprises the user mode code (1) accessing financial market data from the first shared memory, (2) normalizing the accessed financial market data, and (3) writing the normalized financial market data to the second shared memory.

48. The method of claim 47 wherein the processor executing step further comprises:

a second driver executing within the kernel space of the operating system while the operating system is in the kernel mode, the second driver executing step comprising the second driver facilitating the DMA transfers of the normalized financial market data from the second shared memory to the reconfigurable logic device.

49. The method of claim 48 wherein the user mode code comprises a plurality of threads, each thread normalizing a different group of the accessed financial market data independently of the other threads.

50. The method of claim 46 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA).

51. The method of claim 46 wherein the shared memory comprises a ring buffer.

59. The method of claim 22 wherein the changing step comprises changing the content of the data dictionary entries to be accessed and read by the firmware logic, wherein the changing step is performed without altering the firmware logic.

61. The method of claim 22 wherein the normalizing step comprises the processor executing software to normalize the financial market data.