System and method for low latency market data

• US 20070025351A1
• Filed: 04/24/2006
• Published: 02/01/2007
• Est. Priority Date: 06/27/2005
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Merrill Lynch & Co. Incorporated
• Inventors: Cohen, David
• Granted Patent: US 8,130,758 B2
• Time in Patent Office: Days
• US Class Current: 370/390
• CPC Class Codes: G06F 2209/544 Remote
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: H04L 67/55 Push-based network services

Abstract

A method for low latency data comprises receiving a data stream comprising data from a data source, receiving the data via one or more queues, where each of the one or more queues is associated with one or more filters adapted to filter the data based at least upon a time predicate to create filtered data. Each of the one or more filters is established by one or more subscribers. The method also comprises receiving a portion of the filtered data that was filtered by a filter established by a particular one or more subscribers, and publishing the portion of the filtered data for the particular one or more subscribers.

Claims

1. An apparatus for low latency data, comprising:

one or more feed handlers, each of said one or more feed handlers adapted to receive a data stream comprising data from a data source;

one or more queues adapted to receive said data from said one or more feed handlers, each of said one or more queues associated with one or more filters adapted to filter said data based at least upon a time predicate to create filtered data, each of said one or more filters established by one or more subscribers; and

one or more publishers, each of said one or more publishers adapted to;

receive a portion of said filtered data that was filtered by a filter established by a particular one or more subscribers; and

publish said portion of said filtered data for said particular one or more subscribers.

2. The apparatus of claim 1 wherein said particular one or more subscribers comprises an IP multicast group.

3. The apparatus of claim 1 wherein said particular one or more subscribers comprises a single subscriber.

4. The apparatus of claim 1 wherein said particular one or more subscribers comprises a plurality of subscribers.

5. The apparatus of claim 1 wherein said one or more queues and said one or more publishers comprise a plurality of independent processes coupled by an asynchronous communication mechanism.

6. The apparatus of claim 5 wherein each of said plurality of independent processes are assigned to a cluster partition.

7. The apparatus of claim 6 wherein said cluster partition is configured with cluster resources based at least in part on an advance reservation facility.

8. The apparatus of claim 1 wherein said feed handler is adapted to receive one or more direct feeds from one or more markets.

9. The apparatus of claim 1 wherein said feed handler is adapted to route market data to a cluster partition based at least in part on the symbol for an exchange.

10. The apparatus of claim 1 wherein said apparatus is further adapted to maximize data parallelism using a farmer/worker programming pattern.

11. The apparatus of claim 1 wherein said apparatus is further adapted to maximize data parallelism using a data parallelism programming pattern.

12. The apparatus of claim 1 wherein said apparatus is further adapted to maximize data parallelism using a functional/task programming pattern.

13. The apparatus of claim 1 wherein said one or more feed handlers and said one or more queues are coupled via direct connection.

14. The apparatus of claim 13 wherein said direct connection comprises a cluster interconnect of the Beowulf style.

15. The apparatus of claim 13 wherein said one or more publishers are further adapted to send a notification to said particular one or more subscribers when said one or more publishers receives said portion of said filtered market that was filtered by said filter established by said particular one or more subscribers.

16. The apparatus of claim 15 wherein said one or more publishers are further adapted to send a notification to said particular one or more subscribers via a callback mechanism.

17. The apparatus of claim 1 wherein said apparatus is further adapted to provide value-added analytics.

18. The apparatus of claim 17 wherein said apparatus is further adapted to provide said value-added analytics by:

applying a new algorithm to a metric calculation; and

making the resulting stream available to one or more subscribers.

19. The apparatus of claim 18 wherein said metric comprises VWAP (Volume Weighted Average Price).

20. The apparatus of claim 18 wherein said apparatus is further adapted to receive one or more subscriptions from said one or more subscribers, said one or more subscriptions expressed as an SQL-like query.

21. The apparatus of claim 20 wherein said one or more subscriptions are issued as continuous data stream queries which provide an asynchronous callback method.

22. The apparatus of claim 1 wherein said one or more publishers are further adapted to invoke a callback method provided by said particular one or more subscribers when said one or more publishers receives said portion of said filtered market that was filtered by said filter established by said particular one or more subscribers.

23. The apparatus of claim 21 wherein asynchronous callback method is implemented using one or more of Linux Asynchronous I/O (AIO), NET Delegate and J2EE Message-Driven Beans (MDB).

24. The apparatus of claim 21 wherein said apparatus is further adapted to maximize data parallelism, said task parallelism implemented using one or more of C/C++ using Asynchronous I/O (AIO), .NET Delegate, and J2EE.

25. The apparatus of claim 1 wherein a StreamBase™

stream-processing engine comprises said one or more queues and said one or more publishers.

26. The apparatus of claim 20 wherein said apparatus is further adapted to register said one or more subscriptions using a remote procedure call (RPC) mechanism that includes a partition identifier, said apparatus further adapted to return a subscription ID for subsequent use in subscription management.

27. The apparatus of claim 1 wherein said apparatus resides at the root of a sensor network.

28. The apparatus of claim 1 wherein said apparatus is further adapted to allocate costs to one or more consuming applications associated with said one or more subscribers, said allocation of costs being based at least in part on queries submitted by said one or more subscribers, and on the underlying resources consumed by said queries.

29. The apparatus of claim 1 wherein said one or more feed handlers, said one or more queues, and said one or more publishers share a single monotonic clock, said clock for use in synchronizing a sampling interval across said one or more queues.

30. The apparatus of claim 1 wherein said data comprises market data.

31. A method for low latency data, comprising:

receiving a data stream comprising data from a data source;

receive said data via one or more queues, each of said one or more queues associated with one or more filters adapted to filter said data based at least upon a time predicate to create filtered data, each of said one or more filters established by one or more subscribers;

receiving a portion of said filtered data that was filtered by a filter established by a particular one or more subscribers; and

publishing said portion of said filtered data for said particular one or more subscribers.

32. The method of claim 31 wherein said particular one or more subscribers comprises an IP multicast group.

33. The method of claim 31 wherein said particular one or more subscribers comprises a single subscriber.

34. The method of claim 31 wherein said particular one or more subscribers comprises a plurality of subscribers.

35. The method of claim 31 wherein said one or more queues and said one or more publishers comprise a plurality of independent processes coupled by an asynchronous communication mechanism.

36. The method of claim 35 wherein each of said plurality of independent processes are assigned to a cluster partition.

37. The method of claim 36 wherein said cluster partition is configured with cluster resources based at least in part on an advance reservation facility.

38. The method of claim 31 wherein said feed handler is adapted to receive one or more direct feeds from one or more markets.

39. The method of claim 31 wherein said feed handler is adapted to route market data to a cluster partition based at least in part on the symbol for an exchange.

40. The method of claim 31, further comprising maximizing data parallelism using a farmer/worker programming pattern.

41. The method of claim 31, further comprising maximizing data parallelism using a data parallelism programming pattern.

42. The method of claim 31, further comprising maximizing data parallelism using a functional/task programming pattern.

43. The method of claim 31, further comprising sending a notification to said particular one or more subscribers upon said receiving said portion of said filtered market that was filtered by said filter established by said particular one or more subscribers.

44. The method of claim 43, further comprising sending a notification to said particular one or more subscribers via a callback mechanism.

45. The method of claim 31, further comprising providing value-added analytics.

46. The method of claim 45, further comprising providing said value-added analytics by:

applying a new algorithm to a metric calculation; and

making the resulting stream available to one or more subscribers.

47. The method of claim 46 wherein said metric comprises VWAP (Volume Weighted Average Price).

48. The method of claim 46, further comprising receiving one or more subscriptions from said one or more subscribers, said one or more subscriptions expressed as an SQL-like query.

49. The method of claim 50, further comprising issuing said one or more subscriptions as continuous data stream queries which provide an asynchronous callback method.

50. The method of claim 31, further comprising invoking a callback method provided by said particular one or more subscribers when said one or more publishers receives said portion of said filtered market that was filtered by said filter established by said particular one or more subscribers.

51. The method of claim 49 wherein asynchronous callback method is implemented using one or more of Linux Asynchronous I/O (AIO), NET Delegate and J2EE Message-Driven Beans (MDB).

52. The method of claim 49, further comprising maximizing data parallelism using one or more of C/C++ using Asynchronous I/O (AIO), NET Delegate, and J2EE.

53. The method of claim 31 wherein a StreamBase™

stream-processing engine comprises said one or more queues and said one or more publishers.

54. The method of claim 48, further comprising:

registering said one or more subscriptions using a remote procedure call (RPC) mechanism that includes a partition identifier; and

returning a subscription ID for subsequent use in subscription management.

55. The method of claim 31, further comprising allocating costs to one or more consuming applications associated with said one or more subscribers, said allocation of costs being based at least in part on queries submitted by said one or more subscribers, and on the underlying resources consumed by said queries.

56. The method of claim 31 wherein said data comprises market data.

57. An apparatus for low latency data, comprising:

means for receiving a data stream comprising data from a data source;

means for receive said data via one or more queues, each of said one or more queues associated with one or more filters adapted to filter said data based at least upon a time predicate to create filtered data, each of said one or more filters established by one or more subscribers;

means for receiving a portion of said filtered data that was filtered by a filter established by a particular one or more subscribers; and

means for publishing said portion of said filtered data for said particular one or more subscribers.

58. The apparatus of claim 57 wherein said particular one or more subscribers comprises an IP multicast group.

59. The apparatus of claim 57 wherein said particular one or more subscribers comprises a single subscriber.

60. The apparatus of claim 57 wherein said particular one or more subscribers comprises a plurality of subscribers.

61. The apparatus of claim 57 wherein said one or more queues and said one or more publishers comprise a plurality of independent processes coupled by an asynchronous communication mechanism.

62. The apparatus of claim 61 wherein each of said plurality of independent processes are assigned to a cluster partition.

63. The apparatus of claim 62 wherein said cluster partition is configured with cluster resources based at least in part on an advance reservation facility.

64. The apparatus of claim 57 wherein said feed handler is adapted to receive one or more direct feeds from one or more markets.

65. The apparatus of claim 57 wherein said feed handler is adapted to route market data to a cluster partition based at least in part on the symbol for an exchange.

66. The apparatus of claim 57, further comprising means for maximizing data parallelism using a farmer/worker programming pattern.

67. The apparatus of claim 57, further comprising means for maximizing data parallelism using a data parallelism programming pattern.

68. The apparatus of claim 57, further comprising means for maximizing data parallelism using a functional/task programming pattern.

69. The apparatus of claim 57, further comprising means for sending a notification to said particular one or more subscribers upon said receiving said portion of said filtered data that was filtered by said filter established by said particular one or more subscribers.

70. The apparatus of claim 69, further comprising means for sending a notification to said particular one or more subscribers via a callback mechanism.

71. The apparatus of claim 57, further comprising means for providing value-added analytics.

72. The apparatus of claim 71, further comprising means for providing said value-added analytics by:

applying a new algorithm to a metric calculation; and

making the resulting stream available to one or more subscribers.

73. The apparatus of claim 72 wherein said metric comprises VWAP (Volume Weighted Average Price).

74. The apparatus of claim 72, further comprising means for receiving one or more subscriptions from said one or more subscribers, said one or more subscriptions expressed as an SQL-like query.

75. The apparatus of claim 74, further comprising means for issuing said one or more subscriptions as continuous data stream queries which provide an asynchronous callback method.

76. The apparatus of claim 57, further comprising means for invoking a callback method provided by said particular one or more subscribers when said one or more publishers receives said portion of said filtered market that was filtered by said filter established by said particular one or more subscribers.

77. The apparatus of claim 75 wherein asynchronous callback method is implemented using one or more of Linux Asynchronous I/O (AIO), NET Delegate and J2EE Message-Driven Beans (MDB).

78. The apparatus of claim 75, further comprising means for maximizing data parallelism using one or more of C/C++ using Asynchronous I/O (AIO), NET Delegate, and J2EE.

79. The apparatus of claim 57 wherein a StreamBase™

stream-processing engine comprises said one or more queues and said one or more publishers.

80. The apparatus of claim 74, further comprising:

means for registering said one or more subscriptions using a remote procedure call (RPC) mechanism that includes a partition identifier; and

means for returning a subscription ID for subsequent use in subscription management.

81. The apparatus of claim 57, further comprising means for allocating costs to one or more consuming applications associated with said one or more subscribers, said allocation of costs being based at least in part on queries submitted by said one or more subscribers, and on the underlying resources consumed by said queries.

82. The apparatus of claim 57 wherein said data comprises market data.

83. A program storage device readable by a machine, embodying a program of instructions executable by the machine to perform a method for low latency data, the method comprising:

receiving a data stream comprising data from a data source;

receive said data via one or more queues, each of said one or more queues associated with one or more filters adapted to filter said data based at least upon a time predicate to create filtered data, each of said one or more filters established by one or more subscribers;

receiving a portion of said filtered data that was filtered by a filter established by a particular one or more subscribers; and

publishing said portion of said filtered data for said particular one or more subscribers.

84. The program storage device of claim 83 wherein said particular one or more subscribers comprises an IP multicast group.

85. The program storage device of claim 83 wherein said particular one or more subscribers comprises a single subscriber.

86. The program storage device of claim 83 wherein said particular one or more subscribers comprises a plurality of subscribers.

87. The program storage device of claim 83 wherein said one or more queues and said one or more publishers comprise a plurality of independent processes coupled by an asynchronous communication mechanism.

88. The program storage device of claim 87 wherein each of said plurality of independent processes are assigned to a cluster partition.

89. The program storage device of claim 88 wherein said cluster partition is configured with cluster resources based at least in part on an advance reservation facility.

90. The program storage device of claim 83 wherein said feed handler is adapted to receive one or more direct feeds from one or more markets.

91. The program storage device of claim 83 wherein said feed handler is adapted to route market data to a cluster partition based at least in part on the symbol for an exchange.

92. The program storage device of claim 83, said method further comprising maximizing data parallelism using a farmer/worker programming pattern.

93. The program storage device of claim 83, said method further comprising maximizing data parallelism using a data parallelism programming pattern.

94. The program storage device of claim 83, said method further comprising maximizing data parallelism using a functional/task programming pattern.

95. The program storage device of claim 83, said method further comprising sending a notification to said particular one or more subscribers upon said receiving said portion of said filtered market that was filtered by said filter established by said particular one or more subscribers.

96. The program storage device of claim 95, said method further comprising sending a notification to said particular one or more subscribers via a callback mechanism.

97. The program storage device of claim 83, said method further comprising providing value-added analytics.

98. The program storage device of claim 97, said method further comprising providing said value-added analytics by:

applying a new algorithm to a metric calculation; and

making the resulting stream available to one or more subscribers.

99. The program storage device of claim 97 wherein said metric comprises VWAP (Volume Weighted Average Price).

100. The program storage device of claim 97, said method further comprising receiving one or more subscriptions from said one or more subscribers, said one or more subscriptions expressed as an SQL-like query.

101. The program storage device of claim 100, said method further comprising issuing said one or more subscriptions as continuous data stream queries which provide an asynchronous callback method.

102. The program storage device of claim 83, said method further comprising invoking a callback method provided by said particular one or more subscribers when said one or more publishers receives said portion of said filtered market that was filtered by said filter established by said particular one or more subscribers.

103. The program storage device of claim 101 wherein asynchronous callback method is implemented using one or more of Linux Asynchronous I/O (AIO), NET Delegate and J2EE Message-Driven Beans (MDB).

104. The program storage device of claim 101, said method further comprising maximizing data parallelism using one or more of C/C++ using Asynchronous I/O (AIO), NET Delegate, and J2EE.

105. The program storage device of claim 83 wherein a StreamBase™

stream-processing engine comprises said one or more queues and said one or more publishers.

106. The program storage device of claim 100, said method further comprising:

registering said one or more subscriptions using a remote procedure call (RPC) mechanism that includes a partition identifier; and

returning a subscription ID for subsequent use in subscription management.

107. The program storage device of claim 83, said method further comprising allocating costs to one or more consuming applications associated with said one or more subscribers, said allocation of costs being based at least in part on queries submitted by said one or more subscribers, and on the underlying resources consumed by said queries.

108. The program storage device of claim 83 wherein said data comprises market data.

109. A method for low latency market data, the method comprising:

aggregating, by a first entity, one or more data streams;

registering, by a second entity, one or more queries regarding said one or more data streams;

publishing, by said second entity, one or more results for said one or more queries; and

subscribing, by a third entity, for receipt of at least one of said one or more results.

110. An apparatus for low latency market data, the apparatus comprising:

a feed handler adapted to aggregate one or more data streams;

a message board adapted to;

register one or more queries regarding said one or more data streams; and

publish one or more results for said one or more queries; and

one or more consumer applications adapted to subscribe for receipt of at least one of said one or more results.

111. An apparatus for low latency data, comprising:

one or more feed handlers, each of said one or more feed handlers adapted to receive a data stream comprising data from a data source;

one or more queues adapted to receive said data from said one or more feed handlers, each of said one or more queues associated with one or more filters adapted to filter said data based at least upon a time predicate to create filtered data, each of said one or more filters established by one or more subscribers; and

one or more publishers, each of said one or more publishers adapted to;

receive a portion of said filtered data that was filtered by a filter established by a particular one or more subscribers; and

publish said portion of said filtered data for said particular one or more subscribers.