Device, system, and method of publishing information to multiple subscribers

• US 7,802,071 B2
• Filed: 07/16/2007
• Issued: 09/21/2010
• Est. Priority Date: 07/16/2007
• Status: Active Grant

• Current Assignee: Mellanox Technologies TLV Ltd. (NVIDIA Corporation)
• Original Assignee: Voltaire Limited (NVIDIA Corporation)
• Inventors: Oved, Tzah
• Primary Examiner(s): Nguyen; Hiep T
• Application Number:
  US11/778,109
• Publication Number:
  US 20090024817A1
• Time in Patent Office: 1,163 Days
• Field of Search: None
• US Class Current: 711/170
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence

Abstract

Device, system, and method of publishing information to multiple subscribers. For example, a data publisher capable of communication with one or more subscribers includes: a memory allocator to allocate a memory area of a local memory unit of the data publisher for storing data to be accessible for Remote Direct Memory Access (RDMA) read operations by one or more of said subscribers.

Claims

1. A Remote Direct Memory Access (RDMA) data publisher capable of communication with one or more RDMA subscribers, the data RDMA publisher comprising:

a memory allocator to allocate a memory area of a local memory unit of the RDMA data publisher for storing data to be accessible for RDMA read operations by one or more of said RDMA subscribers,

wherein the memory area comprises one or more message stores able to store a plurality of messages corresponding to said data,

wherein each message store is capable of serving a plurality of subscribers simultaneously,

wherein each message store comprises a cyclic buffer of messages,

wherein the RDMA data publisher is to write messages into the cyclic buffer utilizing a structure which enables each RDMA subscriber (a) to autonomously detect loss of messages, and (b) to autonomously re-read lost messages via RDMA,

wherein the RDMA data publisher is to write messages into the cyclic buffer independently from operations of the RDMA subscribers and independently from a subscriber request to perform RDMA read of one or more particular messages from the message store,

wherein a first RDMA subscriber is authorized by the RDMA data publisher to remotely pull data using RDMA from a first message store and is not authorized by the data publisher to remotely pull data using RDMA from a second message store,

wherein the first RDMA subscriber and a second RDMA subscriber are authorized by the RDMA data publisher to concurrently pull data from the first message store,

wherein the first RDMA subscriber is to remotely pull messages from the first message store at a first pace using RDMA read operations, and wherein the second RDMA subscriber is to concurrently remotely pull the same messages from the first message store at a second, different, pace using RDMA read operations.

2. The RDMA data publisher of claim 1, wherein the memory area is registered towards an RDMA-capable hardware component for RDMA read operations.

3. The RDMA data publisher of claim 1, wherein the RDMA data publisher is to locally write messages to said cyclic buffer to publish a message to the plurality of subscribers without requiring network transaction following a prior connection establishment, and wherein the one or more RDMA subscribers are to remotely read messages from said cyclic buffer using RDMA.

4. The RDMA data publisher of claim 1, wherein the one or more RDMA subscribers are to remotely pull on demand at least some of said messages from at least one of the message stores.

5. The RDMA data publisher of claim 1, wherein at least one of the RDMA subscribers is to read from said memory area by bypassing an Operating System (OS) of the RDMA data publisher.

6. The RDMA data publisher of claim 1, wherein the one or more RDMA subscribers are decoupled from the RDMA data publisher,

wherein the one or more RDMA subscribers are to remotely pull data from said memory area independent of a timing scheme common to the RDMA data publisher and the one or more RDMA subscribers,

wherein the first RDMA subscriber is able to initiate an RDMA message pull operation without any prior request to the RDMA data publisher,

wherein the first RDMA subscriber is able to independently access and read messages from the first message store comprising said cyclic buffer of messages until the first RDMA subscriber encounters a message which the first RDMA subscriber determines, based on the message header, to be invalid;

and, based on encountering the invalid message, the first RDMA subscriber is to determine that the first RDMA subscriber has read all valid and existing messages in the first message store and that the first RDMA subscriber is up-to-date and is synchronized with the RDMA data publisher.

7. The RDMA data publisher of claim 1, wherein at least one of the RDMA subscribers is to perform the RDMA read operation through InfiniBand hardware.

8. The RDMA data publisher of claim 4, wherein at least one of the RDMA subscribers is to remotely pull said messages using RDMA read operation through InfiniBand hardware and by bypassing an Operating System (OS) of the RDMA data publisher.

9. The RDMA data publisher of claim 1, wherein at least one of the RDMA subscribers is to remotely read messages from the message stores over RDMA-capable, kernel driver-bypassing and OS-bypassing capable hardware.

10. The RDMA data publisher of claim 1, wherein the RDMA data publisher comprises:

a publisher application to receive incoming data from an external data provider, to re-format the incoming data into messages, and to write locally the messages into the one or more message stores independently of the operation of the RDMA subscribers,

wherein the publisher application utilizes a write pointer,

wherein the publisher application writes a message to the memory area which the write pointer points to, and then the publisher application advances the write pointer to the following empty space in the same message store,

wherein the write pointer of the publisher application of the RDMA data publisher is completely decoupled from read pointers of the RDMA subscribers,

wherein each message comprises a header which defines whether or not the message is intact.

11. The RDMA data publisher of claim 10, wherein the incoming data comprises stock trading information,

wherein the first message store is to store data related to stock trading information associated with a first set of one or more stocks, and

wherein the second message store is to store data related to stock trading information associated with a second, different, set of one or more stocks.

12. The RDMA data publisher of claim 1, wherein a common structure of said messages comprises a message header and a message body, and wherein the message header comprises a validity parameter field, a message serial number field, and a field indicating the size of the message body.

13. A method of publishing data to one or more Remote Direct Memory Access (RDMA) subscribers, the method comprises:

allocating a memory area of a local memory unit of a RDMA data publisher for storing data to be accessible for RDMA read operations by one or more of said subscribers,

wherein allocating the memory area comprises creating one or more message stores able to store a plurality of messages corresponding to said data,

wherein each message store is capable of serving a plurality of subscribers simultaneously,

wherein each message store comprises a cyclic buffer of messages;

writing messages into the cyclic buffer utilizing a structure which enables each RDMA subscriber (a) to autonomously detect loss of messages, and (b) to autonomously re-read lost messages via RDMA;

writing messages into the cyclic buffer independently from operations of the RDMA subscribers and independently from a subscriber request to perform RDMA read of one or more particular messages from the message store;

authorizing a first RDMA subscriber to remotely pull data using RDMA from a first message store and un-authorizing the first RDMA subscriber to remotely pull data using RDMA from a second message store;

authorizing the first RDMA subscriber and a second RDMA subscriber to concurrently pull data from the first message store;

authorizing the first RDMA subscriber to remotely pull messages from the first message store at a first pace using RDMA read operations, and authorizing the second RDMA subscriber to concurrently remotely pull the same messages from the first message store at a second, different, pace using RDMA read operations.

14. The method of claim 13, wherein creating a message store comprises creating a cyclic buffer to which the RDMA data publisher is able to locally write messages to publish messages to the plurality of subscribers without requiring network transaction following a prior connection establishment, and from which message store the RDMA subscribers are able to read messages.

15. The method of claim 13, comprising:

authorizing to one or more RDMA subscribers to remotely pull on demand at least some of said messages from at least one of the message stores.

16. The method of claim 13, comprising:

authorizing to at least one of the RDMA subscribers to read from said memory area by bypassing an Operating System (OS) of the data publisher.

17. The method of claim 13, comprising:

decoupling the one or more RDMA subscribers from the RDMA data publisher; and

authorizing the one or more RDMA subscribers to remotely pull data from said memory area independent of a timing scheme common to the RDMA data publisher and the RDMA subscribers,

wherein the authorizing comprises authorizing the first RDMA subscriber;

(a) to initiate an RDMA message pull operation without any prior request to the RDMA data publisher;

(b) to independently access and read messages from the first message store comprising said cyclic buffer of messages until the first RDMA subscriber encounters a message which the first RDMA subscriber determines, based on the message header, to be invalid; and

, based on encountering the invalid message, the first RDMA subscriber is to determine that the first RDMA subscriber has read all valid and existing messages in the first message store and that the first RDMA subscriber is up-to-date and is synchronized with the RDMA data publisher.

18. The method of claim 13, comprising:

authorizing to at least one of the RDMA subscribers to perform the RDMA read operation through InfiniBand hardware.

19. The method of claim 13, comprising:

authorizing to at least one of the RDMA subscribers to remotely read messages from the message stores over RDMA-capable, kernel driver-bypassing and OS-bypassing capable hardware.

20. The method of claim 13, comprising:

receiving incoming data from an external data provider;

re-formatting the incoming data into messages; and

writing locally the messages into the one or more message stores independently of the operation of the RDMA subscribers,

wherein the writing utilizes a write pointer of the RDMA data publisher,

wherein the write pointer of the RDMA data publisher is completely decoupled from read pointers of the RDMA subscribers,

wherein the writing comprises;

writing a message to the memory area which the write pointer points to, and then advancing the write pointer to the following empty space in the same message store,

wherein each message comprises a header which defines whether or not the message is intact.

21. The method of claim 20, wherein the incoming data comprises stock trading information, and wherein writing comprises:

writing into a first message store data related to stock trading information associated with a first set of one or more stocks; and

writing into a second message store data related to stock trading information associated with a second, different, set of one or more stocks.

22. The method of claim 13, wherein a common structure of said messages comprises a message header and a message body, and wherein the message header comprises a validity parameter field, a message serial number field, and a field indicating the size of the message body.

23. The method of claim 15, comprising:

authorizing to at least one of the RDMA subscribers to remotely pull said messages using RDMA read operation through InfiniBand hardware and by bypassing an Operating System (OS) of the RDMA data publisher.