HARDWARE MESSAGE QUEUES FOR INTRA-CLUSTER COMMUNICATION

• US 20140096145A1
• Filed: 02/27/2013
• Published: 04/03/2014
• Est. Priority Date: 10/02/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Aingaran, Kathirgamar, Bridge, William H. Jr., Swart, Garret F., Jairath, Sumti, Johnson, John G.
• Granted Patent: US 9,158,810 B2
• Time in Patent Office: Days
• US Class Current: 719/314
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/082 Associative directories G06F12/0822 takes precedence
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 12/1475 in a virtual system, e.g. with translation means
• CPC Class Codes: G06F 13/102 where the programme performs an interfacing function, e.g. device driver G06F13/105 takes precedence; scheduling within device drivers G06F9/52; contention policies within device drivers G06F9/4881
• CPC Class Codes: G06F 13/362 with centralised access control
• CPC Class Codes: G06F 13/4068 Electrical coupling
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 15/7889 Reconfigurable logic implemented as a co-processor instruction execution using a coprocessor G06F9/3877
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/24569 Query processing with adaptation to specific hardware, e.g. adapted for using GPUs or SSDs
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/6209 : to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 9/30065 : Loop control instructions; iterative instructions, e.g. LOOP, REPEAT
• CPC Class Codes: G06F 9/381 : Loop buffering
• CPC Class Codes: G06F 9/3851 : from multiple instruction streams, e.g. multistreaming
• CPC Class Codes: G06F 9/3867 : using instruction pipelines
• CPC Class Codes: G06F 9/546 : Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 69/14 : Multichannel or multilink protocols
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A method and apparatus for sending and receiving messages between nodes on a compute cluster is provided. Communication between nodes on a compute cluster, which do not share physical memory, is performed by passing messages over an I/O subsystem. Typically, each node includes a synchronization mechanism, a thread ready to receive connections, and other threads to process and reassemble messages. Frequently, a separate queue is maintained in memory for each node on the I/O subsystem sending messages to the receiving node. Such overhead increases latency and limits message throughput. Due to a specialized coprocessor running on each node, messages on an I/O subsystem are sent, received, authenticated, synchronized, and reassembled at a faster rate and with lower latency. Additionally, the memory structure used may reduce memory consumption by storing messages from multiple sources in the same memory structure, eliminating the need for per-source queues.

Claims

1. A device comprising:

enqueue circuitry;

said enqueue circuitry being coupled to a memory;

said enqueue circuitry being configured;

to receive an enqueue command that identifies a queue in said memory;

to enqueue a queue entry to the queue identified by the received enqueue command in response to receiving said enqueue command.

2. The device of claim 1, wherein said device includes an integrated circuit chip that includes, separate from said enqueue circuitry, a processor, said processor being coupled to said enqueue circuitry.

3. The device of claim 1, further comprising:

wherein said enqueue command comprises an RKEY;

wherein said RKEY identifies said queue in said memory.

4. The device of claim 3, wherein said enqueue circuitry being further configured:

to authenticate said RKEY, before enqueuing an entry to the queue identified by said RKEY, in response to receiving said enqueue command;

to enqueue a queue entry to said queue identified by said RKEY in response to receiving said enqueue command if said RKEY was successfully authenticated.

5. The device of claim 1, wherein said enqueue circuitry being further configured to execute enqueue commands serially.

6. The device of claim 1, wherein said enqueue command includes a payload received in a plurality of chunks, and said enqueue circuitry being further configured to reassemble said plurality of chunks in response to receiving said enqueue command and said plurality of chunks.

7. The device of claim 1, wherein said enqueue command includes a payload received in a plurality of chunks, wherein said plurality of chunks were received in a different order than said plurality of chunks were sent;

and said enqueue circuitry being further configured to reassemble said plurality of chunks in response to receiving said enqueue command and said plurality of chunks.

8. A method comprising a machine-executed operation involving software instructions, wherein said software instructions which, when executed by a computing device, cause performance of steps including;

transmitting a particular enqueue command to an enqueue circuitry to enqueue a queue entry into a particular queue stored in a memory coupled to said enqueue circuitry, said particular enqueue command identifying said particular queue;

wherein said enqueue circuitry is configured;

to receive an enqueue command that identifies a queue in said memory;

responsive to receipt of the enqueue command, to enqueue a queue entry to the queue identified by the received enqueue command;

dequeueing a queue entry from said queue.

9. The method of claim 8, wherein said enqueue circuitry is located on an integrated circuit chip that includes, separate from said enqueue circuitry, a processor, said processor being coupled to said enqueue circuitry.

10. The method of claim 8, wherein dequeuing a queue entry from said queue comprises setting a value in said queue entry to indicate that the queue entry has been read.

11. The method of claim 8, wherein dequeuing a queue entry from said queue comprises atomically setting a pointer to point to a queue entry that has not been read.

12. The method of claim 8, further comprising receiving an interrupt in response to said queue entry being enqueued.

13. The method of claim 8, wherein dequeuing a queue entry from said queue comprises copying said queue entry in memory.

14. A system comprising:

a first enqueue circuitry;

said first enqueue circuitry being operably coupled to a first memory and a second memory;

said first enqueue circuitry being configured;

to receive a first enqueue command, which comprises a first RKEY, wherein said first RKEY identifies said first memory;

to receive a second enqueue command, which comprises a second RKEY, wherein said second RKEY identifies said second memory.

15. The system of claim 14, wherein said first enqueue circuitry is further configured to:

enqueue a queue entry in said first memory based on said first RKEY identifying said first memory;

enqueue a queue entry in said second memory based on said second RKEY identifying said second memory.

16. The system of claim 15, wherein said first memory and said second memory both reside on the same physical memory.

17. The system of claim 14, wherein said first enqueue circuitry is further configured to:

authenticate said first RKEY;

authenticate said second RKEY;

execute said first enqueue command, if said first RKEY is successfully authenticated, wherein said first enqueue command comprises enqueuing a queue entry in said first memory based on said first RKEY identifying said first memory;

execute said second enqueue command, if said second RKEY is successfully authenticated, wherein said second enqueue command comprises enqueuing a queue entry in said second memory based on said second RKEY identifying said second memory.

18. The method of claim 17, wherein said first enqueue circuitry is further configured to:

execute said first enqueue command and said second enqueue command serially, wherein said second enqueue command is executed after said first enqueue command.

19. The system of claim 14, further comprising:

a second enqueue circuitry;

wherein said first command further comprises a first payload;

wherein said second enqueue circuitry configured to;

disassemble said first payload into numbered chunks;

send said numbered chunks to said first enqueue circuitry;

wherein said first enqueue circuitry is further configured to;

receive said numbered chunks;

reassemble said numbered chunks;

execute said first enqueue command.