Repeatable message streams for message queues in distributed systems
First Claim
Patent Images
1. A computer-implemented method for propagating messages in a distributed system, the method comprising:
- sending, to a plurality of destination queues, a set of messages that are enqueued in a source queue;
wherein the source queue and each destination queue of the plurality of destination queues are maintained at separate nodes of the distributed system;
wherein the set of messages is an ordered sequence of messages;
before all of the set of messages have been consumed at the plurality of destination queues, receiving, from each destination queue of the plurality of destination queues, an acknowledgement indicating which messages of the set of messages have been consumed at said each destination queue, wherein multiple consumers are configured to consume the set of messages from said each destination queue;
wherein the acknowledgement received from said each destination queue indicates a highest numbered message in said each destination queue that was consumed by all of the multiple consumers that consume messages from said each destination queue;
at a node that maintains the source queue, determining a lowest numbered message of all highest numbered messages that are indicated in acknowledgements received from all of the plurality of destination queues;
wherein determining the lowest numbered message comprises comparing to each other the sequence numbers of all highest numbered messages that are indicated in the acknowledgements received from all of the plurality of destination queues;
based on the acknowledgements received from each destination queue of the plurality of destination queues, ceasing to maintain one or more messages of the set of messages enqueued in the source queue;
wherein ceasing to maintain the one or more messages includes ceasing to maintain any message that is numbered lower than the lowest numbered message;
performing a replay operation to re-send from the source queue, and to re-enqueue in at least one destination queue of the plurality of destination queues, at least one message that is numbered higher than the lowest numbered message;
wherein the replay operation is not performed for any messages from the source queue that are numbered lower than the lowest numbered message;
wherein the method is performed by one or more computer systems.
1 Assignment
0 Petitions
Accused Products
Abstract
Techniques for propagating messages in a distributed system are provided. A set of messages enqueued in a source queue are sent to one or more destination queues. An acknowledgement is received from each of the one or more destination queues, where the acknowledgement indicates which messages of the set of messages have been consumed at the particular destination queue. Based on the acknowledgements, one or more messages of the set messages are ceased to be maintained in the source queue.
-
Citations
14 Claims
-
1. A computer-implemented method for propagating messages in a distributed system, the method comprising:
-
sending, to a plurality of destination queues, a set of messages that are enqueued in a source queue; wherein the source queue and each destination queue of the plurality of destination queues are maintained at separate nodes of the distributed system; wherein the set of messages is an ordered sequence of messages; before all of the set of messages have been consumed at the plurality of destination queues, receiving, from each destination queue of the plurality of destination queues, an acknowledgement indicating which messages of the set of messages have been consumed at said each destination queue, wherein multiple consumers are configured to consume the set of messages from said each destination queue; wherein the acknowledgement received from said each destination queue indicates a highest numbered message in said each destination queue that was consumed by all of the multiple consumers that consume messages from said each destination queue; at a node that maintains the source queue, determining a lowest numbered message of all highest numbered messages that are indicated in acknowledgements received from all of the plurality of destination queues; wherein determining the lowest numbered message comprises comparing to each other the sequence numbers of all highest numbered messages that are indicated in the acknowledgements received from all of the plurality of destination queues; based on the acknowledgements received from each destination queue of the plurality of destination queues, ceasing to maintain one or more messages of the set of messages enqueued in the source queue; wherein ceasing to maintain the one or more messages includes ceasing to maintain any message that is numbered lower than the lowest numbered message; performing a replay operation to re-send from the source queue, and to re-enqueue in at least one destination queue of the plurality of destination queues, at least one message that is numbered higher than the lowest numbered message; wherein the replay operation is not performed for any messages from the source queue that are numbered lower than the lowest numbered message; wherein the method is performed by one or more computer systems. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A computer-readable non-transitory storage medium storing one or more sequences of instructions for propagating messages in a distributed system, which instructions, when executed by one or more processors, cause the one or more processors to perform steps comprising:
-
sending, to a plurality of destination queues, a set of messages that are enqueued in a source queue; wherein the source queue and each destination queue of the plurality of destination queues are maintained at separate nodes of the distributed system; wherein the set of messages is an ordered sequence of messages; before all of the set of messages have been consumed at the plurality of destination queues, receiving, from each destination queue of the plurality of destination queues, an acknowledgement indicating which messages of the set of messages have been consumed at said each destination queue, wherein multiple consumers are configured to consume the set of messages from said each destination queue; wherein the acknowledgement received from said each destination queue indicates a highest numbered message in said each destination queue that was consumed by all of the multiple consumers that consume messages from said each destination queue; at a node that maintains the source queue, determining a lowest numbered message of all highest numbered messages that are indicated in acknowledgements received from all of the plurality of destination queues; wherein determining the lowest numbered message comprises comparing to each other the sequence numbers of all highest numbered messages that are indicated in the acknowledgements received from all of the plurality of destination queues; based on the acknowledgements received from each destination queue of the plurality of destination queues, ceasing to maintain one or more messages of the set of messages enqueued in the source queue; wherein ceasing to maintain the one or more messages includes ceasing to maintain any message that is numbered lower than the lowest numbered message; performing a replay operation to re-send from the source queue, and to re-enqueue in at least one destination queue of the plurality of destination queues, at least one message that is numbered higher than the lowest numbered message; wherein the replay operation is not performed for any messages from the source queue that are numbered lower than the lowest numbered message. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
Specification