Detecting and managing flooding of multi-tenant message queues

• US 9,632,852 B2
• Filed: 03/18/2015
• Issued: 04/25/2017
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Modani, Vipul, Kwong, Hui Lam, Wang, Xiaodan, Buisson, John B.
• Primary Examiner(s): Mudrick, Timothy A
• Application Number:
  US14/661,206
• Publication Number:
  US 20160055042A1
• Time in Patent Office: 769 Days
• Field of Search: 719314
• CPC Class Codes: G06F 2209/547 Messaging middleware
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1029 using data related to the state of servers by a load balancer
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A messaging system implements messaging among application servers and databases, utilizing other servers that implement messaging brokers. A large flood of incoming messages can bring down messaging brokers by overflowing the message queues, negatively impacting performance of the overall system. This disclosure in some embodiments detects and identifies “flooders” in a timely manner and isolates their message traffic to dedicated queues to avoid impacting other system users. Subsequently, a preferred system de-allocates the queues and returns the messaging system to normal operation when flooding conditions subside, and “sweeps” up any remaining orphan messages.

Claims

1. A computer-implemented method for managing message queues in a database system having a message queue system that includes queues shared by multiple message types and tenants, the method comprising:

detecting a message flooding condition in the message queue system;

identifying a source of the message flooding condition as a flooder;

allocating a dedicated message queue to the flooder, wherein the dedicated message queue is implemented in local physical memory of a server in the database system;

diverting the flooder'"'"'s message traffic to the dedicated message queue; and

responsive to the flooding condition subsiding, de-allocating the dedicated message queue and routing the flooder'"'"'s message traffic normally.

2. The method of claim 1 wherein the message queue system includes a plurality of message brokers operable in one or more message queue broker hosts, and identifying the flooder includes:

fetching queue depth counts of messages in the message queue broker hosts by message type;

detecting a specific message type for which the queue depth count exceeds a predetermined threshold value;

identifying a sender of the specific message type;

estimating an enqueue rate of new messages coming from the sender of the specific message type; and

in the case that the estimated enqueue rate exceeds a predetermined enqueue threshold value, designating the sender as the source of the message flooding condition on a flooder list.

3. The method of claim 2 wherein the queue depth and enqueue rate thresholds are tuned to fit a collection of workloads.

4. The method of claim 2 wherein the flooder is an individual tenant of the database system.

5. The method of claim 2 wherein the flooder is a specific message type and the specific message type is flooding across multiple tenants.

6. The method of claim 2 wherein the flooder is a combination of one or more message types associated with a specific rac node.

7. The method of claim 2 wherein diverting the flooder'"'"'s message traffic includes storing new messages from the flooder to secondary storage on disk outside of the heap memory of message queue broker hosts.

8. The method of claim 2 including updating a routing table to re-route the flooder'"'"'s message traffic to the dedicated message queue.

9. The method of claim 2 and further comprising:

persisting the flooder list to a database; and

disseminating the flooder list to app servers in the database system.

10. The method of claim 7 including moving messages from the flooding queue into the dedicated queue and routing new messages from the flooder to the secondary storage.

11. A system comprising:

a database;

a plurality of application servers configured for communication with the database;

a message framework interconnecting the database and the application servers;

the message framework including a plurality of message brokers implemented on message queue host servers, each host server having local physical server memory accessible to the message broker to implement a local message queue, and a flood detector component arranged to detect flooding on the message framework; and

wherein the flood detector is arranged to divert message traffic on the message framework from a flooder to a secondary message storage element in the database to relieve flooding on the message framework, the secondary message storage element physically separate from the local message queue.

12. The system of claim 11 including a routing table coupled to the flood detector to enable routing selected messages to a dedicated message queue to avoid flooding.

13. The system of claim 12 including a flooder list coupled to the flood detector and stored in a memory to store flooder data.

14. The system of claim 13 wherein the flood detector is configured to periodically check a flooder listed in the flooder list;

and

if the flooder is no longer flooding the message framework, remove the flooder from the flooder list; and

deallocate the dedicated message queue.

15. A computer program stored on a tangible medium for a database system for managing a message queue system to prevent message flooding from impacting system performance, the computer program comprising a set of instructions operable to:

detect a message flooding condition in the message queue system;

identify a source of the message flooding condition as a flooder;

allocate a dedicated message queue to the flooder;

divert the flooder'"'"'s message traffic to the dedicated message queue to avoid impacting other message traffic; and

responsive to the flooding condition subsiding, de-allocate the dedicated message queue.

16. The computer program as recited in claim 15, wherein the message queue system includes a plurality of message brokers operable in one or more message queue broker hosts, and the instructions for identifying the flooder are operable to:

fetch queue depth counts of messages in the message queue broker hosts by message type;

detect a specific message type for which the queue depth count exceeds a predetermined threshold value;

identify a sender of the specific message type;

estimate an enqueue rate of new messages coming from the sender of the specific message type; and

in the case that the estimated enqueue rate exceeds a predetermined enqueue threshold value, designate the sender on a flooder list as the source of the message flooding condition.

17. The computer program as recited in claim 16 including identifying the flooder as one of a specific message type, a specific tenant, a message type-tenant combination, and a message type-cluster combination, the cluster comprising a specific cluster of servers cooperating as a single system.

18. The computer program as recited in claim 17 wherein the instructions are further operable to store the queue depth counts in a distributed cache and disseminate the queue depth counts to multiple application servers via the distributed cache.

19. The computer program as recited in claim 16 wherein the instructions are further operable to periodically recheck to determine whether an existing flooder is no longer flooding;

and, in the case that the flooder is no longer flooding, remove the flooder from the flooder list.

20. The computer program as recited in claim 19 wherein the periodic recheck includes the steps of:

checking the corresponding queue depth count for a flooder on the flooder list,

checking whether all messages in the secondary store for that flooder were swept in the last sweeper run; and

checking a recent enqueue rate for the flooder; and

then,

in the case that (a) the corresponding queue depth count is below a predetermined low depth count, (b) all messages in the secondary store for that flooder were swept in the last sweeper run, and (c) the recent enqueue rate is below a predetermined maximum enqueue rate, removing the flooder from the flooder list.