Detecting and managing flooding of multi-tenant message queues

• US 10,013,294 B2
• Filed: 03/22/2017
• Issued: 07/03/2018
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Kwong, Hui Lam, Wang, Xiaodan, Modani, Vipul, Buisson, John B.
• Primary Examiner(s): Mudrick, Timothy A
• Application Number:
  US15/465,917
• Publication Number:
  US 20170192828A1
• Time in Patent Office: 468 Days
• Field of Search: 719314
• CPC Class Codes: G06F 2209/547 Messaging middleware
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1029 using data related to the state of servers by a load balancer
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A messaging system implements messaging among application servers and databases, utilizing other servers that implement messaging brokers. A large flood of incoming messages can bring down messaging brokers by overflowing the message queues, negatively impacting performance of the overall system. This disclosure in some embodiments detects and identifies “flooders” in a timely manner and isolates their message traffic to dedicated queues to avoid impacting other system users. Subsequently, a preferred system de-allocates the queues and returns the messaging system to normal operation when flooding conditions subside, and “sweeps” up any remaining orphan messages.

Claims

1. A computer-implemented method for managing message queues in a database system having a message queue system that includes queues shared by multiple message types and tenants, the method comprising:

estimating an enqueue rate of new messages coming from a sender in the message queue system;

based on the estimated enqueue rate exceeding a predetermined enqueue rate threshold value—

identifying the sender as a flooder;

allocating a dedicated message queue to the flooder;

diverting the flooder'"'"'s message traffic to the dedicated message queue; and

responsive to the flooding condition subsiding, de-allocating the dedicated message queue and discontinue said diverting the flooder'"'"'s message traffic.

2. The method of claim 1 wherein estimating the enqueue rate is based on tracking enqueue counts and periodically examining the enqueue counts.

3. The method of claim 1 including collecting local message enqueue data in at least some app servers in the database system.

4. The method of claim 3 including, in each of the app servers, periodically flushing its recent data to a distributed store configured to hold cumulative data for flood detection.

5. The method of claim 1 wherein the flooder is a specific message type and the specific message type is flooding across multiple tenants.

6. The method of claim 1 wherein the flooder is a combination of one or more message types associated with a specific rac node.

7. The method of claim 1 wherein diverting the flooder'"'"'s message traffic includes storing new messages from the flooder to secondary storage on disk outside of a heap memory of message queue broker hosts.

8. The method of claim 1 including updating a routing table to re-route the flooder'"'"'s message traffic to the dedicated message queue.

9. The method of claim 1 and further comprising:

persisting the flooder list to a database; and

disseminating the flooder list to app servers in the database system.

10. The method of claim 7 including moving messages from the flooding queue into the dedicated queue and routing new messages from the flooder to the secondary storage.

11. A system comprising:

a database;

a plurality of application servers configured for communication with the database;

a message framework interconnecting the database and the application servers;

the message framework including a plurality of message brokers implemented on message queue (MQ) host servers, each MQ host server having access to a local message queue, and a flood detector component arranged to detect flooding on the message framework;

wherein the flood detector is arranged to estimate an enqueue rate of new messages coming from a sender in the message queue system, and based on the estimated enqueue rate exceeding a predetermined enqueue rate threshold value, identify the sender as a flooder; and

wherein the flood detector component is arranged to divert message traffic on the message framework from the identified flooder to a secondary message storage element in the database to relieve flooding on the message framework.

12. The system of claim 11 including a routing table coupled to the flood detector component to enable routing selected messages to a dedicated message queue to avoid flooding.

13. The system of claim 12 including a flooder list coupled to the flood detector and stored in a memory to store flooder data.

14. The system of claim 13 wherein the flood detector is configured to periodically check a flooder listed in the flooder list;

and

if the flooder is no longer flooding the message framework, remove the flooder from the flooder list; and

deallocate the dedicated message queue.

15. A computer program stored on a tangible medium for a database system for managing a message queue system to prevent message flooding from impacting system performance, the computer program comprising a set of instructions operable to:

detect a message flooding condition in the message queue system based on estimating an enqueue rate of messages;

identify a source of the message flooding condition as a flooder;

allocate a dedicated message queue to the flooder;

divert the flooder'"'"'s message traffic to the dedicated message queue to avoid impacting other message traffic; and

responsive to the flooding condition subsiding, de-allocate the dedicated message queue.

16. The computer program as recited in claim 15, wherein the message queue system includes a plurality of message brokers operable in one or more message queue broker hosts, and the instructions for identifying the flooder are operable to:

designate a specific message type to check for a flooding condition in the message queue system;

identify a sender of the specific message type;

estimate an enqueue rate of new messages coming from the sender of the specific message type; and

in the case that estimated enqueue rate exceeds a predetermined enqueue threshold value, designate the sender on a flooder list as the source of the message flooding condition.

17. The computer program as recited in claim 15 including identifying the flooder as one of a specific message type, a specific tenant, a message type-tenant combination, and a message type-cluster combination, the cluster comprising a specific cluster of servers cooperating as a single system.

18. The computer program as recited in claim 15 wherein the instructions are further operable to estimate the enqueue rate based on tracking queue depth counts and periodically examining the queue depth counts.

19. The computer program as recited in claim 15 wherein the instructions are further operable to store the queue depth counts in a distributed cache, and disseminate the queue depth counts to multiple application servers via a distributed cache.

20. The computer program as recited in claim 15 wherein the instructions are further operable to periodically recheck to determine whether an existing flooder is no longer flooding;

and, in the case that the flooder is no longer flooding, remove the flooder from the flooder list.