Dynamic tracer message logging based on bottleneck detection

• US 10,416,974 B2
• Filed: 10/06/2017
• Issued: 09/17/2019
• Est. Priority Date: 10/06/2017
• Status: Active Grant

• Current Assignee: Chicago Mercantile Exchange, Inc. (CME Group Incorporated)
• Original Assignee: Chicago Mercantile Exchange, Inc. (CME Group Incorporated)
• Inventors: Kavanagh, Kyle Dennis, Acuna-Rohter, Jose Antonio, Wong, David Michael
• Primary Examiner(s): Chowdhury, Ziaul A
• Assistant Examiner(s): Wei, Zengpu
• Application Number:
  US15/727,097
• Publication Number:
  US 20190108004A1
• Time in Patent Office: 711 Days
• Field of Search: 717151
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 8/443 Optimisation
• CPC Class Codes: G06F 8/70 Software maintenance or management

Abstract

A monitoring system monitors processing of incoming messages by an application, and logs data related to performance of the application. The application includes a plurality of checkpoints, and the monitoring system logs data upon each message traversing the checkpoints in the application. The monitoring system is configured to dynamically modify checkpoints within the application based on latency detection of portions of the application, resulting in improved granularity/resolution of the data collected from congested portions of the application, and reducing the performance penalty of the monitoring system from portions of the application that are not congested.

Claims

1. A computer implemented method of monitoring message processing by an application in a data transaction processing system, the method comprising:

detecting, by a processor coupled with the application, that the application has received a message of a plurality of messages for processing as each message of the plurality of messages is received by the application, the application including a plurality of checkpoints, wherein each of the plurality of checkpoints is enabled or disabled, and further wherein at least first, second and third checkpoints of the plurality of checkpoints are enabled, the third enabled checkpoint being located between the first and second enabled checkpoints in a message processing sequence;

detecting, by the processor, that a message processing latency between the first and second enabled checkpoints of the plurality of checkpoints is less than an amount of time between new messages arriving at a portion of the application defined by the first and second enabled checkpoints and further detecting that a number of messages awaiting processing by the portion of the application defined by the first and second enabled checkpoints exceeds a threshold number of messages, and based thereon, disabling, by the processor, the third enabled checkpoint; and

upon a received message of the plurality of messages traversing an enabled checkpoint of the plurality of checkpoints, storing, by the processor, in a data store, a data entry indicative of the received message, the traversed enabled checkpoint, and a time when the received message traversed the enabled checkpoint.

2. The computer implemented method of claim 1, wherein upon a received message of the plurality of messages traversing a disabled checkpoint of the plurality of checkpoints, a data entry indicative of the traversed disabled checkpoint is not stored in the data store.

3. The computer implemented method of claim 1, further comprising:

detecting that a message processing latency between fourth and fifth enabled checkpoints of the plurality of checkpoints satisfies another condition, the application including a sixth disabled checkpoint between the fourth and fifth enabled checkpoints in a message processing sequence; and

enabling the sixth disabled checkpoint between the fourth and fifth enabled checkpoints, such that upon a received message of the plurality of messages traversing the sixth checkpoint, a data entry indicative of the received message, the sixth checkpoint, and a time when the received message traversed the sixth checkpoint is stored in the data store.

4. The computer implemented method of claim 3, wherein the other condition comprises one of:

(i) the message processing latency between the fourth and fifth enabled checkpoints exceeds a latency threshold associated with the fourth and fifth enabled checkpoints;

(ii) the message processing latency between the fourth and fifth enabled checkpoints exceeds a message processing latency between a different pair of enabled checkpoints of the plurality of checkpoints;

(iii) the message processing latency between the fourth and fifth enabled checkpoints exceeds an amount of time between new messages arriving at a portion of the application defined by the fourth and fifth enabled checkpoints;

or (iv) a number of messages awaiting processing by a portion of the application defined by the first and second enabled checkpoints exceeds a threshold number of messages.

5. The computer implemented method of claim 1, wherein disabling the third checkpoint reduces an amount of data stored in the data store.

6. The computer implemented method of claim 1, wherein disabling the third checkpoint reduces an amount of storage space necessary to store the data store.

7. The computer implemented method of claim 1, the method further comprising, before the third checkpoint is disabled, upon a received message of the plurality of messages traversing the third checkpoint, storing, in the data store, a data entry indicative of the received message, the third checkpoint, and a time when the received message traversed the third checkpoint.

8. The computer implemented method of claim 1, wherein detecting whether a message processing latency between two enabled checkpoints of the plurality of checkpoints is less than an amount of time between new messages arriving at a portion of the application defined by the first and second enabled checkpoints comprises determining an elapse of time between times when a received message traversed the two enabled checkpoints.

9. The computer implemented method of claim 1, wherein the message processing latency between two enabled checkpoints is determined based on data entries stored in the data store and indicative of times when a received message traversed the two enabled checkpoints.

10. The computer implemented method of claim 1, wherein each message is an electronic data transaction request message, and wherein the application processes the electronic data transaction request message by determining whether an attempt to match the electronic data transaction request message with at least one previously received but unsatisfied electronic data transaction request message for a transaction which is counter thereto results in at least partial satisfaction of one or both of the electronic data transaction request message and the at least one previously received but unsatisfied electronic data transaction request message.

11. The computer implemented method of claim 1, further comprising, enabling the third checkpoint when the message processing latency between the first and second enabled checkpoints of the plurality of checkpoints ceases to be less than an amount of time between new messages arriving at a portion of the application defined by the first and second enabled checkpoints or a number of messages awaiting processing by the portion of the application defined by the first and second enabled checkpoints is less than a threshold number of messages.

12. A non-transitory computer-readable medium storing instructions for monitoring message processing by an application in a data transaction processing system that, when executed by a processor, cause the processor to:

detect that the application has received a message of a plurality of messages for processing as each message of the plurality of messages is received by the application, the application including a plurality of checkpoints, wherein each of the plurality of checkpoints is enabled or disabled, and further wherein at least first, second and third checkpoints of the plurality of checkpoints are enabled, the third enabled checkpoint being located between the first and second enabled checkpoints in a message processing sequence;

detect that a message processing latency between the first and second enabled checkpoints of the plurality of checkpoints is less than an amount of time between new messages arriving at a portion of the application defined by the first and second enabled checkpoints and further detect that a number of messages awaiting processing by the portion of the application defined by the first and second enabled checkpoints exceeds a threshold number of messages, and based thereon, disable the third enabled checkpoint; and

upon a received message of the plurality of messages traversing an enabled checkpoint of the plurality of checkpoints, store, in a data store, a data entry indicative of the received message, the traversed enabled checkpoint, and a time when the received message traversed the enabled checkpoint.

13. The computer system of claim 12, wherein the instructions are further configured to cause the processor to, upon a received message of the plurality of messages traversing a disabled checkpoint of the plurality of checkpoints, not store a data entry indicative of the traversed disabled checkpoint in the data store.

14. The computer system of claim 12, wherein disabling the third checkpoint reduces an amount of data stored in the data store and an amount of storage space necessary to store the data store.

15. The computer system of claim 12, wherein detecting whether a message processing latency between two enabled checkpoints of the plurality of checkpoints is less than an amount of time between new messages arriving at a portion of the application defined by the first and second enabled checkpoints comprises determining an elapse of time between times when a received message traversed the two enabled checkpoints.

16. The computer system of claim 12, wherein the message processing latency between two enabled checkpoints is determined based on data entries stored in the data store and indicative of times when a received message traversed the two enabled checkpoints.

17. The computer system of claim 12, wherein the instructions are further configured to cause the processor to enable the third checkpoint when the message processing latency between the first and second enabled checkpoints of the plurality of checkpoints ceases to be less than an amount of time between new messages arriving at a portion of the application defined by the first and second enabled checkpoints or that a number of messages awaiting processing by the portion of the application defined by the first and second enabled checkpoints is less than a threshold number of messages.

18. A computer system for monitoring message processing by an application in a data transaction processing system, the computer system comprising:

means for detecting that the application has received a message of a plurality of messages for processing as each message of the plurality of messages is received by the application, the application including a plurality of checkpoints, wherein each of the plurality of checkpoints is enabled or disabled, and further wherein at least first, second and third checkpoints of the plurality of checkpoints are enabled, the third enabled checkpoint being located between the first and second enabled checkpoints in a message processing sequence;

means for detecting that a message processing latency between the first and second enabled checkpoints of the plurality of checkpoints is less than an amount of time between new messages arriving at a portion of the application defined by the first and second enabled checkpoints and for further detecting that a number of messages awaiting processing by the portion of the application defined by the first and second enabled checkpoints exceeds a threshold number of messages, and based thereon, means for disabling the third enabled checkpoint; and

means for, upon a received message of the plurality of messages traversing an enabled checkpoint of the plurality of checkpoints, storing in a data store, a data entry indicative of the received message, the traversed enabled checkpoint, and a time when the received message traversed the enabled checkpoint.