Batch compression management of messages

• US 10,056,918 B2
• Filed: 08/11/2015
• Issued: 08/21/2018
• Est. Priority Date: 08/12/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Crowther, Fiona M., Geleji, Geza, Poole, Christopher J., Ross, Martin A., Stirling, Craig H.
• Primary Examiner(s): Anyan, Barbara B
• Application Number:
  US14/822,993
• Publication Number:
  US 20160050296A1
• Time in Patent Office: 1,106 Days
• CPC Class Codes: H03M 7/3084 using adaptive string matching, e.g. the Lempel-Ziv method
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/0613 based on the type or category of the network elements
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 69/14 Multichannel or multilink protocols
• CPC Class Codes: H04L 69/18 Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
• CPC Class Codes: H04M 15/43 Billing software details
• CPC Class Codes: H04M 3/00 Automatic or semi-automatic exchanges

Abstract

Batch compression management of messages. A message being sent to a batch for transmission is intercepted. The message is assigned to a batch pool, in which multiple batch pools are provided as sub-sets of the batch. The assigning of the message to the batch pool is based on a similarity of properties of the message with other messages and determined by looking up the properties in a batch pool mapping database. The message is added to the batch pool or a new batch pool is created for the message. A batch pool of messages is compressed using a dictionary of the properties common to at least some of the messages in the batch pool. The compressed batch pools are transmitted over an input/output channel.

Claims

1. A computer-implemented method of batch compression management of messages, comprising:

intercepting, by one or more processors, in middleware between a first distributed system and a second distributed system, a new message originating from the first distributed system being sent to a batch for transmission to the second distributed system;

determining, by the one or more processors, if a message schema of the new message is of a predefined similarity to message schemas of messages previously assigned to multiple batch pools comprising sub-sets of the batch, wherein the similarity in message schemas comprises a similarity between a majority of elements in a payload of the new message to a majority of elements in the payloads of the messages previously assigned to the multiple batch pools comprising the sub-sets of the batch, and wherein the determining comprises;

obtaining, by the one or more processors, from a batch pool mapping database, the message schemas of the messages previously assigned to the multiple batch pools comprising the sub-sets of the batch;

comparing, by the one or more processors, the message schemas of the previously assigned messages the to the message schema of the new message; and

determining, by the one or more processors, if a given batch pool of the multiple batch pools comprises one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message;

based on determining that the given batch pool of the multiple batch pools comprises one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message, assigning, by the one or more processors, the message from the batch to the given batch pool;

based on determining that the given batch pool of the multiple batch pools does not comprise one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message, creating, by the one or more processors, a new batch pool for the new message and assigning the new message to the new batch pool;

performing, by the one or more processors, a customized compression on the given batch pool using a dictionary of properties common to the one or more message schemas of the one or more messages, wherein the customized compression is an optimized compression that minimizes a number of entries utilized in the dictionary and reduces overhead in the one or more processors, based on the messages in the given batch pool comprising similar message schemas; and

transmitting, by the one or more processors, the compressed batch pool over an input/output channel to the second distributed system.

2. The computer-implemented method of claim 1, further including:

providing a batch pool to dictionary mapping database and updating the batch pool to dictionary mapping database with new dictionaries for batch pools and updated dictionaries for additional entries.

3. The computer-implemented method of claim 2, further including:

comparing stored dictionaries of batch pools;

determining whether two dictionaries have a similarity greater than a predefined threshold; and

merging dictionaries and their associated batch pools based on the similarity being greater than the predefined threshold.

4. The computer-implemented method of claim 3, further including:

updating the batch pool to dictionary mapping database to merge two or more dictionaries by updating two or more batch pool mappings to map to a single dictionary; and

merging the two or more batch pools.

5. The computer-implemented method of claim 3, wherein the predefined threshold is user defined or machine learned.

6. The computer-implemented method of claim 1, further including:

scheduling the selected batch pool for compression based on one or more of;

monitoring an arrival time of messages and scheduling batch pools according to earliest arriving messages;

monitoring a size of a batch pool and scheduling batch pools according to a predefined maximum batch pool size; and

monitoring a count of times a batch pool is passed over for compression and scheduling batch pools according to a highest count.

7. A computer program product for batch compression management of messages, comprising:

a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

intercepting, by one or more processors, in middleware between a first distributed system and a second distributed system, a new message originating from the first distributed system being sent to a batch for transmission to the second distributed system;

determining, by the one or more processors, if a message schema of the new message is of a predefined similarity to message schemas of messages previously assigned to multiple batch pools comprising sub-sets of the batch, wherein the determining comprises;

obtaining, by the one or more processors, from a batch pool mapping database, the message schemas of the messages previously assigned to the multiple batch pools comprising the sub-sets of the batch;

comparing, by the one or more processors, the message schemas of the previously assigned messages the to the message schema of the new message; and

determining, by the one or more processors, if a given batch pool of the multiple batch pools comprises one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message;

based on determining that the given batch pool of the multiple batch pools comprises one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message, assigning, by the one or more processors, the message from the batch to the given batch pool;

based on determining that the given batch pool of the multiple batch pools does not comprise one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message, creating, by the one or more processors, a new batch pool for the new message and assigning the new message to the new batch pool;

performing, by the one or more processors, a customized compression on the given batch pool using a dictionary of properties common to the one or more message schemas of the one or more messages at, wherein the customized compression is an optimized compression that minimizes a number of entries utilized in the dictionary and reduces overhead in the one or more processors, based on the messages in the given batch pool comprising similar message schemas; and

transmitting, by the one or more processors, the compressed batch pool over an input/output channel to the second distributed system.

8. The computer program product of claim 7, wherein the method further comprises:

providing a batch pool to dictionary mapping database and updating the batch pool to dictionary mapping database with new dictionaries for batch pools and updated dictionaries for additional entries.

9. The computer program product of claim 8, further including:

comparing stored dictionaries of batch pools;

determining whether two dictionaries have a similarity greater than a predefined threshold; and

merging dictionaries and their associated batch pools based on the similarity being greater than the predefined threshold.

10. The computer program product of claim 9, wherein the method further comprises:

updating the batch pool to dictionary mapping database to merge two or more dictionaries by updating two or more batch pool mappings to map to a single dictionary; and

merging the two or more batch pools.

11. The computer program product of claim 7, wherein the method further comprises:

scheduling the selected batch pool for compression based on one or more of;

monitoring an arrival time of messages and scheduling batch pools according to earliest arriving messages;

monitoring a size of a batch pool and scheduling batch pools according to a predefined maximum batch pool size; and

monitoring a count of times a batch pool is passed over for compression and scheduling batch pools according to a highest count.

12. A computer system for batch compression management of messages, comprising:

a memory; and

a processing device in communication with the memory, wherein the computer system is configured to perform a method, said method comprising;

intercepting, by the processing device, in middleware between a first distributed system and a second distributed system, a new message originating from the first distributed system being sent to a batch for transmission to the second distributed system;

determining, by the processing device, if a message schema of the new message is of a predefined similarity to message schemas of messages previously assigned to multiple batch pools comprising sub-sets of the batch, wherein the determining comprises;

obtaining, by the processing device, from a batch pool mapping database, the message schemas of the messages previously assigned to the multiple batch pools comprising the sub-sets of the batch;

comparing, by the processing device, the message schemas of the previously assigned messages the to the message schema of the new message; and

determining, by the processing device, if a given batch pool of the multiple batch pools comprises one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message;

based on determining that the given batch pool of the multiple batch pools comprises one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message, assigning, by the processing device, the message from the batch to the given batch pool;

based on determining that the given batch pool of the multiple batch pools does not comprise one or more messages with one or more message schemas of the predefined similarity to the message schema of the new message, creating, by the processing device, a new batch pool for the new message and assigning the new message to the new batch pool;

performing, by the processing device, a customized compression on the given a batch pool using a dictionary of properties common to the one or more message schemas of the one or more messages, wherein the customized compression is an optimized compression that minimizes a number of entries utilized in the dictionary and reduces overhead in the one or more processors, based on the at messages in the given batch pool comprising similar message schemas;

and transmitting, by the processing device, the compressed batch pool over an input/output channel to the second distributed system.

13. The computer system of claim 12, wherein the method further comprises:

providing a batch pool to dictionary mapping database and updating the batch pool to dictionary mapping database with new dictionaries for batch pools and updated dictionaries for additional entries.

14. The computer system of claim 13, wherein the method further comprises:

comparing stored dictionaries of batch pools;

determining whether two dictionaries have a similarity greater than a predefined threshold; and

merging dictionaries and their associated batch pools based on the similarity being greater than the predefined threshold.

15. The computer system of claim 14, wherein the method further comprises:

updating the batch pool to dictionary mapping database to merge two or more dictionaries by updating two or more batch pool mappings to map to a single dictionary; and

merging the two or more batch pools.

16. The computer system of claim 12, wherein the method further comprises:

scheduling the selected batch pool for compression based on one or more of;

monitoring an arrival time of messages and scheduling batch pools according to earliest arriving messages;

monitoring a size of a batch pool and scheduling batch pools according to a predefined maximum batch pool size; and

monitoring a count of times a batch pool is passed over for compression and scheduling batch pools according to a highest count.