Methods, apparatus and computer programs for managing persistence

• US 10,528,405 B2
• Filed: 08/17/2016
• Issued: 01/07/2020
• Est. Priority Date: 07/01/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Todd, Stephen J.
• Primary Examiner(s): Saeed, Usmaan
• Assistant Examiner(s): Kim, Paul
• Application Number:
  US15/238,815
• Publication Number:
  US 20160357619A1
• Time in Patent Office: 1,238 Days
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues
• CPC Class Codes: G06Q 10/0635 Risk analysis of enterprise or organisation activities

Abstract

Described are methods, apparatus and computer programs for managing persistence within a data processing system such as a messaging system, database system or file system. The method for managing persistence includes a deferred evaluation (230,330,430) of at least one criterion associated with costs and/or benefits of saving to persistent storage, instead of persistence behaviour being fully predefined. The evaluation may be performed when a disk write is about to be performed (240,340,440), or at various times during processing of a data update and at various points within a data processing network. In a messaging solution, the method for managing persistence includes a dynamic evaluation (230,330,430) of costs and/or benefits of saving to persistent storage, with the evaluation being performed at various points in the messaging network after the message has been created and sent by the originating entity. The method includes determining whether message data and/or log records relating to the message require saving to persistent storage, in accordance with the costs and/or the benefits of saving to persistent storage. The benefits of persisting may be determined with reference to a message value (400).

Claims

1. A data processing apparatus comprising:

a processor;

a volatile data store;

a non-volatile data store;

a persistence manager for managing saving of data to the non-volatile data store, wherein the persistence manager includes means for evaluating a′

least one criterion of a set of criteria, the set of criteria representing costs of saving and risks associated with not saving data to the non-volatile data store, to determine whether a data update held in the volatile data store requires saving to the non-volatile data store; and

,

a messaging manager, wherein the persistence manager includes functions for;

evaluating at least one criterion of a set of criteria, the set of criteria representing costs of saving and risks associated with not saving data relating to a message to the non-volatile data store, during handling of the message by the messaging manager, to determine whether to save a data update to the persistent storage, the evaluating including receiving a value representing a number of messages read from a queue per second, and applying an exponential smoothing algorithm to the value representing the number of messages read from the queue per second to generate a message processing rate value;

determining from said evaluation whether to save the data relating to the message to the non-volatile data store;

in response to a positive determination, initiating a save of the data relating to the message to the non-volatile data store;

in response to determining that the data update should not be saved to the persistent storage, setting, by the persistence manager, a timer for a next evaluation;

determining, by the persistence manager, whether to save the data update to the persistent storage based upon evaluating the at least one criterion of the set of criteria at an end of a set time period resulting from an expiration of the timer; and

,

saving the data update to the persistent storage upon a determination to save the data update to the persistent storage.

2. A data processing apparatus according to claim 1, wherein the evaluation is performed in response to a message attribute indicating that the message is a candidate for saving to the non-volatile data store.

3. A data processing apparatus according to claim 2, including a messaging API providing message persistence options including at least one message attribute value selected from the group consisting of:

persistent;

non-persistent;

persistence determined by reference to message queue definition; and

persistence determined by reference to message value.

4. A data processing apparatus according to claim 3, wherein the messaging API provides means for specifying a message value and provides at least the message persistence option ‘

persistence determined by reference to message value’

.

5. A data processing apparatus according to claim 1, wherein the persistence manager'"'"'s function for evaluating comprises means for evaluating a risk of loss of data if data relating to a message is not saved to the non-volatile data store, and means for evaluating a cost associated with said loss based on the value of the message.

6. A data processing apparatus according to claim 5 comprising a plurality of data processing systems connected for communication via a network, each data processing system comprising:

a processor, a volatile data store, a non-volatile data store, a messaging manager and an associated persistence manager for determining whether to save message related data to the respective non-volatile data store;

wherein a first messaging manager at a first data processing system comprises means for determining a message value for use by a first persistence manager at the first data processing system, and means for sending the determined message value to a second messaging manager at a second data processing system; and

wherein the second messaging manager comprises means for identifying the message value and providing the identified message value to a second persistence manager at the second data processing system, to enable a determination by the second persistence manager of whether to save message related data to a non-volatile data store at the second data processing system.

7. A data processing apparatus according to claim 1, wherein the persistence manager'"'"'s function for evaluating comprises means for evaluating the benefits of saving data relating to the message to the non-volatile data store based on a calculated risk of losing said data relating to the message.

8. A data processing apparatus according to claim 1, wherein the persistence manager is responsive to a message queue reaching a threshold queue depth, to initiate the functions for evaluating, determining and initiating a save to the non-volatile data store.

9. A data processing apparatus according to claim 1, wherein the persistence manager is responsive to the messaging manager performing an operation in relation to the message, to initiate the functions for evaluating, determining and initiating a save to the non-volatile data store.

10. A data processing apparatus according to claim 9, wherein the operation comprises placing the message on a queue managed by the messaging manager.

11. A method for managing persistence of data within a data processing apparatus, comprising the steps of:

evaluating, by a persistence manager executing in a memory of a computer, at least one criterion of a set of criteria, the set of criteria representing costs of saving and risks associated with not saving data to persistent storage, to determine whether to save a data update to persistent storage, the evaluating comprising valuating a risk of loss of data relating to a message and evaluating a potential cost of said loss based on a value of the message, the evaluating additionally comprising receiving a value representing a number of messages read from a queue per second, applying an exponential smoothing algorithm to the value representing the number of messages read from the queue per second to generate a message processing rate value; and

in response to a determination that the data update should be saved to persistent storage, initiating, by the persistence manager, a save of the data update to persistent storage;

in response to determining that the data update should not be saved to the persistent storage, setting, by the persistence manager, a timer for a next evaluation;

determining, by the persistence manager, whether to save the data update to the persistent storage based upon evaluating the at least one criterion of the set of criteria at an end of a set time period resulting from an expiration of the timer; and

,

saving the data update to the persistent storage upon a determination to save the data update to the persistent storage.

12. The method of claim 11, wherein the value of the message is determined from a monetary value within the message content.

13. The method of claim 11, wherein the value of the message is inferred from attributes of a message queue to which the message is sent.

14. The method of claim 11, wherein the value of the message is an attribute of the message specified via a messaging API.

15. The method of claim 11, for use in a messaging system, wherein the step of evaluating at least one criterion comprises determining the current size of a queue of persistent messages.

16. The method of claim 11, for use in a messaging system, wherein the step of evaluating is performed in response to performance of an operation in relation to a message, to determine whether data relating to the performed operation requires saving to persistent storage.

17. The method of claim 11, for use in a messaging system, wherein the step of evaluating comprises comparing the time period for which a message has been enqueued at the messaging system with a threshold time period.

18. The method of claim 11, for use in a messaging system, wherein the step of evaluating is performed in response to identification of a message attribute indicating that the message is a candidate for saving to persistent storage.

19. The method of claim 11, wherein the step of evaluating is performed by a persistence manager that comprises:

a data collector component; and

an evaluator component for applying a stored set of rules to data collected by the data collector component, to determine whether to save a data update to persistent storage.