Workload discovery using real-time analysis of input streams

• US 10,108,689 B2
• Filed: 11/09/2016
• Issued: 10/23/2018
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cadarette, Paul M., Spyker, James D.
• Primary Examiner(s): Liao, Jason G
• Application Number:
  US15/347,713
• Publication Number:
  US 20170177691A1
• Time in Patent Office: 713 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 2201/855 Details of asynchronous mirroring using a journal to transfer not-yet-mirrored changes
• CPC Class Codes: G06F 2201/87 Monitoring of transactions

Abstract

Provided are techniques for workload discovery using real-time analysis of input streams. For a meta workload, changes to data objects made by change operations that are in a replication change stream are stored into a recovery log. Using an analytics engine, one of the recovery log and the replication change stream are analyzed to identify associations between the data objects based on usage and access patterns. The associations are used to identify sub-workloads of the meta workload that form consistency groups for replication.

Claims

1. A method, comprising:

identifying, with a processor of a computer, a single workload for analysis;

storing, with the processor of the computer, changes to data objects in a source copy made by change operations that are in a replication change stream into a recovery log;

identifying, with the processor of the computer, associations between the data objects and a plurality of applications based on usage and access patterns in one of the recovery log and the replication change stream, wherein the associations comprise any combination of;

a) a set of the data objects that are updated by a particular application of the plurality of applications;

b) a set of the data objects that are updated by a given set of applications of the plurality of applications;

c) patterns of data access of the data objects;

d) patterns of shared data access of the data objects; and

e) a set of the data objects that are referenced together in transactions; and

using, with the processor of the computer, the associations to identify change operations that form consistency groups for replication from the source copy to a target copy, and wherein the consistency groups comprise sub-workloads; and

executing, with the processor of the computer, the change operations of the sub-workloads to perform the replication for the single workload, wherein the change operations within each of the sub-workloads are executed such that the target copy is a replica of the source copy at a given point in time, and wherein the sub-workloads provide increased granularity.

2. The method of claim 1, wherein the recovery log includes transactional semantics for recovery log records, and wherein the recovery log includes audit information for the change operations.

3. The method of claim 1, further comprising:

using the recovery log to identify two-phase commit transactions for roll-up synchronization.

4. The method of claim 1, wherein the sub-workloads are used for one of partitioning and optimized processing.

5. The method of claim 1, wherein the sub-workloads are used for replication optimization.

6. The method of claim 1, wherein software is provided as a service in a cloud environment.

7. A computer program product, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by at least one processor to perform:

identifying a single workload for analysis;

storing changes to data objects in a source copy made by change operations that are in a replication change stream into a recovery log;

identifying associations between the data objects and a plurality of applications based on usage and access patterns in one of the recovery log and the replication change stream, wherein the associations comprise any combination of;

a) a set of the data objects that are updated by a particular application of the plurality of applications;

b) a set of the data objects that are updated by a given set of applications of the plurality of applications;

c) patterns of data access of the data objects;

d) patterns of shared data access of the data objects; and

e) a set of the data objects that are referenced together in transactions;

using the associations to identify change operations that form consistency groups for replication from the source copy to a target copy, and wherein the consistency groups comprise sub-workloads; and

executing the change operations of the sub-workloads to perform the replication for the single workload, wherein the change operations within each of the sub-workloads are executed such that the target copy is a replica of the source copy at a given point in time, and wherein the sub-workloads provide increased granularity.

8. The computer program product of claim 7, wherein the recovery log includes transactional semantics for recovery log records, and wherein the recovery log includes audit information for the change operations.

9. The computer program product of claim 7, wherein the program code is executable by the at least one processor to perform:

using the recovery log to identify two-phase commit transactions for roll-up synchronization.

10. The computer program product of claim 7, wherein the sub-workloads are used for one of partitioning and optimized processing.

11. The computer program product of claim 7, wherein the sub-workloads are used for replication optimization.

12. The computer program product of claim 7, wherein a Software as a Service (SaaS) is configured to perform computer program product operations.

13. A computer system, comprising:

one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices; and

program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to perform operations, wherein the operations comprise;

identifying a single workload for analysis;

storing changes to data objects in a source copy made by change operations that are in a replication change stream into a recovery log;

identifying associations between the data objects and a plurality of applications based on usage and access patterns in one of the recovery log and the replication change stream, wherein the associations comprise any combination of;

a) a set of the data objects that are updated by a particular application of the plurality of applications;

b) a set of the data objects that are updated by a given set of applications of the plurality of applications;

c) patterns of data access of the data objects;

d) patterns of shared data access of the data objects; and

e) a set of the data objects that are referenced together in transactions;

using the associations to identify change operations that form consistency groups for replication from the source copy to a target copy, and wherein the consistency groups comprise sub-workloads; and

executing the change operations of the sub-workloads to perform the replication for the single workload, wherein the change operations within each of the sub-workloads are executed such that the target copy is a replica of the source copy at a given point in time, and wherein the sub-workloads provide increased granularity.

14. The computer system of claim 13, wherein the recovery log includes transactional semantics for recovery log records, and wherein the recovery log includes audit information for the change operations.

15. The computer system of claim 13, wherein the operations further comprise:

using the recovery log to identify two-phase commit transactions for roll-up synchronization.

16. The computer system of claim 13, wherein the sub-workloads are used for one of partitioning and optimized processing.

17. The computer system of claim 13, wherein a Software as a Service (SaaS) is configured to perform computer system operations.