LEARNING FROM HISTORICAL LOGS AND RECOMMENDING DATABASE OPERATIONS ON A DATA-ASSET IN AN ETL TOOL

• US 20170308595A1
• Filed: 04/26/2016
• Published: 10/26/2017
• Est. Priority Date: 04/26/2016
• Status: Active Grant

• Current Assignee: Informatica LLC (Informatica, Inc. (California))
• Original Assignee: Informatica LLC (Informatica, Inc. (California))
• Inventors: Dey, Atreyee, Kaluskar, Sanjay, Dhansingh, Udayakumar
• Granted Patent: US 10,324,947 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/90324 using system suggestions
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A data analysis server maintains database operation history data and context data for database operations performed on tables by a set of training users. The data analysis server builds predictive models for using the maintained data to recommend database operations and operands to a set of guided users. The data analysis server trains the predictive models by determining and weighting features derived from context data that are predictive of performing database operations to tables with similar context data. Using the predictive model, the data analysis server generates recommended database operations and operands based on context data received from a data analysis application of a guided user and sends the recommendations to the data analysis application for presentation to the guided user.

Claims

1. A method for recommending a database operation to a user of an instance of a data analysis application, the method comprising:

profiling a context data log, the profiling comprising capturing database operation history data entries and training context data entries from the context data log that can be used to train a predictive model, the context data log comprising entries received from one or more instances of the data analysis application responsive to database operations being performed on tables in the data analysis application;

maintaining, for a first set of users, profiled database operation history data and profiled context data for a plurality of database operations performed on a plurality of tables;

generating a predictive model to recommend database operations to a second set of users of the data analysis application, the predictive model including a plurality of features corresponding to context data fields from the profiled context data and a plurality of classes corresponding to database operations for recommendation;

receiving an application log entry comprising application context data, the application log entry received responsive to a user of the second set of users selecting a column in a table in the instance of the data analysis application, the application context data comprising metadata associated with the table;

profiling the application log entry to capture the application context data in a format that can be used as an input to the predictive model;

determining, using the application context data as inputs to the predictive model, one or more recommended database operations; and

sending the one or more recommended database operations to the instance of the data analysis application for presentation to the user.

2. The method of claim 1, wherein generating the predictive model comprises training the predictive model, the training comprising:

determining the plurality of features by selecting the plurality of context data fields from the profiled context data;

determining the plurality of database operations for recommendation; and

determining, for each of the plurality of database operations, a feature weight for each of the plurality of features, the feature weight corresponding to a measure of predictiveness of the feature with respect to the database operation.

3. The method of claim 1, wherein the training context data comprises at least one of project metadata, worksheet metadata, and user metadata.

4. The method of claim 1, wherein the predictive model is a multinomial logistic classifier.

5. The method of claim 1, wherein the application context data comprises at least one of project metadata, worksheet metadata, and user metadata.

6. The method of claim 1, wherein the one or more recommended database operations comprise at least one of a join operation and a union operation.

7. The method of claim 1, further comprising:

generating a second predictive model to recommend database operands to the second set of users of the data analysis application, the predictive model including a plurality of features corresponding to context data fields and a plurality of classes corresponding to database operands for recommendation;

profiling the application log entry to capture the application context data in a format that can be used as an input to the second predictive model;

determining, using the one or more recommended database operations and the application context data as inputs to the second predictive model, one or more recommended database operands; and

sending the one or more recommended database operands to the instance of the data analysis application for presentation to the user.

8. A system for recommending a database operation to a user of an instance of a data analysis application, the system comprising:

a processor configured to execute modules; and

a memory storing the modules, the modules executable to perform steps comprising;

profiling a context data log, the profiling comprising capturing database operation history data entries and training context data entries from the context data log that can be used to train a predictive model, the context data log comprising entries received from one or more instances of the data analysis application responsive to database operations being performed on tables in the data analysis application;

maintaining, for a first set of users, database operation history data and training context data for a plurality of database operations performed on a plurality of tables;

generating a predictive model to recommend database operations to a second set of users of the data analysis application, the predictive model including a plurality of features corresponding to context data fields and a plurality of classes corresponding to database operations for recommendation;

receiving an application log entry comprising application context data, the application log entry received responsive to a user of the second set of users selecting a column in a table in the instance of the data analysis application, the application context data comprising metadata associated with the table;

profiling the application log entry to capture the application context data in a format that can be used as an input to the predictive model;

determining, using the application context data as inputs to the predictive model, one or more recommended database operations; and

sending the one or more recommended database operations to the instance of the data analysis application for presentation to the user.

9. The system of claim 8, wherein generating the predictive model comprises training the predictive model, the training comprising:

determining the plurality of features by selecting the plurality of context data fields from the profiled context data;

determining the plurality of database operations for recommendation; and

determining, for each of the plurality of database operations, a feature weight for each of the plurality of features, the feature weight corresponding to a measure of predictiveness of the feature with respect to the database operation.

10. The system of claim 8, wherein the training context data comprises at least one of project metadata, worksheet metadata, and user metadata.

11. The system of claim 8, wherein the predictive model is a multinomial logistic classifier.

12. The system of claim 8, wherein the application context data comprises at least one of project metadata, worksheet metadata, and user metadata.

13. The method of claim 8, wherein the one or more recommended database operations comprise at least one of a join operation and a union operation.

14. The system of claim 8, the steps further comprising:

generating a second predictive model to recommend database operands to the second set of users of the data analysis application, the predictive model including a plurality of features corresponding to context data fields and a plurality of classes corresponding to database operands for recommendation;

profiling the application log entry to capture the application context data in a format that can be used as an input to the second predictive model;

determining, using the one or more recommended database operations and the application context data as inputs to the second predictive model, one or more recommended database operands; and

sending the one or more recommended database operands to the instance of the data analysis application for presentation to the user.

15. A non-transitory computer-readable storage medium storing executable computer program instructions for recommending a database operation to a user of an instance of a data analysis application, the instructions executable to perform steps comprising:

profiling a context data log, the profiling comprising capturing database operation history data entries and training context data entries from the context data log that can be used to train a predictive model, the context data log comprising entries received from one or more instances of the data analysis application responsive to database operations being performed on tables in the data analysis application;

maintaining, for a first set of users, profiled database operation history data and profiled context data for a plurality of database operations performed on a plurality of tables;

generating a predictive model to recommend database operations to a second set of users of the data analysis application, the predictive model including a plurality of features corresponding to context data fields from the profiled context data and a plurality of classes corresponding to database operations for recommendation;

receiving an application log entry comprising application context data, the application log entry received responsive to a user of the second set of users selecting a column in a table in the instance of the data analysis application, the application context data comprising metadata associated with the table;

profiling the application log entry to capture the application context data in a format that can be used as an input to the predictive model;

determining, using the application context data as inputs to the predictive model, one or more recommended database operations; and

sending the one or more recommended database operations to the instance of the data analysis application for presentation to the user.

16. The computer-readable storage medium of claim 15, generating the predictive model comprises training the predictive model, the training comprising:

determining the plurality of features by selecting the plurality of context data fields from the profiled context data;

determining the plurality of database operations for recommendation; and

determining, for each of the plurality of database operations, a feature weight for each of the plurality of features, the feature weight corresponding to a measure of predictiveness of the feature with respect to the database operation.

17. The computer-readable storage medium of claim 15, wherein the training context data comprises at least one of project metadata, worksheet metadata, and user metadata.

18. The computer-readable storage medium of claim 15, wherein the predictive model is a multinomial logistic classifier.

19. The computer-readable storage medium of claim 15, wherein the application context data comprises at least one of project metadata, worksheet metadata, and user metadata.

20. The computer-readable storage medium of claim 15, the steps further comprising:

generating a second predictive model to recommend database operands to the second set of users of the data analysis application, the predictive model including a plurality of features corresponding to context data fields and a plurality of classes corresponding to database operands for recommendation;

profiling the application log entry to capture the application context data in a format that can be used as an input to the second predictive model;

determining, using the one or more recommended database operations and the application context data as inputs to the second predictive model, one or more recommended database operands; and

sending the one or more recommended database operands to the instance of the data analysis application for presentation to the user.