USER-DEFINED LOADING OF DATA ONTO A DATABASE

• US 20150178342A1
• Filed: 06/04/2012
• Published: 06/25/2015
• Est. Priority Date: 06/04/2012
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Andrew Allinson Lamb
• Inventors: Bear, Charles Edward, Lamb, Andrew Allinson, Seering, Adam, Venkatesh, Rajat, Lawande, Shipa
• Granted Patent: US 10,474,658 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 7/36 Combined merging and sorting
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561

Abstract

As part of managing the loading of data from a source onto a database, according to an example, an interface through which a user is to define logic related to the loading of the data onto the database is provided. The user-defined logic pertains to at least one of a user-defined location identification of the source, a user-defined filter to be applied on the data, and a user-defined parsing operation to be performed on the data to convert the data into an appropriate format for the database. In addition, the user-defined logic is received and the user-defined logic is implemented to load the data onto the database.

Claims

1. A method for managing loading of data from a source onto a database, said method comprising:

providing an interface through which a user is to define logic related to the loading of the data onto the database, wherein the user-defined logic pertains to a user-defined location identification of the source, a user-defined filter to be applied on the data, and a parsing operation to be performed on the data to convert the data into an appropriate format for the database;

receiving the user-defined logic; and

implementing, by a processor, the user-defined logic to load the data onto the database.

2. The method according to claim 1, wherein providing the interface further comprises providing the interface through which the user is to define the logic by programming the logic, said method further comprising:

storing the programmed logic in a library.

3. The method according to claim 1, wherein providing the interface further comprises providing access to a library on which is stored previously defined logic through the interface, and wherein receiving the user-defined logic further comprises receiving previously stored logic from the library.

4. The method according claim 1, wherein the user-defined filter to be applied on the data is to at least one of uncompress, compress, and convert the data.

5. The method according to claim 1, wherein implementing the user-defined logic further comprises implementing the user-defined logic as part of a pre-configured data-load stack.

6. The method according to claim 1, wherein implementing the user-defined logic further comprises:

retrieving the data from the source using the user-defined location identification of the source;

filtering the data using the user-defined filter;

parsing the data through performance of the parsing operation; and

loading the parsed data onto the database.

7. The method according to claim 6, wherein retrieving, filtering, parsing, and loading the parsed data further comprises retrieving, filtering, parsing, and loading the parsed data as the data is streamed from the source.

8. The method according to claim 1, wherein the user-defined logic further pertains to a user-defined policy that defines a condition to be met prior to the data being retrieved from the source.

9. The method according to claim 1, wherein the user-defined logic further pertains to a user-defined policy that defines how data loaded onto the database is to be managed.

10. The method according to claim 1, wherein implementing the user-defined logic further comprises implementing the user-defined logic in a system that manages the database, wherein the system that manages the database comprises a distributed database system and wherein implementing the user-defined logic further comprises implementing the user-defined logic on a plurality of the systems in the distributed database system while implementing a database optimization operation.

11. An apparatus for managing loading of data from a source onto a database, said apparatus comprising:

at least one module to,

provide an interface through which a user is to define functions related to the loading of the data onto the database, wherein the user-defined functions pertain to a user-defined location identification of the source, a user-defined filter to be applied on the data, and a parsing operation to be performed on the data to convert the data into a format that is compatible with the database;

receive the user-defined functions; and

implement the user-defined functions to load the data onto the database;

a processor to implement the at least one module.

12. The apparatus according to claim 11, wherein the user-defined logic further pertains to a user-defined policy that defines at least one of a condition to be met prior to the data being retrieved from the source and management of the data loaded onto the database, and wherein the at least one module is further to:

retrieve the data from the source using the user-defined location identification of the source according to the user-defined policy;

filter the data using the user-defined filter;

parse the data through performance of the parsing operation; and

load the parsed data onto the database.

13. The apparatus according to claim 11, wherein the apparatus comprises a database management apparatus.

14. The apparatus according to claim 11, wherein the database is a relational database.

15. A non-transitory computer readable storage medium on which is stored machine readable instructions that when executed by a processor, implement a method for managing loading of data from a source onto a database, said machine readable instructions comprising code to:

provide an interface through which a user is to define code related to the loading of the data onto the database, wherein the user-defined code pertains to a user-defined location identification of the source, a user-defined filter to be applied on the data, a user-defined policy that defines at least one of a condition to be met prior to the data being retrieved from the source and management of the data loaded onto the database, and a parsing operation to be performed on the data to convert the data into format that is compatible with the database;

receive the user-defined code;

implement the user-defined code to transform the data into a format suitable for loading the data onto the database; and

implementing the user-defined code to load the data onto the database.