DATA INGESTION PROCESS

• US 20160299957A1
• Filed: 03/04/2016
• Published: 10/13/2016
• Est. Priority Date: 04/09/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: A. C., Charan, Hegde, Santosh, Narayanaswamy, Mohan, Saxena, Gaurav
• Granted Patent: US 10,650,016 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A method, system and computer program product is disclosed for performing data ingestion. A request is received for data ingestion. Based on the request received, a client resource information with a server resource information is compared. After the comparing of client resource information with server resource information, location for formatting of data during data ingestion is determined. Further, in response to the determination of location, formatting of the data is performed at the determined location.

Claims

1. A method for performing data ingestion, the method comprising:

receiving a request for data ingestion;

receiving a plurality of computational resource information respectively providing information about current resource(s) availability for a plurality of computational resources sets;

responsive to the request for data ingestion, comparing the plurality of computational resource information to each other to determine a selected computational resource set based at least in part upon current resource(s) availability;

in response to the determination of a selected computational resource set, sending data for the requested data ingestion to the selected computational resource set;

starting the formatting of the data for the requested data ingestion at the selected resource set;

subsequent to the start of the formatting of the data, re-comparing the plurality of computational resource information to each other to determine a dynamically-selected computational resource set based at least in part upon current resource(s) availability;

dynamically switching a partially performed formatting of the data from the selected computational resource set to the dynamically-selected computational resource set; and

executing remaining formatting of the data at the dynamically-selected computational resource set;

wherein the re-comparison of the plurality of computational resource information sets is performed in response to at least one of the following;

(i) time elapsed since progress of the formatting of the data, (ii) data having been formatted in a commit interval, and/or (iii) a call back event indicating change in resource availability.