Collecting data from data sources

• US 8,738,565 B2
• Filed: 11/07/2005
• Issued: 05/27/2014
• Est. Priority Date: 11/07/2005
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cook, Jonathan Michael, Rendahl, Randy Allan, Robinson, David G., Shi, Shaw-Ben, Todd, Byron Eugene III
• Primary Examiner(s): Uddin, Md. I
• Application Number:
  US11/268,978
• Publication Number:
  US 20070112889A1
• Time in Patent Office: 3,123 Days
• Field of Search: 707/204, 707/200, 707/600, 707/602, 707/661, 707/736
• US Class Current: 707/602
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/972 Access to data in other repository systems, e.g. legacy data or dynamic Web page generation

Abstract

A computer implemented method, apparatus, and computer usable program code for collecting data. The data is collected from a particular data source by an application portion of agent to form collected data. The application portion is associated with the particular data source in response to an event. The collected data stored in a temporary data store for transfer to a data storage system to form stored data. The stored data sent to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system.

Claims

1. A computer implemented method for collecting data, the computer implemented method comprising:

collecting data from a particular data source by an application portion of agent to form collected data, wherein the application portion is associated with the particular data source in response to an event;

storing, by the application portion of the agent, the collected data in a data file within a data store device for transfer to a data storage system to form stored data in the data store device;

sending the stored data to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system, wherein both the application portion of the agent and the generic portion of the agent directly access the data store device by calling a respective application programming interface (API), and wherein the event is an invocation of the application portion of the agent by the generic portion of the agent that occurs responsive to information received by the generic portion of the agent that specifies the particular data source as well as an expected format of the data to collect from the particular data source; and

selectively transforming the collected data into a format for aggregation with other data in the data storage system using an aggregation policy that specifies attributes for aggregating the collected data and the other data into different time-based granularities, wherein the generic portion of the agent comprises (i) a generic application programming interface that the application portion of the agent calls using an application programming interface portion of the application agent in order to store the collected data in the data file, and (ii) an intelligent remote agent application programming interface that is called by generic agent code of the generic portion of the agent to both retrieve the stored data from the data file and send the stored data to the data storage system.

2. The computer implemented method of claim 1 wherein the collected data in the data file is sent to the data storage system upon expiration of a timer.

3. The computer implemented method of claim 1 further comprising receiving the stored data by the data storage system to form received data, and pruning the received data by a pruning agent based on a pruning policy that defines when certain warehoused data in the data storage system is to be deleted from the data storage system, wherein the event is a periodic event.

4. The computer implemented method of claim 1, wherein the collecting step comprises:

establishing a connection to the data source; and

sending requests to the data source for the data.

5. A data processing system comprising a processor coupled to a memory having instructions stored therein that are operable, when executed by the process, to perform a method comprising:

collecting data from a particular data source by an application portion of agent to form collected data, wherein the application portion is associated with the particular data source in response to an event;

storing, by the application portion of the agent, the collected data in a data file within a data store device for transfer to a data storage system to form stored data in the data store device; and

sending the stored data to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system, wherein both the application portion of the agent and the generic portion of the agent directly access the data store device by calling a respective application programming interface (API), and wherein the event is an invocation of the application portion of the agent by the generic portion of the agent that occurs responsive to information received by the generic portion of the agent that specifies the particular data source as well as an expected format of the data to collect from the particular data source, wherein the generic portion of the agent comprises (i) a generic application programming interface that the application portion of the agent calls using an application programming interface portion of the application agent in order to store the collected data in the data file, and (ii) an intelligent remote agent application programming interface that is called by generic agent code of the generic portion of the agent to both retrieve the stored data from the data file and send the stored data to the data storage system, wherein the data is collected according to a collection window and the stored data is sent to the data storage system according to a warehouse window that expires every nth time the collection window expires.

6. An agent stored in a non-transitory computer readable medium of a data processing system and operable by a data processor of the data processing system for collecting data when executed by the data processor, the agent perform the operation of:

collecting data from a particular data source by an application portion of agent to form collected data, wherein the application portion is associated with the particular data source in response to an event;

storing, by the application portion of the agent, the collected data in a data file within a data store device for transfer to a data storage system to form stored data in the data store device;

sending the stored data to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system, wherein both the application portion of the agent and the generic portion of the agent directly access the data store device by calling a respective application programming interface (API), and wherein the event is an invocation of the application portion of the agent by the generic portion of the agent that occurs responsive to information received by the generic portion of the agent that specifies the particular data source as well as an expected format of the data to collect from the particular data source; and

selectively transforming the collected data into a format for aggregation with other data in the data storage system using an aggregation policy that specifies attributes for aggregating the collected data and the other data into different time-based granularities, wherein the generic portion of the agent comprises (i) a generic application programming interface that the application portion of the agent calls using an application programming interface portion of the application agent in order to store the collected data in the data file, and (ii) an intelligent remote agent application programming interface that is called by generic agent code of the generic portion of the agent to both retrieve the stored data from the data file and send the stored data to the data storage system.

7. The agent of claim 6, wherein the generic agent code calls the intelligent remote agent application programming interface to register the table.