Method and apparatus for collecting data from data sources

• US 20070112889A1
• Filed: 11/07/2005
• Published: 05/17/2007
• Est. Priority Date: 11/07/2005
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cook, Jonathan, Robinson, David, Shi, Shaw-Ben, Rendahl, Randy, Todd, Byron
• Granted Patent: US 8,738,565 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/972 Access to data in other repository systems, e.g. legacy data or dynamic Web page generation

Abstract

A computer implemented method, apparatus, and computer usable program code for collecting data. The data is collected from a particular data source by an application portion of agent to form collected data. The application portion is associated with the particular data source in response to an event. The collected data stored in a temporary data store for transfer to a data storage system to form stored data. The stored data sent to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system.

Claims

1. A computer implemented method for collecting data, the computer implemented method comprising:

collecting data from a particular data source by an application portion of agent to form collected data, wherein the application portion is associated with the particular data source in response to an event;

storing the collected data in a temporary data store for transfer to a data storage system to form stored data; and

sending the stored data to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system.

2. The computer implemented method of claim 1 further comprising:

selectively transforming the collected data into a format for aggregation with other data if the collected data is not in a format useable by the data warehouse.

3. The computer implemented method of claim 2, wherein the format is a table.

4. The computer implemented method of claim 1, wherein the storing step comprises:

storing the collected data in a binary flat data file for sending to the data storage system.

5. The computer implemented method of claim 1, wherein the event is a periodic event.

6. The computer implemented method of claim 1, wherein the collecting step comprises:

establishing a connection to the data source; and

sending requests to the data source for the data.

7. The computer implemented method of claim 1, wherein the transformed data is sent to the data warehouse through a warehouse proxy.

8. The computer implemented method of claim 1, wherein the agent comprises a generic agent and an application agent.

9. A computer program product comprising:

A computer usable medium having computer usable program code for collecting data, the computer program product including;

computer usable program code for collecting data from a particular data source by an application portion of agent to form collected data, wherein the application portion is associated with the particular data source in response to an event;

computer usable program code for storing the collected data in a temporary data store for transfer to a data storage system to form stored data; and

computer usable program code for sending the stored data to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system.

10. The computer program product of claim 9 further comprising:

computer usable program code for selectively transforming the collected data into a format for aggregation with other data if the collected data is not in a format useable by the data warehouse.

11. The computer program product of claim 10, wherein the format is a table.

12. The computer program product of claim 9, wherein the computer usable program code for storing the collected data in a temporary data store for transfer to a data storage system to form stored data comprises:

computer usable program code for storing the collected data in a binary flat data file for sending to the data storage system.

13. The computer program product of claim 9, wherein the event is a periodic event.

14. The computer program product of claim 9, wherein the computer usable program code for collecting data from a data source by an agent associated with the data source in response to an event to form collected data comprises:

computer usable program code for establishing a connection to the data source; and

computer usable program code for sending requests to the data source for the data.

15. The computer program product of claim 9, wherein the transformed data is sent to the data warehouse through a warehouse proxy.

16. The computer program product of claim 9, wherein the agent comprises a generic agent and an application agent.

17. An agent for collecting data, the apparatus comprising:

an application agent, wherein the application agent collects the data from a data source to form collected data; and

a generic agent, wherein the generic agent communicates with the application agent using an interface system, wherein the generic agent receives the collected data from the application agent and transfers the collected data to a data storage system.

18. A data processing system for collecting data, the data processing system comprising:

collecting means for collecting data from a particular data source by an application portion of agent to form collected data, wherein the application portion is associated with the particular data source in response to an event;

storing means for storing the collected data in a temporary data store for transfer to a data storage system to form stored data; and

sending means for sending the stored data to the data storage system using a generic portion of the agent, wherein the generic portion of the agent is configured to send data to the data storage system.