Data extraction system and method

• US 20030182283A1
• Filed: 03/22/2002
• Published: 09/25/2003
• Est. Priority Date: 03/22/2002
• Status: Abandoned Application

• Current Assignee: NCR Corporation (NCR Voyix Corporation)
• Original Assignee: NCR Corporation (NCR Voyix Corporation)
• Inventors: Browning, James L., Bean, Thomas A., Smith, Tucker L., Carty, Scott D.
• Application Number:
  US10/104,659
• Publication Number:
  US 20030182283A1
• Time in Patent Office: Days
• US Class Current: 707/6
• CPC Class Codes: G06F 16/972 Access to data in other repository systems, e.g. legacy data or dynamic Web page generation

Abstract

One embodiment of the present invention relates to a system for extracting online data from a variety of web-sources in real-time and transmitting the data to a data warehouse in real-time. The system comprises an extractor plug-in having instructions configured to integrate with a pre-determined type of host server and extracting data in real-time from any variety of web source in communication with the host server. The system also comprises a transformer engine in communication with the extractor plug-in and configured to transmit the extracted data in real-time into a data warehouse for analysis thereof.

Claims

1. A system for extracting data from a variety of web-sources, the system comprising:

an extractor plug-in comprising instructions configured to integrate with a plurality of pre-determined types of host servers and extract data in real-time from a plurality of types of web sources in communication with the host server; and

a transformer engine in communication with the extractor plug-in and configured to transmit the extracted data into a data warehouse for analysis thereof.

2. The system of claim 1, further comprising:

a configurator comprising instructions in communication with the extractor, the configurator operable to allow a user to configure parameters of the extractor for identifying data to be extracted.

3. The system of claim 1, wherein the host server comprises a web server.

4. The system of claim 1, further comprising:

an output pipe in communication with said extractor plug-in, wherein the data extracted from the web-source is transmitted to the output pipe utilizing the format of the pre-determined web-source.

5. The system of claim 4, wherein the output pipe comprises at least one of a named pipe and a message queue.

6. The system of claim 1, wherein the transformer engine comprises a continuous load utility.

7. The system of claim 1, wherein the transformer engine comprises a parallel load utility.

8. The system of claim 1, further comprising:

a buffer storage area in communication with the output pipe, the buffer storage area configured to temporarily store data transmitted from the output pipe before being transmitted to a data warehouse.

9. A system for extracting data from a variety of web-sources, the system comprising:

a batch extractor comprising instructions configured to integrate with a plurality of pre-determined types of host servers and extract data in batch from a plurality of types of web sources in communication with the host server; and

a transformer engine in communication with the extractor and configured to transmit the extracted data in batch into a data warehouse for analysis thereof.

10. The system of claim 9, further comprising:

a configurator comprising instructions in communication with the extractor-in, the configurator operable to allow a user to configure parameters of the batch extractor for identifying data to be extracted.

11. The system of claim 10, further comprising:

an output pipe in communication with said batch extractor, wherein the data extracted from the web-source is transmitted to the output pipe utilizing the format of the pre-determined web-source.

12. The system of claim 9, wherein the output pipe comprises one of the following:

a named pipe and a message queue.

13. The system of claim 9, further comprising:

a buffer storage area in communication with said batch extractor configured to receive data.

14. The system of claim 9, wherein the transformer engine comprises a parallel load utility.

15. A method of extracting data from a variety of web-sources, the method comprising:

identifying a type of web source in communication with a host server;

selecting an extraction protocol based on the identified type of web source; and

executing the extraction protocol to extract data from the web source.

16. The method of claim 15, further comprising the step of:

transmitting the extracted data in real time to a data warehouse.

17. The method of claim 15, further comprising the step of:

transmitting the data in batch to a data warehouse.

18. The method of claim 15, further comprising the step of:

receiving extracted data in an output pipe utilizing the data format of the web source.

19. The method of claim 15, further comprising the step of:

temporarily storing extracted data in a buffer storage area before transmitting the data to a data warehouse.

20. The method of claim 15, further comprising the step of:

allowing a user to configure parameters identifying data to be extracted.