System and method for data retrieval and collection in a structured format

• US 20040015523A1
• Filed: 07/18/2002
• Published: 01/22/2004
• Est. Priority Date: 07/18/2002
• Status: Active Grant

• Current Assignee: Uab Oxylabs
• Original Assignee: International Business Machines Corporation
• Inventors: Sharp, Frederick Thomas, Guest, Michael Alan
• Granted Patent: US 7,020,667 B2
• Time in Patent Office: Days
• US Class Current: 707/204
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: Y10S 707/951 Calendar or scheduling
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

A data collection and retrieval system and associated method allow the capture and replication of data presented at various Web pages into a database application through text parsing of the HTML source code of that document. The system allows the user to select one or more Web pages containing data of interest; to specify exactly which data within any page is to be captured; to specify how frequently data is to be collected; and to specify the conditions for collection and retrieval. The advantage of the system is realized through efficient, automated data collection that would otherwise be impractical. The system includes an initialization stage and an automatic execution stage. The initialization stage provides the user interface which allows the user to select the source file that contains data the user wishes to copy, target database that will receive the data, and timing criteria for automatic transfer of data. The automatic execution stage automatically transfers the data from the source file to the target database as instructed by the user in the initialization stage.

Claims

1. A method of selectively retrieving data from a source file and automatically collecting retrieved data in a target database, comprising:

initializing a data retrieval stage by;

selecting the source file;

parsing selected data elements in the source file into a list of data elements;

selecting at least one of the data elements from the list to be stored in the target database;

selecting the target database and a schema in which the at least one of the data elements is to be stored in the target database; and

automatically collecting at least one of the parsed data elements by;

transferring at least one of the parsed data elements from the source file to the target database; and

storing at least one of the transferred data elements in the target database in the schema.

2. The method of claim 1, wherein initializing the data retrieval stage further comprising setting a schedule frequency for transferring the at least one of the parsed data elements from the source file to the target database.

3. The method of claim 1, further comprising processing the at least one of the transferred data elements stored in the target database.

4. The method of claim 3, wherein processing the at least one of the transferred data elements is executed automatically.

5. The method of claim 4, wherein initializing the data retrieval stage is executed automatically.

6. The method of claim 1, wherein the data structure format is different from the schema of the target database.

7. The method of claim 6, wherein the data structure format is a network-based format.

8. The method of claim 7, wherein the data structure format is a markup language.

9. The method of claim 8, wherein the data structure format is HTML.

10. The method of claim 6, wherein the target database is a relational database.

11. The method of claim 1, wherein initializing the data retrieval stage further comprises presenting selection options to a user.

12. The method of claim 1, wherein initializing the data retrieval stage further comprises presenting options for the design of the schema to a user.

13. The method of claim 1, wherein the at least one of the data elements is dynamic.

14. The method of claim 13, wherein initializing the data retrieval stage further comprises analyzing the at least one of the data elements.

15. The method of claim 14, wherein the at least one of the data elements comprises data that changes periodically.

16. The method of claim 1, wherein processing the at least one of the transferred data is executed manually.

17. The method of claim 1, further comprising automatically updating the at least one of the transferred data elements that are stored in the target database.

18. The method of claim 17, wherein automatically updating the at least one of the transferred data elements comprises reading control information from database tables;

and

comparing at least one of the selected data elements to the corresponding at least one of the data elements stored in the database.

19. The method of claim 1, wherein automatically updating the at least one of the transferred data elements comprises periodically updating the at least one of the transferred data elements.

20. The method of claim 1, further comprising reinitializing the data retrieval stage.

21. A computer program product having instruction codes for selectively retrieving data from a source file and for automatically collecting retrieved data in a target database in a schema, comprising:

a first set of instruction codes for initializing a data retrieval stage by;

selecting the source file;

parsing selected data elements in the source file into a list of data elements;

selecting at least one of the data elements from the list to be stored in the target database;

selecting the target database and the schema in which the at least one of the data elements is to be stored in the target database; and

a second set of instruction codes for automatically collecting at least one of the parsed data elements by;

transferring at least one of the parsed data elements from the source file to the target database; and

storing at least one of the transferred data elements in the target database in the schema.

22. The computer program product of claim 21, wherein the data structure format is different from the schema of the target database.

23. The computer program product of claim 22, wherein the data structure format is a network-based format.

24. The computer program product of claim 23, wherein the data structure format is a markup language.

25. The computer program product of claim 24, wherein the data structure format is HTML.

26. The computer program product of claim 21, wherein the target database is a relational database.

27. The computer program product of claim 21, wherein the at least one of the data elements is dynamic.

28. The computer program product of claim 21, wherein the at least one of the data elements comprises data that changes periodically.

29. The computer program product of claim 21, further comprising a third set of instruction codes for presenting transfer options to a user.

30. The computer program product of claim 21, further comprising a fourth set of instruction codes for reinitializing the data retrieval stage.

31. A system for selectively retrieving data from a source file and for automatically collecting retrieved data in a target database in a schema, comprising:

means for initializing a data retrieval stage by;

selecting the source file;

parsing selected data elements in the source file into a list of data elements;

selecting at least one of the data elements from the list to be stored in the target database;

selecting the target database and the schema in which the at least one of the data elements is to be stored in the target database; and

means for automatically collecting at least one of the parsed data elements by;

transferring at least one of the parsed data elements from the source file to the target database; and

storing at least one of the transferred data elements in the target database in the schema.

32. The system of claim 31, wherein the data structure format is different from the schema of the target database.

33. The system of claim 32, wherein the data structure format is a network-based format.

34. The system of claim 33, wherein the data structure format is a markup language.

35. The system of claim 34, wherein the data structure format is HTML.

36. The system of claim 31, wherein the target database is a relational database.

37. The system of claim 31, wherein the at least one of the data elements is dynamic.

38. The system of claim 31, wherein the at least one of the data elements comprise data that changes periodically.

39. The system of claim 31, further comprising means for presenting transfer options to a user.

40. The system of claim 31, further comprising means for reinitializing the data retrieval stage.