System and Methodology for Extraction and Aggregation of Data from Dynamic Content

• US 20040199497A1
• Filed: 05/07/2004
• Published: 10/07/2004
• Est. Priority Date: 02/08/2000
• Status: Active Grant

• Current Assignee: Sybase Incorporated (SAP SE)
• Original Assignee: Sybase Incorporated (SAP SE)
• Inventors: Timmons, Michael
• Granted Patent: US 7,240,067 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system and methodology for extraction and aggregation of data from dynamic content is described. In one embodiment, for example, a method is described for extracting and structuring items of data from content available via the Internet, the method comprises steps of: receiving input of a user specifying at least one source of content available via the Internet, types of data to be extracted from the at least one source, and fields for structuring extracted items of data; retrieving content from the at least one source; parsing the retrieved content to extract items of data of the types specified by the user; and mapping the extracted items of data to the fields specified by the user so as to transform the extracted items of data into a structured format.

Claims

1. A method for extracting and structuring items of data from content available via the Internet, the method comprising:

receiving input of a user specifying at least one source of content available via the Internet, types of data to be extracted from said at least one source, and fields for structuring extracted items of data;

retrieving content from said at least one source;

parsing the retrieved content to extract items of data of the types specified by the user; and

mapping the extracted items of data to the fields specified by the user so as to transform the extracted items of data into a structured format.

2. The method of claim 1, wherein said at least one source of content includes a Web site.

3. The method of claim 1, wherein said at least one source of content includes an HTML page.

4. The method of claim 1, wherein said receiving step includes receiving a URL specifying a source of content available via the Internet.

5. The method of claim 1, wherein said receiving step includes receiving user input specifying attributes of data to be extracted from said at least one source.

6. The method of claim 1, wherein said retrieving step includes retrieving a Web page.

7. The method of claim 6, wherein said Web page comprises a selected one of an HTML page, an cHTML page, and an XHTML page.

8. The method of claim 6, wherein said parsing step includes parsing container objects of the Web page.

9. The method of claim 8, wherein said parsing step includes creating a new object for a particular container object of the Web page, the new object containing information for the particular container object.

10. The method of claim 8, wherein said parsing step includes creating feature tags for elements of the container objects.

11. The method of claim 10, wherein said creating feature tags step includes creating information for selected one of a headline, a graphic object, a button, and a run of text.

12. The method of claim 10, wherein said creating feature tags includes creating feature tags based on attributes of each element.

13. The method of claim 10, further comprising:

saving at least some of the feature tags; and

subsequently, using a saved feature tag to retrieve an element from the Web page.

14. The method of claim 1, wherein said mapping step includes receiving user input for selecting a particular field for placing a given item of data.

15. The method of claim 14, wherein said mapping step further comprises automatically mapping other items of data similar to the given item of data to the particular field.

16. The method of claim 1, wherein said mapping step includes generating an XML document including the extracted items of data.

17. The method of claim 1, wherein said mapping step includes saving the extracted information to a database table.

18. A computer-readable medium having processor-executable instructions for performing the method of claim 1.

19. A downloadable set of processor-executable instructions for performing the method of claim 1.