System and methodology for extraction and aggregation of data from dynamic content

• US 8,060,518 B2
• Filed: 06/26/2007
• Issued: 11/15/2011
• Est. Priority Date: 02/08/2000
• Status: Expired due to Fees

• Current Assignee: Sybase Incorporated (SAP SE)
• Original Assignee: Sybase Incorporated (SAP SE)
• Inventors: Timmons, Michael
• Primary Examiner(s): Wu; Yicun
• Assistant Examiner(s): DARNO, PATRICK A
• Application Number:
  US11/768,290
• Publication Number:
  US 20090024574A1
• Time in Patent Office: 1,603 Days
• Field of Search: 707/755, 707/756, 707/999.001, 707/999.1, 707/999.101
• US Class Current: 707/755
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system and methodology for extraction and aggregation of data from dynamic content is described. In one embodiment, for example, a method is described for extracting and structuring items of data from content available via the Internet, the method comprises steps of: receiving input of a user specifying at least one source of content available via the Internet, types of data to be extracted from the at least one source, and fields for structuring extracted items of data; retrieving content from the at least one source; parsing the retrieved content to extract items of data of the types specified by the user; and mapping the extracted items of data to the fields specified by the user so as to transform the extracted items of data into a structured format.

Claims

1. A method for extracting and structuring items of data from content available via the Internet, the method comprising:

receiving at a server information from a user'"'"'s computer specifying at least (a) a Web page available via the Internet, (b) types of data to be extracted from the Web page, and (c) fields for structuring extracted items of data;

retrieving the Web page from the Internet yielding retrieved content;

processing the retrieved content including;

(a) parsing one or more container objects in the retrieved content, wherein the container objects correspond to the types of data, and

(b) creating feature tags based on content of respective ones of the container objects, and using at least one of the feature tags to extract data of the types specified according to the information received at the server yielding extracted data;

mapping the extracted data to the fields so as to transform the extracted data into a structured format yielding structured data; and

returning aspects of the structured data to the user'"'"'s computer,

wherein the feature tags each comprise a series of alphanumeric characters that represent (i) a container object type, (ii) a version of the container object, and (iii) an occurrence of the container object within the retrieved content.

2. The method of claim 1, wherein the creating one or more feature tags step includes creating information for an element selected from one of a headline, a graphic object, a button, and a run of text.

3. The method of claim 1, wherein the mapping the extracted data step includes generating an XML document including aspects of the extracted data.

4. The method of claim 1, wherein the mapping the extracted data step includes saving aspects of the extracted data to a repository.

5. The method of claim 1, further comprising using HTML TABLE tags to identify container objects.

6. The method of claim 1, further comprising looping through each container object and looping through all container objects on the Web page to create the feature tags.

7. The method of claim 1, wherein said Web page comprises a selected one of an HTML page, a cHTML page, and an XHTML page.