Intellegent data search engine

• US 8,190,556 B2
• Filed: 09/08/2009
• Issued: 05/29/2012
• Est. Priority Date: 08/24/2006
• Status: Active Grant

• Current Assignee: Data Record Science, Inc.
• Original Assignee: Derek Edwin Pappas
• Inventors: Pappas, Derek Edwin
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US12/555,083
• Publication Number:
  US 20090327249A1
• Time in Patent Office: 994 Days
• Field of Search: 1/1, 707/706, 707/709, 707/710, 707/736, 707/602
• US Class Current: 707/602
• CPC Class Codes: G06F 16/2423 Interactive query statement specification based on a database schema
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Automatically extracting information that matches a predetermined criterion from one or more web pages at one or more web sites and automatically producing one or more extracted data-field names from the information extracted from the one or more web pages at the one or more web sites. The extracted information includes at least one extracted data-field value associated with one of the one or more extracted data-field names. If one of the extracted data-field names matches an existing data-field name in a previously constructed data base including one or more data fields each associated with a data-field name and a data-field value, the method updates an extracted data-field value associated with the data-field name in the data base. If one of the extracted data field names does not match any of the existing data-field names in the data base, the method adds the extracted data-field name to the data base.

Claims

1. A computer-implemented method for extracting product information displayed on a plurality of web pages on a first remote web site representing a first database and storing said product information in a second database, comprising:

requesting and, in response to said request, receiving said plurality of web pages related to one or more products from said first remote website;

automatically identifying a first cluster of web pages associated with said first remote web site that comprises product information from said first database, said product information displayed as data field values (DFVs) related to the one or more products and exhibiting common identification characteristics in each web page of said first cluster of web pages;

structurally comparing a first sample of web pages within said first cluster of web pages and creating an intersection data structure comprising the structural location of said DFVs of database records (DBRs), and inferring data field names (DFNs) associated with keywords, symbols, or patterns, indicating the location of said DFVs in said intersection data structure;

automatically deriving a first extraction template from said intersection data structure associated with said first sample of web pages within said first cluster of web pages;

utilizing said first extraction template for extracting said DFVs from said first cluster of web pages to create, in association with said inferred DFNs, extracted DBRs;

and storing said extracted DBRs in said second database.

2. The method of claim 1, wherein inferring the DFNs include determining a common group of DFVs based on a common relative position of the DFVs in each web page of the first sample of web pages.

3. The method of claim 1, wherein inferring the DFNs include identifying a semantic type associated with each DFV in each web page of the first sample of web pages.

4. The method of claim 1, wherein inferring the DFNs associated with a common group of DFVs which are product values spanning the first cluster of web pages on a web page from the first cluster of web pages from the first remote web site includes correlating a DFN, from a reference database of verified DBRs from second remote web site which contain DFNs and DFVs.

5. The method of claim 1, wherein deriving the first extraction template includes finding an intersection data structure that represents an order of relative positioning across the first sample of web pages.

6. The method of claim 5, wherein finding the intersection includes representing a source code associated with a web page as a data structure to allow a comparison of the order of markup tags, key words, symbols, image links and markers.

7. The method of claim 1, wherein the second database is populated with product information extracted from a plurality of web pages generated from a plurality of databases and displayed on a plurality of remote web sites, where the product information is extracted with an extraction template specific to each remote web site.

8. The method of claim 7, wherein product information extracted from one of the plurality of databases can be used to enhance and verify product information extracted from the first database.

9. The method of claim 1, further comprising:

structurally comparing a second sample of web pages within a second cluster of web pages from the first remote web site and automatically deriving from an intersection of common identification characteristics a second extraction template,

wherein the second cluster of web pages is structurally distinct from the first cluster of web pages.

10. The method of claim 1, further comprising:

structurally comparing a third sample of web pages within a third cluster of web pages from a second remote web site and automatically deriving a third extraction template,

wherein a second extraction template using a second sample of web pages from a second cluster of web pages from the first remote web site has been derived, and the third cluster of web pages is structurally distinct from the first cluster of structured web pages.

11. The method of claim 10, wherein the second remote web site is associated with a dealer or authorized reseller related to products of the first remote web site, and DFVs extracted from the second remote web site can be used to verify the first extraction template and DFVs extracted from the first remote web site.

12. The method of claim 1, wherein populating the second database includes normalizing the DFNs to reduce duplicate product information.

13. The method of claim 1, further comprising:

displaying search results from the second database responsive to a user query.

14. The method of claim 13, wherein displaying search results includes displaying a table which compares and sorts different products based on a criteria.

15. The method of claim 13, wherein displaying search results includes displaying aggregated product information for a product, the aggregated product information being sourced from different web pages or different remote web sites.

16. The method of claim 1, wherein the first cluster of web pages from the first remote web site are automatically created from product information stored in the first database by a database report generator.

17. The method of claim 1, wherein identifying the first cluster of web pages includes finding structurally similar pages that contain markup tags, keywords and semantic types, and image links in the same relative order, and finding structurally similar pages that contain DFVs type information.