Intellegent Data Search Engine

• US 20090327249A1
• Filed: 09/08/2009
• Published: 12/31/2009
• Est. Priority Date: 08/24/2006
• Status: Active Grant

• Current Assignee: Data Record Science, Inc.
• Original Assignee: Derek Edwin Pappas
• Inventors: Pappas, Derek Edwin
• Granted Patent: US 8,190,556 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2423 Interactive query statement specification based on a database schema
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Automatically extracting information that matches a predetermined criterion from one or more web pages at one or more web sites and automatically producing one or more extracted data-field names from the information extracted from the one or more web pages at the one or more web sites. The extracted information includes at least one extracted data-field value associated with one of the one or more extracted data-field names. If one of the extracted data-field names matches an existing data-field name in a previously constructed data base including one or more data fields each associated with a data-field name and a data-field value, the method updates an extracted data-field value associated with the data-field name in the data base. If one of the extracted data field names does not match any of the existing data-field names in the data base, the method adds the extracted data-field name to the data base.

Claims

1. A computer-implemented method for reverse engineering a first database in a second database from product information displayed on a remote web site, comprising:

receiving a first remote web site including a plurality of web pages related to one or more products;

automatically identifying a first cluster of web pages associated with the first remote web site that comprises product information from a first database, the product information displayed as data field values (DFVs) related to the one or more products and located in the same relative order in each web page of the first cluster of web pages;

automatically deriving a first extraction template from a first sample of web pages within the first cluster of web pages;

inferring data field names (DFNs) associated with the DFVs; and

populating a second database, including extracting the DFVs from the first cluster of web pages and storing the DFVs in association with an inferred DFN.

2. The method of claim 1, wherein inferring the DFNs include determining a common group of DFVs based on a common relative position of the DFVs in each web page of the first sample of web pages.

3. The method of claim 1, wherein inferring the DFNs include identifying a semantic type associated with each DFV in each web page of the first sample of web pages.

4. The method of claim 1, wherein inferring the DFNs include correlating a DFN, from a reference database of verified DFNs and DFVs, with a common group of DFVs spanning the first cluster of web pages.

5. The method of claim 1, wherein deriving the first extraction template includes finding an intersection data structure that represents an order of relative positioning across the first sample of web pages.

6. The method of claim 5, wherein finding the intersection includes representing a source code associated with a web page as a data structure to allow a comparison of the order of markup tags, key words, symbols, image links and markers.

7. The method of claim 1, wherein the second database is populated with product information extracted from a plurality of web pages generated from a plurality of databases and displayed on a plurality of remote web sites.

8. The method of claim 7, wherein product information extracted from one of the plurality of databases can be used to enhance and verify product information extracted from the first database.

9. The method of claim 1, further comprising:

deriving a second extraction template using a second sample of web pages from a second cluster of web pages including structured information and associated with the first remote web site,

wherein the second cluster of web pages is structurally distinct from the first cluster of structured web pages, the first and second clusters of web pages both associated with the first remote web site.

10. The method of claim 1, further comprising:

deriving a third extraction template using a third sample of web pages from a third cluster of web pages including structured information and associated with a second remote web site,

wherein the third cluster of web pages is structurally distinct from the first cluster of structured web pages.

11. The method of claim 10, wherein the second remote web site is associated with different products than the first remote web site.

12. The method of claim 11, wherein the second remote web site is associated with a dealer or authorized reseller related to products of the first remote web site, and DFVs extracted from the first remote web site can be used to verify the second extraction template and DFVs extracted from the second remote web site.

13. The method of claim 1, wherein the web pages can be formatted in hypertext markup language (HTML), extensible markup language (XML), portable document format (PDF), a word processor format, or a Flash format.

14. The method of claim 1, wherein populating the second database includes normalizing the DFNs to reduce duplicate product information.

15. The method of claim 1, further comprising:

displaying search results from the second database responsive to a user query.

16. The method of claim 15, wherein displaying search results includes displaying a table which compares and sorts different products based on a criteria.

17. The method of claim 15, wherein displaying search results includes displaying aggregated product information for a product, the aggregated product information being sourced from different web pages or different remote web sites.

18. The method of claim 1, wherein the product information comprises product specification information.

19. The method of claim 1, wherein the first cluster of web pages are automatically created from product information stored in the first database by a database report generator.

20. The method of claim 1, wherein identifying the first cluster of web pages includes finding structurally similar pages that contain markup tags, keywords and semantic types, and image links in the same relative order, and finding structurally similar pages that contain DFV type information.

21. A system to reverse engineer a first database in a second database from product information displayed on a remote web site, comprising:

a first module to receive a first remote web site including a plurality of web pages related to one or more products;

a second module, coupled in communication with the first module, the second module automatically identifying a first cluster of web pages associated with the first remote web site that comprise product information from a first database;

a third module, coupled in communication with the second module, the third module automatically deriving a first extraction template from a first sample of web pages within the first cluster of web pages, wherein the web pages comprise product information displayed as data field values (DFVs) related to the one or more products and located in the same relative positions across the first cluster of web pages;

a fourth module, coupled in communication with the third module, the fourth module inferring data field names (DFNs) of the DFVs; and

a fifth module, coupled in communication with the fourth module, the fifth module populating a second database, including extracting the DFVs from the first cluster of web pages and storing the DFVs in association with an inferred DFN.

22. A computer-implemented method for reverse engineering a plurality of databases in an aggregate database from product information displayed on a plurality of remote web sites, comprising:

receiving a plurality of remote web sites including a plurality of web pages related to a plurality of products;

automatically identifying a plurality of clusters of web pages, each cluster of web pages associated with at least one of the plurality of remote web sites, each remote web site comprising product information from at least one of the plurality of databases, the product information displayed as data field values (DFVs) related at least one of the one of the plurality of products and located in the same relative order in each web page of an identified cluster of web pages;

automatically deriving a plurality of extraction templates, each extraction template derived from a sample of web pages within a cluster of web pages form the plurality of clusters of web pages;

inferring data field names (DFNs) associated with the DFVs; and

populating an aggregation database, including extracting the DFVs from the plurality of clusters of web pages and storing the DFVs in association with an inferred DFN.