System and method for extracting content for submission to a search engine

• US 20040177015A1
• Filed: 02/13/2004
• Published: 09/09/2004
• Est. Priority Date: 08/14/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Advertisingcom LLC (Apollo Global Management, Inc.)
• Inventors: Itzhak, Oded, Galai, Yaron
• Granted Patent: US 7,809,710 B2
• Time in Patent Office: Days
• US Class Current: 705/35
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/81 Indexing, e.g. XML tags; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes

Abstract

A system and a method for automatically submitting Web pages to a search engine, which is preferably used for submitting dynamic Web pages, but may optionally be used for any type of Web page. The present invention features a gateway server for providing these Web pages to the search engine, either directly or optionally through an autonomous software search program. Optionally and more preferably, the gateway server modifies the Web page before serving it to the autonomous software search program and/or search engine.

Claims

1. A method for providing a Web page to a search engine, comprising:

separating non-essential code from essential content of the Web page;

extracting said essential content from the Web page; and

providing said essential content of the Web page to the search engine.

2. The method of claim 1, wherein said extracting includes modifying the Web page by removing one or more links from the Web page.

3. The method of claim 1, wherein said extracting includes modifying the Web page by changing a URL address by deleting, adding or modifying parameters within this URL.

4. The method of any of claims 1-3, wherein the Web page is provided to an autonomous software search program, for being provided to the search engine.

5. The method of any of claims 1-4, wherein said extracting further comprises:

determining a pattern of at least one repetitive element within at least the Web page; and

extracting the unique content from the Web page according to said pattern.

6. The method of claim 5, wherein said pattern is determined by comparing a plurality of Web pages.

7. The method of claim 6, wherein said pattern is a template.

8. The method of claim 7, wherein the unique content is extracted by removing said template.

9. The method of any of claims 1-8, wherein said separating non-essential code from essential content further comprises:

normalizing the URL for the Web page for said search engine.

10. The method of claim 9, wherein said normalizing comprises removing at least one redundant parameter from the URL to form a normalized URL.

11. The method of claim 9 or 10, wherein all redundant parameters are removed.

12. The method of claims 9-11, wherein each redundant parameter is removed by:

removing a parameter from the URL to form a reduced URL;

retrieving a new Web page according to said reduced URL; and

comparing said new Web page and the Web page to determine similarity, such that similarity indicates that said parameter is redundant.

13. The method of claim 12, wherein similarity is determined according to content of said new Web page and the Web page.

14. The method of claim 12 or 13, wherein similarity is determined according to a quantitative comparison, such that if similarity is above a threshold, said parameter is redundant.

15. The method of claims 9-14 where the removal of parameters and the comparison of the content in order to determine redundancy of parameters is done either automatically or manually.

16. The method of any of claims 1-15, wherein said separating further comprises:

providing at least one attribute for an information type, said at least one attribute being determined according to an information schema;

identifying at least one field of the Web page; and

associating said at least one field with at least one attribute.

17. The method of claim 16, wherein said at least one attribute comprises a weight factor for ranking the Web page by the search engine.

18. The method of any of claims 1-17, wherein said separating further comprises:

examining a plurality of Web pages to perform a statistical analysis of said plurality of Web pages; and

identifying at least one field of the Web page according to said statistical analysis.

19. A system for providing a Web page for indexing, comprising:

(a) a gateway Web server for modifying the Web page for enabling indexing to be performed; and

(b) a search engine for performing indexing.

20. The system of claim 19, further comprising:

(c) an autonomous software search program for requesting the Web page from said gateway Web server and for providing the Web page to said search engine.

21. The system of claim 19 or 20, further comprising:

(d) a scheduling process for selecting a Web page to be served to said search engine.

22. The system of claim 21, wherein said gateway Web server comprises said scheduling process.

23. The system of claim 21 or 22, further comprising:

(e) a plurality of Web servers for serving the Web page, wherein said scheduling process selects one of said plurality of Web servers for serving the Web page.

24. A method for extracting unique content from a Web page, comprising:

determining a pattern of at least one repetitive element within at least the Web page; and

extracting the unique content from the Web page according to said pattern.

25. The method of claim 24, wherein said pattern is determined by comparing a plurality of Web pages.

26. The method of claim 25, wherein said pattern is a template.

27. The method of claim 26, wherein the unique content is extracted by removing said template.

28. The method of any of claims 24-27, wherein the unique content is extracted before the Web page is provided to a search engine.

29. A method for extracting an element from a Web page for serving the element to a search engine, the method comprising:

analyzing a structure of the Web page;

learning to extract the element from the Web page;

producing a set of instructions to extract the element;

extracting the element from a plurality of Web pages; and

creating a structured representation of the content of said plurality of Web pages for submission to the search engine.

30. The method of claim 29, wherein said analyzing is performed by automatically scanning a plurality of Web pages having at least a similar structure.

31. The method of claim 30, wherein said at least a similar structure is determined according to at least one of an originating template and a construction process.

32. The method of any of claims 29-31, wherein said learning further comprises parsing the HTML code of the Web page.

33. The method of any of claims 29-32, wherein said structured representation is in XML.

34. A method for feeding information about a plurality of Web pages to a search engine, comprising:

extracting at least one field from the plurality of Web pages;

automatically generating feed information for being fed to the search engine from said at least one field;

receiving information about a template common to the plurality of Web pages;

merging said template with said feed information; and

transmitting said feed information to the search engine.

35. The method of claim 34, wherein said at least one field comprises at least one of information to be displayed in a search result including one of the plurality of Web pages and information to be indexed but not displayed.

36. The method of claim 34 or 35, wherein said automatically generating feed information comprises:

selecting an output format for generating said feed information; and

constructing a script for generating said feed information according to said template.

37. The method of claim 36, wherein said output format comprises XML.