Pattern recognition in web search engine result pages

• US 8,326,830 B2
• Filed: 10/06/2009
• Issued: 12/04/2012
• Est. Priority Date: 10/06/2009
• Status: Active Grant

• Current Assignee: Business Objects Incorporated (SAP SE)
• Original Assignee: Business Objects Incorporated (SAP SE)
• Inventors: Hollingsworth, Daniel
• Primary Examiner(s): GORTAYO, DANGELINO N
• Application Number:
  US12/573,894
• Publication Number:
  US 20110082884A1
• Time in Patent Office: 1,155 Days
• Field of Search: 707/1, 707/100, 707/104, 707/3, 707/705, 707/713, 707/771, 707/776, 707/758, 707/750, 707/748, 707/709, 707/728, 707/730
• US Class Current: 707/730
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Described herein are methods and systems for pattern recognition in web search engine result pages. The input data is a result page from a web search engine as well as an integer number for the results on the page. The output is a regular expression that matches all the results on the page, capturing each result and its individual fields.

Claims

1. A non-transitory computer readable medium comprising computer readable instructions, which, when executed by a computer, cause the computer to perform a method, the method comprising:

receiving a result page from a web search engine, the result page comprising text fields and markup tags, and an integer number for the results on the result page;

generating simplified variations of the result page, the generating comprising;

determining noisy markup tags in the result page;

generating a first variation of the result page by removing the noisy markup tags from the result page;

generating a plurality of other variations of the result page by preserving one noisy markup tag from the noisy markup tags and removing the rest of the noisy markup tags;

stripping inside of remaining markup tags in the first variation and the plurality of other variations; and

simplifying the text fields by marking the text fields with free text markers;

parsing the simplified variations of the result page to determine one or more repeating patterns, the one or more repeating patterns comprising a substring of the simplified variations of the result page, the substring beginning at a start of a remaining markup tag or a free text marker and ending at a close of the remaining markup tag or the free text marker;

selecting the one or more repeating patterns that are repeated the integer number of times in the result page as result patterns;

selecting one of the one or more result patterns as a highest rated result pattern according to predefined rating criteria; and

generating a regular expression from the highest rated result pattern as an output that matches the results on the result page.

2. The medium of claim 1, wherein the result page is an HTML page.

3. The medium of claim 2, wherein the predefined rating criteria include one or more criteria selected from the group consisting of ability to normalize text fields as a date, character length of text fields, position of text fields in relation to other text fields, and location and proximity of HTML anchors in relation to text fields.

4. The medium of claim 1, wherein the method further comprises saving the removed noisy markup tags and their location in the result page in order to be restored.

5. The medium of claim 1, wherein generating the regular expression from the highest rated result pattern comprises:

restoring the removed noisy markup tags during the simplification step;

examining the text fields for markup tags, including the restored noisy markup tags;

setting the found tags in the text fields as allowable; and

setting tags in the pattern that are not in the text fields as required to be matched.

6. A computer implemented method for pattern recognition in web search engine result pages, comprising:

receiving an HTML result page from a web search engine, the HTML result page comprising text fields and markup tags, and an integer value with a number of results on the HTML result page;

generating simplified variations of the HTML result page, the generating comprising;

determining noisy markup tags in the result page;

generating a first variation of the result page by removing the noisy markup tags from the result page;

generating a plurality of other variations of the result page by preserving one noisy markup tag from the noisy markup tags and removing the rest of the noisy markup tags;

stripping inside of remaining markup tags in the first variation and the plurality of other variations; and

simplifying the text fields by marking the text fields with free text markers;

parsing the simplified variations of the HTML result pages to determine one or more repeating patterns, the one or more repeating patterns comprising a substring of the simplified variations of the HTML result page, the substring beginning at a start of a remaining markup tag or a free text marker and ending at a close of the remaining markup tag or the free text marker;

selecting the one or more repeating patterns that are repeated the integer value number of times in the HTML result page as result patterns;

selecting one of the one or more result patterns as a highest rated result pattern according to predefined rating criteria; and

generating a regular expression from the highest rated result pattern as an output that matches the results on the HTML result page.

7. The method of claim 6, wherein the noisy markup tags include one or more tags selected from the group consisting of bold tags, break tags, cite tags, emphasis tags, font tags, image tags, italic tags, no break tags, span tags, strong tags, and underline tags.

8. The method of claim 6, wherein the method further comprises saving the removed noisy markup tags and their location in the HTML result page in order to be restored.

9. The method of claim 6, wherein the predefined rating criteria include one or more selected from the group consisting of ability to normalize text fields as a date, character length of text fields, position of text fields in relation to other text fields, and location and proximity of HTML anchors in relation to text fields.

10. The method of claim 6, wherein generating the regular expression from the highest rated result pattern comprises:

restoring the removed noisy markup tags during the simplification step;

examining the text fields for markup tags, including the restored noisy markup tags;

setting the found tags in the text fields as allowable; and

setting tags in the pattern that are not in the text fields as required to be matched.

11. The method of claim 6, further comprising using the regular expression to extract fields from HTML result pages from the web search engine.