Techniques for approximating the visual layout of a web page and determining the portion of the page containing the significant content

• US 20080033996A1
• Filed: 08/03/2006
• Published: 02/07/2008
• Est. Priority Date: 08/03/2006
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Oath Inc. (Verizon Communications Inc.)
• Inventors: Kesari, Anandsudhakar
• Application Number:
  US11/499,181
• Publication Number:
  US 20080033996A1
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/972 Access to data in other repository systems, e.g. legacy data or dynamic Web page generation
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]

Abstract

To approximate a visual layout of a web page without rendering the page, an object tree representing elements within the page is recursively traversed to determine bounds for the width of the elements, resulting in lower bounds induced for non-leaf nodes by elements within these nodes and upper bounds induced by ancestors and siblings of nodes. For each element, the minimum required width (lower bound), the desired width were there no constraints, and the maximum available width (upper bound) based on constraints of parents are computed, and an approximate width is derived therefrom. A positioning process positions each element within its corresponding parent container by advancing a cursor according to the elements'"'"' approximate width and appropriate constraints. The element that contains the most meaningful content is determined based on the amount of weighted content of elements and their position within the page.

Claims

1. A method comprising performing a machine-executed operation involving instructions for approximating the visual layout of a Web page, wherein the machine-executed operation is at least one of:

A) sending said instructions over transmission media;

B) receiving said instructions over transmission media;

C) storing said instructions onto a machine-readable storage medium; and

D) executing the instructions;

wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of;

constructing an object tree according to a structure of elements within Web page code; and

approximating a visual layout of said Web page without rendering said Web page, wherein said approximating a visual layout comprises;

constraining the geometry of at least a subset of said elements based on geometric properties of corresponding container elements that contain said elements in said subset,

computing an approximate width of each of said elements in said subset, wherein said approximate width of an element may be different than a width for said element as specified in said Web page,

positioning each of said elements in said subset in a corresponding constraining container by logically placing the element at the current position of a locator which is advanced for each subsequent element, and

annotating said object tree in association with each of said elements in said subset, with corresponding coordinates of the element based on the position of said locator corresponding to the element.

2. The method of claim 1, wherein said step of computing an approximate width comprises computing an approximate width for each of said elements in said subset based on (a) a minimum required width for the element, (b) a width the element would occupy if there was no constraint on the element, and (c) a width available for the element based on one or more constraints imposed by the width of the container element in which the element is contained.

3. The method of claim 2, wherein said step of annotating comprises annotating said object tree in association with each of said elements in said subset, with said approximate width of the element and an approximate height of the element.

4. The method of claim 2, wherein said step of computing an approximate width of each of said elements in said subset comprises the steps of:

computing said minimum required width for each container element by recursively computing a corresponding minimum width required to contain all child elements of the container element.

5. The method of claim 4, wherein said step of computing an approximate width comprises computing a minimum required width of a text element as the width of the longest sub-element in said text element, wherein a sub-element is a set of continuous characters without a space, and wherein said longest sub-element is the sub-element having the most characters.

6. The method of claim 4, wherein said step of computing an approximate width of a table element comprises the steps of:

computing a minimum width for each column in said table element as the width of the largest cell in the column; and

computing an initial approximate width of said table element as a sum of the minimum widths of all the columns in said table element.

7. The method of claim 6, wherein said step of computing an approximate width of a table element comprises the steps of:

if a table width for said table element is specified in said Web page, then comparing said initial approximate width of said table element with said specified table width of said table element;

if said initial approximate width of said table element is less than said specified table width, then distributing the difference between said specified table width and said initial approximate width to columns having variable-width type, wherein said distributing is in proportion to a difference between said minimum width for said variable-width column and a width the variable-width column would occupy if there was no constraint on the variable-width column.

8. The method of claim 4, wherein said step of computing an approximate width comprises computing a minimum required width of an image element as the actual width of said image element.

9. The method of claim 4, wherein said step of computing an approximate width comprises:

computing a minimum required width of a text element as the width of the longest sub-element in said text element, wherein a sub-element is a set of continuous characters without a space, and wherein said longest sub-element is the sub-element having the most characters;

computing a minimum required width of a table element as a sum of the minimum widths of all columns in said table element, wherein the minimum width of a column is the width of a largest cell in said column; and

computing a minimum required width of an image element as the actual width of said image element.

10. The method of claim 2, wherein said step of computing an approximate width of each of said elements in said subset comprises the steps of:

computing a desired width of the element as the width the element would occupy if there was no constraint on the element; and

wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of;

recursively computing the desired width of each container element as a sum of the desired widths of all child elements of the container element.

11. The method of claim 2, wherein said step of computing an approximate width of each of said elements in said subset comprises the steps of:

if a width for an element in said subset is specified in the Web page, and if the minimum required width for the element is less than said specified width, then computing the approximate width for the element as said specified width.

12. The method of claim 2, wherein said step of computing an approximate width of each of said elements in said subset comprises the steps of:

if a width for an element in said subset is specified in the Web page, and if the minimum required width for the element is greater than said specified width, then computing the approximate width for the element as said minimum required width.

13. The method of claim 1,

wherein said step of positioning each of said elements comprises positioning each of said elements in said subset in an order, based on the object tree, from a root element down through one or more branches of said object tree to a corresponding leaf element;

and

wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of;

computing the height of each container element based on a movable bottom boundary of said container element, wherein the bottom boundary is based on a final position of the elements contained in said container element.

14. The method of claim 1, wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the step of:

computing a most significant portion of the Web page based on an amount of weighted content in said most significant portion.

15. The method of claim 14, wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of:

computing said amount of weighted content of corresponding elements based on a weighted sum of a number of words and the area of images in said corresponding element and one or more sub-trees of said corresponding element;

computing a content loss between parent elements and child elements as the difference between said amount of weighted content of a parent element and said amount of weighted content of a corresponding child element;

computing a true content of a parent element as said content loss computed for said parent element and a particular child element, of said parent element, that has a maximum amount of weighted content;

traversing the object tree along a path having maximum amount of weighted content until said true content of a particular element is below a threshold value; and

identifying a parent element of said particular element as said most significant portion of said Web page, wherein said most significant portion comprises said parent element and said parent element'"'"'s sub-trees if any.

16. The method of claim 14, wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of:

informing an information extraction process of said most significant portion of said Web page.

17. The method of claim 14, wherein said step of computing a most significant portion of said Web page comprises computing said amount of weighted content in an element of said Web page based on weighting said content of said element based on one or more of (a) text formatting specified for said element in said Web page code, (b) a border or cell spacing specified, for a table element, in said Web page code, and based on weighting the depth of said element in said object tree.

18. The method of claim 1, wherein said Web page is coded at least in part in HTML.

19. The method of claim 18, wherein said Web page resides on a private network.