Eliminating noise in periodicals

• US 9,251,228 B1
• Filed: 04/21/2011
• Issued: 02/02/2016
• Est. Priority Date: 04/21/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Iyer, Deepa Subramanian, Dass, Ramya
• Primary Examiner(s): Hong, Stephen
• Assistant Examiner(s): ROBINSON, MARSHON L
• Application Number:
  US13/092,098
• Time in Patent Office: 1,748 Days
• Field of Search: 715/205, 715/734, 715/255
• US Class Current: 1/1
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/84 Mapping; Conversion
• CPC Class Codes: G06F 16/9577 Optimising the visualization of content, e.g. distillation of HTML documents

Abstract

A method and system for eliminating global and local noise in periodical items is described. An exemplary method may include preprocessing each item in a set of items using one or more rules, removing global noise from the set of items using semantic similarities across items in the set of items, and removing local noise in each item in the set of items based on text content in each item.

Claims

1. A method, comprising:

preprocessing, by a server computer system, each item in a set of items using one or more rules, wherein preprocessing an item in the set of items comprises;

determining that the item includes a print option; and

using a version of the item associated with the print option instead of an alternate version;

removing, by the server computer system, global noise from the set of items using semantic similarities across items in the set of items; and

removing, by the server computer system, local noise in the item of the set of items, wherein removing the local noise in the item of the set of items comprises;

determining an amount of content for a node associated with the item;

calculating a content score for the node based on the amount of content;

calculating a link density for the node based on a number of links in the node as a percentage of the content;

calculating a local noise score for the node based on the content score and the link density; and

removing the node responsive to a determination that the local noise score is above a threshold.

2. The method of claim 1 wherein preprocessing the item further comprises:

determining whether the item includes indicators of a print-friendly version;

responsive to determining that the item includes the indicators of the print-friendly version, removing non-print friendly content from the item; and

removing predefined irrelevant tags from the item.

3. The method of claim 1 wherein removing global noise from the set of items using semantic similarities comprises:

building a section style tree for the set of items using a closeness measure to merge nodes from each item;

calculating a composite importance value for each of a plurality of element nodes in the section style tree;

mapping portions of items in the set of items to element nodes in the section style tree using the closeness measure; and

identifying one or more element nodes in the section style tree as noise based on corresponding composite importance values.

4. The method of claim 3 wherein building the section style tree for the set of items comprises:

creating a plurality of document object model (DOM) trees for the set of items;

creating an element node in the section style tree for each tag node in a first of the plurality of DOM trees;

merging a tag node of a subsequent DOM tree into the created element node responsive to a determination that children of the tag node correspond to children of the created element node and textual content of the tag node corresponds to textual content of the created element node;

responsive to a determination that children of the tag node do not correspond to children of the created element node or textual content of the tag node does not correspond to textual content of the created element node, creating a new style node underneath the created element node and adding the children of the tag node as a sub-tree of the new style node; and

calculating a page count of each style node based on a number of corresponding merged tag nodes.

5. The method of claim 3 wherein calculating the composite importance value for each of the plurality of element nodes in the section style tree comprises:

calculating a leaf node importance value of each leaf element node in the section style tree; and

calculating the composite importance value for each of the plurality of element nodes using leaf node importance values of corresponding leaf element nodes.

6. The method of claim 3 wherein the one or more element nodes are identified as noise when corresponding composite importance values are below a threshold.

7. The method of claim 3 further comprising:

removing noise from the set of items based on the one or more element nodes identified as noise and based on configuration parameters.

8. The method of claim 7 wherein the configuration parameters specify predefined types of element nodes to be removed and predefined types of element nodes not to be removed.

9. The method of claim 1 wherein removing the local noise in the item in the set of items further comprises:

removing a particular node associated with the item of the set of items responsive to a determination that a class name and an identifier of the particular node match a regular expression pattern of a predefined noise node;

adding the content score of a current node to a parent node of the current node and adding a portion of the content score of the current node to a grandparent node of the current node; and

repeating the calculating and the adding for nodes at a number of higher levels.

10. The method of claim 1 wherein the item is any one of a newspaper article, a magazine article, a news feed and a blog feed.

21. The method of claim 1, further comprising:

using a syntax checker to determine whether any hypertext markup language (HTML) tags in each item are not closed properly;

correcting the HTML tags that were not closed properly; and

removing the HTML tags having predefined properties.

22. The method of claim 21, wherein the HTML tags having the predefined properties comprise script tags, noscript tags, style tags, iframe tags, form tags and object tags.

23. The method of claim 1, wherein an increase in the number of links in the node causes the local noise score for that node to increase.

11. A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations comprising:

preprocessing, by the processor, each item in a set of items using one or more rules, wherein preprocessing an item in the set of items comprises;

determining that the item includes a print option; and

using a version of the item associated with the print option instead of an alternate version;

removing, by the processor, global noise from the set of items using semantic similarities across items in the set of items; and

removing, by the processor, local noise in the item of the set of items, wherein removing the local noise in the item of the set of items comprises;

determining an amount of content for a node associated with the item;

calculating a content score for the node based on the amount of content;

calculating a link density for the node based on a number of links in the node as a percentage of the content;

calculating a local noise score for the node based on the content score and the link density; and

removing the node responsive to a determination that the local noise score is above a threshold.

12. The non-transitory computer readable storage medium of claim 11 wherein preprocessing the item further comprises:

determining whether the item includes indicators of a print-friendly version;

responsive to determining that the item includes the indicators of the print-friendly version, removing non-print friendly content from the item; and

removing predefined irrelevant tags from the item.

13. The non-transitory computer readable storage medium of claim 11 wherein removing global noise from the set of items using semantic similarities comprises:

building a section style tree for the set of items using a closeness measure to merge nodes from each item;

calculating a composite importance value for each of a plurality of element nodes in the section style tree;

mapping portions of items in the set of items to element nodes in the section style tree using the closeness measure; and

identifying one or more element nodes in the section style tree as noise based on corresponding composite importance values.

14. The non-transitory computer readable storage medium of claim 13 wherein building the section style tree for the set of items comprises:

creating a plurality of document object model (DOM) trees for the set of items;

creating an element node in the section style tree for each tag node in a first of the plurality of DOM trees;

merging a tag node of a subsequent DOM tree into the created element node responsive to a determination that children of the tag node correspond to children of the created element node and textual content of the tag node corresponds to textual content of the created element node;

responsive to a determination that children of the tag node do not correspond to children of the created element node or textual content of the tag node does not correspond to textual content of the created element node, creating a new style node underneath the created element node and adding the children of the tag node as a sub-tree of the new style node; and

calculating a page count of each style node based on a number of corresponding merged tag nodes.

15. The non-transitory computer readable storage medium of claim 13 wherein calculating the composite importance value for each of the plurality of element nodes in the section style tree comprises:

calculating a leaf node importance value of each leaf element node in the section style tree; and

calculating the composite importance value for each of the plurality of element nodes using leaf node importance values of corresponding leaf element nodes.

16. The non-transitory computer readable storage medium of claim 13 wherein the one or more element nodes are identified as noise when corresponding composite importance values are below a threshold.

17. The non-transitory computer readable storage medium of claim 13, the operations further comprising:

removing noise from the set of items based on the one or more element nodes identified as noise and based on configuration parameters.

18. The non-transitory computer readable storage medium of claim 17 wherein the configuration parameters specify predefined types of element nodes to be removed and predefined types of element nodes not to be removed.

19. The non-transitory computer readable storage medium of claim 11 wherein removing the local noise in the item in the set of items further comprises:

removing a particular node associated with the item of the set of items responsive to a determination that a class name and an identifier of the particular node match a regular expression pattern of a predefined noise node;

adding the content score of a current node to a parent node of the current node and adding a portion of the content score of the current node to a grandparent node of the current node; and

repeating the calculating and the adding for nodes at a number of higher levels.

20. A server computer system comprising:

a memory; and

a processor, coupled to the memory, the processor to;

preprocess each item in a set of items using one or more rules, wherein to preprocess an item in the set of items the processor is to;

determine that the item includes a print option; and

use a version of the item associated with the print option instead of an alternate version;

remove global noise from the set of items using semantic similarities across items in the set of items; and

remove local noise in the item of the set of items, wherein to remove the local noise in an item of the set of items the processor is to;

determine an amount of content for a node associated with the item;

calculate a content score for the node based on the amount of content;

calculate a link density for the node based on a number of links in the node as a percentage of the content;

calculate a local noise score for the node based on the content score and the link density; and

remove the node responsive to a determination that the local noise score is above a threshold.