Document data classification using a noise-to-content ratio

• US 9,773,182 B1
• Filed: 09/13/2012
• Issued: 09/26/2017
• Est. Priority Date: 09/13/2012
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Wolkerstorfer, Bernhard, Li, Lei, Parihar, Narendra S.
• Primary Examiner(s): Hong, Stephen
• Assistant Examiner(s): Robinson, Marshon
• Application Number:
  US13/614,858
• Time in Patent Office: 1,839 Days
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 40/258 Heading extraction; Automatic titling; Numbering

Abstract

A method and system for classifying document data is described. An exemplary method includes identifying a markup language document having a plurality of portions, determining a set of substantive content metrics and a set of noise metrics for each of the plurality of portions, calculating a noise-to-content ratio for each of the plurality of portions based on a corresponding set of substantive content metrics and a corresponding set of noise metrics, and removing noise from the markup language document using the noise-to-content ratio.

Claims

1. A system comprising:

at least one memory;

at least one processing device, operatively coupled to the at least one memory, to;

receive an electronic document having a first portion, a second portion, and a third portion;

determine a first content-to-noise ratio for the first portion;

determine a second content-to-noise ratio for the second portion;

determine a third content-to-noise ratio for the third portion;

sort a list of the first portion, the second portion, and the third portion in order from a highest content-to-noise ratio to a lowest content-to-noise ratio using the first content-to-noise ratio, the second content-to-noise ratio, and the third content-to-noise ratio; and

remove, from the electronic document, a predetermined number of the first portion, the second portion, and the third portion starting with the highest content-to-noise ratio in the list.

2. The system of claim 1, wherein:

the electronic document is a markup language document; and

the first portion and the second portion are nodes within a document object model (DOM) tree of the markup language document or a sub-tree comprising a parent node and one or more child nodes within the DOM tree of the markup language document.

3. The system of claim 2, wherein the at least one processing device is further to remove the first portion from the electronic document when the first portion comprises:

content invisible during rendering of the markup language document;

advertisements;

or

a reference to comment providers.

4. The system of claim 2, wherein the at least one processing device is further to determine the content-to-noise ratio using substantive content metrics, the substantive content metrics comprises at least one of:

a number of plain text characters in the first portion;

a number of inline tag nodes in the first portion;

a number of lines in the first portion;

a number of images in the first portion;

or a number of paragraphs in the first portion.

5. The system of claim 2, wherein the at least one processing device is further to determine the content-to-noise ratio using noise metrics, wherein the noise metrics comprises at least one of:

a ratio of a number of link nodes in the first portion to a number of text nodes in the first portion;

a number of characters in links within the first portion;

a number of images in the first portion, wherein the images comprise advertisements;

a number of link nodes in the first portion, wherein the link nodes are links to nodes comprising advertisements;

a number of valid containers in the first portion;

or

a number of small images in the first portion, wherein a small image is an image that has a size smaller than a threshold.

6. The system of claim 1, wherein the at least one processing device is further to determine the noise-to-content ratio for the first portion by:

determining substantive content metrics and noise metrics of the electronic document;

calculating a substantive content score for the first portion using a ratio of substantive content metrics of the first portion to corresponding substantive content metrics of the electronic document;

calculating a noise score for the first portion using a ratio of noise metrics of the first portion to corresponding noise metrics of the electronic document; and

calculating a ratio of the noise score to the substantive content score.

7. The system of claim 1, wherein the at least one processing device is further to include the second portion in a document model for user devices without including the first portion.

8. A non-transitory computer readable storage medium including instructions that, when executed by at least one processing device, cause the at least one processing device to perform operations comprising:

receiving, by the at least one processing device, an electronic document having a first portion, a second portion, and a third portion;

determining, by the at least one processing device, a first content-to-noise ratio for the first portion;

determining, by the at least one processing device, a second content-to-noise ratio for the second portion;

determining, by the at least one processing device, a third content-to-noise ratio for the third portion;

sort a list of the first portion, the second portion, and the third portion in order from a highest content-to-noise ratio to a lowest content-to-noise ratio using the first

content-to-noise ratio, the second content-to-noise ratio, and the third content-to-noise ratio; and

removing, from the electronic document, a predetermined number of the first portion, the second portion, and the third portion starting with the highest content-to-noise ratio in the list.

9. The non-transitory computer readable storage medium of claim 8, wherein:

the electronic document is a markup language document; and

the first portion, the second portion, and the third portion are nodes within a document object model (DOM) tree of the markup language document or a sub-tree comprising a parent node and one or more child nodes within the DOM tree of the markup language document.

10. The non-transitory computer readable storage medium of claim 9, wherein the operations further comprise removing the first portion when the first portion comprises:

content invisible during rendering of the markup language document;

advertisements;

or a reference to comment providers.

11. The non-transitory computer readable storage medium of claim 9, wherein the operations further comprise determining the content-to-noise ratio using substantive content metrics, the substantive content metrics comprises at least one of:

a number of plain text characters in the first portion;

a number of inline tag nodes in the first portion;

a number of lines in the first portion;

a number of images in the first portion;

or a number of paragraphs in the first portion.

12. The non-transitory computer readable storage medium of claim 9, wherein the operations further comprise determine the content-to-noise ratio using noise metrics, the noise metrics comprises at least one of:

a ratio of a number of link nodes in the first portion to a number of text nodes in the first portion;

a number of characters in links within the first portion;

a number of images in the first portion, wherein the images comprise advertisements;

a number of link nodes in the first portion, wherein the link nodes are links to nodes comprising advertisements;

a number of valid containers in the first portion;

or

a number of small images in the first portion, wherein a small image is an image has a size smaller than a threshold.

13. The non-transitory computer readable storage medium of claim 8, wherein determining the content-to-noise ratio for the first portion further comprises:

determining substantive content metrics and noise metrics of the electronic document;

calculating a substantive content score for the first portion using a ratio of substantive content metrics of the first portion to corresponding substantive content metrics of the electronic document;

calculating a noise score for the first portion using a ratio of noise metrics of the first portion to corresponding noise metrics of the electronic document; and

calculating a ratio of the noise score to the substantive content score.

14. The non-transitory computer readable storage medium of claim 8, the

operations further comprising including the second portion in a document model for user devices without including the first portion.

15. A method comprising:

receiving, by at least one, an electronic document having a first portion, a second portion, and a third portion;

determining, by the at least one processing device, a first content-to-noise ratio for the first portion; and

determining, by the at least one processing device, a second content-to-noise ratio for the second portion;

determining, by the at least one processing device, a third content-to-noise ratio for the third portion;

sort, by the at least one processing device, a list of the first portion, the second portion, and the third portion in order from a highest content-to-noise ratio to a lowest content-to-noise ratio using the first content-to-noise ratio, the second content-to-noise ratio, and the third content-to-noise ratio; and

removing, from the electronic document, a predetermined number of the first portion, the second portion, and the third portion starting with the highest content-to-noise ratio in the list.

16. The method of claim 15, wherein:

the electronic document is a markup language document; and

the first portion and the second portion are nodes within a document object model (DOM) tree of the markup language document or a sub-tree comprising a parent node and one or more child nodes within the DOM tree of the markup language document.

17. The method of claim 16, further comprising removing the first portion from the electronic document when the first portion comprises:

content invisible during rendering of the markup language document;

advertisements;

or

a reference to comment providers.

18. The method of claim 16, further comprising determining the content-to-noise ratio using substantive content metrics, the substantive content metrics comprises at least one of:

a number of plain text characters in the first portion;

a number of inline tag nodes in the first portion;

a number of lines in the first portion;

a number of images in the first portion;

or a number of paragraphs in the first portion.

19. The method of claim 16, further comprising determining the content-to-noise ratio using noise metrics, wherein the noise metrics comprises at least one of:

a ratio of a number of link nodes in the first portion to a number of text nodes in the first portion;

a number of characters in links within the first portion;

a number of images in the first portion, wherein the images comprise advertisements;

a number of link nodes in the first portion, wherein the link nodes are links to nodes comprising advertisements;

a number of valid containers in the first portion;

or

a number of small images in the first portion, wherein a small image is an image has a size smaller than a threshold.

20. The method of claim 15, wherein determining the content-to-noise ratio for the first portion further comprises:

determining substantive content metrics and noise metrics of the electronic document;

calculating a substantive content score for the first portion using a ratio of substantive content metrics of the first portion to corresponding substantive content metrics of the electronic document;

calculating a noise score for the first portion using a ratio of noise metrics of the first portion to corresponding noise metrics of the electronic document; and

calculating a ratio of the noise score to the substantive content score.