Systems and methods for analyzing boilerplate

• US 20080040316A1
• Filed: 03/31/2004
• Published: 02/14/2008
• Est. Priority Date: 03/31/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lawrence, Stephen
• Granted Patent: US 8,041,713 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536

Abstract

Systems and methods for analyzing boilerplate are described. In one described system, an indexer identifies a common element in a plurality of related articles. The indexer then classifies the common element as boilerplate. For example, the indexer may identify a copyright notice appearing in a plurality of related articles. The copyright notice in these articles is considered boilerplate.

Claims

1. A method comprising:

identifying a common element in a plurality of articles;

analyzing a spatial location of the common element in an article of the plurality of articles; and

determining whether the common element is a boilerplate element of the article based at least in part on the spatial location.

2. The method of claim 1, further comprising generating an implicit search query including a search term, the search term comprising a term present in a content element of the article, the content element being distinguishable from the boilerplate element.

3. The method of claim 1, wherein the common element comprises a copyright notice.

4. The method of claim 1, wherein the common element comprises a term having a low inverse document frequency measure.

6. The method of claim 1, wherein analyzing the spatial location of the common element comprises:

determining whether the common element is at the bottom of the article.

7. The method of claim 1, wherein the common element comprises a navigational element of the article.

10. The method of claim 1, further comprising:

analyzing a markup language element proximate to the common element in the article,

wherein determining whether the common element is a boilerplate element comprises determining whether the common element is a boilerplate element of the article based at least in part on the markup language element.

11. The method of claim 1, further comprising:

responding to the common element being the boilerplate element, removing the boilerplate element from the article; and

indexing the article.

12. The method of claim 1, further comprising:

determining weights for elements in the article based at least in part on whether the elements are boilerplate elements.

13. The method of claim 12, further comprising:

receiving a search query;

determining articles relevant to the search query; and

ranking the articles based as least in part on the determined weights.

26. The method of claim 10, wherein the markup language element proximate to the common element comprises a markup language element affecting a display of the common element in the article.

5. A method comprising:

comparing an element in an article to a predetermined list to generate a comparison result;

analyzing a spatial location of the element in the article; and

determining whether the element is a boilerplate element of the article based at least in part on the spatial location and the comparison result.

25. The method of claim 5, wherein the predetermined list comprises terms or phrases, and wherein the comparison result indicates whether the element matches a term or a phrase in the predetermined list.

8. A method comprising:

identifying a common element in a plurality of articles;

analyzing a link associated with the common element in an article of the plurality of articles; and

determining whether the common element is a boilerplate element of the article based at least in part on the link associated with the common element.

9. The method of claim 8, wherein analyzing the link associated with the common element comprises analyzing an address to which the link refers.

14. A tangible computer-readable medium on which is encoded program code, the encoded program code comprising:

program code for identifying a common element in a plurality of articles;

program code for analyzing a spatial location of the common element in an article of the plurality of articles; and

program code for determining whether the common element is a boilerplate element of the article based at least in part on the spatial location.

15. The tangible computer-readable medium of claim 14, wherein the encoded program code further comprises:

program code for generating an implicit search query including a search term, the search term comprising a term present in a content element of the article, the content element being distinguishable from the boilerplate element.

17. The tangible computer-readable medium of claim 14, wherein analyzing the spatial location of the common element comprises:

determining whether the common element is at the bottom of the article.

18. The tangible computer-readable medium of claim 14, wherein common element comprises a navigational element of the article.

21. The tangible computer-readable medium of claim 14, wherein the encoded program code further comprises:

program code for analyzing a markup language element proximate to the common element in the article,

wherein determining whether the common element is a boilerplate element comprises determining whether the common element is a boilerplate element of the article based at least in part on the markup language element.

22. The tangible computer-readable medium of claim 14, wherein the encoded program code further comprises:

program code for responding to the common element being the boilerplate element, removing the boilerplate element from the article; and

program code for indexing the article.

23. The tangible computer-readable medium of claim 14, wherein the encoded program code further comprises:

program code for determining weights for elements in the article based at least in part on whether the elements are boilerplate elements.

24. The tangible computer-readable medium of claim 23, further comprising:

program code for receiving a search query;

program code for determining articles relevant to the search query; and

program code for ranking the articles based as least in part on the determined weights.

16. A tangible computer-readable medium on which is encoded program code, the encoded program code comprising:

program code for comparing an element in an article to a predetermined list to generate a comparison result;

program code for analyzing a spatial location of the element in the article; and

program code for determining whether the element is a boilerplate element of the article based at least in part on the spatial location and the comparison result.

19. A tangible computer-readable medium on which is encoded program code, the encoded program code comprising:

program code for identifying a common element in a plurality of articles;

program code for analyzing a link associated with the common element in an article of the plurality of articles; and

program code for determining whether the common element is a boilerplate element of the article based at least in part on the link associated with the common element.

20. The tangible computer-readable medium of claim 19, wherein analyzing the link associated with the common element of the article comprises analyzing an address to which the link refers.