Mechanism for Separating Content from Noisy Context in Template-Based Documents for Search Indexing

• US 20100306217A1
• Filed: 05/28/2009
• Published: 12/02/2010
• Est. Priority Date: 05/28/2009
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Schneider, James P.
• Granted Patent: US 8,732,137 B2
• Time in Patent Office: Days
• US Class Current: 707/759
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

In one embodiment, a mechanism for separating content from noisy context in template-based documents for search indexing is disclosed. In one embodiment, a method includes selecting a plurality of documents for index comparison, identifying one or more identical elements found in each of the plurality of documents, and removing the one or more identical elements from consideration in an indexing process of the plurality of documents.

Claims

1. A computer-implemented method, comprising:

selecting, by a server computer, a plurality of documents for index comparison;

identifying, by the server computer, one or more identical elements found in each of the plurality of documents; and

removing, by the server computer, the one or more identical elements from consideration in an indexing process of the plurality of documents.

2. The method of claim 1, wherein the plurality of documents are web pages.

3. The method of claim 1, wherein selecting the plurality of documents is based on at least one of occupying a same Uniform Resource Locator (URL) space and location on a same server.

4. The method of claim 1, wherein the one or more identical elements include at least one of an advertisement, site navigation component, header, footer, and copyright notice.

5. The method of claim 1, further comprising, for each document of the plurality of documents:

retrieving more than one updated view of the document;

identifying one or more items that differ between the more than one updated views of the document; and

removing the one or more items from consideration in the indexing process of the plurality of documents.

6. The method of claim 5, wherein the selecting, identifying, and removing are performed by a web search engine as a pre-processing step for the indexing process.

7. The method of claim 1, further comprising retrieving the plurality of documents via a network interface of the comparator.

8. A system, comprising:

a network interface to receive a plurality of documents for index comparison; and

a comparison module communicably coupled to the network interface to receive the plurality of documents, the comparison module operable to;

identify one or more identical elements found in each of the plurality of documents; and

remove the one or more identical elements from consideration in an indexing process of the plurality of documents.

9. The system of claim 8, wherein the plurality of documents are web pages.

10. The system of claim 8, wherein the plurality of documents are selected based on at least one of occupying a same Uniform Resource Locator (URL) space and location on a same server.

11. The system of claim 8, wherein the one or more identical elements include at least one of an advertisement, site navigation component, header, footer, and copyright notice.

12. The system of claim 8, further comprising for each document of the plurality of documents:

the network interface to retrieve more than one updated view of the document; and

the comparison module to;

identify one or more items that differ between the more than one updated views of the document; and

remove the one or more items from consideration in the indexing process of the plurality of documents.

13. The system of claim 12, wherein the selecting, identifying, and removing are performed by a web search engine as a pre-processing step for the indexing process.

14. The system of claim 8, further comprising a document buffer communicably coupled to the network interface and the comparison module to store the plurality of documents and to store a result from the comparison module.

15. An article of manufacture comprising a machine-readable storage medium including data that, when accessed by a machine, cause the machine to perform operations comprising:

selecting a plurality of documents for index comparison;

identifying one or more identical elements found in each of the plurality of documents; and

removing the one or more identical elements from consideration in an indexing process of the plurality of documents.

16. The article of manufacture of claim 15, wherein the plurality of documents are web pages.

17. The article of manufacture of claim 15, wherein the communality between the plurality of documents is at least a same Uniform Resource Locator (URL) space and located on a same server.

18. The article of manufacture of claim 15, wherein the one or more identical elements include at least one of an advertisement, site navigation component, header, footer, and copyright notice.

19. The article of manufacture of claim 15, wherein the machine-readable storage medium includes data that, when accessed by the machine, cause the machine to perform operations further comprising, for each document of the plurality of documents:

retrieving more than one updated view of the document;

identifying one or more items that differ between the more than one updated views of the document; and

removing the one or more items from consideration in the indexing process of the plurality of documents.

20. The article of manufacture of claim 15, wherein the selecting, identifying, and removing are performed by a web search engine as a pre-processing step for the indexing process.