Separating content from noisy context in template-based documents for search indexing

• US 8,732,137 B2
• Filed: 05/28/2009
• Issued: 05/20/2014
• Est. Priority Date: 05/28/2009
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Schneider, James P.
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US12/473,928
• Publication Number:
  US 20100306217A1
• Time in Patent Office: 1,818 Days
• Field of Search: None
• US Class Current: 707/687
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

In one embodiment, a mechanism for separating content from noisy context in template-based documents for search indexing is disclosed. In one embodiment, a method includes selecting a plurality of documents for index comparison, identifying one or more identical elements found in each of the plurality of documents, and removing the one or more identical elements from consideration in an indexing process of the plurality of documents.

Claims

1. A method comprising:

selecting, by a processing device, a plurality of documents for comparison;

identifying, by the processing device, an identical element comprising information common to each of the plurality of documents; and

removing, by the processing device, the identical element from each of the plurality of documents to form modifications to the plurality of documents, prior to a subsequent indexing process of the plurality of documents.

2. The method of claim 1, wherein the plurality of documents are web pages.

3. The method of claim 1, wherein selecting the plurality of documents is based on at least one of occupying a same Uniform Resource Locator (URL) space or location on a same computer.

4. The method of claim 1, wherein the identical element comprises at least one of an advertisement, a site navigation component, a header, a footer, or a copyright notice.

5. The method of claim 1, further comprising, for each document of the plurality of documents:

retrieving a plurality of updated views of the document;

identifying an item that differs between the plurality of updated views of the document; and

removing the item from consideration prior to the subsequent indexing process of the plurality of documents.

6. The method of claim 1, wherein the selecting, identifying, and removing are performed by the processing device of a web search engine as a pre-processing step for the indexing process.

7. The method of claim 1, further comprising retrieving the plurality of documents via a network interface of a computer.

8. A computer system comprising:

a memory; and

a processing device coupled to the memory, the processing device to;

identify an identical element comprising information common to each of the plurality of documents; and

remove the identical element from each of the plurality of documents to form modifications to the plurality of documents, prior to a subsequent indexing process of the plurality of documents.

9. The computer system of claim 8, wherein the plurality of documents are web pages.

10. The computer system of claim 8, wherein the plurality of documents are selected based on at least one of occupying a same Uniform Resource Locator (URL) space or location on a same computer.

11. The computer system of claim 8, wherein the identical element comprises at least one of an advertisement, a site navigation component, a header, a footer, or a copyright notice.

12. The computer system of claim 8, the processing device, for each document of the plurality of documents, to:

retrieve a plurality of updated views of the document;

identify an item that differs between the plurality of updated views of the document; and

remove the item from consideration prior to the subsequent indexing process of the plurality of documents.

13. The computer system of claim 12, wherein the selecting, identifying, and removing are performed by a web search engine as a pre-processing step for the indexing process.

14. The computer system of claim 8, the memory to store the plurality of documents.

15. A non-transitory machine-readable storage medium programmed to include instructions executable by a processing device to cause the processing device to perform operations comprising:

selecting, by the processing device, a plurality of documents for index comparison;

identifying, by the processing device, an identical element comprising information common to each of the plurality of documents; and

removing, by the processing device, the identical element from each of the plurality of documents to form modifications to the plurality of documents, prior to a subsequent indexing process of the plurality of documents.

16. The non-transitory machine-readable storage medium of claim 15, wherein the plurality of documents are web pages.

17. The non-transitory machine-readable storage medium of claim 15, wherein the plurality of documents have at least a same Uniform Resource Locator (URL) space or are located on a same computer.

18. The non-transitory machine-readable storage medium of claim 15, wherein the identical element comprises at least one of an advertisement, a site navigation component, a header, a footer, or a copyright notice.

19. (C) The non-transitory machine-readable storage medium of claim 15, further comprising, for each document of the plurality of documents:

retrieving a plurality of updated views of the document;

identifying an item that differs between the plurality of updated views of the document; and

removing the item from consideration prior to the subsequent indexing process of the plurality of documents.

20. The non-transitory machine-readable storage medium of claim 15, wherein the selecting, identifying, and removing are performed by a web search engine as a pre-processing step for the indexing process.