Method, system, and program for handling redirects in a search engine

• US 8,296,304 B2
• Filed: 01/26/2004
• Issued: 10/23/2012
• Est. Priority Date: 01/26/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Fontoura, Marcus F., Neumann, Andreas, Qi, Runping, Shekita, Eugene J.
• Primary Examiner(s): ALAM, SHAHID AL
• Application Number:
  US10/764,771
• Publication Number:
  US 20050165800A1
• Time in Patent Office: 3,193 Days
• Field of Search: 707 1- 10, 707100-1041, 707200-205, 707/741, 707/748, 715/501.1, 709/201, 719/328
• US Class Current: 707/748
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9558 Details of hyperlinks; Management of linked annotations

Abstract

Disclosed is a method, system, and program for handling redirects in documents. At least one equivalence class that includes documents that are connected through a redirect. Cycles for each equivalence class are detected, wherein documents in a cycle are marked so that they are not indexed. Incomplete chains for each equivalence class are detected, wherein documents in an incomplete chain are marked so that they are not indexed. A representative for each equivalence class is selected.

Claims

1. A method for handling redirects in documents, comprising:

while generating an index,

determining a rank for each of the documents, wherein the rank represents an importance of each document relative to the other documents;

forming at least one equivalence class that includes documents that are connected through a redirect, wherein each equivalence class describes a redirect chain;

detecting cycles for each equivalence class, wherein documents in a cycle are marked so that they are not indexed, and, wherein, for each equivalence class, the cycle is formed when a last document in the redirect chain redirects to a first document in the redirect chain;

detecting incomplete chains for each equivalence class, wherein documents in an incomplete chain are marked so that they are not indexed, and, wherein, for each equivalence class, the indirect chain is formed when a last document in the redirect chain redirects to a document that has not been crawled;

selecting a representative for each equivalence class whose documents are to be indexed, wherein the representative is associated with a path that indicates a location of a document in a data store;

detecting duplicate documents in two different equivalence classes; and

merging the equivalence classes.

2. The method of claim 1, wherein the representative is selected based on a type of redirect in an equivalence class.

3. The method of claim 1, wherein the representative is selected based on the rank of each document in the equivalence class.

4. The method of claim 1, further comprising:

locating each document that contains a redirect; and

creating an entry in a redirect file for each document.

5. The method of claim 4, wherein the entry includes a source path, a target path, and a redirect type.

6. The method of claim 1, wherein documents are duplicates if a certain portion of their content is similar.

7. The method of claim 1, wherein the documents in the at least one equivalence class include a target document and one or more source documents and wherein the selected representative is one of the source documents, further comprising:

propagating the content of the target document to the selected representative.

8. The method of claim 1, wherein the documents in the at least one equivalence class include a target document and one or more source documents, and wherein at least one source document includes a path to the target document.

9. The method of claim 8, further comprising:

indexing the content of the target document with a path of the representative.

10. The method of claim 1, wherein marking documents so that they are not indexed includes marking documents to indicate the documents are to be ignored.

11. An article of manufacture comprising one of hardware logic and a computer readable storage medium including a program for handling redirects in documents, wherein the hardware logic or program causes operations to be performed, the operations comprising:

while generating an index,

determining a rank for each of the documents, wherein the rank represents an importance of each document relative to the other documents;

forming at least one equivalence class that includes documents that are connected through a redirect, wherein each equivalence class describes a redirect chain;

detecting cycles for each equivalence class, wherein documents in a cycle are marked so that they are not indexed, and, wherein, for each equivalence class, the cycle is formed when a last document in the redirect chain redirects to a first document in the redirect chain;

detecting incomplete chains for each equivalence class, wherein documents in an incomplete chain are marked so that they are not indexed, and, wherein, for each equivalence class, the indirect chain is formed when a last document in the redirect chain redirects to a document that has not been crawled;

selecting a representative for each equivalence class whose documents are to be indexed, wherein the representative is associated with a path that indicates a location of a document in a data store;

detecting duplicate documents in two different equivalence classes; and

merging the equivalence classes.

12. The article of manufacture of claim 11, wherein the representative is selected based on a type of redirect in an equivalence class.

13. The article of manufacture of claim 11, wherein the representative is selected based on the rank of each document in the equivalence class.

14. The article of manufacture of claim 11, wherein the operations further comprise:

locating each document that contains a redirect; and

creating an entry in a redirect file for each document.

15. The article of manufacture of claim 14, wherein the entry includes a source path, a target path, and a redirect type.

16. The article of manufacture of claim 11, wherein documents are duplicates if a certain portion of their content is similar.

17. The article of manufacture of claim 11, wherein the documents in the at least one equivalence class include a target document and one or more source documents and wherein the selected representative is one of the source documents, wherein the operations further comprise:

propagating the content of the target document to the selected representative.

18. The article of manufacture of claim 11, wherein the documents in the at least one equivalence class include a target document and one or more source documents, and wherein at least one source document includes a path to the target document.

19. The article of manufacture of claim 18, wherein the operations further comprise:

indexing the content of the target document with a path of the representative.

20. The article of manufacture of claim 11, wherein the operations for marking documents so that they are not indexed include operations for marking documents to indicate the documents are to be ignored.

21. A computer system including logic for handling redirects in documents, comprising:

a processor; and

while generating an index,

determining a rank for each of the documents, wherein the rank represents an importance of each document relative to the other documents;

forming at least one equivalence class that includes documents that are connected through a redirect, wherein each equivalence class describes a redirect chain;

detecting cycles for each equivalence class, wherein documents in a cycle are marked so that they are not indexed, and, wherein, for each equivalence class, the cycle is formed when a last document in the redirect chain redirects to a first document in the redirect chain;

detecting incomplete chains for each equivalence class, wherein documents in an incomplete chain are marked so that they are not indexed, and, wherein, for each equivalence class, the indirect chain is formed when a last document in the redirect chain redirects to a document that has not been crawled;

selecting a representative for each equivalence class whose documents are to be indexed, wherein the representative is associated with a path that indicates a location of a document in a data store;

detecting duplicate documents in two different equivalence classes; and

merging the equivalence classes.

22. The computer system of claim 21, wherein the representative is selected based on a type of redirect in an equivalence class.

23. The computer system of claim 21, wherein the representative is selected based on the rank of each document in the equivalence class.

24. The computer system of claim 21, wherein the logic further comprises:

locating each document that contains a redirect; and

creating an entry in a redirect file for each document.

25. The computer system of claim 24, wherein the entry includes a source path, a target path, and a redirect type.

26. The computer system of claim 21, wherein documents are duplicates if a certain portion of their content is similar.

27. The computer system of claim 26, wherein the documents in the at least one equivalence class include a target document and one or more source documents and wherein the selected representative is one of the source documents, wherein the logic further comprises:

propagating the content of the target document to the selected representative.

28. The computer system of claim 21, wherein the documents in the at least one equivalence class include a target document and one or more source documents, and wherein at least one source document includes a path to the target document.

29. The computer system of claim 28, wherein the logic further comprises:

indexing the content of the target document with a path of the representative.

30. The computer system of claim 21, wherein marking documents so that they are not indexed includes marking documents to indicate the documents are to be ignored.