Method, system, and program for handling redirects in a search engine

• US 20050165800A1
• Filed: 01/26/2004
• Published: 07/28/2005
• Est. Priority Date: 01/26/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Neumann, Andreas, Shekita, Eugene J., Qi, Runping, Fontoura, Marcus F.
• Granted Patent: US 8,296,304 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9558 Details of hyperlinks; Management of linked annotations

Abstract

Disclosed is a method, system, and program for handling redirects in documents. At least one equivalence class that includes documents that are connected through a redirect. Cycles for each equivalence class are detected, wherein documents in a cycle are marked so that they are not indexed. Incomplete chains for each equivalence class are detected, wherein documents in an incomplete chain are marked so that they are not indexed. A representative for each equivalence class is selected.

Claims

1. A method for handling redirects in documents, comprising:

forming at least one equivalence class that includes documents that are connected through a redirect;

detecting cycles for each equivalence class, wherein documents in a cycle are marked so that they are not indexed;

detecting incomplete chains for each equivalence class, wherein documents in an incomplete chain are marked so that they are not indexed; and

selecting a representative for each equivalence class.

2. The method of claim 1, wherein the representative is selected based on a type of redirect in an equivalence class.

3. The method of claim 1, wherein the representative is selected based on a rank of each document in the equivalence class.

4. The method of claim 1, further comprising:

locating each document that contains a redirect; and

creating an entry in a redirect file for each document.

5. The method of claim 4, wherein the entry includes a source path, a target path, and a redirect type.

6. The method of claim 1, further comprising:

detecting duplicate documents in two different equivalence classes; and

merging the equivalence classes.

7. The method of claim 6, wherein documents are duplicates if a certain portion of their content is similar.

8. The method of claim 1, wherein the documents in the at least one equivalence class include a target document and one or more source documents and wherein the selected representative is one of the source documents, further comprising:

propagating the content of the target document to the selected representative.

9. The method of claim 1, wherein the documents in the at least one equivalence class include a target document and one or more source documents, and wherein at least one source document includes a path to the target document.

10. The method of claim 9, further comprising:

indexing the content of the target document with a path of the representative.

11. The method of claim 1, wherein marking documents so that they are not indexed includes marking documents to indicate the documents are to be ignored.

12. The method of claim 1, further comprising:

determining a rank for each of the documents, wherein the rank represents an importance of each document relative to the other documents.

13. An article of manufacture including a program for handling redirects in documents, wherein the program causes operations to be performed, the operations comprising:

forming at least one equivalence class that includes documents that are connected through a redirect;

detecting cycles for each equivalence class, wherein documents in a cycle are marked so that they are not indexed;

detecting incomplete chains for each equivalence class, wherein documents in an incomplete chain are marked so that they are not indexed; and

selecting a representative for each equivalence class.

14. The article of manufacture of claim 13, wherein the representative is selected based on a type of redirect in an equivalence class.

15. The article of manufacture of claim 13, wherein the representative is selected based on a rank of each document in the equivalence class.

16. The article of manufacture of claim 13, wherein the operations further comprise:

locating each document that contains a redirect; and

creating an entry in a redirect file for each document.

17. The article of manufacture of claim 16, wherein the entry includes a source path, a target path, and a redirect type.

18. The article of manufacture of claim 13, wherein the operations further comprise:

detecting duplicate documents in two different equivalence classes; and

merging the equivalence classes.

19. The article of manufacture of claim 18, wherein documents are duplicates if a certain portion of their content is similar.

20. The article of manufacture of claim 13, wherein the documents in the at least one equivalence class include a target document and one or more source documents and wherein the selected representative is one of the source documents, wherein the operations further comprise:

propagating the content of the target document to the selected representative.

21. The article of manufacture of claim 13, wherein the documents in the at least one equivalence class include a target document and one or more source documents, and wherein at least one source document includes a path to the target document.

22. The article of manufacture of claim 21, wherein the operations further comprise:

indexing the content of the target document with a path of the representative.

23. The article of manufacture of claim 13, wherein the operations for marking documents so that they are not indexed include operations for marking documents to indicate the documents are to be ignored.

24. The article of manufacture of claim 13, wherein the operations further comprise:

determining a rank for each of the documents, wherein the rank represents an importance of each document relative to the other documents.

25. A computer system including logic for handling redirects in documents, comprising:

forming at least one equivalence class that includes documents that are connected through a redirect;

detecting cycles for each equivalence class, wherein documents in a cycle are marked so that they are not indexed;

detecting incomplete chains for each equivalence class, wherein documents in an incomplete chain are marked so that they are not indexed; and

selecting a representative for each equivalence class.

26. The computer system of claim 25, wherein the representative is selected based on a type of redirect in an equivalence class.

27. The computer system of claim 25, wherein the representative is selected based on a rank of each document in the equivalence class.

28. The computer system of claim 25, wherein the logic further comprises:

locating each document that contains a redirect; and

creating an entry in a redirect file for each document.

29. The computer system of claim 28, wherein the entry includes a source path, a target path, and a redirect type.

30. The computer system of claim 25, wherein the logic further comprises:

detecting duplicate documents in two different equivalence classes; and

merging the equivalence classes.

31. The computer system of claim 30, wherein documents are duplicates if a certain portion of their content is similar.

32. The computer system of claim 31, wherein the documents in the at least one equivalence class include a target document and one or more source documents and wherein the selected representative is one of the source documents, wherein the logic further comprises:

propagating the content of the target document to the selected representative.

33. The computer system of claim 25, wherein the documents in the at least one equivalence class include a target document and one or more source documents, and wherein at least one source document includes a path to the target document.

34. The computer system of claim 33, wherein the logic further comprises:

indexing the content of the target document with a path of the representative.

35. The computer system of claim 25, wherein marking documents so that they are not indexed includes marking documents to indicate the documents are to be ignored.

36. The computer system of claim 25, wherein the logic further comprises:

determining a rank for each of the documents, wherein the rank represents an importance of each document relative to the other documents.