GLOBAL ANCHOR TEXT PROCESSING

• US 20080162425A1
• Filed: 12/28/2006
• Published: 07/03/2008
• Est. Priority Date: 12/28/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: GRENNAN, Frederick Stephen, LEMPEL, Ronny, NEUMANN, Andreas
• Granted Patent: US 7,788,253 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/134 Hyperlinking

Abstract

Provided are techniques for building a search index. While building the search index and using the search index to respond to one or more search requests, an anchor information store is maintained, wherein each entry of the anchor information store identifies a referring document, a target document, and anchor text associated with a link from the referring document to the target document; a document is received for processing; one or more entries in the anchor information store for which the document to be processed is identified as the target document are located; anchor text is retrieved from each of the identified entries; and the retrieved anchor text is stored in an entry of the search index for the document.

Claims

1. A computer-implemented method for building a search index:

while building the search index and using the search index to respond to one or more search requests,

maintaining an anchor information store, wherein each entry of the anchor information store identifies a referring document, a target document, and anchor text associated with a link from the referring document to the target document;

receiving a document for processing;

locating one or more entries in the anchor information store for which the document to be processed is identified as the target document;

retrieving anchor text from each of the identified entries; and

storing the retrieved anchor text in an entry of the search index for the document.

2. The computer-implemented method of claim 1, wherein the document includes one or more outgoing links, each of which points to a target document, further comprising:

for each of the outgoing links, adding an entry to the anchor information store, wherein the document to be processed is the referring document for each target document.

3. The computer-implemented method of claim 1, wherein the document includes one or more outgoing links, each of which points to a target document, further comprising:

adding an entry to a rebuild agenda, wherein the entry identifies the target document.

4. The computer-implemented method of claim 3, further comprising:

asynchronously, for each target document identified in each entry in the rebuild agenda,

using the anchor information store to find anchor text for that target document; and

updating the anchor text in the entry of the search index for the target document, wherein the anchor text for the target document is updated incrementally.

5. The computer-implemented method of claim 1, further comprising:

when a document is one of missing and unavailable,

locating one or more entries in the anchor information store for which the document is a referring document;

for each of the one or more entries, registering the target document of that entry in a rebuild agenda; and

deleting the one or more entries from the anchor information store.

6. The computer-implemented method of claim 1, further comprising:

when a document is one of missing and unavailable,

locating one or more entries in the anchor information store for which the document is the target document; and

reporting the referring document in each of the identified entries to one of a search engine administrator and a content owner of the referring document.

7. The computer-implemented method of claim 1, further comprising:

generating a graph that shows connections of nodes using the anchor information store to enable studying site connectedness.

8. The computer-implemented method of claim 1, further comprising:

adjusting a search-result weight associated with the document based on an in-link count determined by counting a number of referring documents for the document in the anchor information store.

9. The computer-implemented method of claim 1, wherein the anchor information store includes tags, which are user annotations of documents.

10. The computer-implemented method of claim 1, further comprising:

when the document has been updated with new content,

comparing existing entries in the anchor information store for which the document is a referring document with a current set of anchors in the document;

deleting each entry from the anchor information store that no longer matches the document content;

adding a new entry to the anchor information store for each new anchor in the document; and

registering a target document of any entry that was one of deleted from and added to the anchor information store in a rebuild agenda.

11. A computer-implemented method for maintaining a search index, comprising:

maintaining an anchor information store, wherein each entry of the anchor information store identifies a referring document, a target document, and anchor text associated with a link from the referring document to the target document;

maintaining a rebuild agenda, wherein each entry in the rebuild agenda identifies a target document; and

asynchronously, for each target document identified in each entry in the rebuild agenda,

using the anchor information store to find anchor text for that target document; and

updating the anchor text in an entry of the search index for the target document, wherein the anchor text for the target document is updated incrementally.

12. The computer-implemented method of claim 11, further comprising:

receiving a document for processing;

locating one or more entries in the anchor information store for which the document to be processed is identified as the target document;

retrieving anchor text from each of the identified entries; and

storing the retrieved anchor text in an entry of the search index for the document.

13. A computer program product comprising a computer useable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

while building the search index and using the search index to respond to one or more search requests,

maintain an anchor information store, wherein each entry of the anchor information store identifies a referring document, a target document, and anchor text associated with a link from the referring document to the target document;

receive a document for processing;

locate one or more entries in the anchor information store for which the document to be processed is identified as the target document;

retrieve anchor text from each of the identified entries; and

store the retrieved anchor text in an entry of the search index for the document.

14. The computer program product of claim 13, wherein the document includes one or more outgoing links, each of which points to a target document, and wherein the computer readable program when executed on a computer causes the computer to:

for each of the outgoing links, add an entry to the anchor information store, wherein the document to be processed is the referring document for each target document.

15. The computer program product of claim 13, wherein the document includes one or more outgoing links, each of which points to a target document, and wherein the computer readable program when executed on a computer causes the computer to:

add an entry to a rebuild agenda, wherein the entry identifies the target document.

16. The computer program product of claim 15, wherein the computer readable program when executed on a computer causes the computer to:

asynchronously, for each target document identified in each entry in the rebuild agenda,

use the anchor information store to find anchor text for that target document; and

update the anchor text in the entry of the search index for the target document, wherein the anchor text for the target document is updated incrementally.

17. The computer program product of claim 13, wherein the computer readable program when executed on a computer causes the computer to:

when a document is one of missing and unavailable,

locate one or more entries in the anchor information store for which the document is a referring document;

for each of the one or more entries, register the target document of that entry in a rebuild agenda; and

delete the one or more entries from the anchor information store.

18. The computer program product of claim 13, wherein the computer readable program when executed on a computer causes the computer to:

when a document is one of missing and unavailable,

locate one or more entries in the anchor information store for which the document is the target document; and

report the referring document in each of the identified entries to one of a search engine administrator and a content owner of the referring document.

19. The computer program product of claim 13, wherein the computer readable program when executed on a computer causes the computer to:

generate a graph that shows connections of nodes using the anchor information store to enable studying site connectedness.

20. The computer program product of claim 13, wherein the computer readable program when executed on a computer causes the computer to:

adjust a search-result weight associated with the document based on an in-link count determined by counting a number of referring documents for the document in the anchor information store.

21. The computer program product of claim 13, wherein the anchor information store includes tags, which are user annotations of documents.

22. The computer program product of claim 13, wherein the computer readable program when executed on a computer causes the computer to:

when the document has been updated with new content,

compare existing entries in the anchor information store for which the document is a referring document with a current set of anchors in the document;

delete each entry from the anchor information store that no longer matches the document content;

add a new entry to the anchor information store for each new anchor in the document; and

register a target document of any entry that was one of deleted from and added to the anchor information store in a rebuild agenda.

23. A computer program product comprising a computer useable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

maintain an anchor information store, wherein each entry of the anchor information store identifies a referring document, a target document, and anchor text associated with a link from the referring document to the target document;

maintain a rebuild agenda, wherein each entry in the rebuild agenda identifies a target document; and

asynchronously, for each target document identified in each entry in the rebuild agenda,

use the anchor information store to find anchor text for that target document; and

update the anchor text in an entry of the search index for the target document, wherein the anchor text for the target document is updated incrementally.

24. The computer program product of claim 23, wherein the computer readable program when executed on a computer causes the computer to:

receive a document for processing;

locate one or more entries in the anchor information store for which the document to be processed is identified as the target document;

retrieve anchor text from each of the identified entries; and

store the retrieved anchor text in an entry of the search index for the document.

25. A system for building a search index, comprising:

logic capable of performing operations, the operations comprising;

while building the search index and using the search index to respond to one or more search requests,

maintaining an anchor information store, wherein each entry of the anchor information store identifies a referring document, a target document, and anchor text associated with a link from the referring document to the target document;

receiving a document for processing;

locating one or more entries in the anchor information store for which the document to be processed is identified as the target document;

retrieving anchor text from each of the identified entries; and

storing the retrieved anchor text in an entry of the search index for the document.

26. The system of claim 25, wherein the document includes one or more outgoing links, each of which points to a target document, and wherein the operations further comprise:

for each of the outgoing links, adding an entry to the anchor information store, wherein the document to be processed is the referring document for each target document.

27. The system of claim 25, wherein the document includes one or more outgoing links, each of which points to a target document, and wherein the operations further comprise:

adding an entry to a rebuild agenda, wherein the entry identifies the target document.

28. The system of claim 27, wherein the operations further comprise:

asynchronously, for each target document identified in each entry in the rebuild agenda,

using the anchor information store to find anchor text for that target document; and

updating the anchor text in the entry of the search index for the target document, wherein the anchor text for the target document is updated incrementally.

29. The system of claim 25, wherein the operations further comprise:

when a document is one of missing and unavailable,

locating one or more entries in the anchor information store for which the document is a referring document;

for each of the one or more entries, registering the target document of that entry in a rebuild agenda; and

deleting the one or more entries from the anchor information store.

30. The system of claim 25, wherein the operations further comprise:

when a document is missing or unavailable,

locating one or more entries in the anchor information store for which the document is the target document; and

reporting the referring document in each of the identified entries to one of a search engine administrator and a content owner of the referring document.

31. The system of claim 25, wherein the operations further comprise:

generating a graph that shows connections of nodes using the anchor information store to enable studying site connectedness.

32. The system of claim 25, wherein the operations further comprise:

adjusting a search-result weight associated with the document based on an in-link count determined by counting a number of referring documents for the document in the anchor information store.

33. The system of claim 25, wherein the anchor information store includes tags, which are user annotations of documents.

34. The system of claim 25, wherein the operations further comprise:

when the document has been updated with new content,

comparing existing entries in the anchor information store for which the document is a referring document with a current set of anchors in the document;

deleting each entry from the anchor information store that no longer matches the document content;

adding a new entry to the anchor information store for each new anchor in the document; and

registering a target document of any entry that was one of deleted from and added to the anchor information store in a rebuild agenda.

35. A system for building a search index, comprising:

logic capable of performing operations, the operations comprising;

maintaining an anchor information store, wherein each entry of the anchor information store identifies a referring document, a target document, and anchor text associated with a link from the referring document to the target document;

maintaining a rebuild agenda, wherein each entry in the rebuild agenda identifies a target document; and

asynchronously, for each target document identified in each entry in the rebuild agenda,

using the anchor information store to find anchor text for that target document; and

updating the anchor text in an entry of the search index for the target document, wherein the anchor text for the target document is updated incrementally.

36. The system of claim 35, wherein the operations further comprise:

receiving a document for processing;

locating one or more entries in the anchor information store for which the document to be processed is identified as the target document;

retrieving anchor text from each of the identified entries; and

storing the retrieved anchor text in an entry of the search index for the document.