System and method for maintaining up-to-date link information in the metadata repository of a search engine

• US 6,611,835 B1
• Filed: 05/04/2000
• Issued: 08/26/2003
• Est. Priority Date: 05/04/2000
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Sundaresan, Neelakantan, Huang, Anita Wai-Ling
• Primary Examiner(s): ROBINSON, GRETA LEE
• Application Number:
  US09/565,129
• Time in Patent Office: 1,209 Days
• Field of Search: 707/104.1, 707/3, 707/10, 707/513, 707/100, 707/200, 709/223, 715/513
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A system and method for updating search engine information that is more efficient, less time-consuming, and less costly than prior techniques. In order to carry out the method a uniform resource locator indicating a Web page for which the search engine information is to be updated is selected. The selected uniform resource locator is transmitted to a server on which the Web page is located. A response code indicating a status of the Web page is received and the search engine information is updated based on the response code.

Claims

1. A method for updating search engine information comprising the steps of:

selecting a uniform resource locator indicating a Web page for which the search engine information is to be updated;

contacting a server on which the indicated Web page is located to obtain the Web page;

receiving a response code indicating the status of the Web page; and

updating the search engine information based on the response code,

wherein when the response code indicates that the Web page cannot be found the updating step comprises the step of deleting information relating to the Web page from the search engine information, and

wherein when the response code indicates that the Web page has been moved the updating step comprises the step of modifying information relating to the Web page that is included in the search engine information, and

wherein the deleting step comprises the steps of

receiving a plurality of parent uniform resource locators related to the selected uniform resource locator;

deleting all instances of the selected uniform resource locator from the search engine information;

updating metadata summarizing Web pages indicated by the plurality of parent uniform resource locators, the metadata referencing the selected uniform resource locator; and

deleting metadata summarizing the Web page indicated by the selected uniform resource locator.

2. The method of claim 1, wherein the step of updating metadata summarizing Web pages comprises the step of:

modifying existing RDF summaries for each parent uniform resource locator in the search engine information to remove references to the selected uniform resource locator and annotation information from its list of out-links.

3. The method of claim 1, wherein the step of updating metadata summarizing Web pages comprises the step of:

resummarizing metadata information for each parent uniform resource locator in the search engine information to create new RDF summaries with updated information.

4. A method for updating search engine information comprising the steps of:

selecting a uniform resource locator indicating a Web page for which the search engine information is to be updated;

contacting a server on which the indicated Web page is located to obtain the Web page;

receiving a response code indicating the status of the Web page; and

updating the search engine information based on the response code,

wherein when the response code indicates that the Web page cannot be found the updating step comprises the step of deleting information relating to the Web page from the search engine information, and

wherein when the response code indicates that the Web page has been moved the updating step comprises the step of modifying information relating to the Web page that is included in the search engine information, and

wherein the modifying step comprises the steps of

receiving a uniform resource locator indicating a new location of the Web page indicated by the selected uniform resource locator;

receiving a plurality of parent uniform resource locators and child uniform resource locators related to the selected uniform resource locator;

replacing all instances of the selected uniform resource locator with the uniform resource locator indicating the new location of the Web page;

updating metadata summarizing Web pages indicated by the plurality of parent uniform resource locators and child uniform resource locators, the metadata referencing the selected uniform resource locator;

crawling the uniform resource locator indicating the new location of the Web page to update metadata summarizing the Web page; and

deleting metadata summarizing the Web page indicated by the selected uniform resource locator.

5. The method of claim 4, wherein the step of updating metadata summarizing Web pages comprises the step of:

modifying existing RDF summaries for each parent uniform resource locator in the search engine information by replacing the selected uniform resource locator with the uniform resource locator indicating the new location of the Web page among the plurality of parent uniform resource locators and child uniform resource locators related to the selected uniform resource locator.

6. The method of claim 4, wherein the step of updating metadata summarizing Web pages comprises the step of:

summarizing each parent uniform resource locator and child uniform resource locator among the received parent uniform resource locators and child uniform resource locators to create new RDF summaries.

7. A system for updating search engine information comprising:

means for selecting a uniform resource locator indicating a Web page for which the search engine information is to be updated;

means for contacting a server on which the indicated Web page is located to obtain the Web page;

means for receiving a response code indicating a status of the Web page;

means for updating the search engine information based on the response code,

wherein when the response code indicates that the Web page cannot be found the updating means comprises means for deleting information relating to the Web page from the search engine information, and

wherein when the response code indicates that the Web page has been moved the updating means comprises means for modifying information relating to the Web page that is included in the search engine information, and

wherein the deleting means comprises

means for receiving a plurality of parent uniform resource locators related to the selected uniform resource locator;

means for deleting all instances of the selected uniform resource locator from the search engine information;

means for updating metadata summarizing Web pages indicated by the plurality of parent uniform resource locators, the metadata referencing the selected uniform resource locator; and

means for deleting metadata summarizing the Web page indicated by the selected uniform resource locator.

8. The system of claim 7, wherein the means for updating metadata summarizing Web pages comprises:

means for modifying existing RDF summaries for each parent uniform resource locator in the search engine information to remove references to the selected uniform resource locator and annotation information from its list of outlinks.

9. The system of claim 7, wherein the means for updating metadata summarizing Web pages comprises:

means for resummarizing metadata information for each parent uniform resource locator in the search engine information to create new RDF summaries with updated information.

10. A system for updating search engine information comprising:

means for selecting a uniform resource locator indicating a Web page for which the search engine information is to be updated;

means for contacting a server on which the indicated Web page is located to obtain the Web page;

means for receiving a response code indicating a status of the Web page;

means for updating the search engine information based on the response code,

wherein when the response code indicates that the Web page cannot be found the updating means comprises means for deleting information relating to the Web page from the search engine information, and

wherein when the response code indicates that the Web page has been moved the updating means comprises means for modifying information relating to the Web page that is included in the search engine information, and

wherein the modifying means comprises

means for receiving a uniform resource locator indicating a new location of the Web page indicated by the selected uniform resource locator;

means for receiving a plurality of parent uniform resource locators and child uniform resource locators related to the selected uniform resource locator;

means for replacing all instances of the selected uniform resource locator with the uniform resource locator indicating the new location of the Web page;

means for updating metadata summarizing Web pages indicated by the plurality of parent uniform resource locators and child uniform resource locators, the metadata referencing the selected uniform resource locator;

means for crawling the uniform resource locator indicating the new location of the Web page to update metadata summarizing the Web page; and

means for deleting metadata summarizing the Web page indicated by the selected uniform resource locator.

11. The system of claim 10, wherein the means for updating metadata summarizing Web pages comprises:

means for modifying existing RDF summaries for each parent uniform resource locator in the search engine information by replacing the selected uniform resource locator with the uniform resource locator indicating the new location of the Web page among the plurality of parent uniform resource locators and child uniform resource locators related to the selected uniform resource locator.

12. The system of claim 10, wherein the means for updating metadata summarizing Web pages comprises:

means for summarizing each parent uniform resource locator and child uniform resource locator among the received parent uniform resource locators and child uniform resource locators to create new RDF summaries.

13. A computer program product for updating search engine information, comprising:

a computer readable medium;

computer program instructions, recorded on the computer readable medium, executable by a processor, for performing the steps of

selecting a uniform resource locator indicating a Web page for which the search engine information is to be updated;

contacting a server on which the indicated Web page is located to obtain the Web page;

receiving a response code indicating a status of the Web page; and

updating the search engine information based on the response code,

wherein when the response code indicates that the Web page cannot be found the updating step comprises the step of deleting information relating to the Web page from the search engine information, and

wherein when the response code indicates that the Web page has been moved the updating step comprises the step of modifying information relating to the Web page that is included in the search engine information, and

wherein the deleting step comprises the steps of

receiving a plurality of parent uniform resource locators related to the selected uniform resource locator;

deleting all instances of the selected uniform resource locator from the search engine information;

updating metadata summarizing Web pages indicated by the plurality of parent uniform resource locators, the metadata referencing the selected uniform resource locator; and

deleting metadata summarizing the Web page indicated by the selected uniform resource locator.

14. The computer program product of claim 13, wherein the step of updating metadata summarizing Web pages comprises the step of:

modifying existing RDF summaries for each parent uniform resource locator in the search engine information to remove references to the selected uniform resource locator and annotation information from its list of out-links.

15. The computer program product of claim 13, wherein the step of updating metadata summarizing Web pages comprises the step of:

resummarizing metadata information for each parent uniform resource locator in the search engine information to create new RDF summaries with updated information.

16. A computer program product for updating search engine information, comprising:

a computer readable medium;

computer program instructions, recorded on the computer readable medium, executable by a processor, for performing the steps of

selecting a uniform resource locator indicating a Web page for which the search engine information is to be updated;

contacting a server on which the indicated Web page is located to obtain the Web page;

receiving a response code indicating a status of the Web page; and

updating the search engine information based on the response code,

wherein when the response code indicates that the Web page cannot be found the updating step comprises the step of deleting information relating to the Web page from the search engine information, and

wherein when the response code indicates that the Web page has been moved the updating step comprises the step of modifying information relating to the Web page that is included in the search engine information, and

wherein the modifying step comprises the steps of

receiving a uniform resource locator indicating a new location of the Web page indicated by the selected uniform resource locator;

receiving a plurality of parent;

uniform resource locators and child uniform resource locators related to the selected uniform resource locator;

replacing all instances of the selected uniform resource locator with the uniform resource locator indicating the new location of the Web page;

updating metadata summarizing Web pages indicated by the plurality of parent uniform resource locators and child uniform resource locators, the metadata referencing the selected uniform resource locator;

crawling the uniform resource locator indicating the new location of the Web page to update metadata summarizing the Web page; and

deleting metadata summarizing the Web page indicated by the selected uniform resource locator.

17. The computer program product of claim 16, wherein the step of updating metadata summarizing Web pages comprises the step of:

modifying existing RDF summaries for each parent uniform resource locator in the search engine information by replacing the selected uniform resource locator with the uniform resource locator indicating the new location of the Web page among the plurality of parent uniform resource locators and child uniform resource locators related to the selected uniform resource locator.

18. The computer program product of claim 16, wherein the step of updating metadata summarizing Web pages comprises the step of:

summarizing each parent uniform resource locator and child uniform resource locator among the received parent uniform resource locators and child uniform resource locators to create new RDF summaries.