AVOIDING MASKED WEB PAGE CONTENT INDEXING ERRORS FOR SEARCH ENGINES

• US 20090265342A1
• Filed: 04/16/2009
• Published: 10/22/2009
• Est. Priority Date: 04/16/2008
• Status: Active Grant

• Current Assignee: Gary S. Shuster
• Original Assignee: Gary S. Shuster
• Inventors: Shuster, Gary Stephen
• Granted Patent: US 9,405,831 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching

Abstract

Multiple non-host client sites provide cached user copies of web pages and/or web content, or summaries thereof, to a server. Obtaining data from non-host sources for indexing purposes avoids masked web page content indexing errors for search engines. The server aggregates, summarizes and indexes the web pages and/or web content in an index of cached content, in conjunction with updating, generating and storing a search index using an indexing agent such as a web crawler or spider. In response to receiving search requests from end users, the search engine uses comparisons between the index of cached content and the index of crawled content to identify potential page masking errors for specific search results and appropriately rank or omit results with a high risk of masking errors in a search result list.

Claims

1. A method comprising:

accessing a web page using a server operating a web crawling application to create first index information for the web page;

receiving, at the server, second index information generated from a cached copy of the web page received by a client via a browser application operated at the client; and

ranking the web page in a search results list generated at the server, based on a comparison between the first and second indexes of the web page.

2. The method of claim 1, further comprising:

generating an updated index based on the comparison between the first index information and the second index information of the web page; and

generating a search result based on the updated index for at least one client.

3. The method of claim 1, wherein the cached copy of the web page from the client is identified by a URL that designate a network address other than the web page.

4. The method of claim 1, wherein generating the second index information further comprises summarizing the cached copy of the web page, the cached copy includes a uniform resource locator, network address and one or more key terms.

5. The method of claim 1, wherein generating the second index information further comprises aggregating a plurality of cached copies of the web page from one or more remote clients.

6. The method of claim 1, further comprising measuring popularity for each of the web pages by counting a number of the cached client copies received.

7. The method of claim 1, further comprising distributing an application to the client, the application configured to operate on computer systems of the client and to periodically transmit the cached copy of a web page to a server.

8. The method of claim 7, wherein the application is further configured to summarize the cached copy of the web page prior to transmission.

9. The method of claim 7, wherein the application is further configured to aggregate the cached user copies prior to transmission.

10. The method of claim 7, wherein the application is further configured to generate the second index information from the cached copy of the web page.

11. A system for indexing web pages comprising:

means for accessing a web page to create a first index information of the web page;

means for receiving a cached copy of the web page at a client;

means for generating a second index information of the web page using the cached copy; and

means for ranking the web page based on a comparison between the first index information and the second index information of the web page.

12. The system of claim 11, further comprising:

means for generating an updated index based on the comparison between the first index information and the second index information of the web page; and

means for generating a search result based on the updated index for at least one client.

13. The system of claim 11, wherein the cached copy of the web page from the remote user is identified by a URL that designate a network address other than the web page.

14. The system of claim 11, wherein means for generating the second index further comprises summarizing the cached copy of the web page, the cached copy includes a uniform resource locator, network address and one or more key terms.

15. The system of claim 11, wherein means for generating the second index information further comprises aggregating a plurality of cached copies of the web page from one or more remote clients.

16. The system of claim 11, further comprising means for distributing an application to the client, the application configured to operate on computer systems of the client user and to periodically transmit the cached copy of a web page to a server.

18. The system of claim 11, wherein the second index further comprises summarizing the cached copy of the web page, the cached copy includes a uniform resource locator, network address and one or more key terms.

19. The system of claim 11, wherein the second index information further comprises an aggregate of a plurality of cached copies of the web page from one or more remote users.

17. A system for indexing web pages comprising:

a crawler to access a web page to create a first index information of the web page;

a server to receive second index information generated from a cached copy of the web page obtained by a browser application operating on a remote client;

an index generator to generate a second index information of the web page using the cached copy; and

an analyzer to rank the web page based on a comparison between the first index information and the second index information of the web page.

20. A method for assisting a server to index web pages, the method comprising:

distributing an application to a client, the application configured to operate on the client and to cause the client to periodically transmit the cached copy of a web page to a server;

generating a first index information based on a cached copy of the web page received from the client;

updating a search index based on a comparison between the first index and a second index generated from a sample of the web page by a web crawling application.

21. The method of claim 20, further comprising sending an instruction to the application to cause the application to access the web page at a specified web address and to send the cached version of the web page from the client to a server.

22. A method for avoiding masked web page content indexing errors, the method comprising:

receiving cached user copies of web pages from client sources at a server, wherein the cached user copies are identified by respective URLs that designate network addresses other than the client sources;

updating a search index with the cached user copies; and

storing the updated search index, wherein the updated search index is used to generate search results for at least one client.

23. The method of claim 22, wherein the respective URLs designate corresponding hosts providing masked pages different from corresponding cached user copies.

24. The method of claim 22, wherein updating the search index further comprises summarizing the cached user copies, wherein the summarized cached user copies include uniform resource locators, network addresses and key terms.

25. The method of claim 22, wherein updating the search index further comprises aggregating the cached user copies, wherein the aggregated cached user copies are cached user copies from multiple client sources.