Avoiding masked web page content indexing errors for search engines

• US 9,405,831 B2
• Filed: 04/16/2009
• Issued: 08/02/2016
• Est. Priority Date: 04/16/2008
• Status: Active Grant

• Current Assignee: Gary S. Shuster
• Original Assignee: Gary S. Shuster
• Inventors: Shuster, Gary Stephen
• Primary Examiner(s): Gofman, Alex
• Assistant Examiner(s): Gurmu, Muluemebet
• Application Number:
  US12/425,269
• Publication Number:
  US 20090265342A1
• Time in Patent Office: 2,665 Days
• Field of Search: 707/696
• US Class Current: 1/1
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching

Abstract

Multiple non-host client sites provide cached user copies of web pages and/or web content, or summaries thereof, to a server. Obtaining data from non-host sources for indexing purposes avoids masked web page content indexing errors for search engines. The server aggregates, summarizes and indexes the web pages and/or web content in an index of cached content, in conjunction with updating, generating and storing a search index using an indexing agent such as a web crawler or spider. In response to receiving search requests from end users, the search engine uses comparisons between the index of cached content and the index of crawled content to identify potential page masking errors for specific search results and appropriately rank or omit results with a high risk of masking errors in a search result list.

Claims

1. A method comprising:

accessing a first web page hosted at a network address using a server operating a web crawling application to create first index information for the first web page;

receiving, at the server, second index information generated from a cached copy of a second web page, wherein the cached copy is not hosted at the network address and the second web page is previously received by a client via a browser application operated at the client in response to a request directed to the network address;

comparing the first index information and the second index information to identify masked web page content; and

ranking the first web page in a search results list generated at the server, based on the comparison between the first index information and the second index information, to reduce errors caused by the masked web page content.

2. The method of claim 1, further comprising:

generating an updated index based on the comparison between the first index information and the second index information; and

generating a search result based on the updated index for at least one client.

3. The method of claim 1, wherein the network address comprises a Uniform Resource Locator (URL).

4. The method of claim 1, wherein generating the second index information further comprises summarizing the cached copy of the second web page, wherein the cached copy includes a uniform resource locator and one or more key terms.

5. The method of claim 1, wherein generating the second index information further comprises aggregating a plurality of cached copies of the second web page from one or more remote clients.

6. The method of claim 5, further comprising measuring popularity for the first web page by counting a number of the cached client copies received.

7. The method of claim 5, further comprising distributing an application to the client, the application configured to operate on computer systems of the client and to periodically transmit the cached copies to a server.

8. The method of claim 7, wherein the application is further configured to summarize the cached copies prior to transmission.

9. The method of claim 7, wherein the application is further configured to aggregate the cached copies prior to transmission.

10. The method of claim 7, wherein the application is further configured to generate the second index information from the cached copies.

11. A system for indexing web pages comprising:

means for accessing a first web page hosted at a first network address to create a first index information of the first web page;

means for receiving a cached copy of a second web page at a client, wherein the cached copy is not hosted at the network address, and the client receives the second web page in response to a browser request directed to the network address;

means for generating a second index information of the second web page using the cached copy;

means for comparing the first index information and the second index information to identify masked web page content; and

means for ranking the first web page based on the comparison between the first index information and the second index information, to reduce errors caused by the masked web page content.

12. The system of claim 11, further comprising:

means for generating an updated index based on the comparison between the first index information and the second index information; and

means for generating a search result based on the updated index for at least one client.

13. The system of claim 11, wherein the network address comprises a Uniform Resource Locator (URL).

14. The system of claim 11, wherein means for generating the second index further comprises summarizing the cached copy of the second web page, wherein the cached copy includes a uniform resource locator and one or more key terms.

15. The system of claim 11, wherein means for generating the second index information further comprises aggregating a plurality of cached copies of the second web page from one or more remote clients.

16. The system of claim 11, further comprising means for distributing an application to the client, the application configured to operate on computer systems of the client and to periodically transmit the cached copy of the second web page to a server.

18. The system of claim 11, wherein the index generator generates the second index information by summarizing the cached copy of the web page, and the cached copy includes a uniform resource locator and one or more key terms.

19. The system of claim 11, wherein the second index information further comprises an aggregate of a plurality of cached copies of the second web page from one or more remote clients.

17. A system for indexing web pages comprising:

a crawler to access a first web page hosted at a network address to create a first index information of the first web page;

a server to receive second index information generated from a cached copy of a second web page, wherein the cached copy is not hosted at the network address, and the second web page is obtained by a browser application operating on a remote client using the network address;

an index generator to generate a second index information of the second web page using the cached copy;

an application to compare the first index information and the second index information to identify masked web page content; and

an analyzer to rank the first web page based on the comparison between the first index information and the second index information, to reduce errors caused by the masked web page content.

20. A method for assisting a server to index web pages, the method comprising:

distributing an application to a client, the application configured to operate on the client and to cause the client to periodically transmit a cached copy of a first web page to a server, wherein the client obtains the first web page using a browser application accessing a specified web address and does not host the first web page at the web address;

generating a first index information based on the cached copy of the first web page received from the client;

comparing the first index and a second index, generated from a sample of a second web page hosted at the specified web address and obtained by a web crawling application, to identify masked web page content; and

updating a search index based on the comparison between the first index and the second index, to reduce errors caused by the masked web page content.

21. The method of claim 20, further comprising sending an instruction to the application to cause the application to access the first web page at the specified web address and to send the cached copy from the client to a server.

22. A method for avoiding masked web page content indexing errors, the method comprising:

receiving cached user copies of web pages from client sources at a server, wherein the cached user copies are identified by respective URLs that designate network addresses other than the client sources and are obtained by browser-initiated requests;

comparing the cached user copies to corresponding crawler copies obtained by a web crawler application and also identified by the respective URLs to identify masked web page content;

updating a search index with the cached user copies, to reduce errors caused by the masked web page content; and

storing the updated search index, wherein the updated search index is used to generate search results for at least one client.

23. The method of claim 22, wherein the respective URLs designate corresponding hosts providing the crawler pages as masked pages different from corresponding cached user copies.

24. The method of claim 22, wherein updating the search index further comprises summarizing the cached user copies, wherein the summarized cached user copies include uniform resource locators and key terms.

25. The method of claim 22, wherein updating the search index further comprises aggregating the cached user copies from multiple client sources.