Agent for identifying domains with content arranged for display by a mobile device

• US 8,041,703 B2
• Filed: 11/29/2006
• Issued: 10/18/2011
• Est. Priority Date: 08/03/2006
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Murugesan, Sundaramoorthy, Seth, Siddharth, Rao, Sudhir Kumar Rama, Gupta, Ashima, Prabhakar, Bangalore Subbaramaiah
• Primary Examiner(s): Lovel; Kimberly
• Assistant Examiner(s): Uddin; Mohammed R
• Application Number:
  US11/564,756
• Publication Number:
  US 20080033998A1
• Time in Patent Office: 1,784 Days
• Field of Search: 707/3, 707/709, 707/999.003
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A system, apparatus, and method are disclosed for providing a single source for searchable content by mobile devices. Mobile user agents crawl host sites to determine whether there is content that may be viewed by any of the mobile user agents. If files are identified as viewable for a given mobile user agent, the site is tagged as crawlable. A best mobile user agent may also be identified for that site, based on number of files viewable, response time, file mobileness, or the like. The sites may be discovered by instrumentation of URLs with mobile device keywords, performing a backlink analysis to discover interlinking of mobile device related sites, exploring URLs of a known host name for potential starting points for mobile device sites, or employing a list of known popular conventional sites with a mobile user agent for possible redirections to a potential mobile device web site.

Claims

1. A method of discovering mobile web sites, comprising:

determining a list of host sites for discovery;

determining a plurality of mobile user agents;

crawling the list of host sites using each of the plurality of mobile user agents;

automatically identifying each host site in the list of host sites that is mobile viewable by at least one of the plurality of mobile user agents, wherein if a threshold of documents are retrieved for a host site within the list of host sites and the documents are determined to be mobile viewable by at least one mobile user agent, then identifying the host site as mobile viewable for the at least one mobile user agent;

for each host site that is identified as being mobile viewable by more than one mobile user agent, selecting one of the mobile user agents for the mobile viewable host site as a best mobile user agent based, in part, on a number of documents viewable, a response time to view the documents with a mobile user agent, and a mobileness of the documents; and

providing a list of each mobile viewable host site along with identification of which mobile user agent is useable for each mobile viewable host site in the list, and for each mobile viewable host site viewable by more than one mobile user agent identifying the selected best mobile user agent.

2. The method of claim 1, wherein mobileness of a document is determined based on at least one of a content type of the document, a DTD the document is pointing to, a number of valid tags within the document, a number of negative tags within the document, or a document size.

3. The method of claim 1, wherein crawling the list of host sites further comprises:

if a link within a crawled host site is to another host site not in the list of host sites, adding the host site to the list of host sites.

4. The method of claim 1, wherein the host site is determined to be a mobile viewable site is based on successfully retrieving content using at least one mobile user agent.

5. The method of claim 1, wherein a computer readable storage medium is configured to store program instructions that when installed on a computer enables the computer to perform the method of claim 1.

6. A server that is configured for use in discovering mobile web sites, comprising:

a transceiver for receiving and for sending information over the network; and

program code that is operative to automatically perform actions comprising;

determining a candidate mobile web site for discovery;

crawling the candidate mobile web site;

if a threshold number of mobile viewable content is retrieved from the candidate mobile web site using at least one mobile user agent, designating the candidate mobile web site as mobile viewable for the at least one mobile user agent; and

if the at least one mobile user agent includes more than one mobile user agent for the mobile viewable web site, then selecting one of the mobile user agents as a best mobile user agent for the mobile viewable web site based in part on a number of documents viewable and a response time to view the documents with a mobile user agent; and

providing a list of each mobile viewable host site along with identification of which mobile user agent is useable for each mobile viewable host site in the list, and for each mobile viewable host site viewable by more than one mobile user agent identifying the selected best mobile user agent.

7. The server of claim 6, wherein the automatic actions further comprise:

providing the mobile viewable web site, and identification of the at least one mobile user agent useable for the mobile viewable web site a search crawler for use in indexing mobile viewable web sites.

8. The server of claim 6, wherein the automatic actions further comprise:

if a determined number of attempts to retrieve mobile viewable content is exceeded for a mobile user agent from the-candidate mobile web site before the threshold number of mobile viewable content is retrieved, identifying the candidate mobile web site as uncrawlable by the mobile user agent.

9. The server of claim 6, wherein crawling the candidate mobile web site further comprises:

if a link to another candidate mobile web site is discovered based, in part, on crawling the candidate mobile web site, further determining whether the discovered other candidate mobile web site is to a mobile viewable web site.

10. The server of claim 6, wherein content is determined to be mobile viewable content based, in part, on at least one of the following:

a content type of the content, a DTD for which the content is pointing to, a number of valid tags in the content, a number of negative tags in the content, or a content size.

11. The server of claim 6, wherein determining the candidate mobile web site further comprises, determining the candidate mobile web site using at least one of the following:

performing URL instrumentation on at least one site name by modifying the site name using a mobile related keyword, and attempting to access the modified site name using the at least one mobile user agent to determine the candidate mobile web site;

performing a query using a known mobile site, to obtain at least one other site that includes a link to the known mobile site, and then searching the at least one other site for a candidate mobile web site;

selecting at least one site determined to be popular based on a network traffic determination, attempt to access the site name with mobile specific user agent and a mobile header information, and if redirected to a mobile site, employing the mobile site as the candidate mobile web site;

or

providing candidate mobile sites based on an analysis of keywords for mobile content and URLs of sites crawled by another web crawler.

12. A computer-readable storage medium having computer-executable instructions, the computer-executable instructions when installed onto a computer processor enables the computer processor to automatically perform actions comprising:

determining a plurality of candidate mobile sites for discovery;

crawling each of the plurality of candidate mobile sites;

if a threshold number of mobile viewable content is successfully retrieved from a candidate mobile site in the plurality of candidate mobile sites, using at least one mobile user agent, designating the candidate mobile site as mobile viewable for the at least one mobile user agent; and

if the at least one mobile user agent includes more than one mobile user agent for the mobile viewable site, then selecting one of the mobile user agents as a best mobile user agent for the mobile viewable site based in part on a number of documents viewable and a response time to view the documents with a mobile user agent; and

providing a list of each mobile viewable host site along with identification of which mobile user agent is useable for each mobile viewable host site in the list, and for each mobile viewable host site viewable by more than one mobile user agent identifying the selected best mobile user agent.

13. The computer-readable storage medium of claim 12, wherein content is determined to be mobile viewable for the at least one mobile user agent, based, in part, on a content type, a DTD the content is pointing to, a number of valid tags within the content, a number of negative tags within the content, or a content size.

14. The computer-readable storage medium of claim 12, wherein selecting one of the mobile user agents as a best mobile user agent is further based, in part, on a mobileness of the documents, wherein mobileness is based in part on at least one of a content type, a DTD the content is pointing to, a number of tags determined as valid for the mobile user agent, a number of tags determined as negative for the mobile user agent, or a content size.

15. The computer-readable storage medium of claim 12, further comprising:

if a determined number of attempts to retrieve mobile viewable content is exceeded for a mobile user agent from a candidate mobile site before the threshold number of mobile viewable content is successfully retrieved, identifying the candidate mobile site as uncrawlable by the mobile user agent.

16. A system for use in discovering mobile web sites, comprising:

a candidate site generator executing on at least one processor that is configured to perform actions to identify a plurality of candidate sites;

a mobile device repository residing within a computer readable storage medium and configured to store a plurality of mobile user agents; and

a domain discoverer component executing on at least one processor that is configured to perform actions, comprising;

crawling each of the plurality of candidate sites and attempting to successfully retrieve content using each of the plurality of mobile user agents;

if a threshold number of valid mobile viewable content is successfully retrieved from a candidate site in the plurality of candidate sites, by at least one mobile user agent, designating the candidate site as mobile viewable for the at least one mobile user agent; and

if the at least one mobile user agent includes more than one mobile user agent for the mobile viewable site, then selecting one of the mobile user agents as a best mobile user agent for the mobile viewable site based in part on a number of documents viewable and a response time to view the documents; and

providing a list of each mobile viewable host site along with identification of which mobile user agent is useable for each mobile viewable host site in the list, and for each mobile viewable host site viewable by more than one mobile user agent identifying the selected best mobile user agent.

17. The system of claim 16, wherein the candidate site generator further comprises at least one of the following:

a URL instrumentation generator that is configured to modify a site name using a mobile related keyword, and attempt to access the modified site name using the at least one mobile user agent to identify at least one candidate site;

a backlink generator that is configured to perform a query using a known mobile web site to obtain at least one site that includes a link to the known mobile web site, and then searching the at least one site for at least one candidate site;

a redirect analysis generator that is configured to select at least one site determined to be popular based on a network traffic determination, attempt to access the site with mobile user agent and mobile specific header information and if redirected to a mobile site, to identify the mobile site as at least one candidate site;

or

providing candidate mobile sites based on an analysis of keywords for mobile content and URLs of sites crawled by another web crawler.

18. The system of claim 16, wherein the actions of the domain discoverer component further comprise:

if a link to another candidate site is discovered based, in part, on crawling the plurality of candidate sites, adding the discovered other candidate site to the plurality of candidate sites to be crawled and content attempted to be retrieved using each of the plurality of mobile user agents.

19. An apparatus having one or more processors which discovers mobile web sites, comprising:

means for determining a list of candidate host sites for discovery;

a discoverer component useable to automatically identify each host site in the list of candidate host sites that is mobile viewable by at least one of a plurality of mobile user agents by crawling each candidate host site and, in part, attempting to access content on the candidate host site using each of the plurality of mobile user agents, wherein if a threshold of documents are retrieved for a host site within the list of candidate host sites, and the documents are determined to be mobile viewable by at least one mobile user agent, then identifying the host site as mobile viewable for the at least one mobile user agent; and

means for providing a list of each mobile viewable host site along with identification of which mobile user agent is useable for each mobile viewable host site, wherein for each mobile viewable host site having more than one mobile user agent identified, means for identifying within the list a best mobile user agent for each mobile viewable host site based in part on a number of documents viewable and a response time to view the documents with a mobile user agent.