AGENT FOR IDENTIFYING DOMAINS WITH CONTENT ARRANGED FOR DISPLAY BY A MOBILE DEVICE

• US 20080033998A1
• Filed: 11/29/2006
• Published: 02/07/2008
• Est. Priority Date: 08/03/2006
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Murugesan, Sundaramoorthy, Seth, Siddharth, Rao, Sudhir Kumar Rama, Prabhakar, Bangalore Subbaramaiah, Gupta, Ashima
• Granted Patent: US 8,041,703 B2
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A system, apparatus, and method are disclosed for providing a single source for searchable content by mobile devices. Mobile user agents crawl host sites to determine whether there is content that may be viewed by any of the mobile user agents. If files are identified as viewable for a given mobile user agent, the site is tagged as crawlable. A best mobile user agent may also be identified for that site, based on number of files viewable, response time, file mobileness, or the like. The sites may be discovered by instrumentation of URLs with mobile device keywords, performing a backlink analysis to discover interlinking of mobile device related sites, exploring URLs of a known host name for potential starting points for mobile device sites, or employing a list of known popular conventional sites with a mobile user agent for possible redirections to a potential mobile device web site.

Claims

1. A method of discovering mobile web sites, comprising:

determining a list of host sites for discovery;

determining a plurality of mobile user agents;

crawling the list of host sites using each of the plurality of mobile user agents;

automatically identifying each host site in the list of host sites that is mobile viewable by at least one of the plurality of mobile user agents; and

providing a list of each mobile viewable host site along with identification of which mobile user agent is useable for each mobile viewable host site in the list.

2. The method of claim 1, wherein automatically identifying each mobile viewable host site further comprises:

if a threshold of documents are retrieved for a host site within the list of host sites, and the documents are determined to be mobile viewable by at least one mobile user agent, then identifying the host site as mobile viewable for the at least one mobile user agent; and

determining at least one of the mobile user agents for the mobile viewable host site as a best mobile user agent for the host site based, in part, on at least one of a number of documents viewable, a response time, or a mobileness of the documents.

3. The method of claim 2, wherein mobileness of a document is determined based on at least one of a content type of the document, a DTD the document is pointing to, a number of valid tags within the document, a number of negative tags within the document, or a document size.

4. The method of claim 1, wherein crawling the list of host sites further comprises:

if a link within a crawled host site is to another host site not in the list of host sites, adding the host site to the list of host sites.

5. The method of claim 1, wherein the host site is determined to be a mobile viewable site is based on successfully retrieving content using at least one mobile user agent.

6. A modulated data signal configured to include program instructions for performing the method of claim 1.

7. A server that is configured for use in discovering mobile web sites, comprising:

a transceiver for receiving and for sending information over the network; and

program code that is operative to automatically perform actions comprising;

determining a candidate mobile web site for discovery;

crawling the candidate mobile web site;

if a threshold number of mobile viewable content is retrieved from the candidate mobile web site using at least one mobile user agent, designating the candidate mobile web site as mobile viewable for the at least one mobile user agent; and

if the at least one mobile user agent includes more than one mobile user agent for the mobile viewable web site, then selecting one of the mobile user agents as a best mobile user agent for the mobile viewable web site.

8. The server of claim 7, wherein the automatic actions further comprise:

providing the mobile viewable web site, and identification of the at least one mobile user agent useable for the mobile viewable web site a search crawler for use in indexing mobile viewable web sites.

9. The server of claim 7, wherein the automatic actions further comprise:

if a determined number of attempts to retrieve mobile viewable content is exceeded for a mobile user agent from the candidate mobile web site before the threshold number of mobile viewable content is retrieved, identifying the candidate mobile web site as uncrawlable by the mobile user agent.

10. The server of claim 7, wherein crawling the candidate mobile web site further comprises:

if a link to another candidate mobile web site is discovered based, in part, on crawling the candidate mobile web site, further determining whether the discovered other candidate mobile web site is to a mobile viewable web site.

11. The server of claim 7, wherein content is determined to be mobile viewable content based, in part, on at least one of the following:

a content type of the content, a DTD for which the content is pointing to, a number of valid tags in the content, a number of negative tags in the content, or a content size.

12. The server of claim 7, wherein determining the candidate mobile web site further comprises, determining the candidate mobile web site using at least one of the following:

performing URL instrumentation on at least one site name by modifying the site name using a mobile related keyword, and attempting to access the modified site name using the at least one mobile user agent to determine the candidate mobile web site;

performing a query using a known mobile site, to obtain at least one other site that includes a link to the known mobile site, and then searching the at least one other site for a candidate mobile web site;

selecting at least one site determined to be popular based on a network traffic determination, attempt to access the site name with mobile specific user agent and a mobile header information, and if redirected to a mobile site, employing the mobile site as the candidate mobile web site;

or

providing candidate mobile sites based on an analysis of keywords for mobile content and URLs of sites crawled by another web crawler.

13. A computer-readable medium having computer-executable instructions, the computer-executable instructions when installed onto a computer processor enables the computer processor to automatically perform actions comprising:

determining a plurality of candidate mobile sites for discovery;

crawling each of the plurality of candidate mobile sites;

if a threshold number of mobile viewable content is successfully retrieved from a candidate mobile site in the plurality of candidate mobile sites, using at least one mobile user agent, designating the candidate mobile site as mobile viewable for the at least one mobile user agent; and

if the at least one mobile user agent includes more than one mobile user agent for the mobile viewable site, then selecting one of the mobile user agents as a best mobile user agent for the mobile viewable site.

14. The computer-readable medium of claim 13, wherein content is determined to be mobile viewable for the at least one mobile user agent, based, in part, on a content type, a DTD the content is pointing to, a number of valid tags within the content, a number of negative tags within the content, or a content size.

15. The computer-readable medium of claim 13, wherein selecting one of the mobile user agents as a best mobile user agent is based, in part, on at least one of a number of documents viewable, a response time, or a mobileness of the documents, wherein mobileness is based in part on at least one of a content type, a DTD the content is pointing to, a number of tags determined as valid for the mobile user agent, a number of tags determined as negative for the mobile user agent, or a content size.

16. The computer-readable medium of claim 13, further comprising:

if a determined number of attempts to retrieve mobile viewable content is exceeded for a mobile user agent from a candidate mobile site before the threshold number of mobile viewable content is successfully retrieved, identifying the candidate mobile site as uncrawlable by the mobile user agent.

17. A system for use in discovering mobile web sites, comprising:

a candidate site generator that is configured to perform actions to identify a plurality of candidate sites;

a mobile device repository useable to store a plurality of mobile user agents; and

a domain discoverer component that is configured to perform actions, comprising;

crawling each of the plurality of candidate sites and attempting to successfully retrieve content using each of the plurality of mobile user agents;

if a threshold number of valid mobile viewable content is successfully retrieved from a candidate site in the plurality of candidate sites, by at least one mobile user agent, designating the candidate site as mobile viewable for the at least one mobile user agent; and

if the at least one mobile user agent includes more than one mobile user agent for the mobile viewable site, then selecting one of the mobile user agents as a best mobile user agent for the mobile viewable site.

18. The system of claim 17, wherein the candidate site generator further comprises at least one of the following:

a URL instrumentation generator that is configured to modify a site name using a mobile related keyword, and attempt to access the modified site name using the at least one mobile user agent to identify at least one candidate site;

a backlink generator that is configured to perform a query using a known mobile web site to obtain at least one site that includes a link to the known mobile web site, and then searching the at least one site for at least one candidate site;

a redirect analysis generator that is configured to select at least one site determined to be popular based on a network traffic determination, attempt to access the site with mobile user agent and mobile specific header information and if redirected to a mobile site, to identify the mobile site as at least one candidate site;

or

providing candidate mobile sites based on an analysis of keywords for mobile content and URLs of sites crawled by another web crawler.

19. The system of claim 17, wherein the actions of the domain discoverer component further comprise:

if a link to another candidate site is discovered based, in part, on crawling the plurality of candidate sites, adding the discovered other candidate site to the plurality of candidate sites to be crawled and content attempted to be retrieved using each of the plurality of mobile user agents.

20. An apparatus of discovering mobile web sites, comprising:

means for determining a list of candidate host sites for discovery;

a discoverer component useable to automatically identify each host site in the list of candidate host sites that is mobile viewable by at least one of a plurality of mobile user agents by crawling each candidate host site and, in part, attempting to access content on the candidate host site using each of the plurality of mobile user agents; and

means for providing a list of each mobile viewable host site along with identification of which mobile user agent is useable for each mobile viewable host site.