Automated client sitemap generation

• US 8,126,869 B2
• Filed: 02/08/2008
• Issued: 02/28/2012
• Est. Priority Date: 02/08/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hiemstra, Martina, Hollier, Ian V.
• Primary Examiner(s): Yen, Syling
• Application Number:
  US12/028,502
• Publication Number:
  US 20090204638A1
• Time in Patent Office: 1,481 Days
• Field of Search: 707/707, 707/709, 707/711, 707/712, 707/713, 707/722, 707/736, 707/758, 707/781, 707/999.003, 707/999.004, 707/999.005, 709/218, 709/223, 709/224
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Methods and computer-storage media for automated generation of domain sitemap files are provided. A universal resource locator (URL) for a web site having a plurality of web pages associated therewith is received. Log files and permission controls are analyzed to ascertain whether each web page has been previously crawled and which web pages may be crawled and/or indexed. The permitted, not-previously-crawled web pages are subsequently crawled and the relational structure of the web site is ascertained. Other items of metadata, such as web page modification frequency or priority values, also are determined. Once the structure and metadata are available, a current sitemap is generated that provides the hierarchy and related details in the form of metadata. The sitemap file is then written to a disk and may then be sent to search engines as generated or in a compressed format.

Claims

1. One or more computer storage media having computer-executable instructions embodied thereon for performing a method for automated sitemap generation, the method comprising:

receiving a universal resource locator (URL) of a web site for which a sitemap is to be generated;

analyzing one or more files that log user visits to web pages based on respective URLs to determine if the URL of the web site has been previously crawled;

upon determining the URL of the web site has not been previously crawled, crawling at least one web page having a same domain as the web site URL in accordance with control permissions associated with the at least one web page to determine one or more data items relevant to generating the sitemap for the web site;

determining a relational structure of a plurality of web pages having the same domain as the web site URL, including the at least one web page, wherein the relational structure comprises interconnections between the web pages of the plurality of web pages;

for each respective web page of the plurality of web pages, determining a priority value that is incorporated as an item of metadata, the priority value calculated according to the formula;

2. The media of claim 1, wherein analyzing one or more files includes one or more of determining a number of times each logged web page URL has been visited, and a total number of visits to any URL logged in the one or more files.

3. The media of claim 2, wherein the one or more data items relevant to generating the sitemap for the web site include one or more of a number of links each crawled web page having the same domain as the URL has from other web pages having the same domain as the URL, and a number of web pages having the same domain as the URL.

4. The media of claim 1, further comprising normalizing the respective calculated priority values for each web page having the same domain as the web site URL.

5. The media of claim 1, wherein the priority value for each respective web page is capable of being manually modified.

6. The media of claim 1, wherein analyzing the one or more additional values comprises determining a frequency at which each web page having the same domain as the URL has been modified.

7. The media of claim 6, wherein the frequency is determined by determining a last-modified time, comparing a current time to the last modified time to calculate an elapsed time, and comparing the elapsed time to a set of threshold values.

8. The media of claim 7, wherein the method further comprises analyzing one or more previous sitemap files associated with the web site URL to determine a history of modified dates for each web page having the same domain as the URL and determining if the frequency should be updated in accordance with the history of modified dates.

9. The media of claim 1, wherein the current sitemap file is an extensible markup language (XML) file.

10. A method for automatically generating a sitemap file at a client device, the method comprising:

receiving one or more web server log files associated with a universal resource locator (URL);

receiving one or more files controlling permission for programmed crawling of a plurality of web pages having a same domain as the URL;

analyzing data in the one or more web server log files and one or more permission-control files;

crawling one or more permitted web pages of the plurality of web pages;

determining a sitemap file structure for the URL based upon a relational structure of the plurality of web pages, wherein the relational structure comprises interconnections between the web pages of the plurality of web pages;

for each respective web page of the one or more permitted web pages, determining a priority value that is incorporated as an element of metadata, the priority value calculated according to the formula;

11. The method of claim 10, wherein the sitemap file is an extensible markup language (XML) file.

12. The method of claim 10, wherein the method is performed without user intervention incident to any structural modification to the web site.

13. The method of claim 10, further comprising identifying one or more opportunities for search engine optimization from one or more of analyzing data in the one or more log files and one or more permission-control files, determining the sitemap file structure for the URL, and calculating one or more values that are incorporated as elements of metadata.

14. A computer system embodied on one or more computer-storage media having computer-executable instructions embodied thereon for performing a method for automatically generating sitemap metadata, the system comprising:

receiving a universal resource indicator (URL) for a web site comprising a plurality of web pages having a same domain as the URL for the website;

based on the URL received, crawling each web page of the plurality of web pages that does not restrict programmed crawling;

determining a relational structure of the plurality of web pages, wherein the relational structure comprises interconnections between the web pages of the plurality of web pages;

without intervention from a user, calculating a priority value for each web page having the same domain as the web site URL, the priority value calculated according to the formula;