Web forum crawler

• US 20070208703A1
• Filed: 03/03/2006
• Published: 09/06/2007
• Est. Priority Date: 03/03/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Xu, Gu, Shi, Bin
• Granted Patent: US 7,599,931 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A crawling system crawls a web site initially in a pattern detection phase and subsequently in a pattern usage phase. The pattern detection phase attempts to identify patterns of references to pages that contain informational content of interest and patterns of references to pages that contain little informational content of interest. During the pattern usage phase, the crawling system crawls the web site. When the crawling system encounters a reference contained on an accessed page, the crawling system determines whether the reference matches a reference pattern. If the reference matches a reference pattern associated with pages that contain informational content of interest, the crawling system accesses the referenced page. If, however, the reference matches a reference pattern of pages with little informational content, then the crawling system discards that reference without accessing the referenced page.

Claims

1. A system for crawling a site having pages, each page having a reference that identifies the page, comprising:

a grouping component that identifies groups of pages with similar content;

a pattern component that identifies a reference pattern of a group based on the references of the pages of the group; and

a decision component that, after encountering a reference that matches a reference pattern, decides whether to access the page of the encountered reference based on characteristics of the pages of the group of the matching reference pattern.

2. The system of claim 1 wherein the characteristics include number of pages within the group.

3. The system of claim 2 wherein when a group has less than a certain number of pages, the decision component decides to access pages whose references match a reference pattern of the group.

4. The system of claim 2 wherein when a group has more than a certain number of pages, the decision component decides not to access pages whose references match a reference pattern of the group.

5. The system of claim 1 wherein the grouping component and the pattern component are used to perform an initial crawling of the site and the decision component is used to perform subsequent crawling of the site using the reference patterns.

6. The system of claim 1 wherein the grouping component determines that pages are similar based on the number of shingles that are common to the pages and the total number of unique shingles of the pages.

7. The system of claim 1 wherein the site is a web forum.

8. The system of claim 1 wherein the grouping component detects multiple reference pages.

9. A system for detecting web pages of a web site having multiple references, comprising:

a grouping component that identifies groups of web pages with similar content;

a pattern component that identifies a reference pattern of a group based on the references of the web pages of the group; and

a duplicate detection component that, when reference patterns identify a similar set of web pages, indicates that the reference patterns reference multiple reference web pages.

10. The system of claim 9 wherein one reference of a multiple reference web page is to be followed when crawling the web site and any other reference of the multiple reference web page is not to be followed when crawling the web site.

11. The system of claim 9 wherein the web site is a web forum and wherein a multiple reference web page is referenced by previous and next references.

12. The system of claim 9 wherein the duplicate detection component does not detect duplicates for reference patterns that identify operational web pages of the web site.

13. A system for determining web page type for web pages of a web forum, comprising:

a grouping component that identifies groups of web pages with similar content; and

a typing component that indicates that web pages within a group having more than an operational threshold number of web pages are operational web pages and that web pages within a group having not more than an informational threshold number of web pages are informational web pages.

14. The system of claim 13 wherein the informational web pages contain content provided by users of the web forum.

15. The system of claim 13 wherein the operational threshold number and the informational threshold number are the same number.

16. The system of claim 13 including:

a pattern component that identifies a reference pattern of a group based on the references of the web pages of the group.

17. The system of claim 16 including a duplicate detection component that, when reference patterns identify a similar set of informational web pages, indicates that the reference patterns reference multiple reference web pages.

18. The system of claim 17 including a crawling component that, when an encountered reference matches one reference pattern of a multiple reference web page, follows the reference and, when an encountered reference matches any other reference pattern of the multiple reference web page, does not follow the reference.

19. The system of claim 13 wherein the grouping component determines that web pages are similar based on the number of shingles that are common to the web pages and the total number of unique shingles of the web pages.

20. The system of claim 13 wherein the grouping component uses a dynamic resemblance technique for determining whether web pages are similar.