System and method for semantic analysis of intelligent device discovery

• US 7,809,711 B2
• Filed: 06/02/2006
• Issued: 10/05/2010
• Est. Priority Date: 06/02/2006
• Status: Expired due to Fees

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Gruhl, Daniel Frederick, Bhagwan, Varun, Alba, Alfredo, Srinivasan, Savitha
• Primary Examiner(s): Morrison; Jay A
• Application Number:
  US11/422,055
• Publication Number:
  US 20070282799A1
• Time in Patent Office: 1,586 Days
• Field of Search: None
• US Class Current: 707/709
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

The present invention provides a method, system, and service of analyzing electronic documents in an intranet, where the intranet includes a plurality of web sites. In an exemplary embodiment, the method, system, and service include (1) crawling HTML content and text content in a set of the sites, (2) deep-scanning non-HTML content and non-text content in the set of sites, (3) reverse-scanning the set of sites, (4) performing a semantic analysis of the crawled content and the deep-scanned content, (5) correlating the results of the semantic analysis with the results of the reverse-scanning, and (6) comparing user navigation patterns and content from the members of the set of sites. In a further embodiment, the method, system, and service further include combining the results of the performing, the results of the correlating, and the results of the comparing.

Claims

1. A method of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the method comprising:

crawling HTML content and text content in a set of the sites, wherein the crawling comprises

writing each crawled content to a file-based storage system,

annotating each crawled content, and

generating a first 4-tuple entry for each annotated, crawled content;

deep-scanning non-HTML content and non-text content in the set of sites, wherein the deep-scanning comprises fetching all static non-HTML content and non-text content in the set of sites and generating a second 4-tuple entry;

reverse-scanning the set of sites;

performing a semantic analysis of the crawled content and the deep-scanned content by using the first 4-tuple entry and the second 4-tuple entry;

correlating the results of the semantic analysis with the results of the reverse-scanning; and

comparing user navigation patterns and content from members of the set of sites.

2. The method of claim 1 further comprising combining the results of the performing, the results of the correlating, and the results of the comparing.

3. The method of claim 1 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning comprises:

crawling the first megabyte of the text content of the member; and

using the first megabyte of the text content of the member for hash generation.

4. The method of claim 1 wherein the performing comprises performing exact duplicate detection on the text content of each member of the set of sites.

5. The method of claim 1 wherein the performing comprises performing near-duplicate detection on the text content of each member of the set of sites.

6. The method of claim 1 wherein the performing comprises performing main topics detection on the text content of each member of the set of sites.

7. The method of claim 1 wherein the performing comprises aggregating topics in the text content of each member of the set of sites by organizational structure.

8. The method of claim 2 wherein the combining comprises listing the combined results in descending order of duplicate content.

9. A method of providing a service to analyze electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the method comprising:

crawling HTML content and text content in a set of the sites, wherein the crawling comprises

writing each crawled content to a file-based storage system,

annotating each crawled content, and

generating a first 4-tuple entry for each annotated, crawled content;

deep-scanning non-HTML content and non-text content in the set of sites, wherein the deep-scanning comprises fetching all static non-HTML content and non-text content in the set of sites and generating a second 4-tuple entry;

reverse-scanning the set of sites;

performing a semantic analysis of the crawled content and the deep-scanned content by using the first 4-tuple entry and the second 4-tuple entry;

correlating the results of the semantic analysis with the results of the reverse-scanning; and

comparing user navigation patterns and content from members of the set of sites.

10. The method of claim 9 further comprising combining the results of the performing, the results of the correlating, and the results of the comparing.

11. The method of claim 9 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning comprises:

crawling the first megabyte of the text content of the member; and

using the first megabyte of the text content of the member for hash generation.

12. The method of claim 9 wherein the performing comprises performing exact duplicate detection on the text content of each member of the set of sites.

13. The method of claim 9 wherein the performing comprises performing near-duplicate detection on the text content of each member of the set of sites.

14. The method of claim 9 wherein the performing comprises performing main topics detection on the text content of each member of the set of sites.

15. The method of claim 9 wherein the performing comprises aggregating topics in the text content of each member of the set of sites by organizational structure.

16. The method of claim 10 wherein the combining comprises listing the combined results in descending order of duplicate content.

17. A method of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the method comprising:

crawling HTML content and text content in a set of the sites, wherein the crawling comprises

writing each crawled content to a file-based storage system,

annotating each crawled content, and

generating a first 4-tuple entry for each annotated, crawled content;

deep-scanning non-HTML content and non-text content in the set of sites, wherein the deep-scanning comprises fetching all static non-HTML content and non-text content in the set of sites and generating a second 4-tuple entry;

reverse-scanning the set of sites;

performing a semantic analysis of the crawled content and the deep-scanned content by using the first 4-tuple entry and the second 4-tuple entry;

correlating the results of the semantic analysis with the results of the reverse-scanning;

comparing user navigation patterns and content from members of the set of sites; and

combining the results of the performing, the results of the correlating, and the results of the comparing.

18. The method of claim 17 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning comprises:

crawling the first megabyte of the text content of the member; and

using the first megabyte of the text content of the member for hash generation.

19. The method of claim 17 wherein the performing comprises performing exact duplicate detection on the text content of each member of the set of sites.

20. The method of claim 17 wherein the performing comprises performing near-duplicate detection on the text content of each member of the set of sites.

21. The method of claim 17 wherein the performing comprises performing main topics detection on the text content of each member of the set of sites.

22. The method of claim 17 wherein the performing comprises aggregating topics in the text content of each member of the set of sites by organizational structure.