System and method for semantic analysis of intelligent device discovery

• US 20070282799A1
• Filed: 06/02/2006
• Published: 12/06/2007
• Est. Priority Date: 06/02/2006
• Status: Active Grant

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: SRINIVASAN, SAVITHA, GRUHL, DANIEL FREDERICK, BHAGWAN, VARUN, ALBA, ALFREDO
• Granted Patent: US 7,809,711 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

The present invention provides a method, system, and service of analyzing electronic documents in an intranet, where the intranet includes a plurality of web sites. In an exemplary embodiment, the method, system, and service include (1) crawling HTML content and text content in a set of the sites, (2) deep-scanning non-HTML content and non-text content in the set of sites, (3) reverse-scanning the set of sites, (4) performing a semantic analysis of the crawled content and the deep-scanned content, (5) correlating the results of the semantic analysis with the results of the reverse-scanning, and (6) comparing user navigation patterns and content from the members of the set of sites. In a further embodiment, the method, system, and service further include combining the results of the performing, the results of the correlating, and the results of the comparing.

Claims

1. A method of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the method comprising:

crawling HTML content and text content in a set of the sites;

deep-scanning non-HTML content and non-text content in the set of sites;

reverse-scanning the set of sites;

performing a semantic analysis of the crawled content and the deep-scanned content;

correlating the results of the semantic analysis with the results of the reverse-scanning; and

comparing user navigation patterns and content from the members of the set of sites.

2. The method of claim 1 further comprising combining the results of the performing, the results of the correlating, and the results of the comparing.

3. The method of claim 1 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning comprises:

crawling the first megabyte of the text content of the member; and

using the first megabyte of the text content of the member for hash generation.

4. The method of claim 1 wherein the reverse-scanning comprises:

clustering the members of the set of sites into sitename-based sets;

for each of the sitename-based sets, generating a list of Internet Protocol (IP) addresses with open http ports in the same subnet as the sitename-based set; and

for each of the sitename-based sets, scanning all IP addresses for the members of the sitename-based set.

5. The method of claim 1 wherein the performing comprises performing exact duplicate detection on the text content of each member of the set of sites.

6. The method of claim 1 wherein the performing comprises performing near-duplicate detection on the text content of each member of the set of sites.

7. The method of claim 1 wherein the performing comprises performing main topics detection on the text content of each member of the set of sites.

8. The method of claim 1 wherein the performing comprises aggregating topics in the text content of each member of the set of sites by organizational structure.

9. The method of claim 2 wherein the combining comprises listing the combined results in descending order of duplicate content.

10. A system of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the system comprising:

a crawling module configured to crawl HTML content and text content in a set of the sites;

a deep-scanning module configured to deep-scan non-HTML content and non-text content in the set of sites;

a reverse-scanning module configured to reverse-scan the set of sites;

a performing module configured to perform a semantic analysis of the crawled content and the deep-scanned content;

a correlating module configured to correlate the results of the semantic analysis with the results of the reverse-scanning; and

a comparing module configured to compare user navigation patterns and content from the members of the set of sites.

11. The system of claim 10 further comprising a combining module configured to combine the results of the performing module, the results of the correlating module, and the results of the comparing module.

12. The system of claim 10 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning module comprises:

a crawling module configured to crawl the first megabyte of the text content of the member; and

a using module configured to use the first megabyte of the text content of the member for hash generation.

13. The system of claim 10 wherein the reverse-scanning module comprises:

a clustering module configured to cluster the members of the set of sites into sitename-based sets;

for each of the sitename-based sets, a generating module configured to generate a list of Internet Protocol (IP) addresses with open http ports in the same subnet as the sitename-based set; and

for each of the sitename-based sets, a scanning module configured to scan all IP addresses for the members of the sitename-based set.

14. The system of claim 10 wherein the performing module comprises a performing module configured to perform exact duplicate detection on the text content of each member of the set of sites.

15. The system of claim 10 wherein the performing module comprises a performing module configured to perform near-duplicate detection on the text content of each member of the set of sites.

16. The system of claim 10 wherein the performing module comprises a performing module configured to perform main topics detection on the text content of each member of the set of sites.

17. The system of claim 10 wherein the performing module comprises an aggregating module configured to aggregate topics in the text content of each member of the set of sites by organizational structure.

18. The system of claim 11 wherein the combining module comprises a listing module configured to list the combined results in descending order of duplicate content.

19. A method of providing a service to analyze electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the method comprising:

crawling HTML content and text content in a set of the sites;

deep-scanning non-HTML content and non-text content in the set of sites;

reverse-scanning the set of sites;

performing a semantic analysis of the crawled content and the deep-scanned content,

correlating the results of the semantic analysis with the results of the reverse-scanning; and

comparing user navigation patterns and content from the members of the set of sites.

20. The method of claim 19 further comprising combining the results of the performing, the results of the correlating, and the results of the comparing.

21. The method of claim 19 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning comprises:

crawling the first megabyte of the text content of the member; and

using the first megabyte of the text content of the member for hash generation.

22. The method of claim 19 wherein the reverse-scanning comprises:

clustering the members of the set of sites into sitename-based sets;

for each of the sitename-based sets, generating a list of Internet Protocol (IP) addresses with open http ports in the same subnet as the sitename-based set; and

for each of the sitename-based sets, scanning all IP addresses for the members of the sitename-based set.

23. The method of claim 19 wherein the performing comprises performing exact duplicate detection on the text content of each member of the set of sites.

24. The method of claim 19 wherein the performing comprises performing near-duplicate detection on the text content of each member of the set of sites.

25. The method of claim 19 wherein the performing comprises performing main topics detection on the text content of each member of the set of sites.

26. The method of claim 19 wherein the performing comprises aggregating topics in the text content of each member of the set of sites by organizational structure.

27. The method of claim 20 wherein the combining comprises listing the combined results in descending order of duplicate content.

28. A method of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the method comprising:

crawling HTML content and text content in a set of the sites;

deep-scanning non-HTML content and non-text content in the set of sites;

reverse-scanning the set of sites;

performing a semantic analysis of the crawled content and the deep-scanned content;

correlating the results of the semantic analysis with the results of the reverse-scanning;

comparing user navigation patterns and content from the members of the set of sites; and

combining the results of the performing, the results of the correlating, and the results of the comparing.

29. The method of claim 28 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning comprises:

crawling the first megabyte of the text content of the member; and

using the first megabyte of the text content of the member for hash generation.

30. The method of claim 28 wherein the reverse-scanning comprises:

clustering the members of the set of sites into sitename-based sets;

for each of the sitename-based sets, generating a list of Internet Protocol (IP) addresses with open http ports in the same subnet as the sitename-based set; and

for each of the sitename-based sets, scanning all IP addresses for the members of the sitename-based set.

31. The method of claim 28 wherein the performing comprises performing exact duplicate detection on the text content of each member of the set of sites.

32. The method of claim 28 wherein the performing comprises performing near-duplicate detection on the text content of each member of the set of sites.

33. The method of claim 28 wherein the performing comprises performing main topics detection on the text content of each member of the set of sites.

34. The method of claim 28 wherein the performing comprises aggregating topics in the text content of each member of the set of sites by organizational structure.

35. The method of claim 28 wherein the combining comprises listing the combined results in descending order of duplicate content.

36. A system of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the system comprising:

a crawling module configured to crawl HTML content and text content in a set of the sites;

a deep-scanning module configured to deep-scan non-HTML content and non-text content in the set of sites;

a reverse-scanning module configured to reverse-scan the set of sites;

a performing module configured to perform a semantic analysis of the crawled content and the deep-scanned content;

a correlating module configured to correlate the results of the semantic analysis with the results of the reverse-scanning;

a comparing module configured to compare user navigation patterns and content from the members of the set of sites; and

a combining module configured to combine the results of the performing module, the results of the correlating module, and the results of the comparing module.

37. The system of claim 36 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning module comprises:

a crawling module configured to crawl the first megabyte of the text content of the member; and

a using module configured to use the first megabyte of the text content of the member for hash generation.

38. The system of claim 36 wherein the reverse-scanning module comprises:

a clustering module configured to cluster the members of the set of sites into sitename-based sets;

for each of the sitename-based sets, a generating module configured to generate a list of Internet Protocol (IP) addresses with open http ports in the same subnet as the sitename-based set; and

for each of the sitename-based sets, a scanning module configured to scan all IP addresses for the members of the sitename-based set.

39. The system of claim 36 wherein the performing module comprises a performing module configured to perform exact duplicate detection on the text content of each member of the set of sites.

40. The system of claim 36 wherein the performing module comprises a performing module configured to perform near-duplicate detection on the text content of each member of the set of sites.

41. The system of claim 36 wherein the performing module comprises a performing module configured to perform main topics detection on the text content of each member of the set of sites.

42. The system of claim 36 wherein the performing module comprises an aggregating module configured to aggregate topics in the text content of each member of the set of sites by organizational structure.

43. The system of claim 36 wherein the combining module comprises a listing module configured to list the combined results in descending order of duplicate content.

44. A method of providing a service to analyze electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the method comprising:

crawling HTML content and text content in a set of the sites;

deep-scanning non-HTML content and non-text content in the set of sites;

reverse-scanning the set of sites;

performing a semantic analysis of the crawled content and the deep-scanned content;

correlating the results of the semantic analysis with the results of the reverse-scanning;

comparing user navigation patterns and content from the members of the set of sites; and

combining the results of the performing, the results of the correlating, and the results of the comparing.

45. The method of claim 44 wherein, if the text content for a member of the set of sites is greater than one megabyte, the deep-scanning comprises:

crawling the first megabyte of the text content of the member; and

using the first megabyte of the text content of the member for hash generation.

46. The method of claim 44 wherein the reverse-scanning comprises:

clustering the members of the set of sites into sitename-based sets;

for each of the sitename-based sets, generating a list of Internet Protocol (IP) addresses with open http ports in the same subnet as the sitename-based set; and

for each of the sitename-based sets, scanning all IP addresses for the members of the sitename-based set.

47. The method of claim 44 wherein the performing comprises performing exact duplicate detection on the text content of each member of the set of sites.

48. The method of claim 44 wherein the performing comprises performing near-duplicate detection on the text content of each member of the set of sites.

49. The method of claim 44 wherein the performing comprises performing main topics detection on the text content of each member of the set of sites.

50. The method of claim 44 wherein the performing comprises aggregating topics in the text content of each member of the set of sites by organizational structure.

51. The method of claim 44 wherein the combining comprises listing the combined results in descending order of duplicate content.

52. A computer program product usable with a programmable computer having readable program code embodied therein of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the computer program product comprising:

computer readable code for crawling HTML content and text content in a set of the sites;

computer readable code for deep-scanning non-HTML content and non-text content in the set of sites;

computer readable code for reverse-scanning the set of sites, computer readable code for performing a semantic analysis of the crawled content and the deep-scanned content;

computer readable code for correlating the results of the semantic analysis with the results of the reverse-scanning; and

computer readable code for comparing user navigation patterns and content from the members of the set of sites,

53. A computer program product usable with a programmable computer having readable program code embodied therein of analyzing electronic documents in an intranet, wherein the intranet comprises a plurality of web sites, the computer program product comprising:

computer readable code for crawling HTML content and text content in a set of the sites;

computer readable code for deep-scanning non-HTML content and non-text content in the set of sites;

computer readable code for reverse-scanning the set of sites;

computer readable code for performing a semantic analysis of the crawled content and the deep-scanned content;

computer readable code for correlating the results of the semantic analysis with the results of the reverse-scanning;

computer readable code for comparing user navigation patterns and content from the members of the set of sites; and

computer readable code for combining the results of the performing, the results of the correlating, and the results of the comparing.