Focused search engine and method

• US 20020099700A1
• Filed: 12/12/2000
• Published: 07/25/2002
• Est. Priority Date: 12/14/1999
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Li, Wen-Syan
• Granted Patent: US 6,691,108 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A focused search engine and method are directed to crawling vast search spaces comprising markup language documents, for example. Both topic distillation and site distillation methodologies are incorporated into an integrated topic-focused search strategy. Categorization of search results may be initiated by the search engine itself; alternatively, topic categories of interest may be specified in conjunction with the original request for information.

Claims

1. A method of searching a search space comprising a plurality of pages;

said method comprising;

responsive to a request for information, determining a set of pages wherein each page in said set of pages is relevant to said request;

performing topic distillation using said set of pages to determine a set of important keywords; and

performing site distillation using said set of important keywords to determine a set of important pages.

2. The method according to claim 1 wherein said determining comprises locating said at least one of said plurality of keywords in a selected part of the structure of each page in said set of pages.

3. The method according to claim 1 wherein said performing topic distillation comprises keyword extraction.

4. The method according to claim 1 wherein said performing topic distillation comprises keyword propagation.

5. The method according to claim 4 wherein said keyword propagation is weighted.

6. The method according to claim 1 wherein said performing topic distillation enables implicit query expansion.

7. The method according to claim 1 further comprising selectively repeating said performing topic distillation and said performing site distillation.

8. The method according to claim 1 wherein said request for information further comprises a selected topic category.

9. The method according to claim 1 further comprising displaying results of said performing topic distillation.

10. The method according to claim 1 further comprising displaying results of said performing site distillation.

11. A computer-based system for searching a search space, the search space comprising a plurality of pages;

said system comprising;

means for receiving a request for information; and

a focused search engine determining a set of pages wherein each page in said set of pages is relevant to said request, performing topic distillation using said set of pages to determine a set of important keywords, and further performing site distillation using said set of important keywords to determine a set of important pages.

12. The computer-based system according to claim 11 wherein said focused search engine locates said at least one of said plurality of keywords in a selected part of the structure of each page in said set of pages.

13. The computer-based system according to claim 11 wherein said topic distillation comprises implicit query expansion.

14. The computer-based system according to claim 11 wherein said topic distillation comprises keyword extraction.

15. The computer-based system according to claim 11 wherein said topic distillation comprises keyword propagation.

16. The computer-based system according to claim 15 wherein said keyword propagation is weighted.

17. The computer-based system according to claim 11 wherein said focused search engine selectively repeats said topic distillation and said site distillation.

18. The computer-based system according to claim 11 wherein said request for information further comprises a selected topic category.

19. The computer-based system according to claim 11 further comprising a display for displaying results of said topic distillation.

20. The computer-based system according to claim 11 further comprising a display for displaying results of said site distillation.

21. A method of searching a search space comprising a plurality of pages;

said method comprising;

responsive to a request for information comprising a plurality of keywords and a one or more topic categories, determining a set of pages within the search space wherein each page in said set of pages contains at least one of said plurality of keywords and further contains information relevant to said one or more topic categories;

performing topic distillation using said set of pages to determine a set of important keywords; and

performing site distillation using said set of important keywords.

22. The method according to claim 21 wherein said determining comprises locating said at least one of said plurality of keywords in a selected part of the structure of each page in said set of pages.

23. The method according to claim 21 wherein said performing topic distillation comprises keyword extraction.

24. The method according to claim 21 wherein said performing topic distillation comprises keyword propagation.

25. The method according to claim 24 wherein said keyword propagation is weighted.

26. The method according to claim 21 wherein said performing topic distillation comprises implicit query expansion.

27. The method according to claim 21 further comprising selectively repeating said performing topic distillation and said performing site distillation.

28. The method according to claim 21 wherein the request for information is a keyword query.

29. The method according to claim 21 further comprising displaying results of said performing topic distillation.

30. The method according to claim 21 further comprising displaying results of said performing site distillation.

31. A method of processing a query of a search space;

said query comprising a plurality of keywords and said search space comprising a plurality of pages;

said method comprising;

responsive to said query, determining a set of pages within the search space wherein each page in said set of pages contains at least one of said plurality of keywords;

assigning a score to each page in said set of pages in accordance with relevance to said query; and

distributing said score of each page in said set of pages to neighboring pages in said set of pages in accordance with relevance to each of said plurality of keywords.

32. The method according to claim 31 further comprising selectively repeating said assigning and said distributing.

33. The method according to claim 31 wherein said determining comprises extracting keywords from each page within the search space to identify at least one topic category.

34. The method according to claim 33 wherein said distributing comprises keyword propagation.

35. The method according to claim 34 wherein said keyword propagation is weighted.

37. The method according to claim 36 wherein said determining comprises extracting keywords from said set of pages to identify category-specific keywords associated with said at least one topic category.

38. The method according to claim 37 wherein said distributing comprises keyword propagation.

39. The method according to claim 38 wherein said keyword propagation is weighted.

36. A method of processing a query of a search space;

said query comprising a plurality of keywords and at least one topic category associated with at least one of said plurality of keywords;

said search space comprising a plurality of pages;

said method comprising;

responsive to said query, determining a set of pages within the search space wherein each page in said set of pages contains at least one of said plurality of keywords and further contains information relevant to said at least one topic category;

assigning a score to each page in said set of pages in accordance with relevance to said query; and

distributing said score of each page in said set of pages to neighboring pages in said set of pages in accordance with relevance to each of said plurality of keywords and in accordance with relevance to said at least one topic category.

40. A method of focused crawling of a search space comprising a plurality of pages;

said method comprising;

responsive to a request for information, determining a set of pages within the search space wherein each page in said set of pages contains information relevant to said request;

performing site distillation using said set of pages to determine a set of important pages; and

examining said set of important pages to determine a set of keywords;

scoring each keyword in said set of keywords in accordance with relevance to said request; and

selectively repeating said performing, said examining, and said scoring.

41. The method according to claim 40 wherein said determining comprises extracting keywords from said set of pages to identify at least one topic category relevant to said request.

42. The method according to claim 41 wherein said determining comprises keyword propagation.

43. The method according to claim 42 wherein said keyword propagation is weighted.

44. A method of focused crawling of a search space comprising a plurality of pages;

said method comprising;

responsive to a request for information, determining a set of pages within the search space wherein each page in said set of pages contains information relevant to said request;

performing site distillation using said set of pages to determine a set of important pages, including assigning a relevance score to each page in said set of important pages in accordance with relevance to said request;

examining said set of important pages to determine a set of keywords;

scoring each keyword in said set of keywords in accordance with relevance to said request; and

selectively repeating said performing, said examining, and said scoring.

45. The method according to claim 44 wherein said determining comprises extracting keywords from said set of pages to identify at least one topic category relevant to said request.

46. The method according to claim 45 wherein said assigning comprises weighted keyword propagation.

47. The method according to claim 46 wherein said weighted keyword propagation comprises dividing a relevance score unevenly prior to propagating said relevance score.

48. The method according to claim 44 wherein said request for information includes a keyword.

49. The method according to claim 44 wherein said request for information includes a topic category.

50. The method according to claim 44 further comprising displaying results of said performing site distillation on a display.

51. A method of site distillation for identifying information in a search space comprising a plurality of pages;

said method comprising;

responsive to a request for information, determining a set of pages within the search space wherein each page in said set of pages contains information relevant to said request;

identifying one or more keywords relevant to said request;

assigning a score to each page in said set of pages in accordance with relevance to said request and in accordance with relevance to said one or more keywords;

distributing said score of each page in said set of pages to neighboring pages in said set of pages in accordance with relevance to said one or more keywords; and

responsive to said distributing, computing a weight for each page in said set of pages in accordance with relevance to said request and relevance to said one or more keywords.

52. The method according to claim 51 wherein said identifying comprises extracting keywords from said set of pages.

53. The method according to claim 51 wherein said distributing comprises weighted keyword propagation.

54. The method according to claim 53 wherein said weighted keyword propagation comprises dividing said score unevenly prior to said distributing.

55. The method according to claim 51 wherein said request for information includes a keyword.

56. The method according to claim 51 wherein said request for information includes one or more specified topic categories.

57. The method according to claim 51 further comprising displaying results of said computing on a display.

58. The method according to claim 51 further comprising selectively repeating said distributing and said computing.

60. The method according to claim 59 wherein said plurality of pages in said search space are arranged in accordance with relevance to one or more predetermined categories and wherein said method further comprises:

propagating said score of each of said one or more keywords from one of said one or more predetermined categories to another of said one or more predetermined categories; and

weighting each of said one or more predetermined categories in accordance with said assigning and said propagating.

61. The method according to claim 59 wherein said propagating includes applying a neighboring page decay factor.

62. The method according to claim 60 wherein said propagating includes applying a category decay factor.

63. The method according to claim 60 wherein said request for information includes identifying said one or more predetermined categories.

59. A method of topic distillation for identifying information in a search space comprising a plurality of pages;

said method comprising;

responsive to a request for information, determining a set of pages within the search space wherein each page in said set of pages contains information relevant to said request;

for each page in said set of pages, extracting one or more keywords relevant to said request and assigning a score to each of said one or more keywords in accordance with relevance to said request;

propagating said score of each of said one or more keywords to neighboring pages in said set of pages;

weighting each page in said set of pages in accordance with said assigning and said propagating; and

identifying a set of important keywords associated with said request.

64. A method of processing a query of a search space;

said query comprising a plurality of keywords and said search space comprising a plurality of pages;

said method comprising;

responsive to said query, determining a set of pages within the search space wherein each page in said set of pages contains at least one of said plurality of keywords;

identifying one or more topic categories relevant to said query; and

determining, for each of said one or more topic categories, a set of important pages in accordance with relevance to said query and in accordance with relevance to said one or more topic categories.

65. The method according to claim 64 further comprising:

assigning a score to each page in said set of pages in accordance with relevance to said query; and

distributing said score of each page in said set of pages to neighboring pages in said set of pages in accordance with relevance to each of said plurality of keywords.

66. The method according to claim 64 wherein said query includes said one or more topic categories.

67. The method according to claim 64 wherein said determining a set of important pages includes utilizing a vector model.

68. The method according to claim 67 wherein utilizing said vector model considers relevance to said query more important than relevance to said one or more topic categories.