Subject specific search engine

• US 20030163454A1
• Filed: 02/26/2002
• Published: 08/28/2003
• Est. Priority Date: 02/26/2002
• Status: Active Grant

• Current Assignee: Soren Alain Mortensen
• Original Assignee: Soren Alain Mortensen
• Inventors: Mortensen, Soren Alain, Jacobsen, Brian
• Granted Patent: US 7,949,648 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

A subject-specific search engine utilizes a smart web crawler and includes a capability of filtering out sites not relevant to the particular subject. As the smart crawler traverses the Internet, sites are filtered, and only sites found relevant are indexed and stored in a database for later searching. Sites may be filtered an arbitrary number of times for relevance, and such filtering may, for example, comprise automated, lexicon-based filtering; manual filtering, using a human editor; or both.

Claims

1. A method of compiling and accessing subject-specific information from a computer network, the method comprising the steps of:

traversing links between sites on the computer network;

filtering the contents of each site visited to determine relevancy of content; and

presenting information on each site deemed relevant for indexing.

2. The method according to claim 1, further comprising the step of:

filtering the contents of a site at least a second time for relevancy, prior to the step of presenting.

3. The method according to claim 2, wherein at least one of said filtering steps comprises the steps of:

presenting the contents to a human editor;

approving, by the human editor, if the contents are deemed relevant; and

disapproving, by the human editor, if the contents are not deemed relevant.

4. The method according to claim 2, wherein at least one of said filtering steps comprises the step of:

passing the contents of the site through a lexicon-based filter, the filter comparing contents of the site with terminology found in the lexicon.

5. The method according to claim 4, wherein the step of passing the contents of the site through a lexicon-based filter comprises the steps of:

breaking up a web page corresponding to the site contents into component parts; and

comparing the contents of each component part with the lexicon.

6. The method according to claim 5, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

assigning a weight to each component part based on a result of the step of comparing; and

deeming the component part to be relevant if it achieves a high-enough weight.

7. The method according to claim 6, wherein the step of assigning a weight comprises the steps of:

assigning a weight to each word, term, or expression in the component part that matches a word, term, or expression in the lexicon, according to a weight associated with the word, term, or expression; and

accumulating a sum of assigned weights, the sum forming the weight assigned to the component part.

8. The method according to claim 6, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

saving component parts deemed to be relevant and passing them to the presenting step; and

discarding component parts deemed not to be relevant.

9. The method according to claim 6, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

if at least one component part is deemed to be relevant, passing the web page to the presenting step; and

if no component part is deemed to be relevant, discarding the web page.

10. The method according to claim 4, wherein the step of passing the contents of the site through a lexicon-based filter comprises the step of:

comparing the contents of a web page corresponding to the site with the lexicon.

11. The method according to claim 10, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

assigning a weight to the web page based on a result of the step of comparing; and

deeming the web page to be relevant if it achieves a high-enough weight.

12. The method according to claim 11, wherein the step of assigning a weight comprises the steps of:

assigning a weight to each word, term, or expression in the web page that matches a word, term, or expression in the lexicon, according to a weight associated with the word, term, or expression; and

accumulating a sum of assigned weights, the sum forming the weight assigned to the web page.

13. The method according to claim 11, wherein the step of deeming comprises the steps of:

saving the web page and passing it to the step of presenting if it achieves a high-enough weight; and

discarding the web page if it does not achieve a high-enough weight.

14. The method according to claim 1, wherein the step of filtering the contents comprises the step of:

passing the contents of the site through a lexicon-based filter, the filter comparing contents of the site with terminology found in the lexicon.

15. The method according to claim 14, wherein the step of passing the contents of the site through a lexicon-based filter comprises the steps of:

breaking up a web page corresponding to the site contents into component parts; and

comparing the contents of each component part with the lexicon.

16. The method according to claim 15, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

assigning a weight to each component part based on a result of the step of comparing; and

deeming the component part to be relevant if it achieves a high-enough weight.

17. The method according to claim 16, wherein the step of assigning a weight comprises the steps of:

assigning a weight to each word, term, or expression in the component part that matches a word, term, or expression in the lexicon, according to a weight associated with the word, term, or expression; and

accumulating a sum of assigned weights, the sum forming the weight assigned to the component part.

18. The method according to claim 16, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

saving component parts deemed to be relevant and passing them to the presenting step; and

discarding component parts deemed not to be relevant.

19. The method according to claim 16, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

if at least one component part is deemed to be relevant, passing the web page to the presenting step; and

if no component part is deemed to be relevant, discarding the web page.

20. The method according to claim 14, wherein the step of passing the contents of the site through a lexicon-based filter comprises the step of:

comparing the contents of a web page corresponding to the site with the lexicon.

21. The method according to claim 20, wherein the step of passing the contents of the site through a lexicon-based filter further comprises the steps of:

assigning a weight to the web page based on a result of the step of comparing; and

deeming the web page to be relevant if it achieves a high-enough weight.

22. The method according to claim 21, wherein the step of assigning a weight comprises the steps of:

assigning a weight to each word, term, or expression in the web page that matches a word, term, or expression in the lexicon, according to a weight associated with the word, term, or expression; and

accumulating a sum of assigned weights, the sum forming the weight assigned to the web page.

23. The method according to claim 21, wherein the step of deeming comprises the steps of:

saving the web page and passing it to the step of presenting if it achieves a high-enough weight; and

discarding the web page if it does not achieve a high-enough weight.

24. The method according to claim 14, further comprising the step of:

filtering the contents of a site at least a secondtime for relevancy, prior to the step of presenting.

25. The method according to claim 24, wherein the step of filtering the contents at least a second time comprises the steps of:

presenting the contents to a human editor;

approving, by the human editor, if the contents are deemed relevant; and

disapproving, by the human editor, if the contents are not deemed relevant.

26. The method according to claim 14, further comprising the step of:

replacing the lexicon with a lexicon corresponding to a different subject in order to create a different subject-specific database.

27. The method according to claim 1, further comprising the step of:

compiling a database of searchable relevant information.

28. The method according to claim 27, further comprising the steps of:

permitting a user to enter a query; and

searching the database for information according to the query.

29. The method according to claim 28, further comprising the step of:

displaying information found in said step of searching in a hierarchical format.

30. The method according to claim 28, further comprising the step of:

determining a site ranking for each site associated with information found in said searching step, where the determining is according to how interesting at least one of authors and users of the computer network have found the site associated with the information.

31. The method according to claim 30, further comprising the step of:

displaying the results of the user query using the site ranking of each item of [SAM1] information found in the search to determine an order in which the results are displayed.

32. The method according to claim 31, wherein the step of displaying the results of the user query comprises the step of:

displaying the results of the user query in a hierarchical format according to site ranking.

33. The method according to claim 27, wherein the step of compiling a database comprises the step of:

for each relevant site to be stored in the database, assigning a word score to each word appearing on that site. [SAM3]

34. The method according to claim 33, wherein the step of assigning word scores comprises the steps of:

determining all sites found in the database that contain links to the site;

for each word on the site, assigning a word score for that word based at least in part on its presence on each site containing a link to the site.

35. The method according to claim 34, wherein the step of assigning a word score for that word further comprises the step of increasing the word score for each site containing a link to the site if the word appears in close proximity to the link.

36. The method according to claim 33, wherein the step of assigning word scores comprises the steps of:

determining all sites found in the database that contain links to the site; and

assigning a word score to each word on the site based at least in part on how many sites linking to the site also contain the particular word.

37. The method according to claim 36, wherein the step of assigning a word score for that word further comprises the step of increasing the word score for each site containing a link to the site according to the proximity of the word to the link.

38. The method according to claim 33, further comprising the steps of:

entering a user query;

using the user query to search the database; and

computing a site ranking for each site associated with information found in said searching step, the site ranking being computed based on said word scores.

39. The method according to claim 38, wherein the step of computing a site ranking comprises the steps of:

for each site associated with information found in said searching step, summing the word scores for that site corresponding to words in the user query.

40. A computer-readable medium containing software implementing the method as claimed in claim 1.

41. A system for compiling and accessing information from a computer network, the system comprising:

a processor; and

a computer-readable medium as claimed in claim 40.

42. The method according to claim 1, further comprising the step of:

monitoring a depth for each link, the depth being a reflection of relevance.

43. The method according to claim 42, wherein the step of monitoring comprises the steps of:

for a given site being visited, setting depths of any links leading from that site to other sites to a depth of a link traversed to reach the given site;

if the given site is determined to be relevant in the filtering step, setting the depths of the links leading from that site to zero; and

if the given site is determined not to be relevant in the filtering step, incrementing the depths of the links leading from that site.

44. The method according to claim 43, wherein the step of monitoring further comprises the steps of:

comparing the incremented depths to a predetermined maximum depth value;

if the incremented depths exceed the predetermined maximum depth value, discarding the links leading from the given site;

if the incremented depths do not exceed the predetermined maximum depth value, traversing one of the links leading from the given site.

45. The method according to claim 1, wherein said filtering step comprises the steps of:

presenting the contents to a human editor;

approving, by the human editor, if the contents are deemed relevant; and

disapproving, by the human editor, if the contents are not deemed relevant .

46. A system that compiles and permits accessing of subject-specific information from a computer network, the system comprising:

a host computer executing software from a computer-readable medium, the software comprising;

a smart crawler for traversing the computer network;

a first filter, filtering out irrelevant sites, and permitting only relevant sites to pass; and

an indexer indexing the relevant sites; and

memory, connected to the host computer, for storing indexed subject-specific information.

47. The system according to claim 46, wherein said first filter comprises a lexicon-based filter.

48. The system according to claim 47, wherein the system further comprises an interchangeable computer-readable medium on which is stored the lexicon for the lexicon-based filter, the lexicon containing subject-specific terminology.

49. The system according to claim 46, wherein the software further comprises at least a second filter.

50. The system according to claim 49, wherein the system further comprises a human-computer interface, and wherein at least one of said first filter and said at least a second filter comprises:

a presentation of relevant site information received from the smart crawler to a human editor via the human-computer interface; and

means for receiving input from the human editor, entered via the human-computer interface, as to whether or not to index and store the site in the memory.

51. The system according to claim 49, wherein at least one of said first filter and said at least a second filter comprises a lexicon-based filter.

52. The system according to claim 51, wherein the system further comprises an interchangeable computer-readable medium on which is stored the lexicon for the lexicon-based filter, the lexicon containing subject-specific terminology.

53. The system according to claim 46, wherein the system further comprises a human-computer interface, and wherein said first filter comprises:

a presentation of relevant site information received from the smart crawler to a human editor via the human-computer interface; and

means for receiving input from the human editor, entered via the human-computer interface, as to whether or not to index and store the site in the memory.

54. A method of ranking the relevance of information stored in a database, the information comprising web pages, the method comprising the steps of:

computing and storing a word ranking for each word, except for stop words, found on each web page; and

in response to a user query, computing a site ranking for each web page found in response to the user query based on the word rankings.

55. The method according to claim 54, wherein the step of computing a word ranking is performed according to how interesting at least one of authors and users of a computer network in which each web page is resident have found the web page.

56. The method according to claim 54, wherein the step of computing a word ranking comprises the step of:

for each word, except stop words, on each web page, determining all web pages found in the database that contain links to the web page on which the word appears; and

assigning a word score for that word based at least in part on its presence on each web page containing a link to the web page on which that word appears, the word score constituting the word ranking for that word.

57. The method according to claim 56, wherein the step of assigning a word score for that word further comprises the step of increasing the word score for each web page containing a link to the web page on which that word appears if the word appears in close proximity to the link.

58. The method according to claim 54, wherein the step of computing a site ranking comprises the steps of:

for each web page found in response to the user query, summing the word rankings for that web page corresponding to words in the user query.

59. A computer-readable medium containing software implementing the method of claim 54.