Full Text Query and Search Systems and Method of Use

• US 20080077570A1
• Filed: 04/25/2007
• Published: 03/27/2008
• Est. Priority Date: 10/25/2004
• Status: Abandoned Application

• Current Assignee: DeepDyve, Inc.
• Original Assignee: DeepDyve, Inc.
• Inventors: Mei, Minghua, Chen, Chunnuan, Hu, Qianjin, Yang, Yonghong, Tang, Yuanhua
• Application Number:
  US11/740,247
• Publication Number:
  US 20080077570A1
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Roughly described, a database searching method for searching a database, in which hits are ranked in dependence upon an information measure of itoms shared by both the hit and the query. The information measure can be a Shannon information score, or another measure which indicates the information value of the shared itoms. An itom can be a word or other token, or a multi-word phrase, and can overlap with each other. Synonyms can be substituted for itoms in the query, with the information measure of substituted itoms being derated in accordance with a predetermined measure of the synonyms'"'"' similarity. Indirect searching methods are described in which hit from other search engines are re-ranked in dependence upon the information measures of shared itoms. Structured and completely unstructured databases may be searched, with hits being demarcated dynamically. Hits may be clustered based upon distances in an information-measure-weighted distance space.

Claims

1. A method for searching a database, comprising the steps of:

initiating a first search on the database in response to a first query, the first search generating a plurality of hits;

ranking the hits in dependence upon an information measure of itoms shared by both the hit and the first query; and

identifying to a user at least one of the hits selected in accordance with the ranking.

2. A method according to claim 1, wherein the information measure comprises a cumulative Shannon information score of the itoms shared by both the hit and the first query.

3. A method according to claim 1, further comprising the step of calculating a respective cumulative information score of the itoms shared by the query and each of the hits,

wherein the step of ranking comprises the step of ranking the hits in dependence upon the cumulative information score calculated for each of the hits,

wherein the cumulative information score for each given one of the hits is an accumulation of an information score determined for each particular itom shared by both the query and the given hit,

and wherein the information score determined for each particular shared itom is a monotonically non-increasing function of an at least approximate relative frequency of the particular shared itom in the database.

4. A method according to claim 3, wherein the information score for the particular shared itom is further a function of the information score of all itoms fully contained within the particular shared itom.

5. A method according to claim 3, wherein the information score determined for each particular shared itom is given by −

log₂(f_i/T_w), where f is the approximate number of occurrences of the particular itom in the database, and T_w is an approximate total number of itoms in the database.

6. A method according to claim 3, wherein the information score determined for each particular shared itom is given by an accumulation of −

log₂(f_i/T_w), for all i′

th itoms fully contained within the particular itom including itself, where f_i is the approximate number of occurrences in the database of the i′

th itom, and T_w is an approximate total number of itoms in the database.

7. A method according to claim 3, wherein the cumulative information score for the given hit counts each particular itom shared by both the query and the given hit only min(n,k) times, where n is the number of occurrences of the particular itom in the query and k is the number of occurrences of the particular itom in the given hit.

8. A method according to claim 3, wherein the cumulative information score for the given hit counts n occurrences of the particular itom in the query and k occurrences of the particular itom in the hit, at least where k>

n, as the information score of the particular itom, multiplied by n plus a monotonically increasing function of k, where the monotonically increasing function of k converges on an upper limit as k goes to infinity.

9. A method according to claim 8, wherein the monotonically increasing function of k is the sum, for i=1, . . . , (k−

n), of α

ⁱ⁺¹, where α

is a predetermined damping factor, 0<

=α

<

1.

10. A method according to claim 8, wherein

for k>

n and k % n=0, the monotonically increasing function of k is the sum, for i=1, . . . , [(k−

n)/n], of α

ⁱ,

and wherein for k>

n and k%n!=0, the monotonically increasing function of k is ((k−

n)%n)/n*α

<sup>[(k−

n)/n]+1</sup> plus the sum, for i=1, . . . , [(k−

n)/n], of α

ⁱ,

where α

is a predetermined damping factor, 0<

=α

<

1,

where % is the modulo operator,

and where [(k−

n)/n] means the integer part of (k−

n)/n.

11. A method according to claim 8, wherein the cumulative information score for the given hit counts the n occurrences of the particular itom in the query and the k occurrences of the particular itom in the hit, where k<

=n, as k times the information score of the particular itom.

12. A method according to claim 1, wherein the information measure comprises a percent identity between an information measure of the shared itoms and one of the group consisting of:

(1) an information measure of the query, (2) an information measure of the hit, (3) the lesser of an information measure of the query and an information measure of the hit, and (4) the greater of an information measure of the query and an information measure of the hit.

13. A method according to claim 1, wherein the information measure is inversely related to the at least approximate probability that the itoms will be shared by both the hit and the first query purely by chance.

14. A method according to claim 1, wherein the information measure is inversely related to the at least approximate expected value of the information measure of itoms shared by both the hit and the first query.

15. A method according to claim 1, wherein the step of ranking comprises the step of comparing a predetermined function of the angle in an information-measure weighted vector space that each of the hits makes with the query.

16. A method according to claim 15, wherein the predetermined function is the cosine function.

17. A method according to claim 1, wherein the step of ranking is performed further in inverse dependence upon an information measure of itoms in the hit which are between itoms shared by both the hit and the first query, and which are not in the first query.

18. A method according to claim 1, wherein the step of initiating a first search comprises the steps of:

developing a correlated itom list for a first itom in the first query, each itom in the correlated itom list having an associated similarity score with respect to the first itom; and

initiating a search on the database in response to the first query, allowing for substitutions of itoms from the correlated itom list for the first itom in the first query,

and wherein the step of ranking the hits is performed further in dependence upon the similarity score of any itom substituted for the first itom in the step of initiating a search.

19. A method according to claim 18, wherein the correlated itom list includes synonyms for the first itom.

20. A method according to claim 18, wherein the correlated itom list includes non-synonyms found in the database to be frequently present in proximity to the first itom.

21. A method according to claim 18, further comprising the step of providing a similarity matrix of itoms and synonyms therefor, each synonym for a particular itom having an associated similarity score relative to the particular itom,

and wherein the step of developing a synonym list for each of a plurality of itoms in the first query comprises the step of extracting the similarity list from the similarity matrix.

22. A method according to claim 1, wherein at least a first one of the itoms shared by both the hit and the first query contains more than one token.

23. A method according to claim 22, wherein a second one of the itoms shared by both the hit and the first query overlaps with the first itom, either in the hit or in the first query or both.

24. A method according to claim 22, wherein a second one of the itoms shared by both the hit and the first query is wholly contained in with the first itom, either in the hit or in the first query or both.

25. A method according to claim 1, wherein the step of initiating a first search comprises the step of searching in the database for each of a plurality of query itoms in the first query, at least one of the query itoms containing more than one token.

26. A method according to claim 1, wherein the step of initiating a first search comprises the step of initiating a preliminary Boolean search on the database for at least one keyword present in the first query.

27. A method according to claim 26, wherein the step of ranking the hits comprises the steps of:

identifying a set of at least one query itom in the first query, at least one member of the set containing more than one token;

determining, for each given one of the hits generated in the first search, which of the query itoms are shared by the first query and the given hit; and

ranking the hits in dependence upon the information measure of the shared itoms determined in the step of determining.

28. A method according to claim 27, wherein the step of initiating a first search comprises the steps of:

initiating a plurality of preliminary searches on the database for respective differing subsets of the keywords present in the first query, to generate a respective set of preliminary hits from each of the preliminary searches, one of the preliminary searches being the preliminary Boolean search; and

generating the plurality of hits in dependence upon a combination of the hits in the sets of preliminary hits.

29. A method according to claim 27, wherein the step of initiating a first search comprises the steps of:

initiating a plurality of preliminary searches in response to the first query, through a plurality of different search engines, to generate a respective set of preliminary hits from each of the preliminary searches, one of the preliminary searches being the preliminary Boolean search; and

generating the plurality of hits in dependence upon a combination of the hits in the sets of preliminary hits.

30. A method according to claim 1, wherein the step of initiating a first search comprises the steps of:

forwarding to a set of at least one external search engine, preliminary search queries formed in dependence upon the first query, each combination of a preliminary search query and an external search engine yielding a respective set of preliminary hits; and

generating the plurality of hits in dependence upon a combination of the hits in the sets of preliminary hits.

31. A method according to claim 30, wherein at least two of the preliminary search queries differ from each other.

32. A method according to claim 30, wherein at least two of the external search engines differ from each other.

33. A method according to claim 32, wherein at least two of the preliminary search queries differ from each other.

34. A method according to claim 1, further comprising the step of initiating a preliminary search on the database in response to a predecessor query, the preliminary search generating a plurality of preliminary hits, the first query being at least a subset of the content of a user-selected one of the preliminary hits.

35. A method according to claim 34, wherein the step of initiating a preliminary search comprises a step of initiating a Boolean search on the database for at least one keyword present in the preliminary query.

36. A method according to claim 34, wherein the step of initiating a preliminary search comprises the steps of:

identifying a preliminary set of at least one preliminary itom in the preliminary query, at least one member of the preliminary set containing more than one token; and

identifying, as the plurality of preliminary hits, entries in the database sharing at least one of the preliminary itoms.

37. A method according to claim 36, further comprising the steps of:

detecting, for each particular one of the hits generated in the preliminary search, which of the preliminary itoms are shared by the preliminary query and the particular hit; and

ranking the hits generated in the preliminary search in dependence upon an information measure of the shared itoms determined in the step of detecting.

38. A method according to claim 1, wherein the step of initiating a first search comprises the steps of:

selecting a proper subset of the itoms in said first query in dependence upon a relative information measure of the itoms in said first query; and

initiating the first search in a manner that considers itoms in the subset and ignores the itoms not in the subset.

39. A method according to claim 38, wherein the step of initiating the first search in a manner that considers itoms in the subset and ignores the itoms not in the subset, comprises the step of forwarding to an external search engine itoms in the subset and not itoms not in the subset.

40. A method according to claim 1, wherein the database includes a set of at least one table each having a plurality of rows and a plurality of columns,

wherein the first query identifies a plurality of itoms to search for and a column in which to search for each of the identified itoms.

41. A method according to claim 40, wherein the information measure of each given one of the itoms shared by both the hit and the first query depends inversely on a relative frequency of occurrence of the given itom within the column within which, according to the first query, the given itom was to be searched.

42. A method according to claim 40, wherein the relative frequency of occurrence of the given itom counts multiple occurrences in a single table cell only once.

43. A method according to claim 40, wherein the relative frequency of occurrence of the given itom counts each occurrence in a single table cell separately.

44. A method according to claim 40, wherein the set of tables includes a plurality of tables, wherein two of the columns identified by the first query are located in different ones of the tables, and wherein the information measure of each given one of the itoms shared by both the hit and the first query depends inversely on a relative frequency of occurrence of the given itom within the column within which, according to the first query, the given itom was to be searched, multiplied by the number of times the given itom would be repeated within such column if all tables involved in the first query were to be joined by left outer joins.

45. A method according to claim 40, wherein the first query identifies particular content as being required for a particular one of the columns,

and wherein the plurality of hits excludes any entry in which the particular column does not contain the particular content.

46. A method according to claim 40, wherein the first query identifies content as being required for a particular one of the columns,

and wherein the step of ranking comprises the step of ranking all hits in which the particular column does contain the particular content above all hits in which the particular column does not contain the particular content.

47. A method according to claim 40, further comprising the step of assigning a weighting factor to each of the columns identified in the first query,

and wherein the step of ranking the hits is performed in further dependence upon the weighting factors.

48. A method according to claim 40, wherein the first query identifies content as being desired for exclusion in a particular one of the columns,

and wherein the plurality of hits excludes any entry in which the particular column shares the particular content.

49. A method according to claim 40, wherein the first query identifies content as being desired for exclusion in a particular one of the columns,

and wherein the step of ranking comprises the step of penalizing the rank of each entry in which the particular column shares the particular content.

50. A method according to claim 1, wherein the database comprises unstructured text.

51. A method according to claim 1, wherein the database comprises a plurality of sub-databases,

wherein each sub-database has associated therewith respective itom distribution data,

and wherein the step of ranking the hits comprises the step of, for each given one of the itoms shared by any hit and the first query, determining an overall relative frequency of the given itom in dependence upon the itom distribution data associated with all of the sub-databases, the information measure of the given itom being inversely related to the overall relative frequency of the given itom.

52. A method according to claim 50, wherein at least two of the sub-databases are queried through different nodes on a network.

53. A method according to claim 50, wherein at least two of the sub-databases have different database schemas.

54. A method according to claim 50, wherein the step of, for each given itom, determining an overall relative frequency, comprises the steps of:

merging the itom distribution data associated with all of the sub-databases into an overall itom distribution data set; and

retrieving the overall relative frequency for the given itom from the overall itom distribution data set.

55. A method according to claim 50, wherein the step of initiating the first search comprises the step of initiating the first search separately on each of the sub-databases, each sub-database generating a respective list of zero or more hits,

and wherein the step of, for each given itom, determining an overall relative frequency, comprises the steps of:

retrieving a respective individual frequency of the given itom from the itom distribution data associated with each of the sub-databases;

retrieving a respective individual total itomic number associated with each of the sub-databases;

determining the overall relative frequency for the given itom in dependence upon the individual frequencies and the individual total itomic numbers.

56. A method according to claim 1, wherein the step of initiating a first search comprises the step of initiating a first search in which the sequence in which shared itoms appear in the hit is required to match the sequence in which the shared itoms appear in the first query.

57. A method according to claim 1, wherein the step of initiating a first search comprises the step of initiating a first search in which the sequence in which shared itoms appear in the hit is required to match the sequence in which the shared itoms appear in the first query, with the exception of allowed variations of a type selected from the group consisting of insertions, deletions, substitutions and re-ordering,

and wherein the step of ranking the hits imposes a penalty on the rank of each hit in dependence upon the sequence variations in the hit.

58. A method according to claim 1, wherein the database comprises a plurality of pre-demarcated entries,

and wherein each of the hits comprises a respective one of the entries.

59. A method according to claim 1, wherein a first one of the hits has a starting position and an ending position in the database, at least one of the starting and ending position being determined dynamically in dependence upon the lengths of gaps in the database between itoms shared by both the first hit and the first query.

60. A method according to claim 59, wherein the step of identifying to a user at least one of the hits, comprises the step of displaying to a user, in a side-by-side display, both the first query and the first hit.

61. A method according to claim 59, wherein the step of identifying to a user at least one of the hits, comprises the step of displaying to a user both the first query and the first hit, the shared itoms being highlighted in each.

62. A method according to claim 1, wherein a first one of the hits has starting and ending positions in the database, and wherein the step of initiating a first search comprises the steps of:

locating itoms from the first query in the database;

locating a first gap in which the database contains no itoms from the first query, the first gap having a starting and ending position in the database; and

establishing a member of the group consisting of;

the ending position of the first hit in dependence upon the starting position of the first gap, and

the starting position of the first hit in dependence upon the ending position of the first gap.

63. A method according to claim 62, wherein a second one of the hits has a starting position,

wherein the step of establishing comprises the step of establishing the ending position of the first hit in dependence upon the starting position of the first gap,

and wherein the step of initiating a first search further comprises the step of establishing the starting position of the second hit in dependence upon the ending position of the first gap.

64. A method according to claim 62, wherein the step of locating a first gap comprises the step of locating a gap having at least a predetermined length.

65. A method according to claim 62, wherein the step of locating a first gap comprises the step of locating a gap having a minimum length which depends inversely on an information measure of itoms in the gap.

66. A method according to claim 1, further comprising the step of clustering at least a subset of the hits in dependence upon distances, in an information-measure-weighted distance space, among the itoms in each of the hits in the subset,

wherein the step of identifying the hits comprises the step of showing the hits of the subset in groups dependent upon the clusters determined in the step of clustering.

67. A method for searching a database, comprising the steps of:

initiating a first search on the database in response to a first query, the first search generating a plurality of hits;

ranking the hits in dependence upon an information measure of itoms shared by both the hit and the first query; and

taking automatic action with respect to at least one of the hits selected in accordance with the ranking.

68. A method for searching a database, comprising the steps of:

selecting a proper subset of itoms in a first query in dependence upon a relative information measure of the itoms in said first query;

initiating the first search on the database in response to the first query, in a manner that considers itoms in the subset and ignores the itoms not in the subset, the first search generating a plurality of hits; and

identifying to a user at least one of the hits selected in accordance with the ranking.

69. A method according to claim 68, wherein the step of initiating the first search comprises the step of forwarding to an external search engine itoms in the subset and not itoms not in the subset.

70. A method for preparing a first database for searching, comprising the steps of:

developing a list of itoms present in the first database, at least one of the itoms containing more than one token, and at least one of the itoms constituting less than a full cell in the first database;

associating with each of the itoms in the list an information measure of the respective itom; and

associating with each of the itoms in the list an indication of the location of each occurrence of the respective itom in the first database.

71. A method according to claim 70, wherein the step of developing a list of itoms comprises the steps of:

identifying a plurality of candidate itoms having at least two tokens each; and

including in the list only those candidate itoms occurring more frequently in the database than a threshold.

72. A method according to claim 71, wherein the threshold decreases as the number of tokens in a candidate itom increases.

73. A method according to claim 70, wherein the step of developing a list of itoms comprises the steps of:

identifying a plurality of candidate itoms having at least two tokens each; and

including in the list only

those candidate itoms occurring more frequently in the database than a first threshold multiplier times the frequency that would be expected given the frequency of its components,

and those candidate itoms which occur together more frequently than a threshold percentage of the occurrences of its components in the database.

74. A method according to claim 73, wherein the step of including only certain candidate itoms, excludes candidate itoms which are wholly included in other candidate itoms.

75. A method according to claim 70, wherein the step of developing a list of itoms comprises the steps of:

identifying a plurality of candidate itoms having at least two tokens each; and

including in the list only those candidate itoms which satisfy a chi-square test.

76. A method according to claim 70, wherein the step of developing a list of itoms comprises the steps of:

identifying a plurality of candidate itoms having at least two tokens each; and

including in the list only those candidate itoms which satisfy a predetermined association rule test selected from the group consisting of;

a confidence test, an all confidence test, an interest factor test, a cosine test, a correlation coefficient test, an odds ratio test, a Piatetsky-Shapiro test, a collective strength test, and a Jaccard test.

77. A method according to claim 70, further comprising the step of merging in an additional database for searching, the step of merging comprising the steps of, for each given itom in the list which also appears in the additional database:

updating the information measure for the given itom in dependence upon the information measure of the given itom in the additional database; and

associating with the given itom in the list additional indications of the locations of each occurrence of the given itom in the additional database.

78. A method according to claim 77, further comprising the step of adding to the list an itom present in the additional database but not previously present in the list.

79. A method according to claim 77, wherein the first database and the additional database are accessed through different nodes of a network.

80. A method according to claim 79, further comprising the step of dividing a master database into n sub-databases, n>

=2, the first database and the additional database being two of the sub-databases.

81. A computer readable medium for use in searching a first database, the medium carrying:

a list of itoms present in the first database, at least one of the itoms containing more than one token, and at least one of the itoms constituting less than a full cell in the first database;

for each of the itoms in the list, an information measure of the respective itom; and

for each of the itoms in the list, an indication of the location of each occurrence of the respective itom in the first database.

82. A medium according to claim 81, wherein list include only itoms occurring in the first database more frequently than a threshold.

83. A medium according to claim 82, wherein the threshold decreases as the number of tokens in the itom increases.

84. A medium according to claim 81, wherein the list of itoms includes only:

itoms occurring more frequently in the first database than a first threshold multiplier times the frequency that would be expected given the frequency of its components,

and itoms which occur together in the first database more frequently than a threshold percentage of the occurrences of its components in the first database.

85. A method for searching a database, comprising the steps of:

initiating a first search on the database in response to a first query, the first search generating a plurality of hits, at least a first one of the hits having a starting and ending position determined dynamically in dependence upon an information measure of itoms shared by both the first hit and the first query, the first hit constituting less than a full entry in the database; and

displaying the first hit to a user.

86. A method according to claim 85, wherein the step of displaying further comprises the step of displaying the first query to the user.

87. A method according to claim 86, wherein the first query and the first hit are displayed in a side-by-side display.

88. A method according to claim 86, wherein the step of displaying further comprises the step of highlighting, in both the first query display and the first hit display, all shared itoms.

89. A method according to claim 85, wherein the starting and ending positions of the first hit are determined dynamically in dependence further upon an information measure of itoms in the hit not shared with the first query.

90. A method according to claim 85, wherein the starting and ending positions of the first hit are determined dynamically in dependence further upon an information measure of tokens not within itoms shared with the first query.

91. A method according to claim 85, wherein the step of initiating a first search comprises the steps of:

identifying initial starting and ending positions for the first hit; and

iteratively adjusting at least one of the starting and ending positions by increments until a determination is made that further adjustment in the same direction would reduce a net information value of shared itoms in the first hit, the net information value being positively dependent upon an information measure of the shared itoms included in the hit.

92. A method according to claim 91, wherein the net information value is also negatively dependent upon an information measure of itoms in the hit not shared with the first query.

93. A method according to claim 91, wherein the net information value is also negatively dependent upon an information measure of tokens not within itoms shared with the first query.

94. A method according to claim 85, further comprising the step of displaying to the user a display segment of the first query, the display segment having starting and ending positions in the first query, the starting and ending positions having been determined dynamically in dependence upon an information measure of itoms shared by both the first hit and the display segment of the first query.

95. A method according to claim 94, wherein the starting and ending positions of the display segment are determined dynamically in dependence further upon an information measure of tokens not within itoms shared by both the first hit and the display segment of the first query.

96. A method according to claim 94, wherein the step of displaying a display segment of the first query comprises the steps of:

identifying initial starting and ending positions for the display segment; and

iteratively adjusting at least one of the starting and ending positions by increments until a determination is made that further adjustment in the same direction would reduce a net information value of shared itoms in the display segment, the net information value being positively dependent upon an information measure of the shared itoms included in the display segment.

97. A method for searching a database, comprising the steps of:

developing a synonym list for a first itom in a first query, each synonym in the synonym list having an associated similarity score;

initiating a search on the database in response to the first query, allowing for substitutions of synonyms from the synonym list for the first itom in the first query, the first search generating a plurality of hits;

ranking the hits in dependence upon the similarity score of any synonym substituted for the first itom in the step of initiating a search; and

identifying to a user at least one of the hits selected in accordance with the ranking.

98. A method according to claim 97, further comprising the step of providing a similarity matrix of itoms and synonyms therefor, each synonym for a particular itom having an associated similarity score relative to the particular itom,

and wherein the step of developing a synonym list for each of a plurality of itoms in the first query comprises the step of extracting the similarity list from the similarity matrix.

99. A method for searching a database, comprising the steps of:

initiating a first search on the database in response to a first query, for hits in which at least one itom is shared by both the hit and the first query; and

identifying to a user at least one of the hits,

wherein at least a first one of the itoms shared by both the first query and a first one of the hits contains more than one token,

and wherein a second one of the itoms shared by both the first hit and the first query overlaps with the first itom, either in the hit or in the first query or both.

100. A method according to claim 99, wherein the second itom is wholly contained in with the first itom, either in the hit or in the first query or both.

101. A method for searching a database, comprising the steps of:

developing a plurality of preliminary queries in dependence upon a provided first query, at least two of the preliminary queries differing from each other;

forwarding the preliminary queries to a set of at least one external search engine, each combination of a preliminary search query and an external search engine yielding a respective set of preliminary hits; and

identifying to a user at least one of the hits returned from at least one of the preliminary queries.

102. A method according to claim 101, wherein the step of developing a plurality of preliminary queries comprises the steps of:

identifying a plurality of itoms in the first query;

selecting a subset of the plurality of itoms in dependence upon an information measure of the itoms; and

selecting keywords for each of the preliminary queries from the itoms in the subset.

103. A method according to claim 102, wherein the step of selecting a subset of itoms comprises the step of selecting a predetermined number of the highest information measure itoms from the plurality of itoms.

104. A method according to claim 102, wherein the step of selecting keywords comprises the steps of selecting a respective particular number of the keywords for each of the preliminary queries randomly.

105. A method according to claim 101, for use with a first list of itoms each having an associated information measure, wherein the step of further comprising the steps of:

enhancing the information measures associated with itoms in the first list in dependence upon the frequencies of appearance, in the hits returned from the preliminary queries, of the itoms in the first list; and

ranking the hits returned from the preliminary queries in dependence upon the enhanced information measures.

106. A method according to claim 105, further comprising the step of enhancing the first list of itoms with itoms in the hits returned from the preliminary queries and not previously in the first list.

107. A method according to claim 101, wherein at least two of the external search engines differ from each other.

108. A method for organizing segments in a subject database for presentation to a user, comprising the steps of:

clustering the segments in dependence upon an information measure of itoms in each of the segments; and

identifying the segments to the user in clusters determined in the step of clustering.

109. A method according to claim 108, wherein the step of clustering comprises the steps of:

for each pair of first and second ones of the segments x and y, determining a distance between them given by d(x, y)=Σ

_ix_i+Σ

_jy_j, where x_i are information measures of itoms i that are in the first segment but not in the second segment, and where y_j are information measures of itoms j that are in the second segment but not in the first segment; and

clustering the segments in dependence upon the distances.

110. A method according to claim 108, further comprising the steps of:

initiating a first search on a greater database in response to a first query, the first search generating a plurality of hits,

wherein the segments in the subject database constitute hits in the plurality of hits.

111. A method according to claim 110, further comprising the step of ranking the hits in a particular one of the clusters in dependence upon an information measure of itoms shared by both the hit and the first query,

and wherein the step of identifying the segments to the user in clusters comprises the step of identifying to the user at least one of the hits in the particular cluster selected in accordance with the ranking.

112. A method according to claim 108, wherein the step of clustering the segments includes a step of determining a starting and ending position for a particular one of the segments dynamically in dependence upon an information measure of itoms in the particular segment.

113. A method according to claim 112, wherein the step of determining a starting and ending position for the particular segment comprises the steps of:

identifying initial starting and ending positions for the particular segment;

iteratively extending at least one of the starting and ending positions by increments until a determination is made that further expansion would reduce a net information value of the segment, the net information value being positively dependent upon an information measure of itoms included in the segment, and negatively dependent upon tokens not in an itom.