System and method for improved searching on the internet or similar networks and especially improved MetaNews and/or improved automatically generated newspapers

• US 20050114324A1
• Filed: 09/14/2004
• Published: 05/26/2005
• Est. Priority Date: 09/14/2003
• Status: Active Grant

• Current Assignee: Batya Barhon Mayer
• Original Assignee: Batya Barhon Mayer
• Inventors: Mayer, Yaron
• Granted Patent: US 8,589,373 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Google has recently made available at http://news.google.com an automated “newspaper”, which searches continuously about 4,500 news sources, and lets users view automatically generated headlines in a few general areas or lets users search for news by keywords. The automatic determination of which news items or news stories are most important is done by 3 main criteria: In how many sources the news item appeared, how important are the news sources in which it appeared, and how close it is to the top in each of these news sources. However, many problems still remain, such as for example: a. The choice of a single main news source and a single image for each item seems arbitrary to the user and limits the user. b. If the user clicks on the “related items” link for that item the user always gets a linear list of typically hundreds or even more than a thousand links to related news items, sorted either by relevance or by time, however, the new list is now without any images and without any clustering, so that many times news stories that are about the same event or even identical, may appear at different positions in the list of related links, and various other news items may appear between them and are typically also dispersed in various places. This makes it vary hard for the user to take advantage efficiently of the list of related items. The present invention solves the above problem by creating recursive clustering, so that preferably at any level in the tree the user can preferably either choose a specific news item from the cluster or from the shown sub-clusters or continue in the tree. Another improvement is that searching the Meta News by keywords can generate an automatic newspaper in a way similar to the original automatically generated newspaper. Many additional improvements to the concept of automated newspapers and/or news MetaSearch are also shown. Other improvements are suggested for improved shareware MetaSearch, improved Web pages search, and other types of searches.

Claims

1-20. -20. (canceled)

21. In an online search system, a method of improved News Meta-Search over a large number of Online news sources on the Internet or similar networks, comprising at least one of the following steps:

a. Switching between news items from the same cluster or sub-cluster displayed in a given position in an automatically generated newspaper page, wherein said switching is done automatically or with user intervention;

b. Switching between news images from the same cluster or sub-cluster displayed in a given position in an automatically generated newspaper page, wherein said switching is done automatically or with user intervention, and wherein said images are at least one of still images and streaming data;

c. Creating recursively sub-clusters of the displayed clusters or dub-clusters of news items that are related to a certain event, so that at least one of;

1. For each sub-cluster shown the user can either click on a chosen item from that cub-cluster or click on a link for seeing a list of additional items that belong to the sub-cluster. 2. When the user requests to see the list of additional items of the chosen sub-cluster, the new list can be again clustered similarly. 3. When the user requests to see the list of additional items of the cluster, the new list can be again clustered similarly.

22. The method of claim 21 wherein at least one of the following features exists:

a. The recursive sub-clustering continues until there are sufficiently few items in the final sub-category or until the items are too different to group further;

b. If the user searches for keywords in the News Meta Search, the results are displayed recursively in clusters and sub-cluster in a way similar to the automatically generated newspaper page;

c. If the user searches for keywords in the News Meta Search, the results can have all the features that exist in the automatically generated newspaper page;

d. The user can switch between a mode that displays also images and a mode without images;

e. The same news item or same sub-cluster can belong to more than one cluster or sub-cluster, and thus it is shown and/or can be reached from all the sufficiently relevant clusters or sub-clusters to which it is related;

f. Sorting a list of related items by relevance and/or by time and date can be used to create order between and/or within the sub-clusters, without interfering with the cluster structure itself;

g. The user can request to sort the items by at least one of;

1. The country of the source, so that for the news items are ordered or clustered in addition or instead also according to the country of the news source. 2. The level of reliability of the source, so that for the news items are ordered or clustered in addition or instead also according to the reliability of the news source;

h. The user can view a graphical or textual hierarchical representation which shows simultaneously the multi-level structure of clusters and sub-clusters, showing more than two levels of the hierarchy at the same time, or showing the structure down to the end-nodes;

i. The Meta News system automatically chooses only images that are within a certain reasonable range of sizes;

j. As additional new related news items come in, the headlines and/or the images can be automatically updated even if the user does not click on any refresh button;

k. The user gets a different indication when the items or images themselves have changed or new items or images are brought in (compared to the normal swapping between items), and said indication is at least one of sound indication and visual indication of the item that has changed or the new item that has been inserted;

l. The html protocol and/or the html command set is expanded to allow any image to be requested with a given size limit, so that if the original image is bigger it is either truncated automatically to fit in the allowed window, or is automatically downscaled in order to fit completely into the allowed space;

m. The html protocol and/or the html command set is expanded to allow any image to be requested with a given size limit, so that if the original image is bigger it is truncated automatically to fit in the allowed window and wherein for said truncation the improved html protocol allows the web programmer to specify for each image the x-y coordinates of its central point of interest, and/or various heuristics are used by the browser or by the server in order to find the central point of interest automatically;

n. When the switching images contain also streaming data, at least one of the following is done;

1. The automatic switching of images is disabled so that the user has to click on something in order to view related streaming data from a different source or other still images, and 2. Each streaming source remains in the position for a longer time than still images until switching to the next streaming source or to the next still image;

o. The system determines which item to use as the main item of the general cluster by at least one of;

1. First picking the sub-cluster that has the largest number of items and/or the most recent cluster that is big enough relative to other sub-clusters. 2. Picking the item within the chosen first sub-cluster which has the highest average similarity to other items in that sub-cluster and/or belongs to the largest sub-cluster of that sub-cluster and/or is most relevant within the cluster or within the sub-cluster and/or is most recent within the cluster or within the sub-cluster;

p. When requesting News alerts, instead of being able to request only by specific keywords, the user can also at least one of;

1. Mark a cluster or a specific sub-cluster, so that he/she is notified automatically on any new items that belong to that cluster or after sufficient changes have accumulated in the cluster. 2. Use semantic qualifiers. 3. Mark words in a way that indicates that synonyms should also be checked for these words, so that he/she will be notified also about items that contain synonyms of these marked words.

25. The method of claim 21 wherein the user can request to automatically spread still images and/or streaming images of the same cluster or sub-cluster together next to each other so that they can be viewed simultaneously, and at least one of the following features exists:

a. By clicking on or near one of the simultaneous streaming data images the user is transferred to that source to view it normally there;

b. The user can switch the sound between any of the simultaneous streaming data sources;

c. The group of images is automatically and dynamically generated according to the item of interest and according to availability in the various sources, so that images or streaming date can be automatically added or removed accordingly.

27. The method of claim 21 wherein as additional new related news items come in, the headlines and/or the images can be automatically updated even if the user does not click on any refresh button and wherein said automatic updating is done by partial refresh on a need basis by at least one of the following ways:

a. The refresh command is initiated automatically by the site when there is any change in the page, so that the browser can get a refresh even if it didn'"'"'t ask for it;

b. The browser can ask for refresh, but if nothing has changed then the browser gets just a code that tells it to keep the current page or window as is;

c. When the refresh is sent, it is a smart refresh, which tells the browser only what to change on the page instead of having to send the entire page again.

35. The method of claim 21 wherein at least one of the following features exists:

a. In order to enable the multi-level sub-clustering the same or similar principles are applied similarly at all levels, except that in each step they are applied now to the items of the previous cluster or sub-cluster in order to further divide them into additional sub-clusters;

b. In order to improve the clustering ability, the time each item was published is taken into account, with the assumption that the closer the time of publication between them, the higher the chance that two items are dealing with the same event;

c. The temporal words or phrases used in the news item are used to decide when the event occurred, and this time is used to separate between news items that occurred before this time and items that occurred after this time and/or to help decide the similarity between items that might be referring to the same event;

d. The temporal words or phrases used in the news item are used to decide when the event occurred, and in order to analyze the temporal phrases used in the item, the system is able to perform also at least some minimal type of semantic analysis and/or has at least knowledge of the relevant temporal nouns and relevant verbs;

e. For clustering the system analyses the similarity in the occurrence of combinations of two or more words in the headline and/or in the first 1 or 2 sentences and/or in the entire item;

f. When sorting automatically generated news clusters the number of items in each cluster is normalized by the time factor, since clusters that have exited for a longer time would normally have more items than a newer cluster even if the new cluster is more important.

36. The method of claim 21 wherein the system has at least one of:

a. A knowledge base of at least one of;

country names, city names, and other geographical areas;

b. A knowledge base of at least the most common or most important verbs that typically appear in headlines and/or in the first one or two sentences of news items and/or in entire news items;

c. A knowledge base of verbs that uses semantic trees and/or semantic graphs and/or various rules, so that each verb can be characterized by scores on a number of relevant variables or dimensions;

d. A database of synonyms for the comparisons of nouns and/or of verbs, so that the system can know if two words are different or similar even without “

understanding”

their meaning;

e. A knowledge base of major known political names and organizations;

f. The ability to take into account also similarity in words at least in the headlines, even if they are not exactly identical.

42. The method of claim 21 wherein at least one of the following features exists:

a. If the user chooses one of the top level subject categories, the number of clusters is not limited to a specific number, so the user can continue further to the next issues as long as the next cluster or clusters are sufficiently important and/or as long as there are sufficient items and/or sufficient sub-clusters in the next cluster or clusters;

b. Instead of just a few top categories in the automatically generated newspaper, there are also sub-categories, so that choosing a top category can open a submenu that lets the user choose also a sub-category in one or more additional sub-levels;

c. When the user chooses a certain country in the general choices available for the automatically generated newspaper, the system allows the user to choose if he wants more emphasis on news related to this country, more emphasis on sources related to this country, or both;

d. Automatic recursive clustering and sub-clustering is used also in normal web searches and/or in newsgroup searches;

e. The switching between items and/or between images of the same cluster and/or sub-cluster is done locally on the user'"'"'s browser, so that after the images and the headlines are loaded for the first time, the browser can easily switch between them from the local cache and/or from internal memory, until real new data is sent again;

f. Sending the same data to many users or to many servers or mirror sites at the same time identical data packets are sent only once in each condensed packet by creating automatically multicast groups and/or sub-groups and assigning automatically users to them;

g. Instead of keeping the list of target addresses in the condensed packet, the list of targets is sent first to a server or router in the target area, and the following condensed packets for the same group can be sent to that server or router without the list of targets and instead the condensed packets include a code that identifies the multicast group and/or the desired list of targets that the server or router in that area already has.

50. The method of claim 21 wherein when displaying automatically generated news at least one of the following features exists:

a. The user can define the desired time span to cover;

b. The user can mark certain news items or clusters or sub-clusters as no longer interesting;

c. The user is notified automatically when a sufficient number of new clusters has become available and/or when a sufficient number of new items has become available in existing clusters which the user has not requested to remove, and/or in clusters which the user has marked as especially interesting for him/her;

d. The user can review historically the automatically generated items that were on the automatic newspaper or on a specific section of the automatic newspaper, by at least one of;

1. Specifying the time in hours and/or days to jump back, 2. Specifying the time in hours and/or days to jump back, and then jumping automatically the same interval in the next jumps, 3. Specifying a specific date and/or time, 4. The user can press some link which automatically takes him/her back at each step automatically according the amount of change.

23. (canceled)

24. (canceled)

26. (canceled)

28. In an online search system, an improved Online metasearch method comprising at least one of the following:

a. An improved Shareware Meta Search method wherein shareware programs appear in higher places in the search results according to how many of the included shareware sites list them, and at least one of the following;

In which position they are listed for the given searched keywords;

How important the shareware site is;

How many times they were already downloaded;

The shareware site'"'"'s rating for the shareware.

b. An improved Online MIDI files Meta Search method wherein at least one of the following features exists;

After the system chooses a set of results that are sufficiently close to the search string, the system automatically sorts the song names by the most popular in descending order, and After choosing the desired file name, the system sorts available versions of that sons in descending order by the number of links available for each file size, so the user can reach immediately the desired MIDI file that has the best chance of being the best version of the desired song

29. The method of claim 28 wherein shareware programs appear in higher places in the search results according to how many of the included shareware sites list them, and at least one of the following:

1. In which position they are listed for the given searched keywords, 2. How important the shareware site is, 3. How many times they were already downloaded and 4. The shareware site'"'"'s rating for the shareware, and whertein at least one of the following features exists;

a. The number of downloads data is normalized by the general amount of listed downloads in that included shareware site;

b. The included shareware site'"'"'s rating for the shareware is based on user votes and/or on their own editorial stuff;

c. If the shareware'"'"'s site rating of a given shareware is based on user votes, the shareware site'"'"'s rating is given higher weight than ratings based on editorial decision, if the number of votes is given and is sufficiently large;

d. If the same shareware appears in different versions in various shareware sites then the system at least one of;

1. Uses also the rankings of the previous versions for determining the score for that shareware in general. 2. Uses in this case clusters and sub-clusters like in the meta-news, and 3. Treats each version independently like any other shareware.

30. (canceled)

31. An improved Online web pages search method comprising at least one of the following steps:

a. Taking into account the link relations between web pages for scoring the page but does not reduce the value of a link according to the number of other outgoing links in the linking pages, or reduces the value of a link according to the number of other outgoing links in the linking pages only slightly;

b. Improving slightly the rank for a page that has many outgoing links;

c. Taking into account usage statistics but uses it only for modifying the value of the link in the linking page but not for modifying directly the ranking of a page;

d. Taking into account usage statistics but uses it with one or more thresholds, so that usage lower than a certain factor does not continue to lower the score, and/or usage higher than a certain factor does not continue to increase the score;

e. Using also the anchor text of inbound links to determine the relevance of the linked page to the searched keywords and includes at least some semantic analysis of the anchor href text and/or also at least the surrounding or preceding nearby text, in order to be able to identify at least part of the meaning and/or avoid certain pitfalls that are relevant to the interpretation of the real meaning of the link;

f. Using also the anchor text of inbound links to determine the relevance of the linked page to the searched keywords and at least takes into account some basic language structures such as negation words or modifying words;

g. Allowing the user to define various parameters for scoring the results, wherein said parameters are at least one of;

The relative weight of usage statistics, the amount of reduction of the importance of a link as a result of the total number of links on the linking page, and, the amount of taking into consideration the newness of a web page so that less links to it are required;

h. Automatically identifying if a page is an alphabetic directory and gives higher weight to a link that is closer to the top of the page unless that page is an alphabetic directory;

i. Checking also if incoming links reside on the same IP address (even if the domain name is different) and their domain is owned by the same person or organization, in order to determine the value of the incoming links;

j. Taking into account the number of incoming links for each page and also the time factor of how long the page has existed is taken into account for determining the weight given to the number of links;

k. Taking automatically into account also the synonyms of the requested keywords, by at least one of;

1. Automatically including in the search results also pages that contain synonyms or close synonyms of the requested keywords. 2. Asking the user if he would like to include in the search results automatically also pages that contain close synonyms of the requested search keywords and remembers that as default for that user for following searches, and 3. Checking at least close synonyms of the user'"'"'s search keywords, and if there are more and/or better results with the synonyms then the system asks the user if he wants to switch over to the results of the search that was based on the synonyms, and/or asks the user if he wants to integrate the current results with the results of the search that was based on the synonyms;

l. Using semantic qualifiers when using keyword search for letting the search engine know that certain words are not part of the search string itself but are intended to act as the semantic qualifier;

m. Allowing the user to define words in the search string that are preferred but not necessary.

33. The method of claim 31 wherein the number of incoming links for each page is taken into account and also the time factor of how long the page has existed is taken into account for determining the weight given to the number of links, and at least some threshold is used, so that 0 links or too few links are not compensated by the fact that the page is new, but if the new page has already sufficient valid links, then the newness of the page is taken into account in requiring less links at that stage.

45. The method of claim 31 wherein the user can use semantic qualifiers when using keyword search for letting the search engine know that certain words are not part of the search string itself but are intended to act as the semantic qualifier, and wherein at least one of the following features exists:

a. Brackets or other special marks are used for marking the semantic qualifier words;

b. In order to improve the search results based on the semantic qualifiers, the search engine can at least one of;

1. Use knowledge about synonyms and/or taxonomies. 2. Use information from the link structure and/or other methods in order to identify the relevant results clusters and/or sub-clusters, and then can check how relevant the qualifier word or words is to the clusters and/or sub-clusters and/or how relevant its synonyms or related taxonomy items are to the clusters and/or sub-clusters.

47. The method of claim 31 wherein the user can define words in the search that are preferred but not necessary and wherein at least one of the following features exists:

a. Normal keywords in the search string are by default words that have to be in the page, but specially marked words are only preferably in the page, so that they add to the page'"'"'s rank if they are there, but they do not have to be there;

b. Normal keywords in the search string are by default words that have to be in the page, but the user can also mark words for which synonyms are also automatically acceptable, and at least one of;

1. It doesn'"'"'t matter if the word itself or one of its close synonyms appears on the page. 2. The amount of closeness is also taken into account. 3. The score is still higher if the word itself appears.

48. The method of claim 31 wherein semantic tags can be added to web pages for improved searching abilities and at least one of the following is done:

a. In order to facilitate transfer to explicit semantic tagging and/or structuring developers are supplied with software that tries to automatically convert HTML pages into semantically structured and/or tagged format, so that the process is based on semantic analysis of the page, and the conversion software relies on precompiled databases of world knowledge and asks the user questions when needed or when it is not sure enough;

b. Semantic information is added in tags that do not change anything else in the HTML format and can simply be ignored by ordinary browsers, or ignored except for some information that is useful also for the human readers;

c. Each page contains at least a one or a few basic tags which indicate at least who is the main entity behind the page and what is the page'"'"'s main purpose;

d. When performing the search, the user can tell the search engine if he/she wants to find reviews about the subject of the search, articles, service providers, links, or other types of pages and/or sites, and the search engines tries to use also the information from said tags to answer such queries;

e. Said tags can be generated by at least one of;

1. Automatically by semantically aware search engines during the generation of the index, so that the search engine uses information from the page itself and/or from other pages in the same site in order to understand in general what the site and/or each page of it is about and generate those tags, 2. The tags are suggested to the page owner by automatic conversion software, 3. These tags can be easily manually added by each web page owner, with the incentive that such pages can be more easily cataloged properly by web pages and therefore be more easily found by user.

52. The method of claim 31 wherein synonyms can be automatically included in the results and/or automatically recommended to the user, and at least one of the following features exists:

a. If the search engine finds more than one word in the search string for which synonyms should be offered, the search engine offers in the same step an improved search string which includes the best synonym for each of these keywords;

b. The search engine offers a number of alterative suggested search strings, and near each of them the estimated number of results and/or the number of results with a Page rank beyond a certain value, and/or for the average Page Rank of the top N results, and the user can click on the search string that he/she most likes;

c. The search engine does not display specific alternative synonyms but simply marks to the user for which of the words in the search string the search engine recommends allowing synonyms and then the user can authorize this without having to look at specific suggested synonyms;

d. The search engine can display all the alternative synonyms that are suggested for each of the words for which adding synonyms is recommended, in one new search string which the user can authorize;

e. In the integrated results the search engine can give a higher score to pages that contain a more closer synonym than pages with a less close synonym and/or an even higher score if the page contains the exact synonym;

f. A close synonym does not reduce the score compared to the original word, at least if the user was asked explicitly about adding a specific synonym and agreed.

53. The method of claim 31 wherein if the search string does not produce any results or produces only very few results, the search engine can at least one of:

a. Automatically suggest to the user which alternative of removing one of the search words will lead to the best increase in results;

b. If the search string that produced little or no results contained words that were marked as having to be next to each other, the search engine can automatically suggest to the user which small change can lead to the best improvement in results, wherein said small change is at least one of;

1. Removal of a single word out of the quotation marks, and suggesting the best word for this and/or indicating the estimated number of results and/or the number of highly ranking results and/or the average page ranks of the top N results for each alternative), and 2. Relaxing the nearness requirement so that the words that were supposed to be next to each other can be at larger distance;

c. Automatically suggest to the user that quotation marks will always automatically allow a distance of up to a recommended number of words distance between each two words, and this preferably remains the default for the following searches until the user requests to change this, and/or this is the normal default.

54. The method of claim 31 wherein the search engine can know synonyms from at least one of:

a. From one or more thesauruses;

b. Through automatic learning, based on various patterns and/or statistics;

c. Through automatic learning, based on similar patterns or word combinations that appear in pages that are close on the recursive clustering analysis, wherein the clustering is based on keyword analysis and/or at least partial semantic analysis and/or incoming link analysis and/or outgoing links analysis;

d. Semantic understanding of pages where relevant definitions or connections are used;

e. Understanding from pages where two or more phrases are used for the same thing.

55. The method of claim 31 wherein at least one of the following features exists:

a. In normal web searches and/or in searching the dictionary and/or the thesaurus in the word processor the user can search for synonyms of words with a specific pattern or length, or for the word itself when partial data exists, so that the user can use question marks or any other convenient designation to designate unknown characters;

b. The user can use wild cards in domain names, in the location window of the browser or in the search engine'"'"'s search line, in order to get results only from specific sites or types of sites;

c. The user can use marks for defining the exact maximum distance in words between two or more search keys;

d. The user can use qualifiers such as BEFORE or AFTER, which indicate that one or more keywords must appear in the text before or after a group of one or more other keywords;

e. The user can use a general number designator, which means that the user is expecting a number pattern (without a specific range limit) at a certain position or without limiting the position;

f. The user can use a general number designator and can add a tag that causes the results to be sorted according to this number (or numbers, if more than one number pattern is specified);

g. Instead of using ‘

or’ and

brackets the user can use ‘

/’

to designate ‘

or’

for a single word;

h The search engine'"'"'s response to typing errors is improved by dealing also with typing errors in the first letter, by at least one of;

1. Taking into account the most common types of errors. 2. Talking into account typical errors that can occur because of close characters on the keyboard. 3. Analysis of similar words that lead to significantly more results.

57. The method of claim 31 wherein when the search engine displays advertisements in addition to search results, based on the keywords which the user used, the choice of appropriate advertisements takes into account not just the keywords themselves but also semantic and/or context related information, and this is accomplished by at least one of:

a. Taking into account the order of the keywords which the user used on the search and looking for qualifiers, such as “

all”

, “

not”

, “

most”

, and/or similar qualifiers;

b. Determining the semantic meaning and/or the context based on the search results which the user clicks on, so that the advertisements are updated accordingly when the user requests the next page of results or repeats the search with similar keywords;

c. The advertisements can be dynamically updated also on the same results page which the user already has, so that while the user opens new windows for some of the links and explores them, the original page is already updated based on the links which the user clicked on, by automatic refresh and/or by automatic updates to part or parts of the page and/or streaming data to such parts.

32. (canceled)

34. (canceled)

37-41. -41. (canceled)

43. (canceled)

44. (canceled)

46. (caneled).

49. (canceled)

51. (canceled)

56. (canceled)

58. (canceled)

59. (canceled)