System and method for automating categorization and aggregation of content from network sites

• US 8,271,495 B1
• Filed: 07/09/2004
• Issued: 09/18/2012
• Est. Priority Date: 12/17/2003
• Status: Active Grant

• Current Assignee: Topix
• Original Assignee: Topix
• Inventors: Skrenta, Richard, Dole, Bryn, Markson, Thomas, Truel, Robert
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Khakhar, Nirav K
• Application Number:
  US10/888,787
• Time in Patent Office: 2,993 Days
• Field of Search: None
• US Class Current: 707/738
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

A plurality of content items are retrieved from multiple network sites. Content from each content item is programmatically analyzed in order to associate that content item with one or more categories. The one or more categories may be part of a larger set of predefined categories. A network page is assigned to one or more corresponding categories in the set of predefined categories. At least some content is provided on the network page using one or more content items that were associated with the one or more categories assigned to that network page.

Claims

1. A method for providing aggregated content from a network, the method being implemented by one or more processors that perform steps comprising:

creating a category definition for each of a plurality of categories, the category definition for each category comprising a category name and one or more terms, wherein each of the one or more terms associated with a particular category definition (1) identifies a term that is pertinent in determining whether an article should be associated with a particular category associated with the particular category definition, and (2) excludes any terms that are not pertinent in determining whether an article should be associated with the particular category associated with the particular category definition;

retrieving a plurality of articles from over a network;

analyzing each article of the plurality of articles, in order to associate each article of the plurality of articles, with one or more categories in the plurality of categories, wherein analyzing each article includes associating a particular article of the plurality of articles with a corresponding category based on (1) a presence of one or more character strings that appear in the particular article, wherein each of the one or more character strings (i) correspond to a particular term in the one or more terms in the category definition of the corresponding category, and (ii) is not the category name of the corresponding category, and (2) one or more additional criteria to weight the presence of said one or more character strings over other character strings that correspond to a term of a category definition of another category;

prior to analyzing each article, of the plurality of articles, assigning a portion of each of a plurality of web pages to one or more corresponding categories in the plurality of categories, so that each category in the plurality of categories is assigned to at least a portion of the plurality of web pages;

displaying, on each of the plurality of web pages, at least a portion of individual articles that have been associated with the one or more corresponding categories assigned to the web page, wherein each web page, of the plurality of web pages, displays at least one category name that is assigned thereto.

2. The method of claim 1, wherein the step of retrieving each article, of the plurality of articles, includes crawling network sites on the Internet.

3. The method of claim 1, wherein the step of retrieving each article, of the plurality of articles, includes crawling a plurality of news sites on the Internet to retrieve the plurality of articles, and wherein analyzing each article, of the plurality of articles, includes determining a current event category for each article, of the plurality of articles.

4. The method of claim 3, wherein the step of analyzing each article, of the plurality of articles, includes the step of inspecting text in each article, of the plurality of articles, and using the inspected text to assign the one or more categories to that article.

5. The method of claim 1, wherein the step of assigning at least a portion of each of a plurality of web pages to one or more corresponding categories includes assigning a first category to a first web page, and forming at least part of a network address for the first web page to indicate the first category.

6. The method of claim 1, wherein the step of displaying, on each of the plurality of web pages, at least a portion of individual articles includes displaying, for each of the plurality of web pages, segments of one or more articles that have been analyzed to be associated with a corresponding category that is assigned to that web page.

7. The method of claim 1, wherein the step of displaying, on each of the plurality of web pages, at least a portion of individual articles includes providing, on each web page, links that are selectable by users to access the articles from one or more web sites that host the articles.

8. The method of claim 1, wherein the step of displaying, on each of the plurality of web pages, at least a portion of individual articles includes forming, on each web page, headlines for each of the one or more articles along with a link that is selectable to view the one or more articles from that web page.

9. The method of claim 1, further comprising the step of automatically updating at least some of the articles on each of the plurality of web pages by repeatedly analyzing new articles that are made available at the plurality of web sites in order to associate one or more categories from the plurality of categories with each of the new articles.

10. The method of claim 1, wherein the step of displaying, on each of the plurality of web pages, at least a portion of individual articles includes selecting, for a given one of the plurality of web pages, which articles in the plurality of articles are to be provided on at least a portion of the given web page.

11. The method of claim 10, wherein the step of selecting, for a given one of the plurality of web pages, which articles in the plurality of articles are to be provided on at least a portion of the given web page further comprises programmatically selecting a position on the web page for one or more articles that are selected to be displayed on the portion of the web page.

12. The method of claim 1, wherein the step of analyzing each article, of the plurality of articles, is performed in order to associate each article with a number of categories in a set comprising one thousand or more categories.

13. The method of claim 1, wherein each article, of the plurality of articles, includes associating one or more articles with a genre category and one or more sub-categories.

14. The method of claim 1, wherein the plurality of categories include a specific category selected from (i) a category about a geographic location, (ii) a category about a sport or sport entity, (iii) a category about a celebrity, (iv) a category about a current event topic, and (v) a category about a field of science or medicine.

15. The method of claim 1, further comprising:

on each of the plurality of web pages, selecting the individual articles that are made available on that web page from a larger set of articles that have been associated with the category of that web page.

16. The method of claim 1, wherein displaying, on each of the plurality of web pages, at least portions of individual articles includes displaying, on each of the web pages, a phrase, sentence, or summary of each of the individual articles along with a link to that article.

17. The method of claim 16, wherein the link is in the form of a headline or title.

18. The method of claim 16, wherein displaying the phrase, sentence, or summary of each of the individual articles includes displaying the phrase, sentence, or summary in a separate segment on each of the web pages.

19. The method of claim 1, wherein the step of associating the article with a corresponding category includes excluding a candidate category from being associated with the article, even though the excluded category includes the character string as one of the terms in its category definition.

20. The method of claim 1, wherein creating the category definition includes creating a given category by (i) specifying a set of terms that are required to be present in an analyzed article in order for the given article to be associated with given category, and (ii) specifying a set of terms that, if present in that analyzed article, are indicators that the analyzed article should be associated with the given category when the set of required terms are also present.

21. The method of claim 1, wherein the step of displaying, on each of the plurality of web pages, includes marking individual web pages with a header or keyword that corresponds to the associated category name of that web page.

22. The method of claim 1, wherein the step of displaying, one each of the plurality of web pages, includes enabling the web page to be retrievable or selectable from a website by listing or identifying the web page on the website by its category name.

23. A method for providing aggregated content from a network, the method being implemented by one or more processors that perform steps comprising:

retrieving a plurality of content items from one or more network sites;

for each of the plurality of content items that are retrieved, programmatically analyzing content contained in each content item in order to associate that content item with one or more categories in a plurality of categories,

wherein each of the one or more categories, in the plurality of categories, is associated with a category definition, wherein the category definition for each of the one or more categories comprises a category name and one or more terms that each (1) are pertinent in determining whether a content item should be associated with a particular category associated with the particular category definition, and (2) exclude any terms that are not pertinent in determining whether a content item should be associated with the particular category associated with the particular category definition,

wherein the plurality of categories include categories that correspond to a plurality of geographic locations, and wherein programmatically analyzing content contained in each content item includes determining that at least some of the content items are each associated with a corresponding geographic location by identifying words, terms, or names in the analyzed content other than a proper name or zip code of the geographic location, and wherein analyzing each content item includes evaluating additional criteria to weight the presence of the identified words, terms or names in the analyzed content over other words, terms or names that correspond to a term of a category definition of another category.

24. The method of claim 23, wherein the step of programmatically analyzing content contained in each content item includes inspecting text contained as content in each content item that is separate from a geographical marker displayed in the beginning or end of the content item.

25. The method of claim 24, wherein the step of retrieving a plurality of content items includes crawling a plurality of network sites on the Internet.

26. The method of claim 24, wherein the step of inspecting text includes determining how frequent the particular character string appears in the content item.

27. The method of claim 24, wherein the step of inspecting text includes:

identifying one or more candidate terms in a given content item, and

determining a likelihood that a category associated with the candidate term is an appropriate category for the given content item.

28. The method of claim 27, wherein the step of determining a likelihood that a category associated with the candidate term is an appropriate category for the given content item includes determining a commonality indicator for candidate term.

29. The method of claim 28, wherein the step of determining a likelihood that a category associated with the candidate term is an appropriate category for the given content item includes determining a frequency in which the candidate term appears in the content item.

30. The method of claim 27, wherein the step of determining a likelihood that a category associated with the candidate term is an appropriate category for the given content item includes determining a proximity of the candidate term to the beginning of the text for that content item.

31. The method of claim 23, further comprising the step of displaying content on one or more of the network pages that includes, at least in portions, a cluster of content items that have been retrieved from a plurality of news sites, and assigning each of the one or more network pages to a category of one of the corresponding geographic locations, so that each of the one or more network pages includes at least portions of content retrieved from multiple news sites and pertinent to the corresponding geographic location of that network page.

32. The method of claim 23, wherein the geographic locations are identified by one of a zip code or a name of a location.

33. A method for providing aggregated content from a network, the method comprising the steps of:

(a) assigning each category, in a set of categories, with a corresponding network location, wherein the set of categories includes one thousand or more categories;

(b) retrieving a plurality of content items from one or more network sites;

(c) for each of the plurality of content items, programmatically analyzing each content item in order to associate that content item with one or more categories in the set, wherein programmatically analyzing comprises;

creating a category definition for each category in the set of categories, wherein the category definition for each category in the set of categories includes a category name and one or more terms, wherein each of the one or more terms associated with a particular category definition (1) identifies a term that is pertinent in determining whether a content item should be associated with a particular category associated with the particular category definition, and (2) excludes any terms that are not pertinent in determining whether a content item should be associated with the particular category associated with the particular category definition, and

associating each of the plurality of content items with at least one of the one or more categories based on (1) a presence of one or more character strings that appear in the particular article, wherein each of the one or more character strings (i) correspond to a particular term in the one or more terms in the category definition of the corresponding category, and (ii) is not the category name of the corresponding category, and (2) one or more additional criteria to weight the presence of said one or more character strings over other character strings that correspond to a term of a category definition of another category; and

(d) displaying one or more content items for each category in the set at the corresponding network location for that category.

34. The method of claim 33, wherein programmatically analyzing each content item includes programmatically analyzing text in each content item.

35. The method of claim 34, wherein the step (c) includes associating a plurality of content items with a first category in the set during a given time period, and wherein the method further comprises selecting a subset of the plurality of content items for the first category to be displayed on the corresponding network location of the first category.

36. The method of claim 35, further comprising programmatically selecting a position on a presentation provided at the network location for one or more of the content items in the subset.

37. The method of claim 35, further comprising automatically repeating steps (b), (c) and (d) in order to update what content items appear on one or ore of the corresponding network locations.

38. A non-transitory computer readable medium storing instructions for providing aggregated content from a network, wherein when executed by one or more processors, the instructions cause the one or more processors to perform the steps comprising:

creating a category definition for each of a plurality of categories, the category definition for each category comprising a category name and one or more terms, wherein each of the one or more terms associated with a particular category definition (1) identifies a term that is pertinent in determining whether an article should be associated with a particular category associated with the particular category definition, and (2) excludes any terms that are not pertinent in determining whether an article should be associated with the particular category associated with the particular category definition;

retrieving a plurality of articles from over a network;

analyzing each article of the plurality of articles, in order to associate each article of the plurality of articles, with one or more categories in the plurality of categories, wherein analyzing each article includes associating a particular article of the plurality of articles with a corresponding category based on (1) a presence of one or more character strings that appear in the particular article, wherein each of the one or more character strings (i) correspond to a particular term in the one or more terms in the category definition of the corresponding category, and (ii) is not the category name of the corresponding category, and (2) one or more additional criteria to weight the presence of said one or more character strings over other character strings that correspond to a term of a category definition of another category;

prior to analyzing each article, of the plurality of articles, assigning a portion of each of a plurality of web pages to one or more corresponding categories in the plurality of categories, so that each category in the plurality of categories is assigned to at least a portion of the plurality of web pages;

displaying, on each of the plurality of web pages, at least a portion of individual articles that have been associated with the one or more corresponding categories assigned to the web page, wherein each web page, of the plurality of web pages, displays at least one category name that is assigned thereto.

39. A method for providing aggregated content from a network, the method being implemented by one or more processors that perform steps comprising:

performing an analysis on a text content of each of a plurality of content items, wherein the plurality of content items are provided at a plurality of network locations on one or more network sites, wherein performing the analysis comprises creating a category definition for each category in a set of categories, wherein the category definition for each category in the set of categories includes a category name and one or more terms, wherein each of the one or more terms associated with a particular category definition (1) identifies a term that is pertinent content item should be associated with a particular category associated with the particular category definition, and (2) excludes any terms that are not pertinent in determining whether a content item should be associated with the particular category associated with the particular category definition;

for at least some of the plurality of content items, determining a geographic location that is pertinent to the text content of that content item based at least in part on the analysis, including indentifying one or more words, terms or names that are associated with the geographic location but which are not a proper name of the geographic location, wherein the geographic location pertinent to a particular content item is determined by associating the particular content item which a category in the set of categories using the category definitions for the set of categories;

wherein performing an analysis on a text content includes evaluating additional criteria to weight the presence of the identified words, terms or names in the analyzed content over other words, terms or names that correspond to a term of a category definition of another category; and

generating a presentation for each of a plurality of geographic locations, wherein each presentation makes available at least a portion of one or more content items that have been determined to be pertinent to that geographic location.

40. The method of claim 39, wherein generating a presentation includes generating a webpage.

41. The method of claim 39, further comprising associating the presentation for each of the geographic locations with a corresponding zip code for that geographic location.

42. The method of claim 39, further comprising, for at least some of the plurality of content items, determining a subject category of the content item based at least in part on the analysis, and wherein the method further comprises making available at least a portion of each of the at least some of the plurality of content items available on a presentation that is specific to the subject category.

43. The method of claim 42, further comprising making at least a portion of a given one of the plurality of content items available for both the presentation of the determined geographic location and the presentation of the determined subject category.

44. The method of claim 39, wherein generating a presentation for each of a plurality of geographic locations includes providing a web page that is associated with a zip code for at least some the plurality of geographic locations.

45. A method for providing aggregated content from a network, the method being implemented by one or more processors and comprising the steps of:

retrieving a plurality of content items from one or more network sites;

for each of the plurality of content items that are retrieved, analyzing content contained in each content item in order to associate that content item with one or more categories in a plurality of categories, wherein each category in the plurality of categories is associated with a category definition that comprises a category name and one or more terms, wherein each of the one or more terms associated with a particular category definition (1) identifies a term that is pertinent in determining whether a content item should be associated with a particular category associated with the particular category definition, and (2) excludes any terms that are not pertinent in determining whether a content item should be associated with the particular category associated with the particular category definition, and wherein analyzing content contained in each content item includes (i) comparing words present within each content item with the one or more terms associated with each character definition, and (ii) evaluating additional criteria to weight the presence of said words present within each content item with words that correspond to a term of a category definition of another category; and

wherein the plurality of categories include categories that correspond to a plurality of names of persons or places, and wherein analyzing content contained in each content item includes determining that at least some of the content items are each associated with one or more of the plurality of names, including associating individual content items with a corresponding one of the plurality of names based in part on identification of a character string that (i) corresponds to a term that is defined as being pertinent to the corresponding name, (ii) but not an explicit statement of the corresponding name.

46. The method of claim 45, wherein analyzing a content of each content item in order to associate that content item with one or more categories in a plurality of categories includes associating one or more content items with a genre category and one or more sub-categories.

47. The method of claim 45, further comprising making available, on each of a plurality of network pages, at least portions of individual content items, wherein each network page is associated with a corresponding category from the plurality of categories.

48. The method of claim 47, further comprising:

on each of the plurality of network pages, selecting for display the individual content items that are made available on that network page from a larger set of content items that have been associated with the category of that network page.

49. The method of claim 47, wherein making available, on each of a plurality of network pages, at least portions of individual content items includes displaying, on each of the network pages, a phrase, sentence or summary of each of the individual content items along with a link to that content item.

50. The method of claim 49, wherein the link is in the form of a headline or title.

51. The method of claim 49, wherein displaying the phrase, sentence or summary of each of the individual content items includes displaying the phrase, sentence or summary in a separate segment on each of the network pages.

52. A method for providing aggregated content from a network, the method being implemented using one or more processors that perform steps comprising:

retrieving a plurality of content items from one or more network sites;

for each of the plurality of content items that are retrieved, analyzing content contained in each content item by;

for at least some of the plurality of content items, determining a geographic location that is pertinent to the text content of that content item based at least in part on the analysis, including indentifying one or more words, terms or names that are associated with the geographic location but which are not a proper name of the geographic location, wherein each category in a plurality of categories is associated with a category definition that comprises a category name and one or more terms, wherein each of the one or more terms associated with a particular category definition (1) identifies a term that is pertinent in determining whether a content item should be associated with a particular category associated with the particular category definition, and (2) excludes any terms that are not pertinent in determining whether a content item should be associated with the particular category associated with the particular category definition;

determining that at least some of the content items are associated with one of a plurality of current event topics by comparing words present within each content item with the one or more terms associated with each character definition, wherein at least a portion of the plurality of categories are each associated with each of the plurality of current event topics; and

evaluating additional criteria to weight the presence of the identified words, terms or names that are associated with a geographic location over other words, terms or names that are associated with another geographic location;

and wherein the method further comprises;

generating a presentation for each of a plurality of geographic locations, wherein each presentation makes available at least a portion of one or more content items that have been determined to be pertinent to that geographic location and which are associated with one or more of the current event topics, so that the presentation displays content items that are about current events that pertain to the geographic location of that presentation.

53. The method of claim 52, wherein analyzing a content of each content item in order to associate that content item with one or more categories in a plurality of categories includes associating one or more content items with a genre category and one or more sub-categories.

54. The method of claim 52, further comprising making available, on each of a plurality of network pages, at least portions of individual content items, wherein each network page is associated with a corresponding category from the plurality of categories.

55. The method of claim 54, further comprising:

on each of the plurality of network pages, selecting for display the individual content items that are made available on that network page from a larger set of content items that have been associated with the category of that network page.

56. The method of claim 52, wherein making available, on each of a plurality of network pages, at least portions of individual content items includes displaying, on each of the network pages, a phrase, sentence or summary of each of the individual content items along with a link to that content item.

57. The method of claim 56, wherein the link is in the form of a headline or title.

58. The method of claim 56, wherein displaying the phrase, sentence or summary of each of the individual content items includes displaying the phrase, sentence or summary in a separate segment on each of the network pages.

59. A method for providing aggregated content from a network, the method being implemented using one or more processors that perform step comprising:

for each of a plurality of content items, programmatically analyzing a text of each content item in order to determine a subject of the content item, including identifying at least one term that is required for determining the subject, and one or more terms that are pertinent but not required for determining the subject, wherein each of the one or more terms excludes any terms that are not pertinent in determining the subject, wherein programmatically analyzing includes evaluating one or more additional criteria to weight the presence of the text of each content item that corresponds to a subject over other text that corresponds to a different subject;

associating the content item with at least one of a presentation or a network location that is used to present content about the subject or a category of the subject; and

making at least a portion of the content item available from a presentation provided at the network location.

60. The method of claim 59, wherein making at least a portion of the content item available includes displaying at least one of the portion of the content item, the text of the content item, or a link to the content item.

61. The method of claim 59, wherein the subject corresponds to a person, a place, or a current event topic.

62. A method for providing aggregated content from a network, the method being implemented using one or more processors and comprising the steps of:

defining a plurality of geographic categories, each geographic category corresponding to a geographic location, each geographic category including a category definition that comprises (i) one or more category names that include a proper name of a geographic location that corresponds to that category, and (ii) one or more words, terms, and/or names other than the one or more category names, wherein each of the one or more words, terms, and/or names associated with a particular category definition (1) identifies a term that is pertinent in determining whether a content item should be associated with a particular category associated with the particular category definition, and (2) excludes any terms that are not pertinent in determining whether a content item should be associated with the particular category associated with the particular category definition;

retrieving a plurality of content items from one or more network sites;

for each of the plurality of content items that are retrieved, programmatically analyzing content contained in each content item in order to associate that content item with one or more of the plurality of geographic categories, wherein programmatically analyzing the content of each content item includes identifying, from the content of the analyzed content item, one or more words, terms, and/or names that are part of the definition of the associated geographic category and which are different than the name of the associated geographic category, wherein programmatically analyzing the content of each item includes evaluating additional criteria to weight the presence of the one or more words, terms, and/or names that are part of the definition of the associated category over other words, terms, and/or names that are part of a definition of another category.

63. The method of claim 62, wherein the one or more category names of the geographic category also include a zip code.

64. The method of claim 62, wherein the one or more words, terms, and/or names other than the category name include a historical name or nickname of the geographic location of the geographic category.

65. The method of claim 62, wherein the one or more words, terms, and/or names other than the category name includes a landmark of the geographic category.

66. The method of claim 62, further comprising providing a website comprising a plurality of nodes that each include one or more web pages, wherein individual nodes of the website (i) correspond to individual geographic categories, and (ii) are displayed by the category name of the geographic category.

67. The method of claim 66, wherein the website is navigatable to locate the individual nodes by specification of the category name for the geographic category.

68. The method of claim 67, wherein for at least some of the geographic categories, the category name for the geographic category includes either a proper name or a zip code of the corresponding geographic location that corresponds to that geographic category.

69. The method of claim 66, further comprising assigning the category name of at least some of the geographic categories to include a corresponding zip code, and wherein the method further comprises enabling a user to specify a zip code at the website in order to navigate to a web page that is included in the node that is assigned to the geographic category of the zip code.

70. A method for providing aggregated content from a network, the method being performed using one or more processors and comprising steps of:

for at least some of a plurality of geographic categories, defining that geographic category using one or more words, terms or names that are (i) known to be associated with a corresponding geographic location of that geographic category, (ii) but are not a proper name of the geographic category, wherein each of the one or more words, terms or names associated with a particular category (1) is pertinent in determining whether a content item should be associated with the particular category, and (2) excludes any terms that are not pertinent in determining whether a content item should be associated with the particular category;

performing an analysis on a text content of each of a plurality of content items, wherein the plurality of content items are provided at a plurality of network locations on one or more network sites;

wherein performing the analysis includes identifying, from the text content of individual content items, one of the one or more words, terms or names that are used to define one or more of the geographic categories, and wherein performing the analysis includes evaluating one or more additional criteria to weight the presence of the one or more words, terms or names that are used to define one or more of the geographic categories over other words, terms or names that are used to define another geographic category;

as a result of performing the analysis, associating one or more content items with one or more of the plurality of geographic categories;

generating a presentation for at least some of the plurality of geographic categories, wherein each presentation makes available at least a portion of one or more content items that are associated with that geographic category.