Updating taxonomy based on webpage

• US 8,645,384 B1
• Filed: 05/05/2010
• Issued: 02/04/2014
• Est. Priority Date: 05/05/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Testa, Christopher, Mote, Nicolaus, Juang, Philo
• Primary Examiner(s): HO, BINH VAN
• Application Number:
  US12/774,448
• Time in Patent Office: 1,371 Days
• Field of Search: 707/738
• US Class Current: 707/738
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions

Abstract

According to an example implementation, a computer-implemented method may include extracting, by a computing device, structured content from a website, determining a recent taxonomy by applying category rules to the structured content, the recent taxonomy including multiple categories and a new category, and updating a stored taxonomy based on the determined recent taxonomy by adding the new category to the stored taxonomy.

Claims

1. A computer-implemented method comprising:

extracting, by a computing device, structured content from a website associated with a stored taxonomy, the structured content including at least a first menu and a second menu, contents of the second menu depending on a selection from contents of the first menu;

determining a recent taxonomy by applying category rules to the structured content, the category rules being customized for a structure of a page of the website including the first menu and the second menu, the category rules dictating that contents of the second menu represent subcategories of at least one known category represented by contents of the first menu, the recent taxonomy including multiple known subcategories of the known category and a new subcategory represented by a new item on the second menu; and

updating the stored taxonomy based on the determined recent taxonomy by adding the new subcategory to the stored taxonomy.

2. The method of claim 1, wherein the stored taxonomy did not include the new subcategory before the updating.

3. The method of claim 1, wherein the extracting structured content from the website comprises crawling the website.

4. The method of claim 1, wherein the extracting structured content from the website comprises periodically extracting structured content from the website.

5. The method of claim 1, wherein the determining the recent taxonomy comprises applying the category rules to the structured content, the structured content including structured HyperText Markup Language (HTML) content.

6. The method of claim 1, wherein:

the extracting includes extracting structured HyperText Markup Language (HTML) content from the website; and

the determining the recent taxonomy comprises inferring at least the new subcategory by applying the category rules to the structured HTML content.

7. The method of claim 1, wherein:

the structured content includes significant keywords for at least one of the multiple known subcategories and the new subcategory, and

the determining the recent taxonomy comprises determining categories and/or subcategories based on the category rules dictating that at least some of the significant keywords correspond to categories and/or subcategories.

8. The method of claim 1, wherein the category rules are customized based on the stored taxonomy and a stored version of the website.

9. The method of claim 1, further comprising storing data files based on the updated stored taxonomy.

10. The method of claim 1, further comprising generating a key for accessing data files based on the updated stored taxonomy.

11. The method of claim 1, further comprising presenting advertisements to a user based on the updated stored taxonomy.

12. The method of claim 1, further comprising:

maintaining a first index of websites to crawl for a first stored taxonomy; and

maintaining a second index of websites to crawl for a second stored taxonomy,

wherein the extracting comprises;

extracting structured content from websites identified by the first index; and

extracting structured content from websites identified by the second index;

wherein the determining comprises;

determining a first recent taxonomy by applying first category rules to the structured content from the websites identified by the first index, the first category rules being customized for a structure of at least one page from the websites identified by the first index, the first recent taxonomy including multiple known subcategories associated with the first stored taxonomy and the new subcategory; and

determining a second recent taxonomy by applying second category rules to the structured content from the websites identified by the second index, the second category rules being customized for a structure of at least one page from the websites identified by the second index, the second recent taxonomy including only multiple known categories and/or categories associated with the second stored taxonomy;

wherein the updating includes updating the stored first taxonomy based on the determined recent taxonomy by adding the new subcategory to the stored second taxonomy.

13. The method of claim 1, further comprising:

maintaining a first index of websites to crawl for a first stored taxonomy; and

maintaining a second index of websites to crawl for a second stored taxonomy,

wherein the extracting comprises;

extracting structured content from websites identified by the first index; and

extracting structured content from websites identified by the second index;

wherein the determining comprises;

determining a first recent taxonomy by applying first category rules to the structured content from the websites identified by the first index, the first category rules being customized to;

associate each hyperlinked text within an object on a first page within the identified websites with a category associated with the first stored taxonomy; and

associate each tooltip on a second page within the identified websites with a category, the tooltips on the second page within the identified websites including multiple known subcategories associated with the first stored taxonomy and the new subcategory; and

determining a second recent taxonomy by applying second category rules to the structured content from the websites identified by the second index, the second category rules being customized to associate each selection from a drop-down menu on a first page within the identified websites with a category associated with the second stored taxonomy;

wherein the updating includes updating the stored first taxonomy based on the determined recent taxonomy by adding the new subcategory to the stored second taxonomy.

14. The method of claim 13, further comprising removing at least one subcategory from the stored second taxonomy based on a selection associated with the at least one subcategory not being included in the drop-down menu.

15. The method of claim 1, further comprising removing at least one subcategory from the stored taxonomy based on a selection associated with the at least one subcategory not being included in the page of the website.

16. The method of claim 1, wherein all of the items within the second menu are subcategories of the known category.

17. A method comprising:

extracting, by a computing device, structured content from a website associated with a stored taxonomy, the structured content of the website including a table and table entries within the table;

determining a recent taxonomy by applying category rules to the structured content, the recent taxonomy including multiple known subcategories of a known category and a new subcategory of the known category, the category rules being customized for the structure of the website to dictate that the table entries indicate subcategories of the known category and a table entry within the table must be the new subcategory; and

updating the stored taxonomy based on the determined recent taxonomy by adding the new subcategory of the known category to the stored taxonomy.

18. The method of claim 17, wherein the determining the recent taxonomy includes determining the recent taxonomy based on the new subcategory not being included in the stored taxonomy.

19. The method of claim 17, wherein the determining the recent taxonomy includes determining that the new subcategory is a subcategory of the known category based on a location of a term associated with the new subcategory within the structured content.

20. The method of claim 17, wherein the determining the recent taxonomy includes determining that the new subcategory is a subcategory of the known category based on comparing data from the structured content to previously stored and classified data, the previously stored and classified data being associated with the known category.

21. The method of claim 17, further comprising:

comparing the recent taxonomy to the stored taxonomy; and

presenting a candidate subcategory to an administrator based on the recent taxonomy including the candidate subcategory and the stored taxonomy not including the candidate subcategory,

wherein the updating the stored taxonomy comprises updating the stored taxonomy in response to the administrator accepting the candidate subcategory.

22. The method of claim 21, wherein the extracting, determining, comparing, and presenting are performed automatically and without human intervention.

23. A non-transitory computer-readable medium including executable code tangibly embodied thereon, the executable code being configured to, when executed, cause a data processing apparatus to:

extract structured content from a website associated with a stored taxonomy, the structured content including at least a first menu and a second menu, contents of the second menu depending on a selection from contents of the second menu;

determine a recent taxonomy by applying category rules to the structured content, the category rules being customized for a structure of a page of the website including the first menu and the second menu, the category rules dictating that contents of the second menu represent subcategories of at least one known category represented by contents of the first menu, the recent taxonomy including multiple known subcategories of the known category and a new subcategory represented by a new item on the second menu; and

update the stored taxonomy based on the determined recent taxonomy by adding the new subcategory to the stored taxonomy.

24. The non-transitory computer-readable storage medium of claim 23, wherein the stored taxonomy did not include the new category before the updating.

25. An apparatus comprising:

at least one processor; and

at least one memory device, the at least one memory device comprising executable code stored thereon that, when executed by the at least one processor, is configured to cause the apparatus to;

extract structured content from a website associated with a stored taxonomy, the structured content including at least a first menu and a second menu, contents of the second menu depending on a selection from contents of the first menu;

determine a recent taxonomy by applying category rules to the structured content, the category rules being customized for a structure of a page of the website including the first menu and the second menu, the category rules dictating that contents of the second menu represent subcategories of at least one known category represented by contents of the first menu, the recent taxonomy including multiple known subcategories of the known category and a new subcategory represented by a new item on the second menu; and

update the stored taxonomy based on the determined recent taxonomy by adding the new subcategory to the stored taxonomy.

26. The apparatus of claim 25, wherein the stored taxonomy did not include the new category before the updating.