LARGE SCALE CONCEPT DISCOVERY FOR WEBPAGE AUGMENTATION USING SEARCH ENGINE INDEXERS

• US 20110252045A1
• Filed: 04/07/2010
• Published: 10/13/2011
• Est. Priority Date: 04/07/2010
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Vasudevan, Sudharsan, Monga, Rohan, Sambrani, Hemanth, Garg, Priyank Shanker
• Granted Patent: US 8,886,623 B2
• Time in Patent Office: Days
• US Class Current: 707/750
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Disclosed is a method and system for retrieving data; extracting information from the data; learning to disambiguate the extracted information such that a particular sense of each phrase within the extracted information is determined; generating a disambiguation classifier from the learning to disambiguate step, the disambiguation classifier configured to determine a sense of a phrase within a document; learning to select a portion of the information as being relevant to a theme of the data; generating a selection classifier from the learning to select step, the selection classifier configured to select a topic in a document that is relevant to a theme of the document; and using the disambiguation classifier and the selection classifier by an indexing computer to determine a set of topics from a web document retrieved by the indexing computer.

Claims

1. A method comprising:

retrieving, by a training computer, training data comprising a plurality of web documents;

extracting, by the training computer, information from the training data, the extracted information comprising a plurality of phrases extracted from each document of said plurality of web documents;

learning, by the training computer, to disambiguate the extracted information by analysis of a context derived from words proximate each phrase such that a particular sense of each phrase of the plurality of phrases is determined for each web document;

generating, by the training computer as a result of the learning to disambiguate step, a disambiguation classifier capable of determining a sense of a phrase within a document to be analyzed;

learning, by the training computer using the disambiguated extracted information from each web document, to select a portion of the extracted information of each web document as being relevant to a theme of the each web document;

generating, by the training computer as a result of the learning to select step, a selection classifier capable of selecting a topic in a document that is relevant to the theme of the document; and

using, by an indexing computer, the disambiguation classifier and the selection classifier to determine a set of topics from a new web document that is not a part of the training data.

2. The method of claim 1 wherein the using the disambiguation classifier and the selection classifier to determine a set of topics from a new web document further comprises using the disambiguation classifier and the selection classifier to determine a set of topics and a set of categories from the web document.

3. The method of claim 1 further comprising determining, by the training computer, link data associated with the extracted information.

4. The method of claim 3 wherein the learning to disambiguate the extracted information further comprises learning, from the link data, to disambiguate the extracted information.

5. The method of claim 1 wherein the retrieving the training data further comprises retrieving a plurality of articles from a knowledge collection website.

6. The method of claim 3 wherein the determining link data further comprises determining inlinks.

7. The method of claim 3 wherein the determining link data further comprises determining outlinks.

8. The method of claim 3 wherein the determining link data further comprises determining redirects.

9. The method of claim 3 wherein the determining link data further comprises determining category hierarchy.

10. The method of claim 1 wherein the training computer and the indexing computer are the same computer.

11. The method of claim 1 wherein the training computer comprises a plurality of computers.

12. The method of claim 1 wherein the indexing computer comprises a plurality of computers.

13. A computer readable storage medium storing computer program instructions capable of being executed by a computer processor on a computing device, the computer program instructions defining the steps of:

extracting, by the training computer, information from retrieved training data comprising a plurality of web documents, the extracted information comprising a plurality of phrases extracted from each document of said plurality of web documents;

learning, by the training computer, to disambiguate the extracted information by analysis of a context derived from words proximate each phrase such that a particular sense of each phrase of the plurality of phrases is determined for each web document;

generating, by the training computer as a result of the learning to disambiguate step, a disambiguation classifier capable of determining a sense of a phrase within a document to be analyzed;

learning, by the training computer using the disambiguated extracted information from each web document, to select a portion of the extracted information of each web document as being relevant to a theme of the each web document;

generating, by the training computer as a result of the learning to select step, a selection classifier capable of selecting a topic in a document that is relevant to the theme of the document; and

using, by an indexing computer, the disambiguation classifier and the selection classifier to determine a set of topics from a new web document that is not a part of the training data.

14. The computer readable storage medium of claim 13 wherein the step of using the disambiguation classifier and the selection classifier to determine a set of topics from a web document further comprises using the disambiguation classifier and the selection classifier to determine a set of topics and a set of categories from the web document.

15. A method comprising:

retrieving, by an indexing computer, a web document;

tokenizing, by the indexing computer, the web document to determine phrases in the web document that correspond with phrases in stored data;

applying, by the indexing computer, a disambiguation classifier on each determined phrase in the web document to obtain a sense for the each determined phrase; and

applying, by the indexing computer, a selection classifier on the sense for the each determined phrase to obtain a set of topics for the web document.

16. The method of claim 15 wherein the applying a selection classifier on the sense for the each determined phrase to obtain a set of topics for the web document further comprises applying the selection classifier on the sense for the each determined phrase to obtain a set of topics and a set of categories for the web document.

17. The method of claim 16 further comprising ranking, by the indexing computer, the obtained categories.

18. The method of claim 15 further comprising performing topic ordering on the obtained set of topics.

19. The method of claim 18 wherein the performing of topic ordering further comprises performing aboutness and “

search-inducing”

ness.

20. A computer readable storage medium storing computer program instructions capable of being executed by a computer processor on a computing device, the computer program instructions defining the steps of:

tokenizing, by an indexing computer, a retrieved web document to determine phrases in the web document that correspond with phrases in stored data;

applying, by the indexing computer, a disambiguation classifier on each determined phrase in the web document to obtain a sense for the each determined phrase; and

applying, by the indexing computer, a selection classifier on the sense for the each determined phrase to obtain a set of topics for the web document.

21. The computer readable storage medium of claim 20 wherein the step of applying a selection classifier on the sense for the each determined phrase to obtain a set of topics for the web document further comprises applying the selection classifier on the sense for the each determined phrase to obtain a set of topics and categories for the web document.

22. A method comprising:

retrieving, by a server computer over a network from a computing device, a Uniform Resource Locator of a web document navigated to by a user of the computing device;

determining, by the server computer, a set of topics associated with the web document, the determining of the set of topics being based on results of a classifier previously applied to stored data; and

transmitting, from the server computer to the computing device, an item associated with the set of topics for display by the computing device.

23. The method of claim 22 wherein the determining the set of topics further comprises determining the set of topics and a set of categories associated with the web document.

24. The method of claim 22 wherein the determining a set of topics further comprises ranking the topics in the set of topics.

25. The method of claim 24 wherein the determining a set of topics and a set of categories further comprises ranking the categories in the set of categories for each topic.

26. The method of claim 22 wherein the transmitting of the item further comprises transmitting text, an icon, a graphic, a video, or a link associated with a topic in the set of topics.

27. A computer readable storage medium storing computer program instructions capable of being executed by a computer processor on a computing device, the computer program instructions defining the steps of:

determining, by a server computer, a set of topics associated with a web document to which a user using a computing device has navigated, the determining of the set of topics being based on results of a classifier previously applied to stored data; and

transmitting, from the server computer to the computing device, an item associated with the set of topics.

28. The computer readable storage medium of claim 27 wherein the step of determining the set of topics furl her comprises determining the set of topics and a set of categories for the data.

29. A system comprising:

a training module executing on a server computer and configured to generate

a disambiguation classifier, the disambiguation classifier configured to determine a sense of a phrase within a document, and

a selection classifier, the selection classifier configured to select a topic in the document that is relevant to a theme of the document;

an indexing module executing on the server computer and configured to apply the disambiguation classifier and the selection classifier on a web document to determine a set of topics from the web document; and

a run-time module executing on the server computer and configured to transmit the set of topics determined by the indexing module for the web document to a computing device when the computing device has navigated to the web document.

30. The system of claim 29 wherein the indexing module is further configured to apply the disambiguation classifier and the selection classifier on the web document to determine a set of categories from the web document.

31. A system comprising:

a run-time module executing on a server computer and configured to transmit a topic previously determined for a particular web document over a network to a computing device when the computing device has navigated to the web document.

32. The system of claim 31 wherein the transmitting of the topic further comprises transmitting instructions to display the topic in a second content area of a browser executing on the computing device while the web document is being displayed in a first content area of the browser.