System and method for determining concepts in a content item using context

• US 20080033982A1
• Filed: 12/15/2006
• Published: 02/07/2008
• Est. Priority Date: 08/04/2006
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Parikh, Jignashu, Thrall, John
• Granted Patent: US 8,856,145 B2
• Time in Patent Office: Days
• US Class Current: 707/102
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The present invention is directed towards systems and methods for indexing one or more items of content. The method of the present invention comprises extracting one or more items of text from a given item of content. The one or more items of extracted text are tokenized into one or more concepts. One or more related concepts associated with the one or more concepts are identified. A support score is generated for the one or more concepts, and the item of content is index with the one or more concepts and the one or more associated support scores.

Claims

1. A method for indexing one or more items of content, the method comprising:

extracting one or more items of text from a given item of content;

tokenizing the one or more items of extracted text into one or more concepts;

identifying one or more related concepts associated with the one or more concepts;

calculating a support score for the one or more concepts; and

indexing the item of content with the one or more concepts and the one or more associated support scores.

2. The method of claim 1 wherein an item of content comprises a web page.

3. The method of claim 1 wherein an item of content comprises a document.

4. The method of claim 1 wherein an item of content comprises a video file.

5. The method of claim 1 wherein an item of content comprises an audio file.

6. The method of claim 1 wherein extracting one or more items of text comprises extracting one or more terms from an item of content.

7. The method of claim 1 wherein extracting one or more items of text comprises extracting one or more items of metadata describing the given item of content.

8. The method of claim 1 wherein tokenizing the one or more items of extracted text into one or more concepts comprises tokenizing the one or more items of extracted text into one or more keywords or phrases.

9. The method of claim 8 wherein tokenizing the one or more items of extracted text into one or more keywords or phrases comprises:

identifying a frequency with which one or more keywords or phrases appear in a corpus of content items; and

tokenizing the one or more items of extracted text into the one or more keywords or phrases that appear with the greatest frequency in the corpus of content items.

10. The method of claim 9 wherein identifying the frequency with which one or more keywords or phrases appear in the corpus of content items comprises identifying a frequency with which one or more keywords or phrases appear in a corpus of content items through use of ngram detection.

11. The method of claim 9 wherein identifying the frequency with which one or more keywords or phrases appear in a corpus of content items comprises identifying a frequency with which one or more keywords or phrases appear in a corpus of content items through use of units technology.

12. The method of claim 9 wherein identifying the frequency with which one or more keywords or phrases appear in a corpus of content items comprises identifying a frequency with which one or more keywords or phrases appear in one or more query logs.

13. The method of claim 1 wherein identifying one or more related concepts associated with the one or more concepts comprises identifying one or more keywords or phrases associated with the one or more concepts.

14. The method of claim 13 wherein identifying one or more related concepts comprises identifying one or more query refinement keywords or phrases associated with a given concept.

15. The method of claim 13 wherein identifying one or more related concepts comprises identifying one or more keywords or phrases submitted by a user during a query session.

16. The method of claim 13 wherein identifying one or more related concepts comprises identifying one or more frequently co-occurring keywords or phrases in a corpus of content items.

17. The method of claim 13 wherein identifying one or more related concepts comprises identifying one or more keywords or phrases associated with a given concept as specified by a human editor.

18. The method of claim 1 wherein calculating a support score for the one or more concepts comprises:

identifying the frequency with which the one or more concepts appear in the item of content;

determining the presence or absence of the one or more related concepts associated with the one or more concepts in the item of content; and

calculating a support score for the one or more concepts based upon the frequency with which the one or more concepts appear in the item of content and the presence or absence of the one or more related concepts.

19. The method of claim 1 wherein indexing the item of content comprises storing the item of content, the one or more concepts associated with the item of content, and the support scores associated with the one or more concepts in an index.

20. The method of claim 19 wherein storing the item of content in an index comprises storing the item of content and the one or more concepts associated with the item of content with corresponding support scores exceeding a given support score threshold in an index.

21. The method of claim 1 comprising:

receiving a search query comprising one or more terms;

accessing an index maintaining one or more items of content with associated concepts and support scores;

selecting one or more items of content from the index with associated concepts responsive to the one or more terms comprising the search query; and

transmitting the one or more selected items of content sorted according to the support scores corresponding to the one or concepts with which the one or more items of content are associated.

22. The method of claim 21 wherein transmitting comprises transmitting to a client device.

23. A system for indexing one or more items of content, the system comprising:

a text extractor operative to extract one or more items of text from an item of content;

a concept dictionary operative to maintain one or more concepts;

a context dictionary operative to maintain one or more related concepts associated with the one or more concepts maintained in the concept dictionary; and

an aboutness extractor operative to;

tokenize the one or more items of text extracted from the item of content according to the one or more concepts maintained in the concept dictionary;

identify one or more related concepts associated with the one or more concepts in the item of content;

generate support scores for the one or more concepts associated with the item of content; and

index the item of content, the one or more concepts associated with the item of content, and the one or more corresponding support scores.

24. The system of claim 23 wherein an item of content comprises a web page.

25. The system of claim 23 wherein an item of content comprises a document.

26. The system of claim 23 wherein an item of content comprises a video file.

27. The system of claim 23 wherein an item of content comprises an audio file.

28. The system of claim 23 wherein the text extractor is operative to extract the one or more terms comprising an item of content.

29. The system of claim 23 wherein the text extractor is operative to extract one or more items of metadata associated with an item of content.

30. The system of claim 23.wherein the concept dictionary is operative to maintain one or more concepts comprising keywords or phrases.

31. The system of claim 30 wherein the concept dictionary is operative to maintain one or more keywords or phrases frequently appearing in a corpus of content items.

32. The system of claim 30 wherein the concept dictionary is operative to maintain one or more keywords or phrases frequently appearing in one or more query logs.

33. The system of claim 23 wherein the context dictionary is operative to maintain one or more related concepts comprising keywords or phrases associated with one or more concepts maintained in the concept dictionary.

34. The system of claim 33 wherein the context dictionary is operative to maintain one or more query refinement keywords or phrases associated with a given concept maintained in the concept dictionary.

35. The system of claim 33 wherein the context dictionary is operative to maintain one or more keywords or phrases submitted by a user during a query session.

36. The system of claim 33 wherein the context dictionary is operative to maintain one or more frequently co-occurring keywords or phrases appearing in a corpus of content items.

37. The system of claim 33 wherein the context dictionary is operative to maintain one or more keywords or phrases associated with a given concept as specified by a human editor.

38. The system of claim 23 wherein the aboutness extractor is operative to:

identify the frequency with which one or more concepts appear in the item of content; and

identify the presence or absence of the one or more related concepts associated with the one or more concepts in the item of content.

39. The system of claim 38 wherein the aboutness extractor is operative to generate a support score for the one or more concepts associated with the item of content through use of the frequency with which one or more concepts appear in the item of content and the presence or absence of the one or more related concepts.

40. The system of claim 23 wherein the aboutness extractor is operative to:

select one or more concepts associated with the item of content with support scores exceeding a given support score threshold; and

index the item of content, the one or more selected concepts associated with the item of content, and the one or more corresponding support scores.

41. The system of claim 23 comprising an other information data store.

42. The system of claim 41 wherein the other information data store is operative to maintain information indicating the reliability of one or more items of content.

43. The system of claim 41 wherein the other information data store is operative to maintain information indicating the reliability of one or more concepts maintained in the concept dictionary.

44. The system of claim 41 wherein the aboutness extractor is operative to generate a support score for the one or more concepts associated with an item of content using the information maintained in the other information data store.

45. The system of claim 23 comprising a dictionary manager.

46. The system of claim 45 wherein the dictionary manager is operative to provide updated information associated with one or more concepts to the concept dictionary.

47. The system of claim 45 wherein the dictionary manager is operative to provide updated information associated with one or more related concepts to the context dictionary.