Contextual search on multimedia content

• US 9,852,188 B2
• Filed: 06/23/2014
• Issued: 12/26/2017
• Est. Priority Date: 06/23/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bakir, Gkhan Hasan, Csalogny, Kroly, Behzadi, Behshad
• Primary Examiner(s): KUDDUS, DANIEL A
• Application Number:
  US14/312,630
• Publication Number:
  US 20150370859A1
• Time in Patent Office: 1,282 Days
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Techniques for contextual search on multimedia content are provided. An example method includes extracting entities associated with multimedia content, wherein the entities include values characterizing one or more objects represented in the multimedia content, generating one or more query rewrite candidates based on the extracted entities and one or more terms in a query related to the multimedia content, providing the one or more query rewrite candidates to a search engine, scoring the one or more query rewrite candidates, ranking the scored one or more query rewrite candidates based on their respective scores, rewriting the query related to the multimedia content based on a particular ranked query rewrite candidate and providing for display, responsive to the query related to the multimedia content, a result set from the search engine based on the rewritten query.

Claims

1. A method comprising:

extracting entities associated with multimedia content, wherein the entities include values characterizing one or more objects represented in the multimedia content;

receiving a query related to the multimedia content, the query including one or more terms, being in addition to the multimedia content, and being provided by a user during consumption of the multimedia content by the user;

automatically generating, for rewriting the query related to the multimedia content, one or more query rewrite candidates based on the extracted entities and one or more of the terms in the query, wherein the generating is performed when the query related to the multimedia content is received from the user, and wherein generating the one or more query rewrite candidates comprises;

scoring the one or more extracted entities based on one or more of;

a time at which the extracted entities are annotated in the multimedia content or based on co-occurrences of n-grams in a query repository, the n-grams in the query repository each being a contiguous sequence of n items from a given sequence of text or speech, and the n-grams in the query repository including unigrams, bigrams, trigrams, and four-grams,

ranking the extracted entities based on the scoring, and

combining one or more of the terms of the query with one or more scored extracted entities to generate the one or more query rewrite candidates;

providing the one or more query rewrite candidates to a search engine;

scoring the one or more query rewrite candidates based on characteristics of respective result sets resulting from the providing;

ranking the scored one or more query rewrite candidates based on their respective scores;

rewriting the query related to the multimedia content based on a particular ranked query rewrite candidate; and

providing for display, responsive to the query related to the multimedia content, a result set from the search engine based on the rewritten query.

2. The method of claim 1, wherein scoring the one or more query rewrite candidates based on characteristics of respective result sets resulting from the providing, comprises:

determining a number of results in each of the result sets;

determining a relevance of the results in each of the result sets to the one or more terms in the query; and

determining diversity of the results in each of the result sets, wherein scores of each of the query rewrite candidates are based on one or more of the number, the relevance or the diversity.

3. The method of claim 1, wherein rewriting the query related to the multimedia content includes replacing the query with the particular ranked query rewrite candidate.

4. The method of claim 1, wherein the particular ranked candidate has a highest score relative to other ranked query rewrite candidates.

5. The method of claim 1, wherein the multimedia content is a streaming video.

6. The method of claim 1, wherein the objects include human or non-human representations appearing in the multimedia content.

7. The method of claim 1, wherein the values characterizing the one or more objects included in the multimedia content include a name or date associated with the objects.

8. The method of claim 1, wherein the extracting comprises:

reading metadata associated with the multimedia content, wherein the metadata includes user generated content.

9. The method of claim 1, wherein the scoring is based on a correspondence between the terms in the query and entities that are represented in the multimedia content at the time of the query.

10. The method of claim 1, wherein the multimedia content is a video and wherein extracting the entities comprises extracting at least one of the entities based on the at least one of the entities appearing in the video at a time of the query.

11. A non-transitory machine-readable medium comprising instructions stored therein, which when executed by a processor, causes the processor to perform operations comprising:

receiving a query related to multimedia content, the query including one or more terms, being in addition to the multimedia content, and being provided by a user during consumption of the multimedia content by the user;

identifying one or more terms in the query;

automatically generating, for rewriting the query related to the multimedia content, one or more query rewrite candidates based on entities associated with the multimedia content and the one or more terms in the query, wherein the entities include values characterizing one or more objects represented in the multimedia content, and wherein generating the one or more query rewrite candidates comprises;

scoring the one or more extracted entities based on one or more of;

a time at which the extracted entities are annotated in the multimedia content or based on co-occurrences of n-grams in a query repository, the n-grams in the query repository each being a contiguous sequence of n items from a given sequence of text or speech, and the n-grams in the query repository including unigrams, bigrams, tri grams, and four-grams,

ranking the extracted entities based on the scoring, and

combining one or more of the terms of the query with one or more scored extracted entities to generate the one or more query rewrite candidates;

providing the one or more query rewrite candidates to a search engine;

scoring the one or more query rewrite candidates based on characteristics of respective result sets resulting from the providing;

ranking the scored one or more query rewrite candidates based on their respective scores;

rewriting the query related to the multimedia content based on a particular ranked query rewrite candidate; and

providing for display, responsive to the query related to the multimedia content, a result set from the search engine based on the rewritten query.

12. The non-transitory machine-readable medium of claim 11, wherein rewriting the query related to the multimedia content includes replacing the query with the particular ranked query rewrite candidate.

13. The non-transitory machine-readable medium of claim 11, wherein the particular ranked candidate has a highest score relative to other ranked query rewrite candidates.

14. The non-transitory machine-readable medium of claim 11, wherein the multimedia content is a streaming video.

15. The non-transitory machine-readable medium of claim 11, wherein the objects include human or non-human representations appearing in the multimedia content.

16. The non-transitory machine-readable medium of claim 11, wherein the values characterizing the one or more objects included in the multimedia content include a name or date associated with the objects.

17. The non-transitory machine-readable medium of claim 11, wherein the scoring is based on a correspondence between the terms in the query and entities that are represented in the multimedia content at the time of the query.

18. A system comprising:

a memory comprising instructions; and

a processor configured to execute the instructions to;

receive a query related to streaming multimedia content, wherein the query includes one or more terms and is provided by a user through voice input during consumption of the streaming multimedia content by the user;

automatically generate, for rewriting the query related to the streaming multimedia content, one or more query rewrite candidates based on entities associated with the multimedia content and the one or more terms in the query, wherein the entities include values characterizing one or more objects represented in the multimedia content wherein in executing the instructions to generate the one or more query rewrite candidates, the processor is further to;

score the one or more extracted entities based on one or more of;

a time at which the extracted entities are annotated in the multimedia content or based on co-occurrences of n-grams in a query repository, the n-grams in the query repository each being a contiguous sequence of n items from a given sequence of text or speech, and the n-grams in the query repository including unigrams, bigrams, tri grams, and four-grams,

rank the extracted entities based on the scoring, and

combine one or more of the terms of the query with one or more scored extracted entities to generate the one or more query rewrite candidates;

provide the one or more query rewrite candidates to a search engine;

score the one or more query rewrite candidates based on characteristics of respective result sets resulting from the providing;

rank the scored one or more query rewrite candidates based on their respective scores;

rewrite the query related to the multimedia content based on a particular ranked query rewrite candidate; and

provide, without interrupting consumption of the multimedia content, a result set from the search engine based on the rewritten query.

19. The system of claim 18, wherein the rewrite of the query related to the multimedia content includes replacing the query with the particular ranked query rewrite candidate.

20. The system of claim 18, wherein the particular ranked candidate has a highest score relative to other ranked query rewrite candidates.