Identifying media content

• US 8,484,017 B1
• Filed: 09/25/2012
• Issued: 07/09/2013
• Est. Priority Date: 09/10/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Postelnicu, Gheorghe
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US13/626,351
• Time in Patent Office: 287 Days
• Field of Search: 704/231, 704/270, 704/275, 704/277, 704/257, 704/251, 704/240, 704/235, 704/233, 704/226, 704/222, 704/205, 725/133, 705/14.73, 379/114.14
• US Class Current: 704/205
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G10H 2210/031 Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
• CPC Class Codes: G10H 2240/141 Library retrieval matching, i.e. any of the steps of matching an inputted segment or phrase with musical database contents, e.g. query by humming, singing or playing; the steps may include, e.g. musical analysis of the input, musical feature extraction, query formulation, or details of the retrieval process
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: G10L 25/54 for retrieval

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving (i) audio data that encodes a spoken natural language query, and (ii) environmental audio data, obtaining a transcription of the spoken natural language query, determining a particular content type associated with one or more keywords in the transcription, providing at least a portion of the environmental audio data to a content recognition engine, and identifying a content item that has been output by the content recognition engine, and that matches the particular content type.

Claims

1. A computer-implemented method comprising:

receiving data including (i) audio data that encodes a spoken natural language query, and (ii) environmental audio data;

obtaining a transcription of the spoken natural language query;

determining a particular content type associated with one or more keywords in the transcription;

providing at least a portion of the environmental audio data to a content recognition engine; and

identifying a content item that (i) has been output by the content recognition engine, and (ii) matches the particular content type associated with the one or more keywords in the transcription.

2. The computer-implemented method of claim 1, wherein the particular content type is a movie content type, a music content type, a television show content type, an audio podcast content type, a book content type, an artwork content type, a trailer content type, a video podcast content type, an Internet video content type, or a video game content type.

3. The computer-implemented method of claim 1, wherein receiving the data including the environmental audio data further includes receiving additional audio data that includes background noise.

4. The computer-implemented method of claim 1, further including receiving additional environmental data that includes video data or image data.

5. The computer-implemented method of claim 1, further including outputting data identifying the content item.

6. The computer-implemented method of claim 1, wherein providing at least the portion of the environmental audio data to the content recognition engine further includes providing the portion of the environmental audio data to an audio fingerprinting engine.

7. The computer-implemented method of claim 1, wherein determining the particular content type further includes identifying the one or more keywords using one or more databases that, for each of multiple content types, maps at least one of the keywords to at least one of the multiple content types.

8. The computer-implemented method of claim 7, wherein the multiple content types includes the particular content type, and wherein mapping further includes mapping at least one of the keywords to the particular content type.

9. The computer-implemented method of claim 1, wherein providing further includes providing data identifying the particular content type to the content recognition engine, and

wherein identifying the content item further includes receiving data identifying the content item from the content recognition engine.

10. The computer-implemented method of claim 1, further including receiving two or more content recognition candidates from the content recognition system, and

wherein identifying the content item further includes selecting a particular content recognition candidate based on the particular content type.

11. The computer-implemented method of claim 10, wherein each of the two or more content recognition candidates is associated with a ranking score, the method further including adjusting the ranking scores of the two or more content recognition candidates based on the particular content type.

12. The computer-implemented method of claim 11, further including ranking the two or more content recognition candidates based on the adjusted ranking scores.

13. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving data including (i) audio data that encodes a spoken natural language query, and (ii) environmental audio data;

obtaining a transcription of the spoken natural language query;

determining a particular content type associated with one or more keywords in the transcription;

providing at least a portion of the environmental audio data to a content recognition engine; and

identifying a content item that (i) has been output by the content recognition engine, and (ii) matches the particular content type associated with the one or more keywords in the transcription.

14. The system of claim 13, wherein receiving the data including the environmental audio data further includes receiving additional audio data that includes background noise.

15. The system of claim 13, the operations further including receiving additional environmental data that includes video data or image data.

16. The system of claim 13, wherein providing at least the portion of the environmental audio data to the content recognition engine further includes providing the portion of the environmental audio data to an audio fingerprinting engine.

17. The system of claim 13, wherein determining the particular content type further includes identifying the one or more keywords using one or more databases that, for each of multiple content types, maps at least one of the keywords to at least one of the multiple content types.

18. The system of claim 17, wherein the multiple content types includes the particular content type, and wherein mapping further includes mapping at least one of the keywords to the particular content type.

19. The system of claim 13, wherein providing further includes providing data identifying the particular content type to the content recognition engine, and

wherein identifying the content item further includes receiving data identifying the content item from the content recognition engine.

20. The system of claim 13, the operations further including receiving two or more content recognition candidates from the content recognition system, and

wherein identifying the content item further includes selecting a particular content recognition candidate based on the particular content type.

21. The system of claim 20, wherein each of the two or more content recognition candidates is associated with a ranking score, the method further including adjusting the ranking scores of the two or more content recognition candidates based on the particular content type.

22. The system of claim 21, the operations further including ranking the two or more content recognition candidates based on the adjusted ranking scores.

23. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving data including (i) audio data that encodes a spoken natural language query, and (ii) environmental audio data;

obtaining a transcription of the spoken natural language query;

determining a particular content type associated with one or more keywords in the transcription;

providing at least a portion of the environmental audio data to a content recognition engine; and

identifying a content item that (i) has been output by the content recognition engine, and (ii) matches the particular content type associated with the one or more keywords in the transcription.

24. The computer-readable medium of claim 23, wherein providing at least the portion of the environmental audio data to the content recognition engine further includes providing the portion of the environmental audio data to an audio fingerprinting engine.

25. The computer-readable medium of claim 23, wherein determining the particular content type further includes identifying the one or more keywords using one or more databases that, for each of multiple content types, maps at least one of the keywords to at least one of the multiple content types.

26. The computer-readable medium of claim 25, wherein the multiple content types includes the particular content type, and wherein mapping further includes mapping at least one of the keywords to the particular content type.

27. The computer-readable medium of claim 23, the operations further including outputting data identifying the content item.

28. The computer-readable medium of claim 23, wherein providing further includes providing data identifying the particular content type to the content recognition engine, and

wherein identifying the content item further includes receiving data identifying the content item from the content recognition engine.

29. The computer-readable medium of claim 23, the operations further including receiving two or more content recognition candidates from the content recognition system, and

wherein identifying the content item further includes selecting a particular content recognition candidate based on the particular content type.

30. The computer-readable medium of claim 29, wherein each of the two or more content recognition candidates is associated with a ranking score, the method further including adjusting the ranking scores of the two or more content recognition candidates based on the particular content type.