PRESENTING MOBILE CONTENT BASED ON PROGRAMMING CONTEXT

• US 20110289098A1
• Filed: 05/19/2010
• Published: 11/24/2011
• Est. Priority Date: 05/19/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Oztaskent, Ant, Volovich, Yaroslav
• Granted Patent: US 8,694,533 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/266 Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4826 using recommendation lists, e.g. of programs or channels sorted out according to their score
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating search queries in response to obtaining audio samples on a client device. In one aspect, a method includes the actions of i) receiving audio data from a client device, ii) identifying specific content from captured media based on the received audio data, wherein the identified specific content is associated with the received audio data and the captured media includes at least one of audio media or audio-video media, iii) obtaining additional metadata associated with the identified content, iv) generating a search query based at least in part on the obtained additional metadata, and v) returning one or more search results to the client device, the one or more search results responsive to the search query and associated with the received audio data.

Claims

1. A method comprising:

receiving audio data from a client device;

identifying specific content from captured media based on the received audio data, wherein the identified specific content is associated with the received audio data and the captured media includes at least one of audio media or audio-video media;

obtaining additional metadata associated with the identified content;

generating a search query based at least in part on the obtained additional metadata; and

returning one or more search results to the client device, the one or more search results responsive to the search query and associated with the received audio data.

2. The method of claim 1, wherein the client device includes a mobile device.

3. The method of claim 2, wherein the mobile device includes one of a mobile phone, a smartphone, a laptop, a network, or a tablet PC.

4. The method of claim 1, wherein the client device includes a set-top box associated with a presentation of audio media or audio-video media.

5. The method of claim 1, where the audio data received from the client device includes an audio fingerprint of media recorded at the client device.

6. The method of claim 1, further comprising obtaining an audio fingerprint of the audio data.

7. The method of claim 6, wherein obtaining the audio fingerprint includes generating an audio fingerprint from the audio data received from the client device.

8. The method of claim 1, wherein the captured media includes a collection of captured digital television broadcasts, the method further comprising:

monitoring at least one digital television channel;

extracting an audio stream from each of the at least one digital television channels;

generating, for each of the monitored at least one digital television channels, an audio fingerprint from at least a portion of the extracted audio stream; and

storing each of the generated audio fingerprints in an audio fingerprint repository, each generated audio fingerprint associated with a unique content identifier.

9. The method of claim 8 further comprising:

deriving at least one additional set of information associated with each extracted audio stream;

generating at least one set of content-related metadata associated with each extracted audio stream based on the derived at least one additional set of information;

associating each of the at least one set of content-related metadata with the unique content identifier associated with the generated audio fingerprint associated with the corresponding extracted audio stream; and

storing each of the at least one set of content-related metadata in a metadata repository.

10. The method of claim 9, wherein:

identifying specific content from captured media includes;

matching an audio fingerprint generated from the audio data received from the client device with at least one audio fingerprint stored in the audio fingerprint repository; and

identifying the unique content identifier associated with the at least one audio fingerprint stored in the audio fingerprint repository; and

obtaining additional metadata associated with the identified content includes retrieving content-related metadata associated with the identified unique content identifier from the metadata repository.

11. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving audio data from a client device;

identifying specific content from captured media based on the received audio data, wherein the identified specific content is associated with the received audio data and the captured media includes at least one of audio media or audio-video media;

obtaining additional metadata associated with the identified content;

generating a search query based at least in part on the obtained additional metadata; and

returning one or more search results to the client device, the one or more search results responsive to the search query and associated with the received audio data.

12. The medium of claim 11, wherein the client device includes a mobile device.

13. The medium of claim 12, wherein the mobile device includes one of a mobile phone, a smartphone, a laptop, a network, or a tablet PC.

14. The medium of claim 11, wherein the client device includes a set-top box associated with the presentation of audio media or audio-video media.

15. The medium of claim 11, wherein the audio data received from the client device includes an audio fingerprint of media recorded at the client device.

16. The medium of claim 11, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations further comprising obtaining an audio fingerprint of the audio data.

17. The medium of claim 16, wherein obtaining the audio fingerprint includes generating an audio fingerprint from the audio data received from the client device.

18. The medium of claim 11, wherein the captured media includes a collection of captured digital television broadcasts, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations further comprising:

monitoring at least one digital television channel;

extracting an audio stream from each of the at least one digital television channels;

generating, for each of the monitored at least one digital television channels, an audio fingerprint from at least a portion of the extracted audio stream; and

storing each of the generated audio fingerprints in an audio fingerprint repository, each generated audio fingerprint associated with a unique content identifier.

19. The medium of claim 18, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations further comprising:

deriving at least one additional set of information associated with each extracted audio stream;

generating at least one set of content-related metadata associated with each extracted audio stream based on the derived at least one additional set of information;

associating each of the at least one set of content-related metadata with the unique content identifier associated with the generated audio fingerprint associated with the corresponding extracted audio stream; and

storing each of the at least one set of content-related metadata in a metadata repository.

20. The medium of claim 19, wherein:

identifying specific content from captured media includes;

matching an audio fingerprint generated from the audio data received from the client device with at least one audio fingerprint stored in the audio fingerprint repository; and

identifying the unique content identifier associated with the at least one audio fingerprint stored in the audio fingerprint repository; and

obtaining additional metadata associated with the identified content includes retrieving content-related metadata associated with the identified unique content identifier from the metadata repository.

21. A system comprising:

one or more front-end servers adapted to receive one or more sets of audio data from one or more client devices;

one or more match servers adapted to identify a set of program data from a plurality of sets of program data, wherein each set of program data corresponds to one or more captured media programs, based on the audio data;

one or more service information servers adapted to;

store additional information associated with captured media programs; and

identify specific additional information associated with the identified set of program data; and

one or more search query builder servers adapted to generate a search query based at least in part on the identified specific additional information associated with the identified program data;

the one or more front-end servers further adapted to send at least one search query result identified based on the generated search query to the one or more client devices.

22. The system of claim 21,

the one or more front-end servers further adapted to obtain an audio fingerprint based on the one or more sets of received audio data;

the one or more match servers further adapted to;

compare the obtained audio fingerprint with a plurality of program data-based audio fingerprints; and

identify at least one of the plurality of program data-based audio fingerprints matching the obtained audio fingerprint.

23. The system of claim 21, further comprising one or more search engine servers adapted to receive the generated search query and identify at least one search query result responsive to the generated search query.

24. The system of claim 21, wherein the one or more client devices include a mobile phone, a smartphone, a laptop, a network, or a tablet PC.

25. A method comprising:

obtaining an audio sample from at least one media source at a client device;

transmitting audio data associated with at least a portion of the audio sample from the client device to a non-keyword-based search system;

receiving a set of search results associated with the transmitted portion of the audio data at the client device, wherein the set of search results are generated at the non-keyword-based search system by identifying media programming associated with the obtained audio sample using the transmitted audio data, selecting one or more search terms based on the identified media programming, and submitting the one or more search terms to a search engine; and

presenting the set of search results at the client device.

26. The method of claim 25, wherein the at least one media source includes a local media source, and further wherein obtaining the audio sample from at least one media source at the client device includes recording ambient audio associated with the local media source.

27. The method of claim 25, wherein the at least one media source includes at least one of a television broadcast, a radio broadcast, an Internet-based multimedia presentation, a DVD presentation, or a Blu-ray presentation.

28. The method of claim 25, wherein the non-keyword-based search system includes one or more servers, and further wherein:

identifying media programming associated with the obtained audio sample using the transmitted audio data includes obtaining additional metadata associated with the identified media programming at the non-keyword-based search system;

selecting one or more search terms based on the identified media programming includes generating a search query based at least in part on the obtained additional metadata at the non-keyword-based search system; and

receiving a set of search results associated with the transmitted portion of the audio data at the client device includes transmitting a set of search results responsive to the generated search query from the non-keyword-based search system to the client device.

29. The method of claim 25, wherein the client device includes a mobile device.

30. The method of claim 29, wherein the mobile device includes one of a mobile phone, a smartphone, a laptop, a network, or a tablet PC.

31. The method of claim 25 further comprising obtaining an audio fingerprint of at least a portion of the audio sample at the client device, and wherein the transmitted audio data associated with the at least a portion of the audio sample includes the obtained audio fingerprint.

32. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

obtaining an audio sample from at least one media source;

transmitting audio data associated with at least a portion of the audio sample to a non-keyword-based search system;

receiving a set of search results associated with the transmitted portion of the audio data, wherein the set of search results are generated at the non-keyword-based search system by identifying media programming associated with the obtained audio sample using the transmitted audio data, selecting one or more search terms based on the identified media programming, and submitting the one or more search terms to a search engine; and

presenting the set of search results.

33. The medium of claim 32, wherein the at least one media source includes a local media source, and further wherein obtaining the audio sample from at least one media source includes recording ambient audio associated with the local media source.

34. The medium of claim 32, wherein the at least one media source includes at least one of a television broadcast, a radio broadcast, an Internet-based multimedia presentation, a DVD presentation, or a Blu-ray presentation.

35. The medium of claim 32, wherein the non-keyword-based search system includes one or more servers, and further wherein:

identifying media programming associated with the obtained audio sample using the transmitted audio data includes obtaining additional metadata associated with the identified media programming at the non-keyword-based search system;

selecting one or more search terms based on the identified media programming includes generating a search query based at least in part on the obtained additional metadata at the non-keyword-based search system; and

receiving a set of search results associated with the transmitted portion of the audio data includes transmitting a set of search results responsive to the generated search query from the non-keyword-based search system.

36. The medium of claim 32, wherein the computer storage medium includes a mobile device.

37. The medium of claim 36, wherein the mobile device includes one of a mobile phone, a smartphone, a laptop, a network, or a tablet PC.

38. The medium of claim 32, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations further comprising obtaining an audio fingerprint of at least a portion of the audio sample, and wherein the transmitted audio data associated with the at least a portion of the audio sample includes the obtained audio fingerprint.