System and method for matching a query against a broadcast stream

• US 9,563,699 B1
• Filed: 04/21/2015
• Issued: 02/07/2017
• Est. Priority Date: 07/29/2010
• Status: Active Grant

• Current Assignee: Soundhound AI IP LLC
• Original Assignee: SoundHound Incorporated
• Inventors: Aung, Joe Kyaw Soe, Mohajer, Keyvan, Mont-Reynaud, Bernard
• Primary Examiner(s): Elbin, Jesse
• Application Number:
  US14/692,310
• Time in Patent Office: 658 Days
• Field of Search: 340/4.37, 340/4.4, 381/56, 700/94, 707/746, 707/747, 715/727, 725/18, 725/19
• US Class Current: 1/1
• CPC Class Codes: G06F 16/639 using playlists
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
• CPC Class Codes: G06F 16/687 using geographical or spatial information, e.g. location
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: H04H 20/14 for monitoring programmes
• CPC Class Codes: H04H 60/13 Arrangements for device control affected by the broadcast information
• CPC Class Codes: H04H 60/27 Arrangements for recording or accumulating broadcast information or broadcast-related information
• CPC Class Codes: H04H 60/35 Arrangements for identifying or recognising characteristics with a direct linkage to broadcast information or to broadcast space-time, e.g. for identifying broadcast stations or for identifying users
• CPC Class Codes: H04H 60/372 Programme
• CPC Class Codes: H04H 60/43 for identifying broadcast channels
• CPC Class Codes: H04H 60/44 for identifying broadcast stations
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00

Abstract

A method for matching a query against a broadcast stream includes receiving one or more broadcast streams, from which it generates and stores an audio fingerprint of a selected portion of each received broadcast stream. A query is received from which the method generates an audio fingerprint. From that point, the method continues by identifying audio content from the query, using the query audio fingerprint and a database of indexed audio content. The method concludes by identifying the source of the query using the query audio fingerprint and the stored audio fingerprints. Embodiments of the method further include predictively caching audio fingerprint sequences and corresponding audio item identifiers from a server after storing audio fingerprints extracted from the broadcast stream; and using the predictively cached audio fingerprint sequences to identify an audio item within the audio signal based on at least some additional audio fingerprints of the audio signal.

Claims

1. A method of identifying an audio content sample that originates from an actual broadcast, including:

monitoring a plurality of broadcast stations, fingerprinting and saving the fingerprints of broadcast audio content in a database of unidentified broadcast content as it is received;

accessing playlists, comprising portions of identified broadcast audio content from the plurality of monitored broadcast stations, and fingerprints corresponding to the identified broadcast audio content;

receiving data representing sampled audio content from a portable device and searching for a match between fingerprints of the sampled audio content and the fingerprints corresponding to at least parts of multiple playlists, further including;

upon finding a fingerprint match against the fingerprints corresponding to a particular playlist for a particular monitored broadcast station, reporting the particular monitored broadcast station as a source of the broadcast audio content, and a description of the broadcast audio content back to the portable device; and

upon not finding a fingerprint match against the fingerprints corresponding to any of the multiple playlists, further searching for a match of the fingerprints of the sampled audio content, against at least one of;

parts of the database of unidentified broadcast content from the monitored broadcast stations, to identify a source of the broadcast audio content; and

a reference database of identified audio content not associated with a particular broadcast station, to identify the sampled audio content; and

reporting back to the portable device at least one of the source of the broadcast audio content and the identity of the sampled audio content.

2. The method of claim 1, further including:

receiving a location from the portable device; and

using the location to validate an identity of the source of the sampled audio content.

3. The method of claim 1, further including:

receiving a location from the portable device; and

using at least the location and the identified sampled audio content to target the portable device for advertising or offers.

4. The method of claim 1, wherein:

the monitoring of at least one broadcast station relies on a web-based feed that is delayed from an over-the-air feed from the same broadcast station; and

matching the fingerprints of the sampled audio content against one or more playlists for the at least one monitored broadcast station is limited by a predetermined delay bound.

5. The method of claim 1, further including:

saving playlists of the broadcast audio content from the monitored broadcast stations; and

sending the portable device at least a recent portion of a playlist that identifies a plurality of recent items broadcast by a monitored broadcast station source.

6. The method of claim 1, wherein:

the identified sampled audio content is an advertisement;

further including using at least the identified sampled audio content and the identified broadcast station source to target the portable device for advertising or offers.

7. A non-transitory computer-readable storage medium storing instructions for identifying an audio content sample, the instructions which when executed by a processor cause the processor to:

monitor a plurality of broadcast stations, fingerprint and save the fingerprints of broadcast audio content in a database of unidentified broadcast content as it is received;

access playlists, comprising portions of identified broadcast audio content from the plurality of monitored broadcast stations, and fingerprints corresponding to the identified broadcast audio content;

receive data representing sampled audio content from a portable device and search for a match between fingerprints of the sampled audio content and the fingerprints corresponding to at least parts of multiple playlists, further including;

upon finding a fingerprint match against the fingerprints corresponding to a particular playlist for a particular monitored broadcast station, report the particular monitored broadcast station as a source of the sampled audio content, and identification of the sampled audio content back to the portable device; and

upon not finding a fingerprint match against the fingerprints corresponding to any of the multiple playlists, further search for a match the fingerprints of sampled audio content, against at least one of;

parts of the database of unidentified broadcast content from the monitored broadcast stations, to identify a source of the sampled audio content; and

a reference database of identified audio content not associated with a particular broadcast station, to identify the sampled audio content; and

report back to the portable device at least one of the source of the sampled audio content and the identity of the sampled audio content.

8. The non-transitory computer-readable storage medium of claim 7, further including instructions, which when executed by the processor, cause the processor to:

receive a location from the portable device; and

use the location to validate an identity of the source of the broadcast audio content.

9. The non-transitory computer-readable storage medium of claim 7, further including instructions, which when executed by the processor, cause the processor to:

receive a location from the portable device; and

use at least the location and the identified sampled audio content to target the portable device for advertising or offers.

10. The non-transitory computer-readable storage medium of claim 7, wherein:

the monitoring of at least one broadcast station relies on a web-based feed that is delayed from an over-the-air feed from the same broadcast station; and

matching the fingerprints of the sampled audio content against one or more playlists for the at least one monitored broadcast station is limited by a predetermined delay bound.

11. The non-transitory computer-readable storage medium of claim 7, further including instructions, which when executed by the processor, cause the processor to:

save playlists of the broadcast audio content from the broadcast stations monitored; and

send the portable device at least a recent portion of a playlist that identifies a plurality of recent items broadcast by a monitored broadcast station source.

12. The non-transitory computer-readable storage medium of claim 7, wherein:

the identified sampled audio content is an advertisement;

further including using at least the identified sampled audio content and the identified broadcast station source to target the portable device for advertising or offers.

13. A device for identifying an audio content sample, comprising:

a processor;

a computer-readable storage medium storing instructions, the instructions which when executed by the processor cause the processor to;

monitor a plurality of broadcast stations, fingerprint and save the fingerprints of broadcast audio content in a database of unidentified broadcast content as it is received;

access playlists, comprising portions of identified broadcast audio content from the plurality of monitored broadcast stations, and fingerprints corresponding to the identified broadcast audio content;

receive data representing sampled audio content from a portable device and search for a match between fingerprints of the sampled audio content and the fingerprints corresponding to at least parts of multiple playlists, further including;

upon finding a fingerprint match against the fingerprints corresponding to a particular playlist for a particular monitored broadcast station, report the particular monitored broadcast station as a source of the sampled audio content, and identification of the sampled audio content back to the portable device; and

upon not finding a fingerprint match against the fingerprints corresponding to the multiple playlists, further search for a match the fingerprints of sampled audio content, against at least one of;

parts of the database of unidentified broadcast content from the monitored broadcast stations, to identify a source of the sampled audio content; and

a reference database of identified audio content not associated with a particular broadcast station, to identify the sampled audio content; and

report back to the portable device at least one of the source of the sampled audio content and the identity of the sampled audio content.

14. The device of claim 13, wherein the non-transitory computer-readable storage medium further includes instructions, which when executed by the processor, cause the processor to:

receive a location from the portable device; and

use the location to validate an identity of the source of the sampled audio content.

15. The device of claim 13, wherein the non-transitory computer-readable storage medium further includes instructions, which when executed by the processor, cause the processor to:

receive a location from the portable device; and

use at least the location and the identified sampled audio content to target the portable device for advertising or offers.

16. The device of claim 13, wherein:

the monitoring of at least one broadcast station relies on a web-based feed that is delayed from an over-the-air feed from the same broadcast station; and

matching the fingerprints of the sampled audio content against one or more playlists for the at least one monitored broadcast station is limited by a predetermined delay bound.

17. The device of claim 13, wherein the non-transitory computer-readable storage medium further includes instructions, which when executed by the processor, cause the processor to:

save playlists of the broadcast audio content from the monitored broadcast stations; and

send the portable device at least a recent portion of a playlist that identifies a plurality of recent items broadcast by a monitored broadcast station source.

18. The device of claim 13, wherein:

the identified sampled audio content is an advertisement;

further including using at least the identified sampled audio content and the identified broadcast station source to target the portable device for advertising or offers.