System and methods for continuous audio matching

• US 9,390,167 B2
• Filed: 07/28/2011
• Issued: 07/12/2016
• Est. Priority Date: 07/29/2010
• Status: Active Grant

• Current Assignee: Soundhound AI IP LLC
• Original Assignee: SoundHound Incorporated
• Inventors: Mont-Reynaud, Bernard, Master, Aaron, Stonehocker, Timothy P., Mohajer, Keyvan
• Primary Examiner(s): Tsang, Fan
• Assistant Examiner(s): SIEGEL, DAVID F
• Application Number:
  US13/193,514
• Publication Number:
  US 20120029670A1
• Time in Patent Office: 1,811 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/639 using playlists
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content

Abstract

The present invention relates to the continuous monitoring of an audio signal and identification of audio items within an audio signal. The technology disclosed utilizes predictive caching of fingerprints to improve efficiency. Fingerprints are cached for tracking an audio signal with known alignment and for watching an audio signal without known alignment, based on already identified fingerprints extracted from the audio signal. Software running on a smart phone or other battery-powered device cooperates with software running on an audio identification server.

Claims

1. A method of using a battery powered device to monitor an audio signal and identify audio items within the audio signal, the method including:

responsive to the battery powered device having sent initial audio fingerprints extracted from the audio signal, identifying an initial audio item in the initial audio fingerprints;

responsive to identifying the initial audio item, updating a cache from a server with one or more audio fingerprint sequences from one or more audio items that are predicted to follow the identified initial audio item, and updating the cache with respective audio item identifiers for the one or more audio items; and

matching additional audio fingerprints extracted from the audio signal to the cached audio fingerprint sequences, of the one or more audio items predicted to follow the identified initial audio item, to identify an audio item within the audio signal as one of the one or more audio items.

2. The method of claim 1 wherein the one or more audio items are predicted to follow the identified initial audio item based on an observed sequence using previously identified sequences of songs in a multiplicity of audio items.

3. The method of claim 1, where the one or more audio fingerprint sequences from one or more audio items are stored in a local cache on the battery powered device.

4. The method of claim 1, wherein the one or more audio fingerprint sequences from one or more audio items are stored in a local cache on the server.

5. The method of claim 2, wherein the identified initial audio item is a first song, and the one or more audio items that are predicted to follow the identified initial audio item are one or more songs on the same album as the first song.

6. The method of claim 2, wherein the identified initial audio item is a first song, and the one or more audio items that are predicted to follow the identified initial audio item are one or more songs on a known playlist including the first song.

7. A method of efficiently using a battery powered device to continuously monitor an audio signal and identify audio items within the audio signal, the method including:

receiving into a local cache on the battery powered device predictive audio fingerprints and corresponding audio item identifiers appropriate to a tracking mode;

wherein the tracking mode includes a known or suspected alignment of at least one fingerprint extracted from the audio signal with at least one reference fingerprint, and the predictive audio fingerprints for the tracking mode are sequences of fingerprints relative to the known or suspected alignment that represent multiple candidate audio items for a single current audio item.

8. The method of claim 7, wherein the tracking mode includes resolving the single current audio item from among the multiple candidate audio items.

9. The method of claim 7, further including:

receiving into the local cache on the battery powered device predictive audio fingerprints and corresponding audio item identifiers appropriate to a watching mode as the device switches between the tracking and watching modes, wherein the watching mode is invoked from the tracking mode when an end of an audio item is detected or a user skips past the end of the audio item, and the predictive audio fingerprints for the watching mode are sequences of fingerprints that represent other audio items predicted as likely to follow an identified audio item; and

switching between the watching mode in which a transition has occurred between the identified audio item and the single current audio item, and the tracking mode in which the multiple candidate audio items for the single current audio item have been determined, but ambiguities have not been resolved to identify the single current audio item.

10. The method of claim 9, wherein the switching includes switching between the watching mode, the tracking mode and an identifying mode in which the battery powered device relies upon a server to act upon fingerprints sent from the battery powered device which cannot be resolved using the predictive audio fingerprints.

11. The method of claim 9, including switching from the watching mode to the tracking mode upon a determination one or more of the multiple candidate audio items match the single current audio item.

12. The method of claim 10, including receiving into the local cache updated predictive audio fingerprints and corresponding updated audio item identifiers upon a successful identification of one or more audio items for the fingerprints which were not resolved using the predictive audio fingerprints.

13. A method of managing resources in a server to continuously monitor an audio signal and identify audio items within the audio signal, the method including:

receiving into a local cache on the server predictive audio fingerprints and corresponding audio item identifiers appropriate to a tracking mode;

wherein the tracking mode includes a known or suspected alignment of at least one fingerprint extracted from the audio signal with at least one reference fingerprint, and the predictive audio fingerprints for the tracking mode are sequences of fingerprints relative to the known or suspected alignment that represent multiple candidate audio items for a single current audio item.

14. The method of claim 13, wherein the tracking mode includes resolving the single current audio item from among the multiple candidate audio items.

15. The method of claim 13, further including:

receiving into the local cache on the server predictive audio fingerprints and corresponding audio item identifiers appropriate to a watching mode as the server switches between the tracking and watching modes, wherein the watching mode is invoked from the tracking mode when an end of an audio item is detected or a user skips past the end of the audio item, and the predictive audio fingerprints for the watching mode are sequences of fingerprints that represent other audio items predicted as likely to follow an identified audio item; and

switching between the watching mode in which a transition has occurred between the identified audio item and the single current audio item, and the tracking mode in which the multiple candidate audio items for the single current audio item have been determined, but ambiguities have not been resolved to identify the single current audio item.

16. The method of claim 15, wherein the switching includes switching between the watching mode, the tracking mode and an identifying mode in which the server acts upon additional fingerprints of the audio signal which cannot be resolved using the predictive audio fingerprints.

17. The method of claim 15, including switching from the watching mode to the tracking mode upon a determination that one or more of the multiple candidate audio items match the single current audio item.

18. The method of claim 16, including receiving into the local cache updated predictive audio fingerprints and corresponding updated audio item identifiers upon a successful identification of one or more audio items for the fingerprints which were not resolved using the predictive audio fingerprints.

19. A method of using a battery powered device to efficiently monitor an audio signal and identify audio items that match the audio signal, the method including:

responsive to the battery powered device having sent initial audio fingerprints extracted from the audio signal corresponding to an initial audio item, identifying one or more candidate audio items including fingerprints matching the initial audio fingerprints extracted from the audio signal;

updating a first cache from a server with a plurality of audio fingerprint sequences from the one or more candidate audio items and updating the first cache with respective audio item identifiers for the one or more candidate audio items; and

matching one or more additional audio fingerprints, extracted from the audio signal after the initial audio fingerprints, to the audio fingerprint sequences of the one or more candidate audio items in the first cache to identify the initial audio item as one of the one or more candidate audio items.

20. The method of claim 19, wherein the initial audio item is a song;

and

wherein the one or more candidate audio items include different mixes of the song.

21. The method of claim 19, where the first cache is stored on the battery powered device.

22. The method of claim 19, where the first cache is stored on the server.

23. The method of claim 19, wherein matching the one or more additional audio fingerprints to the cached audio fingerprint sequences includes computing first alignments of the additional audio fingerprints with the one or more candidate audio items.

24. The method of claim 19, further including:

in response to identifying the one or more candidate audio items, identifying one or more additional candidate audio items that are predicted to follow the one or more candidate audio items;

updating a second cache from the server with a plurality of audio fingerprint sequences from the one or more additional candidate audio items and updating the second cache with respective audio item identifiers for the one or more additional candidate audio items; and

matching the one or more additional audio fingerprints to the audio fingerprint sequences of the one or more additional candidate audio items in the second cache to identify a second audio item, in the audio signal after the initial audio item, as one of the one or more additional candidate audio items.

25. The method of claim 23, further comprising adjusting the first alignments of the additional audio fingerprints with the one or more candidate audio items.

26. The method of claim 23, further comprising attempting to maintain the computed first alignments, and

wherein if the attempt to maintain the computed alignment fails, computing second alignments of the additional audio fingerprints with the one or more candidate audio items, different from the first alignments.

27. The method of claim 24, wherein one or more additional candidate audio items are predicted to follow the one or more candidate audio items based on previously identified sequences of songs in a multiplicity of audio items.

28. The method of claim 27, wherein the first audio item is a first song, and the additional candidate audio items are songs on a known playlist including the first song.

29. The method of claim 27, wherein the first audio item is a first song, and the additional candidate audio items are songs on the same album as the first song.

30. A method of using a battery powered device to efficiently monitor an audio signal and identify audio items that match the audio signal, the method including:

responsive to the battery powered device having sent initial audio fingerprints extracted from the audio signal corresponding to an initial audio item, identifying one or more candidate audio items including fingerprints matching the initial audio fingerprints extracted from the audio signal;

updating a first cache from a server with a plurality of audio fingerprint sequences from the one or more candidate audio items and updating the first cache with respective audio item identifiers for the one or more candidate audio items;

in response to identifying the one or more candidate audio items, identifying one or more additional candidate audio items that are predicted to follow the one or more candidate audio items;

updating a second cache from the server with a plurality of audio fingerprint sequences from the one or more additional candidate audio items and updating the second cache with respective audio item identifiers for the one or more additional candidate audio items; and

matching one or more additional audio fingerprints, extracted from the audio signal after the initial audio fingerprints, to the audio fingerprint sequences of the one or more additional candidate audio items in the second cache to identify a second audio item, in the audio signal after the initial audio item, as one of the one or more additional candidate audio items.