Real-time audio recognition using multiple recognizers

• US 9,384,734 B1
• Filed: 02/24/2012
• Issued: 07/05/2016
• Est. Priority Date: 02/24/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wiseman, Matthew, Postelnicu, Gheorghe M., Roblek, Dominik, Bernstein, Yaniv, Sharifi, Matthew, Chen, Annie
• Primary Examiner(s): Armstrong, Angela A
• Application Number:
  US13/404,971
• Time in Patent Office: 1,593 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/60 of audio data
• CPC Class Codes: G06F 16/637 Administration of user profiles, e.g. generation, initialization, adaptation or distribution
• CPC Class Codes: G06F 16/638 Presentation of query results
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/0033 Recording/reproducing or transmission of music for electrophonic musical instruments
• CPC Class Codes: G10H 2240/141 Library retrieval matching, i.e. any of the steps of matching an inputted segment or phrase with musical database contents, e.g. query by humming, singing or playing; the steps may include, e.g. musical analysis of the input, musical feature extraction, query formulation, or details of the retrieval process
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

An audio recognition service recognizes an audio sample across multiple content types. At least a partial set of results generated by the service are returned to a client while the audio sample is still being recorded and/or transmitted. The client additionally displays the results in real-time or near real-time to the user. The audio sample can be sent over a first HTTP connection and the results can be returned over a second HTTP connection. The audio recognition service further processes check-in selections received from the client for content items indicated by the results. Responsive to receiving the check-in selections, the service determines whether a user is eligible for a reward. If the user is eligible, the service provides the reward.

Claims

1. A method, comprising:

receiving, by a system including a processor from a client device over a first communication connection, a transmission of an audio recording made by the client device;

while receiving the transmission, concurrently sending, by the system, a representation of an audio sample derived from the audio recording to a plurality of recognition modules, wherein respective recognition modules are configured to recognize distinct types of content;

receiving, by the system prior to receiving all of the audio recording, from the plurality of recognition modules, a set of results identifying content items matching the representation of the audio sample; and

transmitting, by the system to the client device over a second communication connection that is different from the first communication connection and prior to receiving all of the audio recording, the set of results.

2. The method of claim 1, wherein the audio sample comprises multiple types of content.

3. The method of claim 1, further comprising:

receiving, by the system from the client device, an indication of desired types of content; and

wherein the sending comprises sending the representation of the audio sample to only recognition modules corresponding to the desired types of content.

4. The method of claim 1, wherein the transmitting to the client device the set of results occurs while receiving from the client device the transmission of the audio recording.

5. The method of claim 1, wherein the types of content comprise at least two of music, video, melody, television, movie, environmental sound, speech, speaker, or quote.

6. The method of claim 1, further comprising:

identifying, by the system, at least one available reward associated with at least one content item; and

providing, by the system to the client device, a first indication of the at least one available reward.

7. The method of claim 6, further comprising:

receiving, by the system from the client device, a second indication of interaction with the at least one content item in compliance with terms of the at least one available reward; and

in response to receiving the second indication, providing, by the system to the client device, a third indication of awarding the at least one available reward.

8. The method of claim 1, wherein the transmitting the set of results comprises providing results received from the plurality of recognition modules in batches at specified intervals.

9. The method of claim 1, wherein the transmitting the set of results comprises providing results received from the plurality of recognition modules in real time upon receipt.

10. A system, comprising:

a processor; and

a memory communicatively coupled to the processor, the memory having stored therein computer-executable instructions, comprising;

an audio recognition component configured to;

receive, from a client device over a first communication channel, a transmission of an audio recording made by the client device;

while receiving the transmission, concurrently communicate a representation of an audio sample derived from the audio recording to a plurality of recognition modules, wherein respective recognition modules are configured to recognize distinct types of content;

receive, from the plurality of recognition modules, a set of recognition results identifying content items matching the representation of the audio sample prior to receiving all of the audio recording; and

transmit, to the client device over a second communication channel that is different from the first communication channel and prior to receiving all of the audio recording, the set of recognition results.

11. The system of claim 10, wherein the audio sample comprises multiple types of content.

12. The system of claim 10, wherein the audio recognition component is further configured to transmit to the client device the set of recognition results while receiving the transmission of the audio recording from the client device.

13. The system of claim 10, wherein the audio recognition component is further configured to:

identify at least one available reward associated with at least one content item; and

provide, to the client device, a first indication of the at least one available reward.

14. The system of claim 13, wherein the audio recognition component is further configured to:

receive, from the client device, a second indication of interaction with the at least one content item in compliance with terms of the at least one available reward; and

in response to receipt of the second indication, provide, to the client device, a third indication of awarding the at least one available reward.

15. A non-transitory computer-readable medium having instructions stored thereon that, in response to execution, cause a system including a processor to perform operations comprising:

receiving, from a client device over a first connection, a transmission of an audio recording made by the client device;

while receiving the transmission, concurrently providing a representation of an audio sample derived from the audio recording to a plurality of recognition modules, wherein respective recognition modules are configured to recognize distinct types of content;

receiving, from the plurality of recognition modules, a set of results identifying content items matching the representation of the audio sample prior to receiving all of the audio recording; and

communicating, to the client device over a second connection that is different from the first connection and prior to receiving all of the audio recording, the set of results.

16. The non-transitory computer-readable medium of claim 15, wherein the communicating to the client device the set of results occurs while receiving from the client device the transmission of the audio recording.

17. The non-transitory computer-readable medium of claim 15, the operations further comprising:

identifying at least one available reward associated with at least one content item; and

communicating, to the client device, a first indication of the at least one available reward.

18. The non-transitory computer-readable medium of claim 17, the operations further comprising:

receiving, from the client device, a second indication of interaction with the at least one content item in compliance with terms of the at least one available reward; and

in response to receiving the second indication, communicating, to the client device, a third indication of awarding the at least one available reward.

19. A system, comprising:

means for receiving, from a client device over a first channel, a transmission of an audio recording made by the client device;

means for concurrently communicating, while receiving the transmission, a representation of an audio sample derived from the audio recording to a plurality of recognition modules, wherein respective recognition modules are configured to recognize distinct types of content;

means for receiving, from the plurality of recognition modules, a set of results identifying content items matching the representation of the audio sample prior to receiving all of the audio recording; and

means for transmitting, to the client over a second channel that is different from the first channel and prior to receiving all of the audio recording, the set of results.

20. The system of claim 19, wherein the transmitting to the client device the set of results occurs while receiving from the client device the transmission of the audio recording.

21. The system of claim 19, further comprising:

means for identifying at least one available reward associated with at least one content item; and

means for providing, to the client device, a first indication of the at least one available reward.

22. The system of claim 21, further comprising:

means for receiving, from the client device, a second indication of interaction with the at least one content item in compliance with terms of the at least one available reward; and

means for in response to receiving the second indication, providing, to the client device, a third indication of awarding the at least one available reward.