Periodic ambient waveform analysis for enhanced social functions

• US 8,706,499 B2
• Filed: 08/16/2011
• Issued: 04/22/2014
• Est. Priority Date: 08/16/2011
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Papakipos, Matthew Nicholas, Garcia, David Harry
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): THOMAS-HOMESCU, ANNE L
• Application Number:
  US13/211,182
• Publication Number:
  US 20130046542A1
• Time in Patent Office: 980 Days
• Field of Search: 704/243, 704/249, 704/273, 704235-236, 704/275, 704/246, 707/751, 707/748, 707/724, 707769-770, 707/722, 707/739, 701/410, 701/469, 701/31.5, 701/455, 715/745
• US Class Current: 704/270
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/306 User profiles

Abstract

Client devices periodically capture ambient audio waveforms, generate waveform fingerprints, and upload the fingerprints to a server for analysis. The server compares the waveforms to a database of stored waveform fingerprints, and upon finding a match, pushes content or other information to the client device. The fingerprints in the database may be uploaded by other users, and compared to the received client waveform fingerprint based on common location or other social factors. Thus a client'"'"'s location may be enhanced if the location of users whose fingerprints match the client'"'"'s is known. In particular embodiments, the server may instruct clients whose fingerprints partially match to capture waveform data at a particular time and duration for further analysis and increased match confidence.

Claims

1. A method comprising, by one or more computer systems:

capturing audio data from a sensor of a client device;

performing waveform analysis on the audio data;

generating a fingerprint for the audio data;

sending the fingerprint and a self-determined location of the client device to a server;

receiving, from the server, location information associated with the location of the client device, wherein the location information is determined by;

comparing the fingerprint to a database of stored audio waveform fingerprints comprising uploaded audio waveform fingerprints from one or more client devices;

finding one or more matching audio waveform fingerprints from a comparison set of audio waveform fingerprints obtained from the database, the comparison set determined based at least in part on the self-determined location of the client device; and

determining, based at least in part on the matching audio waveform fingerprints, the location information associated with the location of the client device; and

updating the self-determined location of the client device based at least in part on the received location information.

2. The method of claim 1, further comprising:

receiving, from the server, information relating to content of the audio data; and

displaying the received information to a user of the client device.

3. The method of claim 2, wherein the information is a name, artist, album, or image relating to a song detected in the audio data.

4. The method of claim 1, further comprising:

receiving an instruction from the server, the instruction comprising at least a specific time to begin capturing audio data and a capture duration; and

capturing audio data in accordance with the received instruction.

5. The method of claim 4, further comprising:

receiving, from the server, information relating to content of the audio data; and

displaying the received information to a user of the client device.

6. A method, by one or more computing systems, comprising:

receiving an audio waveform fingerprint and a client-determined location from a client device;

comparing the received audio waveform fingerprint to a database of stored audio waveform fingerprints, each stored audio waveform fingerprint associated with an object in an object database;

finding one or more matching audio waveform fingerprints from a comparison set of audio waveform fingerprints obtained from the audio waveform fingerprint database;

determining, based at least in part on the matching audio waveform fingerprints, location information associated with a location of the client device; and

sending the location information to the client device, wherein the client device is operable to update the client-determined location based at least in part on the location information;

wherein the database of stored audio waveform fingerprints comprises uploaded audio waveform fingerprints from one or more other client devices; and

the comparison set of audio waveform fingerprints is determined based at least in part on the received client-determined location.

7. The method of claim 6, further comprising:

sending content from an object associated with one or more of the matching audio waveform fingerprints to the client device.

8. The method of claim 7, wherein the objects is a song and the content from the song comprises information about the song.

9. The method of claim 6, wherein the one or more other client devices are associated with one or more users of a social-networking system.

10. The method of claim 9, wherein the matching audio waveform fingerprints comprise uploaded audio waveform fingerprints from one or more of the users.

11. The method of claim 10, wherein the location information is determined based at least in part on information associated with the one or more of the users in the social-networking system.

12. The method of claim 9, wherein the objects comprise voices of one or more of the users.

13. The method of claim 9, wherein the comparison set of audio waveform fingerprints from the audio waveform fingerprint database is based at least in part on social factors between a user of the client device and one or more of the users of the social-networking system.

14. A method, by one or more computing systems, comprising:

receiving a first waveform fingerprints and a client-determined location from a first client device;

comparing the received first waveform fingerprints to a database of stored waveform fingerprints, each stored waveform fingerprint being uploaded from one of a plurality of second client devices;

finding one or more matching waveform fingerprints from the waveform fingerprint database;

classifying each match as a high-confidence or low-confidence match; and

for each high-confidence match;

determining, based at least in part on the matching waveform fingerprints, location information associated with a location of the first client device; and

sending the location information to the first client device, wherein the first client device is operable to update the client-determined location based at least in part on the location information;

for each low confidence match, sending an instruction to the first client device and the one or more second client devices whose uploaded waveform fingerprints match the received first waveform fingerprint to capture audio data at a particular time and for a particular duration.

15. The method of claim 14, further comprising:

receiving in response to the instruction a plurality of second audio waveform fingerprints from the first client device and the one or more second client devices;

determining whether the received second audio waveform fingerprint from the first client device matches any of the one or more received second audio waveform fingerprints from the one or more second client devices; and

in response to a positive determination, determining that the first client device is in the same location as the second client devices whose received second audio waveform fingerprints match the received second audio waveform fingerprint from the first client device.

16. The method of claim 15, further comprising:

in response to a positive determination, determining that the first client device and the second client devices whose received second audio waveform fingerprints match the received second audio waveform fingerprint from the first client device are in the same location as a particular user whose voice is associated with the matching received second audio waveform fingerprints; and

storing the location of the particular user.

17. The method of claim 16, further comprising automatically tagging the particular user at the location.

18. The method of claim 14, wherein the comparison set of audio waveform fingerprints from the audio waveform fingerprint database is based at least in part on social factors between a user of the first client device and one or more users of the second client devices.