Device, method, and medium for generating audio fingerprint and retrieving audio data

• US 9,589,283 B2
• Filed: 01/16/2013
• Issued: 03/07/2017
• Est. Priority Date: 11/11/2005
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Hyounggook, She, Yuan Yuan, Eom, Kiwan, Zhu, Xuan, Kim, Jiyeun
• Primary Examiner(s): Vo, Huyen
• Application Number:
  US13/742,858
• Publication Number:
  US 20130132210A1
• Time in Patent Office: 1,511 Days
• Field of Search: 704/270, 704/272, 704/270.1, 704200-207, 704 1- 10, 704/251, 704/255, 704/257, 704/273, 725/32, 713/180, 713/176, 380/201, 370/352, 382/100, 382/124, 455/2.01, 714/700, 707/3
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: G10L 19/0212 using orthogonal transformation
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Provided are device, method, and medium for generating an audio fingerprint and retrieving audio data. The device for generating an audio fingerprint includes: a coefficient extracting section partially decoding audio data in a compression area and extracting MDCT (Modified Discrete Cosine Transform) coefficients; a coefficient selecting section selecting an MDCT coefficient robust to noises from the extracted MDCT coefficients; a modulation spectrum generating section transforming the selected MDCT coefficient by the use of a Fourier transform method and generating a modulation spectrum; and a bit conversion section quantizing the generated modulation spectrum and generating an audio fingerprint. As a result, it is possible to accurately and rapidly retrieve the audio data recorded in a variety of environments. Since elements based on MP3 are used, it is possible to apply to MP3 applications in various manners. In addition, it is possible to apply to classification of audio data such as classification of music moods and classification of music genres and various other fields such as extraction of a specific event from moving images of sports.

Claims

1. A method of providing a mobile device with information associated with audio data, the method comprising:

obtaining, performed by at least one processor, a segment of the audio data;

generating, performed by the at least one processor, a modulation spectrum by performing Fourier transform on Modified Discrete Cosine Transform (MDCT) coefficients of the obtained segment;

generating, performed by the at least one processor, audio fingerprint data from the generated modulation spectrum;

identifying, performed by the at least one processor, the audio data by comparing the generated audio fingerprint data with a plurality of audio fingerprint data stored in a database;

retrieving, performed by the at least one processor, information corresponding to the identified audio data from a database; and

providing, performed by the at least one processor, the retrieved information to the mobile device user.

2. The method of claim 1, wherein the retrieved information further comprises classification information, and the classification information comprises at least one of genre information and mood information of music.

3. The method of claim 2, wherein the segment is obtained in a mobile device connected to a wireless network.

4. The method of claim 2, further comprising:

transmitting, performed by the at least one processor, the obtained segment from a mobile device to a search server,

wherein the generating the audio fingerprint data comprises generating the audio fingerprint data based on the transmitted segment.

5. The method of claim 2, further comprising:

transmitting, performed by the at least one processor, the retrieved information corresponding to the identified audio data from a search server to a mobile device.

6. The method of claim 2, further comprising:

generating, performed by the at least one processor, a hashing value corresponding to the generated audio fingerprint data,

wherein the identifying the audio data comprises matching a hashing value from a hashing table storing hashing values corresponding to a plurality of audio fingerprint data with the generated hashing value.

7. The method of claim 1, wherein the segment is obtained in a mobile device connected to a wireless network.

8. The method of claim 1, further comprising:

transmitting, performed by the at least one processor, the obtained segment from a mobile device to a search server,

wherein the generating the audio fingerprint data comprises generating the audio fingerprint data based on the transmitted segment.

9. The method of claim 1, further comprising:

transmitting, performed by the at least one processor, the retrieved information corresponding to the identified audio data from a search server to a mobile device.

10. The method of claim 1, wherein the audio fingerprint data is generated in a mobile device connected to a wireless network.

11. The method of claim 1, further comprising:

transmitting, performed by the at least one processor, the generated audio fingerprint data from a mobile device to a search server; and

transmitting, performed by the at least one processor, the retrieved information corresponding to the identified audio data from the search server to the mobile device,

wherein the identifying the audio data comprises identifying the audio data based on the transmitted audio fingerprint data.

12. The method of claim 1, further comprising:

generating, performed by the at least one processor, a hashing value corresponding to the generated audio fingerprint data,

wherein the identifying the audio data comprises matching a hashing value from a hashing table storing hashing values corresponding to a plurality of audio fingerprint data with the generated hashing value.

13. The method of claim 1, wherein the audio fingerprint data is generated based on a Fourier Transform.

14. The method of claim 1, wherein the audio fingerprint data is generated by considering characteristics of the audio data so that the generated audio fingerprint data is robust to noise.

15. A method of providing a mobile device user with information associated with audio data, the method comprising:

obtaining, performed by the at least one processor, a segment of audio data;

generating, performed by the at least one processor, a modulation spectrum by performing Fourier transform on Modified Discrete Cosine Transform (MDCT) coefficients of the obtained segment;

generating, performed by the at least one processor, audio fingerprint data from the obtained modulation spectrum;

generating, performed by the at least one processor, a hashing value corresponding to the generated audio fingerprint data;

identifying, performed by the at least one processor, the audio data by matching a hashing value from a hashing table storing hashing values corresponding to a plurality of audio fingerprint data with the generated hashing value, based on an adjustable threshold value;

retrieving, performed by the at least one processor, information corresponding to the identified audio data from a database; and

providing, performed by the at least one processor, the retrieved information to the mobile device user.

16. The method of claim 15, wherein the retrieved information comprises classification information, and the classification information comprises at least one of genre information and mood information of music.

17. The method of claim 16, wherein the segment is obtained in a mobile device connected to a wireless network.

18. The method of claim 16, further comprising:

transmitting, performed by the at least one processor, the obtained segment from a mobile device to a search server,

wherein the generating the audio fingerprint data comprises generating the audio fingerprint data based on the transmitted segment.

19. The method of claim 16, further comprising:

transmitting, performed by the at least one processor, the retrieved information corresponding to the identified audio data from a search server to a mobile device.

20. The method of claim 15, wherein the segment is obtained in a mobile device connected to a wireless network.

21. The method of claim 15, further comprising:

transmitting, performed by the at least one processor, the obtained segment from a mobile device to a search server,

wherein the generating the audio fingerprint data comprises generating the audio fingerprint data based on the transmitted segment.

22. The method of claim 15, further comprising:

transmitting, performed by the at least one processor, the retrieved information corresponding to the identified audio data from a search server to a mobile device.

23. The method of claim 15, wherein the audio fingerprint data is generated in a mobile device connected to a wireless network.

24. The method of claim 15, further comprising:

transmitting, performed by the at least one processor, the generated audio fingerprint data from a mobile device to a search server; and

transmitting, performed by the at least one processor, the retrieved information corresponding to the identified audio data from the search server to the mobile device,

wherein the identifying the audio data comprises identifying the audio data based on the transmitted audio fingerprint data.

25. The method of claim 15, wherein the audio fingerprint data is generated based on a Fourier Transform.

26. The method of claim 15, wherein the audio fingerprint data is generated by considering characteristics of the audio data so that the generated audio fingerprint data is robust to noise.

27. The method of claim 15, wherein the identifying the audio data comprises identifying by matching the hashing value from the hashing table storing hashing values corresponding to the plurality of audio fingerprint data with the generated hashing value, based on an adjustable threshold value and a distance.

28. A method of providing a mobile device user with information associated with audio data, the method comprising:

obtaining, performed by the at least one processor, a segment of audio data;

generating, performed by the at least one processor, audio fingerprint data from the obtained segment;

generating, performed by the at least one processor, a hashing value corresponding to the generated audio fingerprint data;

identifying, performed by the at least one processor, the audio data by matching a hashing value from a hashing table storing hashing values corresponding to a plurality of audio fingerprint data with the generated hashing value, based on an adjustable threshold value;

retrieving, performed by the at least one processor, information corresponding to the identified audio data; and

providing, performed by the at least one processor, the retrieved information to the mobile device user,

wherein the retrieved information corresponding to the identified audio data comprises at least one of event information and classification information, and

wherein the adjustable threshold value is adjusted until a predetermined number of the hashing values from the hashing table storing the hashing values corresponding to the plurality of audio fingerprint data match with the generated hashing value.

29. A method of providing a mobile device user with information associated with audio data, the method comprising:

obtaining, performed by the at least one processor, a segment of audio data;

generating, performed by the at least one processor, a modulation spectrum by performing Fourier transform on Modified Discrete Cosine Transform (MDCT) coefficients of the obtained segment;

generating, performed by the at least one processor, audio fingerprint data from the obtained modulation spectrum;

generating, performed by the at least one processor, a hashing value corresponding to the generated audio fingerprint data;

identifying, performed by the at least one processor, audio data by matching a hashing value from a hashing table storing hashing values corresponding to a plurality of audio fingerprint data with the generated hashing value;

filtering, performed by the at least one processor, the identified audio data;

retrieving, performed by the at least one processor, information corresponding to the filtered audio data; and

providing, performed by the at least one processor, the retrieved information to the mobile device user,

wherein the retrieved information corresponding to the identified audio data comprises at least one of event information and classification information, and

wherein the identifying and the filtering are performed based on threshold values.

30. The method of claim 29, wherein the identifying and the filtering are performed based on the threshold values.

31. A method of providing a mobile device user with information associated with audio data, the method comprising:

obtaining, performed by the at least one processor, a segment of audio data;

generating, performed by the at least one processor, a modulation spectrum by performing Fourier transform on Modified Discrete Cosine Transform (MDCT) coefficients of the obtained segment;

generating, performed by the at least one processor, audio fingerprint data from the obtained modulation spectrum;

generating, performed by the at least one processor, a hashing value corresponding to the generated audio fingerprint data;

identifying, performed by the at least one processor, audio data by matching a hashing value from a hashing table storing hashing values corresponding to a plurality of audio fingerprint data with the generated hashing value;

filtering, performed by the at least one processor, the identified audio data;

retrieving, performed by the at least one processor, information corresponding to the filtered audio data; and

providing, performed by the at least one processor, the retrieved information to the mobile device user,

wherein the retrieved information corresponding to the identified audio data comprises product information including at least one of audio data, image data, and text data corresponding to a specific event of a broadcast moving image.

32. The method of claim 31, wherein the identifying and the filtering are performed based on the threshold values.

33. A method of providing a mobile device with information associated with audio data, the method comprising:

obtaining, performed by at least one processor, a segment of the audio data;

generating, performed by the at least one processor, a modulation spectrum by performing Fourier transform on Modified Discrete Cosine Transform (MDCT) coefficients of the obtained segment;

generating, performed by the at least one processor, audio fingerprint data from the obtained modulation spectrum;

identifying, performed by the at least one processor, the audio data by comparing the generated audio fingerprint data with a plurality of audio fingerprint data stored in a database;

retrieving, performed by the at least one processor, information corresponding to the identified audio data from a database; and

providing, performed by the at least one processor, the retrieved information to the mobile device user,

wherein the retrieved information corresponding to the identified audio data comprises product information including at least two of audio data, image data, and text data corresponding to a specific event of a broadcast moving image.