Methods, systems, and sub-combinations useful in media identification

• US 7,930,546 B2
• Filed: 01/02/2007
• Issued: 04/19/2011
• Est. Priority Date: 05/16/1996
• Status: Expired due to Fees

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Levy, Kenneth L., Rhoads, Geoffrey B., Brunk, Hugh L.
• Primary Examiner(s): Chawan; Vijay B
• Application Number:
  US11/619,123
• Publication Number:
  US 20070174059A1
• Time in Patent Office: 1,568 Days
• Field of Search: 713/176, 713/167, 713/170, 713/180, 713/181, 713/182, 713/189, 713/168, 713/179, 382/100, 382/232, 726/10, 726/26, 726/30, 726/32, 726/27, 726/17, 380/217, 380/201, 380/28, 380/51, 705/58, 705/57, 705/59, 340/5.86, 358/3.28, 704/273
• US Class Current: 713/176
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Identification of media content, such as audio, can be performed through use of watermarking or fingerprinting (aka content signature) technologies. Aspects of these technologies may be combined to advantageous effect. For example, in dealing with the problem of fingerprint errors arising from object distortion, operations known from digital watermarking systems can be employed.

Claims

1. A method useful in identifying media content, the content including audio data, the method comprising:

pre-processing the audio data;

deriving characteristic audio fingerprint data from the pre-processed data; and

sending the characteristic audio fingerprint data to a database, which can identify earlier-stored data that corresponds to the characteristic audio fingerprint data to thereby identify content that matches the media content;

wherein the pre-processing comprises performing a log-mapping process based on a frequency-domain representation of the audio data to yield logarithmically-sampled data, prior to deriving the characteristic audio fingerprint data.

2. The method of claim 1, wherein the pre-processing further comprises transforming the audio data to a frequency domain representation.

3. The method of claim 1, wherein the pre-processing further comprises generating audio power spectrum data corresponding to the frequency-domain representation, and performing a log-mapping process based on the audio power spectrum data.

4. The method of claim 1, wherein the pre-processing comprises identifying segments of the audio data, and wherein the deriving comprises deriving characteristic audio fingerprint data for each of the segments.

5. The method of claim 1, further comprising determining, from the logarithmically-sampled data, a distortion to which the audio data has been subjected.

6. The method of claim 5, further comprising compensating the audio data for the determined distortion prior to deriving the characteristic audio fingerprint data.

7. The method of claim 5, further comprising taking the determined distortion into account when deriving the characteristic fingerprint data.

8. The method of claim 1, wherein the logarithmically-sampled data comprises a hash.

9. The method of claim 1, further comprising correlating the logarithmically-sampled data with one or more pre-existing signals.

10. A method of processing media content data, the media content data including audio, the method comprising:

providing frequency domain data corresponding to the media content data;

performing a log-mapping process on the frequency domain data to yield logarithmically-sampled data; and

using the logarithmically-sampled data in a process that produces an identifier associated with the media content data.

11. The method of claim 10, further comprising generating power spectrum data corresponding to the logarithmically-sampled data, and using the power spectrum data in a process that produces an identifier associated with the media content data.

12. The method of claim 10, further comprising checking a database to identify content corresponding to the identifier.

13. The method of claim 10, further comprising:

generating plural identifiers, wherein the plural identifier are associated with plural non-identical subsets of the media content data; and

storing the plural identifiers in a database for later matching against identifiers derived from unknown content data;

wherein the method is characterized by identifying plural non-identical subsets of data from the media content data, but storing identifiers in the database for only every Nth of the subsets.

14. The method of claim 13, further comprising generating identifiers only for every Nth of the subsets.

15. A method of processing digital audio data to yield an identifier relating to the digital audio data, the method comprising:

providing frequency domain data corresponding to the digital audio data;

producing power spectrum data from the frequency domain data; and

performing a log-mapping process on the power spectrum data to yield logarithmically-sampled data.

16. The method of claim 15, further comprising using the logarithmically-sampled data in a process that produces an identifier corresponding to the digital audio data.

17. The method of claim 15, further comprising processing the logarithmically-sampled data, and then matching the processed data with one or more pre-existing signals.

18. A method of compiling a fingerprint database for identifying media content, wherein the media content data represents at least one of audio and video, the method comprising:

generating plural content fingerprints for plural non-identical excerpts of known media content data; and

storing the plural content fingerprints in a database for later matching against fingerprint data derived from unknown content data;

wherein the method is characterized by identifying plural non-identical subsets of data from the known media content data, and storing fingerprint data in the database for only every Nth of the subsets.

19. The method of claim 18, further comprising generating content fingerprints only for every Nth of the subsets.

20. A method employing data representing image, video, or audio content data, the method comprising:

determining a distortion of the content data from an original state;

calculating a fingerprint identifier from the content data, taking into account the determined distortion; and

using the calculated fingerprint identifier to identify information related to the content data.

21. The method of claim 20, further comprising capturing audio, image, or video content data using a sensor in a handheld wireless device.

22. The method of claim 21, further comprising performing the determining and calculating in the handheld wireless device.

23. The method of claim 20, further comprising compensating the content data for the determined distortion prior to calculating the fingerprint.

24. The method of claim 20, further comprising determining a match between the calculated fingerprint identifier and a reference set of previously-calculated fingerprint identifiers.

25. The method of claim 20, wherein the determining employs calibration data conveyed with the content data.

26. The method of claim 25, wherein the calibration data is included in header information accompanying the content data, hidden in an unused channel or data area, or embedded in a digital watermark.

27. A method of deriving a fingerprint from data representing image, video, or audio content, the method comprising:

determining perceptually relevant features of the content; and

producing a fingerprint for the content based, at least in part, on the perceptually relevant features;

wherein the perceptually relevant features are of at least two different types, and one of the types comprises edges, center of mass, magnitude peaks, frequency characteristics, or Hough transform data.

28. A method of deriving a fingerprint from data representing image, video, or audio content, the method comprising:

applying a Hough transform to the content, to produce Hough output data therefrom; and

producing a fingerprint for the content based, at least in part, on the Hough output data.

29. A method of identifying content based on fingerprint data derived therefrom, the content comprising data representing audio, image, or video content, the method comprising:

computing a fingerprint from the content;

storing the fingerprint in a database;

producing one or more variants of the content by applying one or more manipulation processes thereto, wherein at least one process comprises compression, transformation, or D/A conversion;

computing one or more fingerprints corresponding to the one or more variants of the content;

storing the one or more fingerprints in the database; and

associating the stored fingerprints in the database;

wherein the content corresponds to a set of plural associated fingerprints in the database.

30. The method of claim 29, further comprising:

computing a fingerprint from unknown content;

producing one or more variants of the unknown content, by applying the one or more manipulation processes thereto;

computing one or more fingerprints corresponding to the one or more variants of the unknown content; and

by reference to the fingerprints of the unknown content, determining a match to a set of associated fingerprints stored in the database.

31. The method of claim 29, wherein the associating comprises linking the stored fingerprints by a linked list data structure.

32. The method of claim 29, wherein the computing comprises identifying perceptually relevant features of at least two different types in the content, and computing the fingerprints based thereon.

33. A method of processing audio, image, or video content, the method comprising:

receiving the content in a first format;

encoding the content into a second, different format;

storing the encoded content in a data structure on a storage medium; and

storing fingerprint-related data corresponding to the content, in the same data structure.

34. The method of claim 33, further comprising:

obtaining calibration data associated with the content in the first format;

using the calibration data in computing fingerprint data corresponding to the content; and

storing the fingerprint data in the data structure.

35. The method of claim 33, further comprising storing the fingerprint-related data in a header field of the data structure.

36. The method of claim 33, further comprising deriving fingerprint data from data representing audio, image, or video information, and storing the derived fingerprint data in the data structure.

37. A method of processing audio, image, or video content, the method comprising:

applying a watermark decoding process to the content to derive plural-bit digital watermark data steganographically encoded in the content;

applying a fingerprinting process to the content to derive fingerprint data corresponding to the content;

transmitting both the digital watermark data and the fingerprint data;

receiving response data based on at least some of the transmitted data; and

taking an action based on the response data.

38. The method of claim 37, wherein the digital watermark data and the fingerprint data are both transmitted to a remote computer, wherein the remote computer forwards the fingerprint data to a further computer identified by reference to the digital watermark data.

39. The method of claim 37, wherein the action comprises storing the response data in a data structure that also stores the content.

40. A method of pattern recognition using image, or video input data, the method comprising:

determining an affine transformation of imagery represented in the input data; and

taking the determined affine transformation into account in applying a pattern recognition process to the input data.

41. The method of claim 40, further comprising detecting calibration information steganographically encoded in the input data.

42. The method of claim 41, further comprising determining affine transformation by reference to the calibration information, and counter-acting the affine transformation prior to applying the image recognition process.

43. A method of processing audio, image, or video content, the method comprising:

detecting watermark data steganographically encoded in audio, image, or video information of the content; and

controlling a fingerprinting operation on the content, or use of data resulting therefrom, in accordance with the detected watermark data.

44. The method of claim 43, wherein the controlling comprises triggering application of a fingerprinting operation, depending on a state of the watermark data.

45. The method of claim 43, further comprising performing the fingerprinting operation on the content, and performing one of plural possible operations with resulting fingerprint data, in accordance with the watermark data.

46. The method of claim 43, further comprising:

performing the fingerprinting operation on the content, to produce content fingerprint data;

using the detected watermark data to narrow a field of search within a database of reference fingerprint data; and

searching for a match with the content fingerprint data in the narrowed field of search.

47. A method of processing audio, image, or video content, the method comprising:

deriving essentially unique identification information for the audio, image, or video content, based at least in part on information—

comprising data representing audio, image, or video information—

associated with the content;

searching a memory for a reference data which corresponds with the derived identification information;

identifying content metadata associated with the reference data; and

taking an action based on the content metadata;

wherein the searching comprises interrogating a content addressable memory with the derived information.

48. The method of claim 47, wherein the content is stored on a handheld consumer electronic wireless device, and the action comprises presenting information related to the audio, image, or video content on a display of the device.

49. The method of claim 47, wherein the deriving comprises applying an algorithm based on perceptually significant features of the content.

50. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

pre-processing audio data;

deriving characteristic audio fingerprint data from the pre-processed data; and

sending the characteristic audio fingerprint data to a database, which can identify earlier-stored data that corresponds to the characteristic audio fingerprint data to thereby identify content that matches the media content;

wherein the pre-processing comprises performing a log-mapping process based on a frequency-domain representation of the audio data to yield logarithmically-sampled data, prior to deriving the characteristic audio fingerprint data.

51. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

pre-process audio data;

derive characteristic audio fingerprint data from the pre-processed data; and

send the characteristic audio fingerprint data to a database, which can identify earlier- stored data that corresponds to the characteristic audio fingerprint data to thereby identify content that matches the media content;

wherein the pre-processing comprises performing a log-mapping process based on a frequency-domain representation of the audio data to yield logarithmically-sampled data, prior to deriving the characteristic audio fingerprint data.

52. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

providing frequency domain data corresponding to media content data;

performing a log-mapping process on the frequency domain data to yield logarithmically-sampled data; and

using the logarithmically-sampled data in a process that produces an identifier associated with the media content data.

53. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

provide frequency domain data corresponding to media content data;

perform a log-mapping process on the frequency domain data to yield logarithmically-sampled data; and

use the logarithmically-sampled data in a process that produces an identifier associated with the media content data.

54. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

providing frequency domain data corresponding to digital audio data

producing power spectrum data from the frequency domain data; and

performing a log-mapping process on the power spectrum data to yield logarithmically-sampled data.

55. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

provide frequency domain data corresponding to digital audio data;

produce power spectrum data from the frequency domain data; and

perform a log-mapping process on the power spectrum data to yield logarithmically-sampled data.

56. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

generating plural content fingerprints for plural non-identical excerpts of known media content data; and

storing the plural content fingerprints in a database for later matching against fingerprint data derived from unknown content data;

wherein the method is characterized by identifying plural non-identical subsets of data from the known media content data, and storing fingerprint data in the database for only every Nth of the subsets.

57. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

generate plural content fingerprints for plural non-identical excerpts of known media content data; and

store the plural content fingerprints in a database for later matching against fingerprint data derived from unknown content data;

wherein the method is characterized by identifying plural non-identical subsets of data from the known media content data, and storing fingerprint data in the database for only every Nth of the subsets.

58. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

determining a distortion of content data from an original state;

calculating a fingerprint identifier from the content data , taking into account the determined distortion; and

using the calculated fingerprint identifier to identify information related to the content data.

59. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

determine a distortion of the content data from an original state;

calculate a fingerprint identifier from the content data, taking into account the determined distortion; and

use the calculated fingerprint identifier to identify information related to the content data.

60. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

determining perceptually relevant features of image, video, or audio content; and

producing a fingerprint for the content based, at least in part, on the perceptually relevant features;

wherein the perceptually relevant features are of at least two different types, and one of the types comprises edges, center of mass, magnitude peaks, frequency characteristics, or Hough transform data.

61. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

determine perceptually relevant features of image, video, or audio content; and

produce a fingerprint for the content based, at least in part, on the perceptually relevant features;

wherein the perceptually relevant features are of at least two different types, and one of the types comprises edges, center of mass, magnitude peaks, frequency characteristics, or Hough transform data.

62. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

applying a Hough transform to image, video, or audio content, to produce Hough output data therefrom; and

producing a fingerprint for the content based, at least in part, on the Hough output data.

63. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

apply a Hough transform to image, video, or audio content, to produce Hough output data therefrom; and

produce a fingerprint for the content based, at least in part, on the Hough output data.

64. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

computing a fingerprint from audio, image, or video content;

storing the fingerprint in a database;

producing one or more variants of the content by applying one or more manipulation processes thereto, wherein at least one process comprises compression, transformation, or D/A conversion;

computing one or more fingerprints corresponding to the one or more variants of the content;

storing the one or more fingerprints in the database; and

associating the stored fingerprints in the database;

wherein the content corresponds to a set of plural associated fingerprints in the database.

65. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

compute a fingerprint from audio, image, or video content;

store the fingerprint in a database;

produce one or more variants of the content by applying one or more manipulation processes thereto, wherein at least one process comprises compression, transformation, or D/A conversion;

compute one or more fingerprints corresponding to the one or more variants of the content;

store the one or more fingerprints in the database; and

associate the stored fingerprints in the database;

wherein the content corresponds to a set of plural associated fingerprints in the database.

66. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

applying a watermark decoding process to audio, image, or video content to derive plural-bit digital watermark data steganographically encoded in the content;

applying a fingerprinting process to the content to derive fingerprint data corresponding to the content;

transmitting both the digital watermark data and the fingerprint data;

receiving response data based on at least some of the transmitted data;

and taking an action based on the response data.

67. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

apply a watermark decoding process to audio, image, or video content to derive plural-bit digital watermark data steganographically encoded in the content;

apply a fingerprinting process to the content to derive fingerprint data corresponding to the content;

transmit both the digital watermark data and the fingerprint data;

receive response data based on at least some of the transmitted data; and

take an action based on the response data.

68. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

applying a watermark decoding process to audio, image, or video content to derive plural-bit digital watermark data steganographically encoded in the content;

applying a fingerprinting process to the content to derive fingerprint data corresponding to the content;

transmitting both the digital watermark data and the fingerprint data;

receiving response data based on at least some of the transmitted data; and

taking an action based on the response data.

69. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

apply a watermark decoding process to audio, image, or video content to derive plural- bit digital watermark data steganographically encoded in the content;

apply a fingerprinting process to the content to derive fingerprint data corresponding to the content;

transmit both the digital watermark data and the fingerprint data;

receive response data based on at least some of the transmitted data; and

take an action based on the response data.

70. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

determining an affine transformation of imagery represented in image or video input data; and

taking the determined affine transformation into account in applying a pattern recognition process to the input data.

71. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

determine an affine transformation of imagery represented in image or video input data; and

apply a pattern recognition process to the input data taking the determined affine transformation into account.

72. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

detecting watermark data steganographically encoded in audio, image, or video information of audio, image, or video content; and

controlling a fingerprinting operation on the content, or use of data resulting therefrom, in accordance with the detected watermark data.

73. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

detect watermark data steganographically encoded in audio, image, or video information of audio, image, or video content; and

control a fingerprinting operation on the content, or use of data resulting therefrom, in accordance with the detected watermark data.

74. An article of manufacture including a computer-readable medium having instructions stored thereon that, if executed by a computing device, cause the computing device to perform operations comprising:

deriving essentially unique identification information for the audio, image, or video content, based at least in part on information—

comprising data representing audio, image, or video information—

associated with audio, image, or video content;

searching a memory for a reference data which corresponds with the derived identification information;

identifying content metadata associated with the reference data; and

taking an action based on the content metadata;

wherein the searching comprises interrogating a content addressable memory with the derived information.

75. An apparatus comprising:

a processor; and

a computer-readable medium operatively connected to the processor having instructions stored thereon that, if executed by the processor, cause the apparatus to;

derive essentially unique identification information for the audio, image, or video content, based at least in part on information—

comprising data representing audio, image, or video information—

associated with audio, image, or video content;

search a memory for a reference data which corresponds with the derived identification information;

identify content metadata associated with the reference data; and

take an action based on the content metadata;

wherein the searching comprises interrogating a content addressable memory with the derived information.