Methods and systems for processing a sample of a media stream

• US 9,275,141 B2
• Filed: 05/04/2011
• Issued: 03/01/2016
• Est. Priority Date: 05/04/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Entertainment Limited (Apple Inc.)
• Inventors: Wang, Avery Li-Chun, Revill, Adrian Allan
• Primary Examiner(s): Kuntz, Curtis
• Assistant Examiner(s): MAUNG, THOMAS H
• Application Number:
  US13/101,051
• Publication Number:
  US 20110276157A1
• Time in Patent Office: 1,763 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Methods and systems for processing a sample of a media stream are provided. In one example, a method includes receiving a sample of a media stream rendered by a media rendering source, and for matching media recordings from which the sample was obtained, the method also includes receiving a signature file including a temporally mapped collection of features describing content of the matching media recordings. Each received signature file may be stored in memory. In addition, a signature stream may be received that includes features extracted from the media stream, and a number of matching features between the signature stream of the media stream and the signature file for at least one matching media recording can be determined. In one example, when substantially matching features between the signature stream of the media stream and the signature file for a matching media recording stop occurring, a discontinuity can be identified.

Claims

1. A method comprising:

receiving, at a computing device, a signature stream including a temporally mapped collection of features describing content of a media stream, wherein the features are extracted from the media stream;

performing a content identification of content in a first portion of the signature stream against a reference database of media content resulting in an identification of reference media recordings including the content so as to determine a recognition result of the content identification;

for the reference media recordings identified from the content identification, receiving at the computing device a signature file including a temporally mapped collection of features describing content of the identified reference media recordings, wherein the features in the signature file are anticipated to appear in the signature stream at predetermined times;

comparing, by the computing device, features between a second portion of the signature stream and a given signature file for at least one identified reference media recording, wherein the second portion of the signature stream differs from the first portion; and

based on the comparison, verifying the continuity of match between the media stream and the identified reference media recording.

2. The method of claim 1, wherein receiving the signature file comprises retrieving the signature file from a database.

3. The method of claim 2, wherein performing the content identification comprises:

sending a content identification query including a sample of the first portion of the signature stream to a server; and

receiving identifications of the sample.

4. The method of claim 3, further comprising receiving at the computing device information associated with the identifications of the sample and the signature file for each media recording matching the sample.

5. The method of claim 1, wherein the method is performed by the computing device, and wherein receiving the signature file comprises receiving the signature file from a server.

6. The method of claim 1, wherein the method is performed by the computing device, and wherein receiving the signature file comprises retrieving the signature file from memory of the computing device.

7. The method of claim 1, wherein the method is performed by the computing device, and wherein receiving the signature stream comprises receiving the signature stream based on a recording by the computing device from an ambient environment using a microphone.

8. The method of claim 1, further comprising:

determining a number of matching features between the second portion of the signature stream of the media stream and the signature file for at least one identified media recording; and

based on the number of matching features identifying a matching media recording.

9. The method of claim 1, further comprising:

determining whether features between the second portion of the signature stream of the media stream and the signature file for at least one identified media recording are substantially matching over time.

10. The method of claim 9, further comprising receiving a time offset (T_s,k) corresponding to a sampling time of the second portion of the signature stream within the media stream.

11. The method of claim 10, further comprising:

aligning a time position within the signature file for each identified media recording to the time offset; and

comparing the aligned signature file for each identified media recording with the second portion of the signature stream.

12. The method of claim 9, wherein determining whether features between the second portion of the signature stream of the media stream and the signature file for at least one identified media recording are substantially matching over time comprises:

for a given time segment (T₁,T₂), retrieving from the second portion of the signature stream a set of features F_j with corresponding times t_j such that T₁<

t_j<

T₂;

within a k-th signature file, determining whether a feature (F′

_j,k) substantially similar to F_j occurs at a time (t′

_j,k) substantially similar to T_r,k(t_j)=T_s,k+R_k*(t_j−

T₀), wherein T₀ is a timestamp indicating a time that the sample was recorded, T_r,k(t) is a relative time offset of content within the k-th identified media recording corresponding to content of the media stream at time t, T_s,k is a time offset of a sampling time of the sample within the k-th identified media recording, and R_k is a timescale ratio that indicates a speed at which the media stream is rendered by a media rendering source based on a reference speed of the k-th identified media recording; and

determining a match when F′

_j,k is within a predetermined deviation from F_j and the f′

_j,k is within a predetermined deviation from T_r,k(t_j).

13. The method of claim 9, wherein a given signature file indicates a length L_k of a corresponding reference media recording for which the given signature file includes the temporally mapped collection of features, and the method comprises determining whether features between the second portion of the signature stream of the media stream and the signature file for at least one identified media recording are substantially matching over a time segment (T₁,T₂) of the reference media recording, wherein the time segment (T₁,T₂) is determined according to:

for a given time segment (T₁,T₂) that spans the length L_k, and the time offset T_s,k may enable the determination for the segment (T1,T2);

T₁=T₀−

T_s,k/R_k

T₂=T₀+(L_k−

T_s,k)/R_k

where T₁ corresponds to a beginning time of the reference media recording in the media stream, T₂ corresponds to an end time of the reference media recording in the media stream, T_s,k is a time offset of a sampling time of the sample within the k-th identified media recording, and R_k is a timescale ratio that indicates a speed at which the media stream is rendered by a media rendering source based on a reference speed of the k-th identified media recording.

14. The method of claim 9, wherein determining whether features between the second portion of the signature stream of the media stream and the signature file for at least one identified media recording are substantially matching over time comprises:

iteratively comparing incremental segments of the second portion of the signature stream with corresponding time aligned portions of the signature file.

15. The method of claim 9, further comprising when substantially matching features between the second portion of the signature stream of the media stream and the signature file for a identified media recording stop occurring, identifying a discontinuity.

16. The method of claim 15, wherein substantially matching features stop occurring when a density of matching features falls below a predetermined threshold over time.

17. The method of claim 15, further comprising:

based on the discontinuity, determining that one or more of an identification of the identified media recordings is no longer valid.

18. The method of claim 15, further comprising:

sending a content identification query including a sample of the media stream rendered by a media rendering source;

receiving information associated with identifications of the identified media recordings and the signature file for each of the identified media recordings; and

based on the discontinuity, receiving a notification that one or more of the identifications of the identified media recordings is no longer valid.

19. The method of claim 15, wherein performing the content identification comprises the computing device performing the content identification on the media stream rendered by a media rendering source.

20. The method of claim 9, further comprising:

identifying a given signature file that includes a highest number of matching features to features of the signature stream to be a match; and

identifying an identification of a media recording corresponding to the given signature file.

21. The method of claim 9, further comprising:

receiving a signature stream increment including additional features extracted from the media stream;

determining that a given signature file of multiple signature files includes matching features to the features in the signature stream increment; and

identifying an identification of a media recording corresponding to the given signature file.

22. The method of claim 9, further comprising:

for each of a number of signature files, comparing features to features extracted from the media stream forward and backward in time until continuity is lost in both directions;

identifying a given signature file that includes a longest continuity length of matching features to features of the signature stream to be a match, wherein the continuity length is determined from a time difference between two endpoints of matching features; and

determining an identification of a media recording corresponding to the given signature file.

23. The method of claim 9, further comprising:

identifying a given signature file that includes a longest continuity length of matching features to features of the signature stream to be a match, wherein the continuity length is determined from a time difference between two endpoints of matching features; and

determining an identification of a media recording corresponding to the given signature file.

24. The method of claim 9, further comprising:

determining a given signature file that has features substantially matching the features in the signature stream; and

identifying a second media recording that is related to at least one of the identified media recordings.

25. The method of claim 24, further comprising determining an alignment of the second media recording with the media stream being rendered by a media rendering source.

26. The method of claim 24, further comprising:

receiving the second media recording at the computing device for rendering the second media recording substantially in synchrony to the media stream being rendered by a media rendering source.

27. The method of claim 26, wherein the media stream is a musical recording and the second media recording are lyrics of the musical recording, and wherein the computing device is configured to display the lyrics of the musical recording.

28. The method of claim 27, further comprising when substantially matching features between the second portion of the signature stream of the media stream and the signature file for a given identified media recording stop occurring, discontinuing displaying of upcoming lines of lyrics of the musical recording.

29. The method of claim 26, further comprising:

when substantially matching features between the second portion of the signature stream of the media stream and the signature file for a given identified media recording stop occurring, discontinuing rendering of the second media recording to be substantially in synchrony to the media stream being rendered by the media rendering source.

30. The method of claim 29, further comprising performing a position identification using the signature file of the given identified media recordings to identify a time offset.

31. The method of claim 30, further comprising rendering the second media recording at a position corresponding to the identified time offset so that the second media recording and the media recording being rendered in the media stream are substantially in synchrony.

32. The method of claim 26, further comprising:

determining that synchrony between the second media recording and the media stream has been lost;

determining a time position of an identified media recording in the media stream using the signature file; and

rendering the second media recording at the determined time position.

33. The method of claim 32, wherein determining a time position of an identified media recording in the media stream using the signature file comprises periodically scanning the signature file for matches to features in the media stream.

34. The method of claim 26, further comprising:

determining that synchrony between the second media recording and the media stream has been lost;

performing a content identification on the media stream to obtain a new given signature file that has features substantially matching the features in the signature stream;

receiving a third media recording that is related to the new signature file; and

rendering the third media recording substantially in synchrony to the media stream being rendered by the media rendering source.

35. The method of claim 1, further comprising, storing each received signature file in memory.

36. The method of claim 9, wherein based on determining whether features between the second portion of the signature stream of the media stream and the signature file for at least one identified media recording are substantially matching over time, the method further comprises:

determining a mismatch of a timescale ratio between the media stream and a reference matching media recording, wherein the timescale ratio indicates a speed at which a given media stream is rendered by a given media rendering source based on a reference speed of a reference media recording; and

determining an adjusted timescale ratio based on the mismatch.

37. The method of claim 36, further comprising

determining a given signature file that has features substantially matching the features in the signature stream;

identifying a second media recording that is related to the identified media recording;

receiving the second media recording at the computing device;

rendering the second media recording substantially in synchrony to the media stream being rendered by the media rendering source; and

using the adjusted timescale ratio to perform a resynchronization of the second media recording with the media recording.

38. The method of claim 1, further comprising identifying a given signature file for a given media recording to be substantially matching over time to features in the signature stream of the media stream.

39. An article of manufacture including a tangible non-transitory computer-readable media having computer-readable instructions encoded thereon, the instructions comprising:

instructions for receiving a signature stream including a temporally mapped collection of features describing content of a media stream, wherein the features are extracted from the media stream;

instructions for performing a content identification of content in a first portion of the signature stream against a reference database of media content resulting in an identification of reference media recordings including the content so as to determine a recognition result of the content identification;

for the reference media recordings identified from the content identification, instructions for receiving a signature file including a temporally mapped collection of features describing content of the identified reference media recordings, wherein the features in the signature file are anticipated to appear in the signature stream at predetermined times;

instructions for comparing features between a second portion of the signature stream and a given signature file for at least one identified reference media recording, wherein the second portion of the signature stream differs from the first portion; and

instructions for, based on the comparison, verifying the continuity of match between the media stream and the identified reference media recording.

40. The article of manufacture of claim 39, further comprising instructions for:

determining whether features between the second portion of the signature stream of the media stream and the signature file for at least one identified media recording are substantially matching over time.

41. The article of manufacture of claim 39, further comprising instructions for:

determining a given signature file that has features substantially matching the features in the signature stream; and

identifying a second media recording that is related to the identified media recording.

42. The article of manufacture of claim 41, further comprising instructions for:

receiving the second media recording at the computing device; and

rendering the second media recording substantially in synchrony to the media stream being rendered by a media rendering source.

43. A system comprising:

an input interface configured to receive a signature stream including a temporally mapped collection of features describing content of a media stream, wherein the features are extracted from the media stream;

a module that performs a content identification of content in a first portion of the signature stream against a reference database of media content resulting in an identification of reference media recordings including the content so as to determine a recognition result of the content identification; and

a continuity tracker configured to receive, for the reference media recordings identified from the content identification, a signature file including a temporally mapped collection of features describing content of the identified reference media recordings, wherein the features in the signature file are anticipated to appear in the signature stream at predetermined times, wherein the continuity tracker is further configured to compare features between a second portion of the signature stream and a given signature file for at least one identified reference media recording, and based on the comparison, verify the continuity of match between the media stream and the identified reference media recording, wherein the second portion of the signature stream differs from the first portion.

44. The system of claim 43, wherein the continuity tracker is configured to determine whether features between the signature stream of the media stream and the signature file for at least one identified media recording are substantially matching over time.

45. The system of claim 43, wherein the continuity tracker is configured to determine a given signature file that has features substantially matching the features in the signature stream, and to identify a second media recording that is related to the identified media recording.

46. The system of claim 45, further comprising a media player configured to render the second media recording substantially in synchrony to the media stream being rendered by a media rendering source.