AUDIO VIDEO OFFSET DETECTOR

• US 20130057761A1
• Filed: 09/02/2011
• Published: 03/07/2013
• Est. Priority Date: 09/02/2011
• Status: Active Grant

• Current Assignee: Dialogic, Inc. (Enghouse Systems Limited)
• Original Assignee: Dialogic, Inc. (Enghouse Systems Limited)
• Inventors: Bloom, Jeffrey A., Zou, Dekun, Ding, Ran
• Granted Patent: US 8,717,499 B2
• Time in Patent Office: Days
• US Class Current: 348/515
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Systems and methods of measuring a temporal offset between audio content and video content that employ audio fingerprints from an audio signal in the audio content, and video fingerprints from video frames in the video content. The systems obtain reference audio and video fingerprints prior to transmission of video over a media channel, and obtain target audio and video fingerprints subsequent to transmission of the video over the media channel. Each fingerprint has an associated time stamp. Using the reference and target audio fingerprints and their associated time stamps, the systems determine an audio time stamp offset. Using the reference and target video fingerprints and their associated time stamps, the systems determine a video time stamp offset. Using the audio and video time stamp offsets, the systems determine a temporal offset between the video content and the audio content introduced by the media channel.

Claims

1. A method of measuring a temporal offset between video content and audio content in a system for transmitting video, the video being subject to transmission over a media channel, the method comprising the steps of:

prior to transmission of the video over the media channel;

obtaining reference audio fingerprint data from the audio content and reference video fingerprint data from the video content, wherein each of the reference audio fingerprint data and the reference video fingerprint data includes associated time stamp data; and

subsequent to the transmission of the video over the media channel;

obtaining target audio fingerprint data from the audio content and target video fingerprint data from the video content, wherein each of the target audio fingerprint data and the target video fingerprint data includes associated time stamp data; and

determining a temporal offset between the video content and the audio content based at least on the time stamp data associated with the reference video fingerprint data, the time stamp data associated with the target video fingerprint data, the time stamp data associated with the reference audio fingerprint data, and the time stamp data associated with the target audio fingerprint data.

2. The method of claim 1 wherein the video content and the audio content have an associated synchronization, and wherein the method further comprises:

subsequent to the transmission of the video over the media channel;

adjusting the synchronization of the video content and the audio content based at least on the temporal offset between the video content and the audio content.

3. The method of claim 1 wherein the obtaining of the reference audio fingerprint data comprises:

sampling at least one audio signal included in the audio content to obtain a sampled audio signal.

4. The method of claim 3 wherein the obtaining of the reference audio fingerprint data further comprises:

performing a first short term Fourier transform on the sampled audio signal.

5. The method of claim 4 wherein the obtaining of the reference audio fingerprint data further comprises:

performing a second short term Fourier transform over a frequency response of the first short term Fourier transform.

6. The method of claim 5 wherein the obtaining of the reference audio fingerprint data further comprises:

obtaining, via at least adjacent frequencies from the frequency response of the first short term Fourier transform and a frequency response of the second short term Fourier transform, a series of audio fingerprint vectors, wherein each of the audio fingerprint vectors corresponds to a predetermined time window of the audio signal.

7. The method of claim 1 wherein the obtaining of the target audio fingerprint data comprises:

sampling at least one audio signal included in the audio content to obtain a sampled audio signal.

8. The method of claim 7 wherein the obtaining of the target audio fingerprint data further comprises:

performing a first short term Fourier transform on the sampled audio signal.

9. The method of claim 8 wherein the obtaining of the target audio fingerprint data further comprises:

performing a second short term Fourier transform over a frequency response of the first short term Fourier transform.

10. The method of claim 9 wherein the obtaining of the target audio fingerprint data further comprises:

obtaining, via at least adjacent frequencies from the frequency response of the first short term Fourier transform and a frequency response of the second short term Fourier transform, a series of audio fingerprint vectors, wherein each of the audio fingerprint vectors corresponds to a predetermined time window of the audio signal.

11. The method of claim 1 wherein the obtaining of the reference video fingerprint data comprises:

dividing at least one video frame included in the video content into a plurality of M×

N non-overlapping regions.

12. The method of claim 11 wherein the obtaining of the reference video fingerprint data further comprises:

obtaining a mean luminance for each of the M×

N non-overlapping regions of the video frame to provide a video fingerprint vector that corresponds to an M×

N block mean luminance vector.

13. The method of claim 1 wherein the obtaining of the target video fingerprint data comprises:

dividing at least one video frame included in the video content into a plurality of M×

N non-overlapping regions.

14. The method of claim 13 wherein the obtaining of the target video fingerprint data further comprises:

obtaining a mean luminance for each of the M×

N non-overlapping regions of the video frame to provide a video fingerprint vector that corresponds to an M×

N block mean luminance vector.

15. The method of claim 1 wherein the reference audio fingerprint data includes one or more reference audio fingerprints that are characteristic of the audio content prior to the transmission of the video over the media channel, wherein the target audio fingerprint data includes at least one target audio fingerprint that is characteristic of the audio content subsequent to the transmission of the video over the media channel, and wherein the determining of the temporal offset between the video content and audio content comprises:

performing fingerprint matching of the target audio fingerprint against one or more of the reference audio fingerprints to obtain one or more reference audio fingerprints that at least substantially match the target audio fingerprint.

16. The method of claim 15 wherein the performing of the fingerprint matching of the target audio fingerprint against one or more of the reference audio fingerprints comprises:

searching for one or more nearest neighbors of the target audio fingerprint from among the reference audio fingerprints to obtain the one or more reference audio fingerprints that at least substantially match the target audio fingerprint.

17. The method of claim 16 wherein the searching for one or more nearest neighbors of the target audio fingerprint from among the reference audio fingerprints comprises:

obtaining an Euclidean distance between the target audio fingerprint and each of the reference audio fingerprints.

18. The method of claim 17 wherein the searching for one or more nearest neighbors of the target audio fingerprint from among the reference audio fingerprints further comprises:

identifying the reference audio fingerprint that has the shortest Euclidean distance to the target audio fingerprint.

19. The method of claim 15 wherein the determining of the temporal offset between the video content and audio content further comprises:

through use of at least the time stamp data associated with the target audio fingerprint data and the time stamp data associated with the reference audio fingerprint data, obtaining one or more audio time stamp offsets indicative of one or more temporal offsets between the target audio fingerprint and one or more of the reference audio fingerprints that at least substantially match the target audio fingerprint.

20. The method of claim 19 wherein the reference video fingerprint data includes one or more reference video fingerprints that are characteristic of the video content prior to the transmission of the video over the media channel, wherein the target video fingerprint data includes at least one target video fingerprint that is characteristic of the video content subsequent to the transmission of the video over the media channel, and wherein the determining of the temporal offset between the video content and audio content further comprises:

performing fingerprint matching of the target video fingerprint against one or more of the reference video fingerprints to obtain one or more reference video fingerprints that at least substantially match the target video fingerprint.

21. The method of claim 20 wherein the performing of the fingerprint matching of the target video fingerprint against one or more of the reference video fingerprints comprises:

searching for one or more nearest neighbors of the target video fingerprint from among the reference video fingerprints to obtain one or more reference video fingerprints that at least substantially match the target video fingerprint.

22. The method of claim 21 wherein the searching for one or more nearest neighbors of the target video fingerprint from among the reference video fingerprints comprises:

obtaining an Euclidean distance between the target video fingerprint and each of the reference video fingerprints.

23. The method of claim 22 wherein the searching for one or more nearest neighbors of the target video fingerprint from among the reference video fingerprints further comprises:

identifying the reference video fingerprint that has the shortest Euclidean distance to the target video fingerprint.

24. The method of claim 20 wherein the determining of the temporal offset between the video content and audio content further comprises:

through use of at least the time stamp data associated with the target video fingerprint data and the time stamp data associated with the reference video fingerprint data, obtaining one or more video time stamp offsets indicative of one or more temporal offsets between the target video fingerprint and one or more of the reference video fingerprints that at least substantially match the target video fingerprint.

25. The method of claim 24 wherein the determining of the temporal offset between the video content and audio content further comprises:

through use of at least (a) one or more of the audio time stamp offsets, and (b) one or more of the video time stamp offsets, obtaining a value indicative of the temporal offset between the video content and audio content.

26. The method of claim 25 wherein the obtaining of the value indicative of the temporal offset comprises:

through use of a histogram count, determining a number of occurrences within a predetermined moving window of each of the one or more audio time stamp offsets.

27. The method of claim 26 wherein the obtaining of the value indicative of the temporal offset further comprises:

identifying, from among the one or more audio time stamp offsets, the audio time stamp offset that has the greatest number of occurrences within the predetermined moving window.

28. The method of claim 27 wherein the obtaining of the value indicative of the temporal offset further comprises:

through use of a histogram count, determining a number of occurrences within the predetermined moving window of each of the one or more video time stamp offsets.

29. The method of claim 28 wherein the obtaining of the value indicative of the temporal offset further comprises:

identifying, from among the one or more video time stamp offsets, the video time stamp offset that has the greatest number of occurrences within the predetermined moving window.

30. The method of claim 29 wherein the obtaining of the value indicative of the temporal offset further comprises:

determining a difference between the video time stamp offset that has the greatest number of occurrences within the predetermined moving window and the audio time stamp offset that has the greatest number of occurrences within the predetermined moving window, such that the obtained value is indicative of the temporal offset between the video content and audio content.

31. A system for measuring a temporal offset between video content and audio content, the video content and the audio content being included in video that is subject to transmission over a media channel, the system comprising:

a reference audio fingerprint extractor operative, prior to the transmission of the video over the media channel, to obtain reference audio fingerprint data from the audio content;

a reference video fingerprint extractor operative, prior to the transmission of the video over the media channel, to obtain reference video fingerprint data from the video content,

wherein each of the reference audio fingerprint data and the reference video fingerprint data includes associated time stamp data;

a target audio fingerprint extractor operative, subsequent to the transmission of the video over the media channel, to obtain target audio fingerprint data from the audio content;

a target video fingerprint extractor operative, subsequent to the transmission of the video over the media channel, to obtain target video fingerprint data from the video content,

wherein each of the target audio fingerprint data and the target video fingerprint data includes associated time stamp data,

wherein the reference audio fingerprint data includes one or more reference audio fingerprints characteristic of the audio content prior to the transmission of the video over the media channel, and

wherein the target audio fingerprint data includes at least one target audio fingerprint characteristic of the audio content subsequent to the transmission of the video over the media channel;

an audio fingerprint matcher operative (a) to perform fingerprint matching of the target audio fingerprint against one or more of the reference audio fingerprints to obtain one or more reference audio fingerprints that at least substantially match the target audio fingerprint, and, (b) through use of at least the time stamp data associated with the target audio fingerprint data and the time stamp data associated with the reference audio fingerprint data, to obtain one or more audio time stamp offsets indicative of one or more temporal offsets between the target audio fingerprint and one or more of the reference audio fingerprints that at least substantially match the target audio fingerprint,

wherein the reference video fingerprint data includes one or more reference video fingerprints characteristic of the video content prior to the transmission of the video over the media channel, and

wherein the target video fingerprint data includes at least one target video fingerprint characteristic of the video content subsequent to the transmission of the video over the media channel;

a video fingerprint matcher operative (a) to perform fingerprint matching of the target video fingerprint against one or more of the reference video fingerprints to obtain one or more reference video fingerprints that at least substantially match the target video fingerprint, and, (b) through use of at least the time stamp data associated with the target video fingerprint data and the time stamp data associated with the reference video fingerprint data, to obtain one or more video time stamp offsets indicative of one or more temporal offsets between the target video fingerprint and one or more of the reference video fingerprints that at least substantially match the target video fingerprint; and

an offset estimator operative, through use of at least (a) one or more of the audio time stamp offsets and (b) one or more of the video time stamp offsets, to obtain a value indicative of the temporal offset between the video content and audio content.

32. A system for measuring a temporal offset between video content and audio content, the video content and the audio content being included in video that is subject to transmission over a media channel, the system comprising:

a reference fingerprint database operative, prior to the transmission of the video over the media channel, to store reference audio fingerprint data from the audio content and reference video fingerprint data from the video content,

wherein each of the reference audio fingerprint data and the reference video fingerprint data includes associated time stamp data;

a target audio fingerprint extractor operative, subsequent to the transmission of the video over the media channel, to obtain target audio fingerprint data from the audio content;

a target video fingerprint extractor operative, subsequent to the transmission of the video over the media channel, to obtain target video fingerprint data from the video content,

wherein each of the target audio fingerprint data and the target video fingerprint data includes associated time stamp data,

wherein the reference audio fingerprint data includes one or more reference audio fingerprints characteristic of the audio content prior to the transmission of the video over the media channel, and

wherein the target audio fingerprint data includes at least one target audio fingerprint characteristic of the audio content subsequent to the transmission of the video over the media channel;

an audio fingerprint matcher operative (a) to perform fingerprint matching of the target audio fingerprint against one or more of the reference audio fingerprints to obtain one or more reference audio fingerprints that at least substantially match the target audio fingerprint, and, (b) through use of at least the time stamp data associated with the target audio fingerprint data and the time stamp data associated with the reference audio fingerprint data, to obtain one or more audio time stamp offsets indicative of one or more temporal offsets between the target audio fingerprint and one or more of the reference audio fingerprints that at least substantially match the target audio fingerprint,

wherein the reference video fingerprint data includes one or more reference video fingerprints characteristic of the video content prior to the transmission of the video over the media channel, and

wherein the target video fingerprint data includes at least one target video fingerprint characteristic of the video content subsequent to the transmission of the video over the media channel;

a video fingerprint matcher operative (a) to perform fingerprint matching of the target video fingerprint against one or more of the reference video fingerprints to obtain one or more reference video fingerprints that at least substantially match the target video fingerprint, and, (b) through use of at least the time stamp data associated with the target video fingerprint data and the time stamp data associated with the reference video fingerprint data, to obtain one or more video time stamp offsets indicative of one or more temporal offsets between the target video fingerprint and one or more of the reference video fingerprints that at least substantially match the target video fingerprint; and

an offset estimator operative, through use of at least (a) one or more of the audio time stamp offsets and (b) one or more of the video time stamp offsets, to obtain a value indicative of the temporal offset between the video content and audio content.