TECHNIQUES FOR SYNCHRONIZING AUDIO AND VIDEO DATA IN AN IMAGE SIGNAL PROCESSING SYSTEM

• US 20120081567A1
• Filed: 09/30/2010
• Published: 04/05/2012
• Est. Priority Date: 09/30/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Côté, Guy, Frederiksen, Jeffrey E.
• Granted Patent: US 8,736,700 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4302 Content synchronisation processes, e.g. decoder synchronisation

Abstract

The present disclosure provides techniques for performing audio-video synchronization using an image signal processing system. In one embodiment, a time code register provides a current time stamp when sampled. The value of the time code register may be incremented at regular intervals based on a clock of the image signal processing system. At the start of a current frame acquired by an image sensor, the time code register is sampled, and a timestamp is stored into a timestamp register associated with the image sensor. The timestamp is then read from the time stamp register and written to a set of metadata associated with the current frame. The timestamp stored in the frame metadata may then be used to synchronize the current frame with a corresponding set of audio data.

Claims

1. A method for synchronizing audio and video data on an electronic device comprising:

using an image signal processor to receive an image frame of the video data, the image frame being acquired using a digital image sensor of an electronic device;

detecting the start of the image frame;

upon detecting the start of the image frame, reading a current timestamp value from a time code register configured to provide timestamp values corresponding to a current time;

associating the timestamp value with a set of metadata associated with the image frame; and

using the timestamp value associated with the set of metadata to synchronize the playback of the image frame with a corresponding audio sample from a set audio data acquired concurrently with the video data on the electronic device.

2. The method of claim 1, wherein associating the timestamp value with the set of metadata comprises storing the timestamp value in a timestamp register associated with the digital image sensor and writing the timestamp value stored in the timestamp register to the set of metadata.

3. The method of claim 1, wherein the timestamp values provided by the time code register are incremented at intervals defined by a particular number of clock cycles based on a clock domain of the image signal processor.

4. The method of claim 3, wherein the particular number of clock cycles is determined based at least partially upon on a time configuration value, n.

5. The method of claim 1, wherein the particular number of clock cycles is determined as being 2̂

n.

6. The method of claim of claim 4, wherein detecting the start of the image frame comprises detecting a rising edge in a vertical synchronization signal.

7. The method of claim 6, wherein a programmed delay is added to the timestamp value prior to associating the timestamp value with the set of metadata.

8. A system comprising:

an image signal processor comprising an interface configured to receive a plurality of image frames acquired using a digital image sensor, wherein the plurality of image frames correspond to a set of video data;

an audio sub-system comprising an input device configured to receive audio data concurrently with the image data acquired by the digital image sensor; and

audio-video synchronization logic configured to, for each of the plurality of image frames, identify the start of the image frame, determine a timestamp value corresponding to the start of the image frame, and write the timestamp value to metadata associated with the image frame;

wherein the audio-video synchronization logic is configured to synchronize the plurality of image frames with the audio data by correlating the timestamps values stored in the metadata associated with each of the plurality of image frames to corresponding timestamps associated with the audio data.

9. The system of claim 8, wherein the audio-video synchronization logic comprises a time code register configured to provide current timestamp values, wherein the time stamp values provided by the time code register are incremented at regular intervals.

10. The system of claim 9, wherein the image signal processor and the audio-subsystem operate based on independent first and second clocks, respectively.

11. The system of claim 10, wherein the audio-video synchronization logic comprises a timer configuration register configured to store a time configuration code, wherein the regular intervals correspond to a number of clock cycles of the first clock determined as a function of the time configuration code.

12. The system of claim 11, wherein the time configuration code comprises a 4-bit binary value having a range of 0 to 15, and wherein the number of clock cycles is determined as 2 raised to the power of the value expressed by the 4-bit time configuration code.

13. The system of claim 10, wherein the audio-video synchronization logic is configured to synchronize the first clock with a third clock of a main processor of the system, synchronize the second clock with the third clock of the main processor of the system, determine a time difference between the first and second clocks based upon the synchronization of the first and second clocks to the third clock, and correlate the timestamp values stored in the metadata of each of the plurality of image frames to corresponding timestamps associated with the audio data using the time difference.

14. The system of claim 9, wherein the audio-video synchronization logic is configured to synchronize the plurality of image frames with the audio data by adding or dropping image frames until the timestamps corresponding to the image frames are aligned with timestamps corresponding to the audio data.

15. One or more tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to detect the beginning of an image frame from a set of video data acquired by an image processing system of an electronic device using a digital image sensor;

code to associate a timestamp value with metadata associated with the image frame, wherein the timestamp value corresponds to the beginning of the image frame;

code to align the image frame with a corresponding audio sample from a set of audio data acquired by the electronic device using the timestamp value.

16. The one or more tangible computer-readable storage media of claim 15, wherein the code to associate the timestamp value with the metadata associated with the image frame comprises:

code to read the timestamp value from a first register configured provide a current time value that is incremented periodically;

code to write the timestamp value to a second register associated with the digital image sensor; and

code to store the timestamp value from the second register as part of the metadata associated with the image frame.

17. The one or more tangible computer-readable storage media of claim 16, comprising:

code to read a timer configuration value, n, from a time configuration register;

code to determine a value equal to 2̂

n; and

code for incrementing the current time value of the first register for every 2̂

n clock cycles of a clock of the image processing system.

18. The one or more tangible computer-readable storage media of claim 16, comprising:

code to detect when the current time value of the first register exceeds a maximum bit-resolution of the first register;

code to, if the current time value of the first register exceeds the maximum bit resolution of the first register, determine a higher-resolution current time value having a bit-resolution higher than the bit-resolution of the first register; and

code to write the higher-resolution current time value to the metadata associated with the image frame until the first register is reset.

19. An electronic device comprising:

a first image sensor;

a first image sensor interface;

an image signal processing sub-system configured to receive a first set of video data acquired using the first image sensor via the first sensor interface;

an audio processing sub-system configured to receive a first set of audio data acquired concurrently with the first set of video data; and

audio-video synchronization circuitry comprising;

a time code register configured to provide a current time value;

a first timestamp register associated with the first image sensor; and

logic configured to, for each image frame of the first set of video data, detect the start of an image frame of the first set of video data, sample the time code register upon detecting the start of the image frame to acquire a timestamp value corresponding to the start of the image frame, store the timestamp value in the first timestamp register, write the timestamp value from the first timestamp register to a set of metadata associated with the image frame, and synchronize the first set of video data and the first set of audio data by correlating the timestamp values in the metadata of each image frame of first set of the video data to corresponding timestamps associated with the first set of audio data.

20. The electronic device of claim 19, comprising:

a second image sensor;

a second image sensor interface, wherein the image signal processing sub-system is configured to receive video data acquired using the second image sensor via the second sensor interface, and wherein the audio processing sub-system is configured to receive a second set of audio data acquired concurrently with the second set of video data; and

wherein the audio-video synchronization circuitry comprises a second timestamp register associated with the second image sensor and logic configured to, for each image frame of the second set of video data, detect the start of an image frame of the second set of video data, sample the time code register upon detecting the start of the image frame to acquire a timestamp value corresponding to the start of the image frame, store the timestamp value in the second timestamp register, write the timestamp value from the second timestamp register to a set of metadata associated with the image frame, and synchronize the second set of video data and the second set of audio data by correlating the timestamp values in the metadata of each image frame of the second set of video data to timestamps associated with the second set of audio data.

21. The electronic device of claim 20, comprising a display device configured to display the first and second sets of video data, wherein the display device comprises at least one of an LCD display device, a plasma display device, an LED display device, or an OLED display device or some combination thereof.

22. The electronic device of claim 20, comprising:

an audio input device configured to acquire the first set and second sets of audio data, wherein the audio input device comprises at least one microphone; and

an audio output device configured to playback the first and second sets of audio data.

23. The electronic device of claim 20, wherein first and second image sensors comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

24. The electronic device of claim 20, wherein the first and second sensor interfaces comprises at least one of a Standard Mobile Imaging Architecture (SMIA) interface, a serial sensor interface, a parallel sensor interface, or some combination thereof.

25. The electronic device of claim 19, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

26. The electronic device of claim 19, wherein if the first set of video data and the first set of audio data are not aligned, the audio-video synchronization circuitry is configured to repeat or drop image frames to bring the timestamps of the first set of video data and the timestamps associated with the first set of audio data into alignment.