Techniques for synchronizing audio and video data in an image signal processing system

• US 8,736,700 B2
• Filed: 09/30/2010
• Issued: 05/27/2014
• Est. Priority Date: 09/30/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Ct, Guy, Frederiksen, Jeffrey E.
• Primary Examiner(s): Tran, Nhan T
• Assistant Examiner(s): NGUYEN, CHAN T H
• Application Number:
  US12/895,299
• Publication Number:
  US 20120081567A1
• Time in Patent Office: 1,335 Days
• Field of Search: 348/222.1, 348/231.99, 348/231.3, 348/231.4, 348/231.5, 348/231.6, 725/105, 386/201, 386/203
• US Class Current: 348/222.1
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4302 Content synchronisation processes, e.g. decoder synchronisation

Abstract

The present disclosure provides techniques for performing audio-video synchronization using an image signal processing system. In one embodiment, a time code register provides a current time stamp when sampled. The value of the time code register may be incremented at regular intervals based on a clock of the image signal processing system. At the start of a current frame acquired by an image sensor, the time code register is sampled, and a timestamp is stored into a timestamp register associated with the image sensor. The timestamp is then read from the time stamp register and written to a set of metadata associated with the current frame. The timestamp stored in the frame metadata may then be used to synchronize the current frame with a corresponding set of audio data.

Claims

1. A method for synchronizing audio and video data on an electronic device comprising:

using an image signal processor to receive a plurality of image frames corresponding to the video data, the plurality of image frames being acquired using a digital image sensor of an electronic device;

detecting the start of each image frame of the plurality of image frames;

upon detecting the start of each image frame, reading a respective current timestamp value from a time code register configured to provide timestamp values corresponding to a current time;

associating each respective current timestamp value with a respective set of metadata associated with the respective image frame, wherein each respective current timestamp value corresponds to the start of each respective image frame; and

aligning each image frame of the plurality of image frames with a corresponding audio sample from a set of audio data acquired concurrently with the video data by the electronic device using the respective current timestamp value associated with the respective set of metadata to synchronize the playback of each image frame.

2. The method of claim 1, wherein associating each respective current timestamp value with the respective set of metadata comprises storing each respective current timestamp value in a timestamp register associated with the digital image sensor and writing each respective current timestamp value stored in the timestamp register to the respective set of metadata.

3. The method of claim 1, wherein each respective current timestamp value provided by the time code register is incremented at intervals defined by a particular number of clock cycles based on a clock domain of the image signal processor.

4. The method of claim 3, wherein the particular number of clock cycles is determined based at least partially upon on a time configuration value, n.

5. The method of claim 3, wherein the particular number of clock cycles is determined as being 2^n.

6. The method of claim of claim 1, wherein detecting the start of the image frame comprises detecting a rising edge in a vertical synchronization signal.

7. The method of claim 6, wherein a programmed delay is added to each respective current timestamp value prior to associating each respective current timestamp value with the respective set of metadata.

8. A system comprising:

an image signal processor comprising an interface configured to receive a plurality of image frames acquired using a digital image sensor, wherein the plurality of image frames correspond to a set of video data;

an audio sub-system comprising an input device configured to receive audio data concurrently with the image data acquired by the digital image sensor; and

audio-video synchronization logic configured to, for each of the plurality of image frames, identify the start of the image frame, determine a timestamp value corresponding to the start of the image frame, and write the timestamp value to metadata associated with the image frame;

wherein the audio-video synchronization logic is configured to synchronize the plurality of image frames with the audio data by correlating the timestamps values stored in the metadata associated with each of the plurality of image frames to corresponding timestamps associated with the audio data.

9. The system of claim 8, wherein the audio-video synchronization logic comprises a time code register configured to provide current timestamp values, wherein the time stamp values provided by the time code register are incremented at regular intervals.

10. The system of claim 9, wherein the image signal processor and the audio-subsystem operate based on independent first and second clocks, respectively.

11. The system of claim 10, wherein the audio-video synchronization logic comprises a timer configuration register configured to store a time configuration code, wherein the regular intervals correspond to a number of clock cycles of the first clock determined as a function of the time configuration code.

12. The system of claim 11, wherein the time configuration code comprises a 4-bit binary value having a range of 0 to 15, and wherein the number of clock cycles is determined as 2 raised to the power of the value expressed by the 4-bit time configuration code.

13. The system of claim 10, wherein the audio-video synchronization logic is configured to synchronize the first clock with a third clock of a main processor of the system, synchronize the second clock with the third clock of the main processor of the system, determine a time difference between the first and second clocks based upon the synchronization of the first and second clocks to the third clock, and correlate the timestamp values stored in the metadata of each of the plurality of image frames to corresponding timestamps associated with the audio data using the time difference.

14. The system of claim 9, wherein the audio-video synchronization logic is configured to synchronize the plurality of image frames with the audio data by adding or dropping image frames until the timestamps corresponding to the image frames are aligned with timestamps corresponding to the audio data.

15. One or more non-transitory tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to detect the beginning of a plurality of image frames from a set of video data acquired by an image processing system of an electronic device using a digital image sensor;

code to associate a timestamp value with metadata associated with each image frame of the plurality of image frames, wherein each respective timestamp value corresponds to the beginning of each respective image frame;

code to align each respective image frame with a corresponding audio sample from a set of audio data acquired by the electronic device using each respective timestamp value.

16. The non-transitory one or more tangible computer-readable storage media of claim 15, wherein the code to associate the timestamp value with the metadata associated with the image frame comprises:

code to read the timestamp value from a first register configured provide a current time value that is incremented periodically;

code to write the timestamp value to a second register associated with the digital image sensor; and

code to store the timestamp value from the second register as part of the metadata associated with a respective image frame of the plurality of image frames.

17. The non-transitory one or more tangible computer-readable storage media of claim 16, comprising:

code to read a timer configuration value, n, from a time configuration register;

code to determine a value equal to 2^n; and

code for incrementing the current time value of the first register for every 2^n clock cycles of a clock of the image processing system.

18. The non-transitory one or more tangible computer-readable storage media of claim 16, comprising:

code to detect when the current time value of the first register exceeds a maximum bit-resolution of the first register;

code to, if the current time value of the first register exceeds the maximum bit resolution of the first register, determine a higher-resolution current time value having a bit-resolution higher than the bit-resolution of the first register; and

code to write the higher-resolution current time value to the metadata associated with the image frame until the first register is reset.

19. An electronic device comprising:

a first image sensor;

a first image sensor interface;

an image signal processing sub-system configured to receive a first set of video data acquired using the first image sensor via the first sensor interface, wherein the first set of video data comprises a first plurality of image frames;

an audio processing sub-system configured to receive a first set of audio data acquired concurrently with the first set of video data; and

audio-video synchronization circuitry comprising;

a time code register configured to provide a current time value;

a first timestamp register associated with the first image sensor; and

logic configured to, for each image frame of the first set of video data, detect the start of an image frame of the first set of video data, sample the time code register upon detecting the start of the image frame to acquire a timestamp value corresponding to the start of the image frame, store the timestamp value in the first timestamp register, write the timestamp value from the first timestamp register to a set of metadata associated with the image frame, and synchronize the first set of video data and the first set of audio data by correlating the timestamp values in the metadata of each image frame of first set of the video data to corresponding timestamps associated with the first set of audio data.

20. The electronic device of claim 19, comprising:

a second image sensor;

a second image sensor interface, wherein the image signal processing sub-system is configured to receive a second set of video data acquired using the second image sensor via the second sensor interface, wherein the second set of video data comprises a second plurality of image frames, and wherein the audio processing sub-system is configured to receive a second set of audio data acquired concurrently with the second set of video data; and

wherein the audio-video synchronization circuitry comprises a second timestamp register associated with the second image sensor and logic configured to, for each image frame of the second set of video data, detect the start of an image frame of the second set of video data, sample the time code register upon detecting the start of the image frame to acquire a timestamp value corresponding to the start of the image frame, store the timestamp value in the second timestamp register, write the timestamp value from the second timestamp register to a set of metadata associated with the image frame, and synchronize the second set of video data and the second set of audio data by correlating the timestamp values in the metadata of each image frame of the second set of video data to timestamps associated with the second set of audio data.

21. The electronic device of claim 20, comprising a display device configured to display the first and second sets of video data, wherein the display device comprises at least one of an LCD display device, a plasma display device, an LED display device, or an OLED display device or some combination thereof.

22. The electronic device of claim 20, comprising:

an audio input device configured to acquire the first set and second sets of audio data, wherein the audio input device comprises at least one microphone; and

an audio output device configured to playback the first and second sets of audio data.

23. The electronic device of claim 20, wherein first and second image sensors comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

24. The electronic device of claim 20, wherein the first and second sensor interfaces comprises at least one of a Standard Mobile Imaging Architecture (SMIA) interface, a serial sensor interface, a parallel sensor interface, or some combination thereof

25. The electronic device of claim 19, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof

26. The electronic device of claim 19, wherein if the first set of video data and the first set of audio data are not aligned, the audio-video synchronization circuitry is configured to repeat or drop image frames to bring the timestamps of the first set of video data and the timestamps associated with the first set of audio data into alignment.