Combining data from multiple image sensors

• US 8,896,668 B2
• Filed: 04/04/2011
• Issued: 11/25/2014
• Est. Priority Date: 04/05/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Goma, Sergiu R., Aleksic, Milivoje, Hwang, Hau, Cheung, Joseph
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): KALAPODAS, DRAMOS
• Application Number:
  US13/079,640
• Publication Number:
  US 20120081519A1
• Time in Patent Office: 1,331 Days
• Field of Search: 348/47, 348/E13.074, 348/51, 348/36, 348142-187, 348/189, 348/48, 348/495, 348/584, 348/588, 348/590, 348/589, 348/705, 348/706, 382/294, 359/403, 345/418, 345/419, 358/183, 358/181, 358/182, 358/180, 358/160, 358/22, 358/142, 358/133, 358/12, 358/147
• US Class Current: 348/47
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method of combining data from multiple sensors is disclosed. The method includes receiving lines of image data at an image processor having an input for a single camera. Each line of the image data includes first line data from a first image captured by a first camera and second line data from a second image captured by a second camera. The method also includes generating an output frame having a first section corresponding to line data of the first image and having a second section corresponding to line data of the second image. The first section and the second section are configured to be used to generate a three-dimensional (3D) image format or a 3D video format.

Claims

1. A method comprising:

receiving lines of image data at an image processor having an input for a single camera, wherein each line of the image data is a combined line including a first data line from a first image captured by a first camera and a second data line from a second image captured by a second camera, wherein a combiner coupled to the first camera and to the second camera generates each line of the image data by receiving and aligning one first data line with one second data line to form one combined line, and wherein the combiner and the image processor are included in a system-on-chip; and

generating an output frame based on the combined lines, the output frame having a first section corresponding to the first data lines of the combined lines and having a second section corresponding to the second data lines of the combined lines, wherein the first section and the second section are configured to be used to generate data in a three-dimensional (3D) image format or in a 3D video format.

2. The method of claim 1, further comprising:

synchronizing, using the combiner, line by line readout of first image data from the first camera and second image data from the second camera to generate each line of the image data; and

storing each combined line as one line of the image data in a separate row of a memory.

3. The method of claim 1, further comprising:

storing each combined line as one line of the image data in a separate row of a memory.

4. The method of claim 1, further comprising processing the output frame to generate 3D image data and sending the 3D image data to a display device, wherein the display device is a component of at least one of a communication device, a third camera, a personal digital assistant, and a computer.

5. The method of claim 1, wherein each line of the image data includes a gap section between the first data line and the second data line, wherein the gap section is between the first section and the second section in the output frame, and wherein the gap section comprises non-image data.

6. The method of claim 1, wherein the input for the single camera comprises a single camera parallel interface or serial interface and an inter-integrated circuit interface.

7. The method of claim 1, wherein a control signal is provided by the image processor to the first camera and to the second camera.

8. The method of claim 7, wherein the control signal is provided by the image processor via a sensor synchronizer, and wherein the image processor corresponds to a video front end.

9. An apparatus comprising:

an image processor having an input for a single camera, wherein the image processor is configured to receive lines of image data via the input, each line of the image data being a combined line including a first data line from a first image captured by a first camera and a second data line from a second image captured by a second camera,

wherein the image processor is configured to generate an output frame based on the combined lines, the output frame having a first section corresponding to the first data lines of the combined lines and having a second section corresponding to the second data lines of the combined lines, and wherein the first section and the second section are configured to be used to generate data with a three-dimensional (3D) image format or to generate data with a 3D video format; and

a combiner coupled to the first camera and to the second camera, wherein the combiner is configured to generate each line of the image data by receiving and aligning one first data line with one second data line to form one combined line, and wherein the image processor and the combiner are included in a system-on-chip.

10. The apparatus of claim 9, wherein the combiner is configured to align line by line readout of first image data from the first camera and second image data from the second camera to generate each line of the image data, and wherein a synchronization between data streams from the first and second cameras received at the combiner occurs within a single line of the image data.

11. The apparatus of claim 9, further comprising a graphics processing unit configured to process the output frame to generate 3D image data or 3D video data.

12. The apparatus of claim 9, further comprising;

a graphics processing unit configured to process the output frame to generate 3D image data or 3D video data and

a display device coupled to display the 3D image data or the 3D video data, wherein the display device is a component of at least one of a communication device, a third camera, a personal digital assistant, and a computer.

13. The apparatus of claim 9, further comprising a graphics processing unit configured to process the output frame to generate 3D image data or 3D video data, wherein the input for the single camera comprises a single camera parallel interface or serial interface and an inter-integrated circuit interface.

14. The apparatus of claim 9, further comprising:

a sensor synchronizer coupled to the first camera and to the second camera, wherein the sensor synchronizer generates a common control clock signal, a common control data signal, a common camera clock signal and a common camera reset signal; and

a memory configured to store each combined line as one line of the image data in a separate row of the memory.

15. The apparatus of claim 14, wherein the image processor is configured to provide the common camera clock signal via the sensor synchronizer to the first camera and to the second camera, and wherein the image processor corresponds to a video front end.

16. A non-transitory computer readable-medium comprising instructions, which when executed by a processor cause the processor to:

receive lines of image data via an input for a single camera, wherein each line of the image data is a combined line including a first data line from a first image captured by a first camera and a second data line from a second image captured by a second camera, wherein a combiner coupled to the first camera and to the second camera generates each line of the image data by receiving and aligning one first data line with one second data line to form one combined line, and wherein the combiner and the processor are included in a system-on-chip; and

generate an output frame based on the combined lines, the output frame having a first section corresponding to the first data lines of the combined lines and having a second section corresponding to the second data lines of the combined lines, wherein the first section and the second section are configured to be used to generate data in a three-dimensional (3D) image format or in a 3D video format.

17. The non-transitory computer readable-medium of claim 16, wherein a synchronization between data streams from the first and second cameras received at the combiner occurs within a single line of the image data, and wherein a graphics processor is caused to:

process the output frame to generate 3D image data; and

send the 3D image data to a display device.

18. The non-transitory computer readable-medium of claim 16, further comprising instructions that, when executed by the processor, cause the processor to store each combined line as one line of the image data in a separate row of a memory.

19. The non-transitory computer readable-medium of claim 18, wherein instructions further cause the processor to provide a control signal to the first camera and to the second camera, and wherein the processor corresponds to a video front end.

20. An apparatus comprising:

means for receiving lines of image data at an input for a single camera, wherein each line of the image data is a combined line including a first data line from a first image captured by a first camera and a second data line from a second image captured by a second camera,

wherein the means for receiving lines is configured to generate an output frame based on the combined lines, the output frame having a first section corresponding to the first data lines of the combined lines and having a second section corresponding to the second data lines of the combined lines, wherein the first section and the second section are configured to be used to generate data in a three-dimensional (3D) image format or in a 3D video format; and

means for combining the first data lines and the second data lines, wherein the means for combining is coupled to the first camera and to the second camera, wherein the means for combining is configured to generate each line of the image data by receiving and aligning one first data line with one second data line to form one combined line, and wherein the means for combining and the means for receiving lines are included in a system-on-chip.

21. The apparatus of claim 20, wherein the means for receiving is configured to provide a control signal to the first and second cameras, and wherein the apparatus further comprises:

means for processing the output frame to generate 3D image data; and

means for sending the 3D image data to a display device.

22. The apparatus of claim 20, further comprising means for synchronizing sensors coupled to the first camera and to the second camera, wherein the first camera and the second camera are coupled to receive signals including a common startup or reset signal and a common clock input, and wherein the signals are provided to the first camera and to the second camera from the means for synchronizing sensors.