Combining data from multiple image sensors

• US 9,001,227 B2
• Filed: 04/04/2011
• Issued: 04/07/2015
• Est. Priority Date: 04/05/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Aleksic, Milivoje, Goma, Sergiu R., Hwang, Hau, Cheung, Joseph
• Primary Examiner(s): Berhan, Ahmed A
• Application Number:
  US13/079,629
• Publication Number:
  US 20110242356A1
• Time in Patent Office: 1,464 Days
• Field of Search: 348/159, 348/143, 348/157, 348/207.99, 348/218.1, 348/262, 348/264, 348/265, 348 36- 50, 348/222, 348/1, 348/221.1
• US Class Current: 348/218.1
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/665 involving internal camera communication with the image sensor, e.g. synchronising or multiplexing SSIS control signals
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A method of combining data from multiple sensors is disclosed. The method includes providing a common control signal to multiple image sensors. Each of the multiple image sensors is responsive to the common control signal to generate image data. The method also includes receiving synchronized data output from each of the multiple image sensors.

Claims

1. A method comprising:

providing a common control signal to multiple image sensors, wherein each of the multiple image sensors is responsive to the common control signal to generate data lines, and wherein each data line corresponds to a line of image data;

receiving synchronized data output from each of the multiple image sensors;

rectifying the synchronized data output using a calibration matrix, wherein the calibration matrix accounts for non-ideal position and alignment of the multiple image sensors; and

forming a combined image frame, wherein the combined image frame comprises image sections including image data from the multiple image sensors and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the multiple image sensors.

2. The method of claim 1, wherein the synchronized data output includes first data lines received from a first image sensor and second data lines received from a second image sensor, and wherein a phase offset between each received data line from the first image sensor and each corresponding data line from the second image sensor is substantially constant.

3. The method of claim 1, wherein the multiple image sensors capture image data from a common focal plane of a main lens to generate the synchronized data output.

4. The method of claim 1, wherein the common control signal is provided by a sensor synchronizer that is coupled to each of the multiple image sensors via an inter-integrated circuit (I2C) control interface.

5. The method of claim 1, wherein the synchronized data output includes multiple data lines, wherein each data line includes first data corresponding to a first line of a first image sensor, second data corresponding to a first line of a second image sensor, and multiple bits of non-image data between the first data and the second data.

6. The method of claim 1, wherein the common control signal is provided by a sensor synchronizer that is coupled to each of the multiple image sensors via an interface compliant with a camera serial interface (CSI) specification or a camera parallel interface (CPI) specification.

7. The method of claim 1, wherein the gap sections correspond to black pixels.

8. A method comprising:

receiving a common control signal at multiple image sensors, wherein each of the multiple image sensors is responsive to the common control signal to generate data lines, and wherein each data line corresponds to a line of image data; and

generating synchronized data output from each of the multiple image sensors, wherein the synchronized data output from each of the multiple image sensors corresponds to image data, and wherein each of the multiple image sensors captures corresponding image data from a different portion of a common focal plane of a main lens,

wherein a combined image frame comprises image sections including image data from the multiple image sensors and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the multiple image sensors.

9. The method of claim 8, wherein the synchronized data output includes first data lines generated by a first image sensor and second data lines generated by a second image sensor, and wherein a phase offset between each data line output from the first image sensor and each corresponding data line output from the second image sensor is substantially constant.

10. The method of claim 8, wherein the common control signal is received from a sensor synchronizer that is coupled to each of the multiple image sensors.

11. The method of claim 8, wherein the common control signal is received at each of the multiple image sensors via an inter-integrated circuit (I2C) control interface, and wherein the common control signal includes a camera reset signal, a camera clock signal, a control clock signal, and a control data signal.

12. The method of claim 8, wherein the gap sections correspond to black pixels.

13. The method of claim 8, wherein the common control signal is received at each of the multiple image sensors via an interface compliant with a camera serial interface (CSI) specification.

14. The method of claim 8, wherein the common control signal is received at each of the multiple image sensors via an interface compliant with a camera parallel interface (CPI) specification.

15. An apparatus comprising:

a sensor synchronizer configured to provide a common control signal to multiple image sensors to cause the multiple image sensors to generate data lines, wherein each data line corresponds to a line of image data;

a sensor data interface configured to receive synchronized data output from each of the multiple image sensors;

a memory device to store a calibration matrix, wherein the calibration matrix is usable to rectify the synchronized data output to account for non-ideal position and alignment of the multiple image sensors; and

an image processor configured to form a combined image frame, wherein the combined image frame comprises image sections including image data from the multiple image sensors and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the multiple image sensors.

16. The apparatus of claim 15, wherein the sensor synchronizer is coupled to each of the multiple image sensors via an inter-integrated circuit (I2C) control interface.

17. The apparatus of claim 15, wherein the common control signal includes a camera reset signal, a camera clock signal, a control clock signal, and a control data signal.

18. The apparatus of claim 15, wherein the sensor synchronizer is coupled to each of the multiple image sensors via an interface compliant with a camera serial interface (CSI) specification or a camera parallel interface (CPI) specification.

19. The apparatus of claim 15, wherein the gap sections correspond to black pixels.

20. An apparatus comprising:

a first image sensor;

a second image sensor, wherein the first image sensor and the second image sensor are configured to receive a common control signal and are responsive to the common control signal to generate synchronized data output, wherein the synchronized data output from the first image sensor and the second image sensor corresponds to data lines, wherein each data line corresponds to a line of image data, and wherein the first image sensor and the second image sensor capture corresponding image data from different portions of a common focal plane of a main lens; and

an image processor configured to form a combined image frame, wherein the combined image frame comprises image sections including image data from the first image sensor and the second image sensor and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the first image sensor and the second image sensor.

21. The apparatus of claim 20, wherein the synchronized data output includes first data lines generated by the first image sensor and second data lines generated by the second image sensor, and wherein a phase offset between each data line output from the first image sensor and each corresponding data line output from the second image sensor is substantially constant.

22. The apparatus of claim 20, wherein the first image sensor and the second image sensor are configured to receive the common control signal via an inter-integrated circuit (I2C) control interface.

23. The apparatus of claim 20, wherein the first image sensor and the second image sensor are configured to receive the common control signal via an interface compliant with a camera serial interface (CSI) specification.

24. The apparatus of claim 20, wherein the first image sensor and the second image sensor are configured to receive the common control signal via an interface compliant with a camera parallel interface (CPI) specification.

25. An apparatus comprising:

means for providing a common control signal to multiple image sensors to cause the multiple image sensors to generate data lines, wherein each data line corresponds to a line of image data;

means for receiving synchronized data output from each of the multiple image sensors;

means for rectifying the synchronized data output using a calibration matrix, wherein the calibration matrix accounts for non-ideal position and alignment of the multiple image sensors; and

means for forming a combined image frame, wherein the combined image frame comprises image sections including image data from the multiple image sensors and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the multiple image sensors.

26. The apparatus of claim 25, wherein the means for providing the common control signal is coupled to each of the multiple image sensors via an inter-integrated circuit (I2C) control interface.

27. The apparatus of claim 25, wherein the means for providing the common control signal is coupled to each of the multiple image sensors via an interface compliant with a camera serial interface (CSI) specification.

28. The apparatus of claim 25, wherein the means for providing the common control signal is coupled to each of the multiple image sensors via an interface compliant with a camera parallel interface (CPI) specification.

29. An apparatus comprising:

means for generating first image data;

means for generating second image data, wherein the first image data and the second image data comprise data lines, wherein the means for generating the first image data and the means for generating the second image data are configured to receive a common control signal and are responsive to the common control signal to generate synchronized data output; and

means for focusing light on a common focal plane, wherein the first image data corresponds to a first portion of the common focal plane and the second image data corresponds to a second portion of the common focal plane,

wherein a combined image frame comprises image sections including image data from the means for generating the first image data and the means for generating the second image data and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the means for generating first image data and the means for generating second image data.

30. The apparatus of claim 29, wherein a phase offset between each data line output from the means for generating the first image data and each corresponding data line output from the means for generating the second image data is substantially constant.

31. The apparatus of claim 29, wherein the means for generating the first image data and the means for generating the second image data are configured to receive the common control signal via an inter-integrated circuit (I2C) control interface.

32. A non-transitory computer readable-medium comprising instructions, which when executed by a processor cause the processor to:

provide a common control signal to multiple image sensors, wherein each of the multiple image sensors is responsive to the common control signal to generate data lines, and wherein each data line corresponds to a line of image data;

receive synchronized data output from each of the multiple image sensors;

rectify the synchronized data output using a calibration matrix, wherein the calibration matrix accounts for non-ideal position and alignment of the multiple image sensors; and

form a combined image frame, wherein the combined image frame comprises image sections including image data from the multiple image sensors and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the multiple image sensors.

33. The non-transitory computer readable-medium of claim 32, wherein the synchronized data output includes first data lines received from a first image sensor and second data lines received from a second image sensor, and wherein a phase offset between each received data line from the first image sensor and each corresponding data line from the second image sensor is substantially constant.

34. A non-transitory computer readable-medium comprising instructions, which when executed by a processor cause the processor to:

receive a common control signal at multiple image sensors, wherein each of the multiple image sensors is responsive to the common control signal to generate data lines, and wherein each data line corresponds to a line of image data; and

generate synchronized data output from each of the multiple image sensors, wherein the synchronized data output from each of the multiple image sensors corresponds to image data, and wherein each of the multiple image sensors captures corresponding image data from a different portion of a common focal plane of a main lens,

wherein a combined image frame comprises image sections including image data from the multiple image sensors and gap sections including non-image data that is disposed between the image sections and that is configured to align the image sections, wherein the combined image frame has multiple rows, and wherein each row includes at least one of the gap sections and includes a data line from each of the multiple image sensors.

35. The non-transitory computer readable-medium of claim 34, wherein the synchronized data output includes first data lines generated by a first image sensor and second data lines generated by a second image sensor, and wherein a phase offset between each data line output from the first image sensor and each corresponding data line output from the second image sensor is substantially constant.