Systems and methods for an imaging system using multiple image sensors

• US 8,619,128 B2
• Filed: 09/30/2009
• Issued: 12/31/2013
• Est. Priority Date: 09/30/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bilbrey, Brett, Cote, Guy
• Primary Examiner(s): Meky, Moustafa M
• Assistant Examiner(s): RUBIN, BLAKE J
• Application Number:
  US12/570,966
• Publication Number:
  US 20110074931A1
• Time in Patent Office: 1,553 Days
• Field of Search: 348/48
• US Class Current: 348/48
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

Systems and methods may employ separate image sensors for collecting different types of data. In one embodiment, separate luma, chroma and 3-D image sensors may be used. The systems and methods may involve generating an alignment transform for the image sensors, and using the 3-D data from the 3-D image sensor to process disparity compensation. The systems and methods may involve image sensing, capture, processing, rendering and/or generating images. For example, one embodiment may provide an imaging system, including: a first image sensor configured to obtain luminance data of a scene; a second image sensor configured to obtain chrominance data of the scene; a third image sensor configured to obtain three-dimensional data of the scene; and an image processor configured to receive the luminance, chrominance and three-dimensional data and to generate a composite image corresponding to the scene from that data.

Claims

1. An imaging system, comprising:

a first image sensor configured to obtain luminance data of a scene;

a second image sensor configured to obtain chrominance data of the scene;

a third image sensor configured to obtain three-dimensional data of the scene; and

an image processor configured to receive the luminance data, the chrominance data and the three-dimensional data and to generate a composite image corresponding to the scene from the luminance data, the chrominance data and the three-dimensional data;

wherein the processor is configured to;

generate an alignment transform that accounts for differences in alignment of the first, second and third image sensors;

generate the alignment transform as a calibration of the system and thereafter to apply the alignment transform to provide epipolar geometry alignment and disparity compensation; and

process the disparity compensation in real time using the three-dimensional data received in real time wherein the three-dimensional data includes at least one of distance data of the scene or depth data of the scene.

2. The system of claim 1, wherein the first image sensor is configured to obtain only the luminance data, the second image sensor is configured to obtain only the chrominance data, and the third image sensor is configured to obtain only the three-dimensional data.

3. The system of claim 2, wherein the first image sensor comprises an infrared sensor.

4. The system of claim 2, further comprising an infrared sensor configured to obtain only luminance data.

5. The system of claim 1, wherein the third image sensor comprises lidar.

6. The system of claim 1, wherein the third image sensor is an acoustic sensor.

7. The system of claim 1, wherein the processor is configured to apply the alignment transform to provide epipolar geometry alignment and disparity compensation for each frame of a time series of frames of the scene.

8. An imaging method, comprising:

obtaining luminance data of a scene using a first image sensor;

obtaining chrominance data of the scene using a second image sensor;

obtaining three-dimensional data of the scene using a third image sensor;

processing the luminance data, the chrominance data and the three-dimensional data to generate a composite image corresponding to the scene;

generating an alignment transform that accounts for differences in alignment of the first, second and third image sensors wherein generating the alignment transform comprises a calibration; and

applying the alignment transform after the calibration to provide epipolar geometry alignment and disparity compensation wherein applying the alignment transform comprises processing the disparity compensation in real time using the three-dimensional data received in real time wherein the three-dimensional data includes at least one of distance data of the scene or depth data of the scene.

9. The method of claim 8, wherein only the luminance data type is obtained using the first image sensor, only the chrominance data type is obtained using the second image sensor, and only three-dimensional data is obtained using the third image sensor.

10. The method of claim 8, wherein the first image sensor comprises an infrared sensor.

11. The method of claim 9, wherein obtaining the luminance data, the second chrominance data and the three-dimensional data of the scene comprises obtaining such data for each frame of a time series of frames of the scene, and applying the alignment transform provides epipolar geometry alignment and disparity compensation for each frame of the time series of frames.

12. A non-transitory computer-readable storage medium including stored instructions that, when executed by a computer, cause the computer to:

obtain luminance data of a scene;

obtain chrominance data of of the scene;

obtain three-dimensional data of the scene;

process the luminance data, the chrominance data and the three-dimensional data to generate a composite image corresponding to the scene;

generate an alignment transform that accounts for differences in alignment of the first, second and third image sensors wherein generating the alignment transform comprises a calibration; and

apply the alignment transform after the calibration to provide epipolar geometry alignment and disparity compensation wherein applying the alignment transform comprises processing the disparity compensation in real time using the three-dimensional data received in real time wherein the three-dimensional data includes at least one of distance data of the scene or depth data of the scene.

13. The medium of claim 12, wherein the stored instructions, when executed by the computer, cause the computer to

separately obtain the luminance data, the chrominance data, and the three-dimensional data.

14. The medium of claim 12, wherein the stored instructions, when executed by the computer, cause the computer to:

obtain the luminance data, the chrominance data and the three-dimensional data of the scene for each frame of a time series of frames of the scene; and

apply the alignment transform to provide epipolar geometry alignment and disparity compensation for each frame of the time series of frames.