SYSTEMS AND METHODS FOR AN IMAGING SYSTEM USING MULTIPLE IMAGE SENSORS

• US 20110074931A1
• Filed: 09/30/2009
• Published: 03/31/2011
• Est. Priority Date: 09/30/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Bilbrey, Brett
• Granted Patent: US 8,619,128 B2
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

Systems and methods may employ separate image sensors for collecting different types of data. In one embodiment, separate luma, chroma and 3-D image sensors may be used. The systems and methods may involve generating an alignment transform for the image sensors, and using the 3-D data from the 3-D image sensor to process disparity compensation. The systems and methods may involve image sensing, capture, processing, rendering and/or generating images. For example, one embodiment may provide an imaging system, including: a first image sensor configured to obtain luminance data of a scene; a second image sensor configured to obtain chrominance data of the scene; a third image sensor configured to obtain three-dimensional data of the scene; and an image processor configured to receive the luminance, chrominance and three-dimensional data and to generate a composite image corresponding to the scene from that data.

Claims

1. An imaging system, comprising:

a first image sensor configured to obtain first data of a first data type of a scene;

a second image sensor configured to obtain second data of a second data type of the scene;

a third image sensor configured to obtain three-dimensional data of the scene; and

an image processor configured to receive the first data, the second data and the three-dimensional data and to generate a composite image corresponding to the scene from the first data, the second data and the three-dimensional data.

2. The system of claim 1, wherein the first image sensor is configured to obtain only luminance data, the second image sensor is configured to obtain only chrominance data, and the third image sensor is configured to obtain only the three-dimensional data.

3. The system of claim 2, wherein the first image sensor comprises an infrared sensor.

4. The system of claim 2, further comprising an infrared sensor configured to obtain only luminance data.

5. The system of claim 1, wherein the first image sensor is configured to obtain only the first data, the second image sensor is configured to obtain only the second data, and the third image sensor is configured to obtain only the three-dimensional data.

6. The system of claim 1, wherein the third image sensor comprises lidar.

7. The system of claim 1, wherein the third image sensor is an acoustic sensor.

8. The system of claim 1, wherein the processor is configured to generate an alignment transform that accounts for differences in alignment of the first, second and third image sensors.

9. The system of claim 8, wherein the processor is configured to generate the alignment transform as a calibration of the system and thereafter to apply the alignment transform to provide epipolar geometry alignment and disparity compensation.

10. The system of claim 9, wherein the processor is configured to process the disparity compensation in real time using the three-dimensional data received in real time.

11. The system of claim 10, wherein the processor is configured to apply the alignment transform to provide epipolar geometry alignment and disparity compensation for each frame of a time series of frames of the scene.

12. An imaging method, comprising:

obtaining first data of a first data type of a scene using a first image sensor;

obtaining second data of a second data type of the scene using a second image sensor, the second data type being different from the first data type;

obtaining three-dimensional data of the scene using a third image sensor; and

processing the first data, the second data and the three-dimensional data to generate a composite image corresponding to the scene.

13. The method of claim 12, wherein only the first data type is obtained using the first image sensor, only the second data type is obtained using the second image sensor, and only three-dimensional data is obtained using the third image sensor.

14. The method of claim 12, wherein only luminance data is obtained using the first image sensor, only chrominance data is obtained using the second image sensor, and only three-dimensional data is obtained using the third image sensor.

15. The method of claim 14, wherein the first image sensor comprises an infrared sensor.

16. The method of claim 13, further comprising generating an alignment transform that accounts for differences in alignment of the first, second and third image sensors.

17. The method of claim 16, wherein generating the alignment transform comprises a calibration, the method further comprising applying the alignment transform after the calibration to provide epipolar geometry alignment and disparity compensation.

18. The method of claim 17, wherein applying the alignment transform comprises processing the disparity compensation in real time using the three-dimensional data received in real time.

19. The method of claim 18, wherein obtaining the first data, the second data and the three-dimensional data of the scene comprises obtaining such data for each frame of a time series of frames of the scene, and applying the alignment transform provides epipolar geometry alignment and disparity compensation for each frame of the time series of frames.

20. A computer-readable storage medium including stored instructions that, when executed by a computer, cause the computer to:

obtain first data of a first data type of a scene;

obtain second data of a second data type of the scene, the second data type being different from the first data type;

obtain three-dimensional data of the scene; and

process the first data, the second data and the three-dimensional data to generate a composite image corresponding to the scene.

21. The medium of claim 20, wherein the stored instructions, when executed by the computer, cause the computer to

separately obtain the first data, the second data, and the three-dimensional data.

22. The medium of claim 20, wherein the stored instructions, when executed by the computer, cause the computer to

generate an alignment transform that accounts for differences in alignment of the first, second and third image sensors.

23. The medium of claim 20, wherein the stored instructions, when executed by the computer, cause the computer to

generate the alignment transform as a calibration;

and

apply the alignment transform after the calibration to provide epipolar geometry alignment and disparity compensation.

24. The medium of claim 20, wherein the stored instructions, when executed by the computer, cause the computer to process the disparity compensation in real time using the three-dimensional data received in real time.

25. The medium of claim 20, wherein the stored instructions, when executed by the computer, cause the computer to:

obtain the first data, the second data and the three-dimensional data of the scene for each frame of a time series of frames of the scene; and

apply the alignment transform to provide epipolar geometry alignment and disparity compensation for each frame of the time series of frames.