Image Processing in a Multi-Channel Camera

• US 20150124083A1
• Filed: 07/03/2013
• Published: 05/07/2015
• Est. Priority Date: 07/04/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: LinX Computational Imaging Ltd. (Apple Inc.)
• Inventors: Attar, Ziv, Tovchigrechko, Andrey
• Granted Patent: US 10,109,063 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An imaging system is configured to identify an object represented by a plurality of preliminary images. The preliminary images are each associated with a different camera imaging channel, and include different image information. An object distance is determined based on the difference in preliminary image information. An object shift is determined based on the object distance and a pre-determined relationship between object shift and object distance. The object shift is applied to the portions of one or more preliminary images representing the object to form shifted preliminary images, and the shifted preliminary images are combined to form a final image.

Claims

1. A system for image processing, comprising:

a non-transitory computer-readable storage medium configured to store a set of preliminary images, each preliminary image captured via an imaging channel comprising a chromatic filter, each of the chromatic filters comprising a spectral transmission passband with a substantially different frequency range; and

an image processor configured to;

access the set of preliminary images;

select a corresponding pixel in each of the accessed preliminary images, the selected pixels associated with an object;

access an error function comprising a function of pixel information for a pixel in each of the accessed preliminary images;

determine an object distance based on the accessed error function and the selected pixels;

determine an object shift based on the determined object distance;

apply the determined object shift to the object associated with the object to form a set of shifted preliminary images;

combine the shifted preliminary images to form a final image; and

store the final image in the non-transitory computer-readable storage medium.

2. A system for image processing, comprising:

a non-transitory computer-readable storage medium configured to store a set of preliminary images representing an object and captured by a plurality of imaging channels; and

an image processor configured to;

access the set of preliminary images;

determine an object distance based on a difference in preliminary image information associated with the object;

access a pre-determined relationship between object distance and object shift;

determine an object shift based on the determined object distance and the accessed relationship between object distance and object shift;

apply the determined object shift to the object represented by the set of preliminary images to form a set of shifted preliminary images;

combine the shifted preliminary images to form a final image; and

store the final image in the non-transitory computer-readable storage medium.

3. The system of claim 2, wherein the plurality of imaging channels comprise four imaging channels, each associated with one lens in a 2×

2 lens configuration.

4. The system of claim 2, wherein a first imaging channel further comprises a red color filter, wherein a second imaging channel further comprises a green color filter, and wherein a third imaging channel further comprises a blue color filter.

5. The system of claim 4, wherein a fourth imaging channel further comprises a broad-band spectral filter.

6. The system of claim 5, wherein determining the object distance comprises determining the difference in preliminary image color information associated with the object of two or more of the preliminary images and determining the object distance based on the determined difference in color information, wherein the two or more preliminary images were captured by at least two of the first imaging channel, the second imaging channel, the third imaging channel, and the fourth imaging channel.

7. The system of claim 2, wherein the pre-determined relationship between object distance and object shift comprises a function for each imaging channel mapping object distances to object shifts.

8. The system of claim 2, wherein the determined object shift comprises an x-coordinate shift and a y-coordinate shift.

9. The system of claim 2, wherein the determined object shift comprises a first shift component associated with a first preliminary image and a second shift component associated with a second preliminary image.

10. The system of claim 2, wherein applying the determined object shift to the object comprises modifying one or more preliminary images by shifting a portion of the one or more preliminary images representing the object by a distance based on the determined object shift.

11. A method for image processing, comprising:

accessing a set of preliminary images captured by a plurality of imaging channels, the preliminary images representing an object;

determining an object distance based on a difference in preliminary image information associated with the object;

accessing a pre-determined relationship between object distance and object shift;

determining an object shift based on the determined object distance and the accessed relationship between object distance and object shift;

applying the determined object shift to the object represented by the set of preliminary images to form a set of shifted preliminary images;

combining the shifted preliminary images to form a final image; and

storing the final image in a format for subsequent retrieval.

12. The method of claim 11, wherein the plurality of imaging channels comprise four imaging channels, each associated with one lens in a 2×

2 lens configuration.

13. The method of claim 11, wherein a first imaging channel further comprises a red color filter, wherein a second imaging channel further comprises a green color filter, and wherein a third imaging channel further comprises a blue color filter.

14. The method of claim 13, wherein a fourth imaging channel further comprises a broad-band spectral filter.

15. The method of claim 14, wherein determining the object distance comprises determining the difference in preliminary image color information of two or more of the preliminary images associated with the object and determining the object distance based on the determined difference in color information, wherein the two or more preliminary images were captured by at least two of the first imaging channel, the second imaging channel, the third imaging channel, and the fourth imaging channel.

16. The method of claim 11, wherein the pre-determined relationship between object distance and object shift comprises a function for each imaging channel mapping object distances to object shifts.

17. The method of claim 11, wherein the determined object shift comprises an x-coordinate shift and a y-coordinate shift.

18. The method of claim 11, wherein the determined object shift comprises a first shift component associated with a first preliminary image and a second shift component associated with a second preliminary image.

19. The method of claim 11, wherein applying the determined object shift to the object comprises modifying one or more preliminary images by shifting a portion of the one or more preliminary images representing the object by a distance based on the determined object shift.

20. A camera system, comprising:

a first imaging channel comprising a red color filter;

a second imaging channel comprising a green color filter;

a third imaging channel comprising a blue color filter;

a fourth imaging channel comprising a broad-band spectral filter;

an image sensor configured to capture a first preliminary image via the first imaging channel, a second preliminary image via the second imaging channel, a third preliminary image via the third imaging channel, and a fourth preliminary image via a fourth imaging channel at substantially the same time;

an image processor configured to identify an object represented by two or more preliminary images, to determine an object distance based on a difference in preliminary image information associated with the object, to apply an object shift to at least one preliminary image based on the object distance, and to combine the preliminary images to form a final image; and

a non-transitory computer-readable storage medium configured to store the final image.