Thin multi-aperture imaging system with auto-focus and methods for using same

• US 10,469,735 B2
• Filed: 01/17/2019
• Issued: 11/05/2019
• Est. Priority Date: 08/01/2013
• Status: Active Grant

• Current Assignee: Corephotonics Ltd. (Samsung Electronics Co. Ltd.)
• Original Assignee: Corephotonics Ltd. (Samsung Electronics Co. Ltd.)
• Inventors: Shabtay, Gal, Cohen, Noy, Geva, Nadav, Gigushinski, Oded, Goldenberg, Ephraim
• Primary Examiner(s): Hsu, Amy R
• Application Number:
  US16/249,937
• Publication Number:
  US 20190149721A1
• Time in Patent Office: 292 Days
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G02B 7/36 using image sharpness techniques , e.g. image processing techniques for generating autofocus signals image data processing per se G06T; in cameras having a solid state image sensor H04N23/67
• CPC Class Codes: G03B 19/22 Double cameras
• CPC Class Codes: G03B 2205/0069 using electromagnetic actuators, e.g. voice coils
• CPC Class Codes: G03B 30/00 Camera modules comprising integrated lens units and imaging units, specially adapted for being embedded in other devices, e.g. mobile phones or vehicles
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

Dual-aperture digital cameras with auto-focus (AF) and related methods for obtaining a focused and, optionally optically stabilized color image of an object or scene. A dual-aperture camera includes a first sub-camera having a first optics bloc and a color image sensor for providing a color image, a second sub-camera having a second optics bloc and a clear image sensor for providing a luminance image, the first and second sub-cameras having substantially the same field of view, an AF mechanism coupled mechanically at least to the first optics bloc, and a camera controller coupled to the AF mechanism and to the two image sensors and configured to control the AF mechanism, to calculate a scaling difference and a sharpness difference between the color and luminance images, the scaling and sharpness differences being due to the AF mechanism, and to process the color and luminance images into a fused color image using the calculated differences.

Claims

1. A dual-aperture digital camera with autofocus (AF) for imaging an object or scene, comprising:

a) a first sub-camera that includes a first optics bloc and a first, color image sensor with a first number of pixels, the first camera operative to provide a first, color image of the object or scene;

b) a second sub-camera that includes a second optics bloc and a second image sensor having a second number of pixels, the second sub-camera operative to provide a second image of the object or scene;

c) an AF mechanism coupled mechanically at least to the first optics bloc and used to perform an AF action on at least the first optics bloc; and

d) a camera controller coupled to the AF mechanism and to the two image sensors and configured, based on calibration data acquired through a calibration step, to control the AF mechanism, to find corresponding points in the first and second images, to estimate a scale difference between the first and second images, and to process the first and second images into a fused color image using the calculated scale difference.

2. The dual-aperture digital camera of claim 1, wherein the camera controller is further configured to preprocess the first and second images, to obtain respective rectified first and second images, to register the rectified first and second images to obtain respective registered first and second images, and to fuse the registered first and second images into a fused color image.

3. The dual-aperture digital camera of claim 2, wherein epipolar lines in the rectified first and second images are more-or-less parallel to horizontal or vertical axes of the rectified first and second images.

4. The dual-aperture digital camera of claim 3, wherein the camera controller configuration is based on calibration data acquired through a calibration step, wherein the calibration data is saved in a one-time programmable memory or an EEPROM in the dual-aperture digital camera.

5. The dual-aperture digital camera of claim 4, wherein the first number of pixels and the second numbers of pixels are different.

6. A dual-aperture digital camera with autofocus (AF) for imaging an object or scene, comprising:

a) a first sub-camera that includes a first optics bloc and a first, color image sensor with a first number of pixels, the first camera operative to provide a first, color image of the object or scene;

b) a second sub-camera that includes a second optics bloc and a second image sensor having a second number of pixels, the second sub-camera operative to provide a second image of the object or scene;

c) an AF mechanism coupled mechanically at least to the first optics bloc and used to perform an AF action on at least the first optics bloc; and

d) a camera controller coupled to the AF mechanism and to the two image sensors and configured, based on calibration data acquired through a calibration step, to control the AF mechanism, to preprocess the first and second images to obtain respective rectified and scale-adjusted first and second images while considering scaling differences caused by the AF action, to register the rectified and scale-adjusted first and second images to obtain respective registered first and second images, and to fuse the registered first and second images into a fused color image.

7. The dual-aperture digital camera of claim 6, wherein the color image sensor includes a non-standard color filter array.

8. The dual-aperture digital camera of claim 6, wherein the camera controller configuration is based on calibration data acquired through a calibration step.

9. The dual-aperture digital camera of claim 8, wherein the calibration data is saved in a one-time programmable memory or an EEPROM in the dual-aperture digital camera.

10. The dual-aperture digital camera of claim 6, wherein the first number of pixels and the second numbers of pixels are the same.

11. The dual-aperture digital camera of claim 6, wherein the first number of pixels and the second numbers of pixels are different.

12. The dual-aperture digital camera of claim 6, wherein the first, color image sensor and the second image sensor have pixels of the same size.

13. The dual-aperture digital camera of claim 6, wherein the first sub-camera includes an infra-red (IR) filter that blocks IR wavelengths from entering the first image sensor and wherein the second sub-camera is configured to allow at least some IR wavelengths to enter the second image sensor.

14. The dual-aperture digital camera of claim 6, wherein the second image sensor is a luminance image sensor and wherein the second image is a luminance image.

15. The dual-aperture digital camera of claim 6, wherein epipolar lines in the rectified first and second images are more-or-less parallel to horizontal or vertical axes of the rectified first and second images.

16. A method, comprising:

in a dual camera;

a) obtaining simultaneously a first, color image of an object or scene with a first sub-camera and a second image of the object or scene with a second sub-camera, wherein the first sub-camera includes a first optics bloc, wherein the second sub-camera includes a second optics bloc and wherein the first and second sub-cameras have substantially the same field of view;

b) using an AF mechanism coupled mechanically at least to the first optics bloc to perform an AF action on at least the first optics bloc;

c) preprocessing the first image and the second image to obtain respective rectified and scale-adjusted first and second images while considering scaling differences caused by the AF action;

d) performing registration between the rectified and scale-adjusted first and second images to obtain registered images; and

e) fusing the registered images into a focused fused color image.

17. The method of claim 16, wherein the fusing of the registered images is not performed in non-focused areas.

18. The method of claim 16, wherein the preprocessing to obtain respective rectified scale-adjusted first and second images includes calculating a set of corresponding points in the first and second images and estimating a scale factor S.

19. The method of claim 18, further comprising using scaling factor S to scale one of the images to match the other image, thereby obtaining the registered images.