MULTI-APERTURE CAMERA SYSTEM USING SCAN LINE PROCESSING

• US 20170148169A1
• Filed: 08/25/2016
• Published: 05/25/2017
• Est. Priority Date: 11/23/2015
• Status: Active Grant

• Current Assignee: Dexelion Incorporated
• Original Assignee: Center For Integrated Smart Sensors Foundation (Center for Integrated Smart Sensors)
• Inventors: KYUNG, Chong Min
• Granted Patent: US 10,051,263 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/214 using spectral multiplexing
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

The multi-aperture camera system includes a first aperture receiving an RGB optical signal, a second aperture distinguished from the first aperture and receiving an optical signal of which a wavelength is different from that of the RGB optical signal, an image sensor, a plurality of frame buffers, wherein the first and second images are respectively stored in the frame buffers, and a scan line-based processor unit (SPU) processing the first and second images using a scan line processing technique to determine a depth at at least one pixel of the first and second images. The first aperture and the second aperture are formed on a single optical system so as to be offset in a horizontal direction of the image sensor and thus have center positions deviated from each other.

Claims

1. A multi-aperture camera system using a scan line processing technique, comprising:

a first aperture receiving an RGB optical signal;

a second aperture distinguished from the first aperture and receiving an optical signal of which a wavelength is different from that of the RGB optical signal;

an image sensor obtaining a first image by processing the RGB optical signal received by the first aperture and obtaining a second image by processing the optical signal, of which the wavelength is different from that of the RGB optical signal, received by the second aperture;

a plurality of frame buffers, wherein the first and second images are respectively stored in the frame buffers; and

a scan line-based processor unit (SPU) processing the first and second images using a scan line processing technique to determine a depth at at least one pixel of the first and second images,

wherein the first aperture and the second aperture are formed on a single optical system so as to be offset in a horizontal direction of the image sensor and thus have center positions deviated from each other.

2. The multi-aperture camera system of claim 1, wherein the SPU comprises:

a scan line buffer storing a scan line window applied to the first image and a scan line window applied to the second image are stored;

a source pattern buffer storing a source pattern window set to comprise the at least one pixel at the scan line window applied to the first image;

a search range buffer storing a search range window set based on the source pattern window at the scan line window applied to the second image;

a target pattern buffer storing a plurality of target pattern windows found at the search range window; and

a processor controlling an operation of each of the plurality of frame buffers, the scan line buffer, the source pattern buffer, the search range buffer, and the target pattern buffer, calculating a correlation between each of the plurality of target pattern windows and the scan pattern window by performing image pattern matching about the plurality of target pattern windows with respect to the source pattern window, and determining a depth at the at least one pixel using a disparity between the source pattern window and a target pattern window, which maximizes the correlation, from among the plurality of target pattern windows.

3. The multi-aperture camera system of claim 2, wherein the processor searches a region of the search range window based on a misalignment angle between an offset direction of the first and second apertures and a horizontal direction of the image sensor and sets the plurality of target pattern windows based on the search result.

4. The multi-aperture camera system of claim 3, wherein the processor searches the region of the search range window in horizontal and vertical directions and sets the plurality of target pattern windows based on the search result.

5. The multi-aperture camera system of claim 2, wherein the processor sets a height of the scan line window based on a misalignment angle between an offset direction of the first and second apertures and a horizontal direction of the image sensor, a maximum disparity generated between the first image and the second image as center positions of the first and second apertures are deviated from each other, a height of the source pattern window, and sets a width of the scan line window based on the first image and a width of the first image.

6. The multi-aperture camera system of claim 2, wherein the processor sets a width of the search range based on the misalignment angle between an offset direction of the first and second apertures and a horizontal direction of the image sensor, a maximum disparity generated between the first image and the second image as center positions of the first and second apertures are deviated from each other, a height of the source pattern window, and sets a height of the search range window based on the height of the scan line window.

7. The multi-aperture camera system of claim 2, wherein the processor changes a height and a width of each of the source pattern window and the plurality of target pattern windows into a plurality of values and searches for a region of the search range window to set the plurality of target pattern windows.

8. The multi-aperture camera system of claim 2, wherein the processor sets a source pattern window with respect to the remaining region other than a predetermined boundary region at the scan line window applied to the first image.

9. The multi-aperture camera system of claim 2, wherein the processor determines a height and width of the source pattern window such that the image pattern matching is performed with respect to the at least one pixel.

10. The multi-aperture camera system of claim 8, wherein the processor determines a height and a width of each of the plurality of target pattern windows based on the height and width of the source pattern window.

11. The multi-aperture camera system of claim 2, wherein the SPU further comprises:

a depth frame buffer storing a depth at the at least one pixel.

12. The multi-aperture camera system of claim 2, wherein the processor calculates a correlation between each of the plurality of target pattern windows and the source pattern window using a similarity/dissimilarity metric.

13. The multi-aperture camera system of claim 2, wherein the processor sequentially applies the scan line window in a vertical direction of each of the first and second images.

14. A method for operating a multi-aperture camera system using a scan line processing technique, the method comprising:

receiving an RGB optical signal through a first aperture;

receiving an optical signal, of which a wavelength is different from that of the RGB optical signal, through a second aperture distinguished from the first aperture;

obtaining first and second images by processing the RGB optical signal received by the first aperture and an signal, of which a wavelength is different from that of the RGB optical signal, received by the second aperture, at an image sensor; and

processing the first and second images in a scan line processing manner to determine a depth at at least one pixel of the first and second images, at a scan line-based processor unit (SPU),

wherein the first aperture and the second aperture are formed on a single optical system so as to be offset in a horizontal direction of an image sensor and thus have center positions deviated from each other.

15. The method of claim 14, wherein the determining of the depth about the at least one pixel comprises:

setting a source pattern window so as to comprise the at least one pixel at a scan line window applied to the first image;

setting a search range window at a scan line window applied to the second image based on the source pattern window;

searching for a plurality of target pattern windows from the search range window;

calculating a correlation between each of the plurality of target pattern windows and the source pattern window by performing image pattern matching about the plurality of target pattern windows with respect to the source pattern window; and

determining a depth at the at least one pixel using a disparity between the source pattern window and a target pattern window, which maximizes the correlation, from among the plurality of target pattern windows.

16. The method of claim 15, wherein the searching for the plurality of target pattern windows comprises:

searching a region of the search range window based on a misalignment angle between an offset direction of the first and second apertures and a horizontal direction of the image sensor and setting the plurality of target pattern windows based on the search result.

17. The method of claim 15, further comprising:

setting the scan line window based on a misalignment angle between an offset direction of the first and second apertures and a horizontal direction of the image sensor, a maximum disparity generated between the first image and the second image as center positions of the first and second apertures are deviated from each other, a height of the source pattern window, the first image, and a width of the first image.

18. The method of claim 15, wherein the setting of the search range window comprises:

setting the search range window based on a misalignment angle between an offset direction of the first and second apertures and a horizontal direction of the image sensor, a maximum disparity generated between the first image and the second image as center positions of the first and second apertures are deviated from each other, a width of the source pattern window, and a height of the scan line window.

19. The method of claim 15, wherein the searching for the plurality of target pattern windows from the search range window comprises:

changing a height and a width of the source pattern window and the plurality of target pattern windows into a plurality of values and searching for the a region of the search range window to set the plurality of target pattern windows.

20. A multi-aperture camera system using a disparity, comprising:

a single optical system comprising a first aperture receiving an RGB optical signal and a second aperture receiving an optical signal of which a wavelength is different from that of the RGB optical signal and moving to be disposed at a plurality of positions;

an image sensor obtaining a plurality of image sets corresponding to the plurality of positions as the single optical system is disposed at the plurality of positions, wherein each of the image sets comprises a first image obtained by processing the RGB optical signal received by the first aperture and a second image obtained by processing the optical signal, of which the wavelength is different from that of the RGB optical signal, received by the second aperture;

a plurality of frame buffers, wherein the first and second images are respectively stored in the frame buffers; and

a scanline-based processor unit (SPU) determining a depth at at least one pixel of the image sets using a disparity of each of the image sets calculated by processing each of the image sets through a scan line processing technique,

wherein the first aperture and the second aperture are formed on a single optical system so as to be offset in a horizontal direction of the image sensor and thus have center positions deviated from each other.