Image Processing Systems

• US 20120169848A1
• Filed: 12/27/2011
• Published: 07/05/2012
• Est. Priority Date: 12/29/2010
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kyung, Kyu-Min, Kim, Tae-Chan, Bae, Kwang-Hyuk
• Granted Patent: US 8,953,021 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4861 Circuits for detection, sampling, integration or read-out
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects

Abstract

An image processing system includes a calculation unit, a reconstruction unit, a confidence map estimation unit and an up-sampling unit. The up-sampling unit is configured to perform a joint bilateral up-sampling on depth information of a first input image based on a confidence map of the first input image and a second input image with respect to an object and increase a first resolution of the first input image to a second resolution to provide an output image with the second resolution.

Claims

1. An image processing system comprising:

a calculation unit configured to calculate a phase difference between an emitted light and a first received light reflected from an object due to the emitted light, the calculation unit being further configured to calculate an amplitude indicating an intensity of the first received light and calculate an offset of the first received light;

a reconstruction unit configured to reconstruct the first received light based on the phase difference and configured to provide a reconstruction error, the reconstruction error corresponding to intensity differences between sampling points of the first received light and corresponding sampling points of the reconstructed first received light;

a confidence map estimation unit configured to determine a confidence map based on the amplitude, the offset and the reconstruction error, the confidence map associated with a first input image generated based on the first received light, the first input image having a first resolution; and

an up-sampling unit configured to perform a joint bilateral up-sampling on depth information of the first input image based on the confidence map and a second input image of the object, the up-sampling unit configured to increase the first resolution of the first input image to a second resolution to provide an output image with the second resolution.

2. The image processing system of claim 1, wherein the second input image has the second resolution.

3. The image processing system of claim 1, wherein the first input image is a three-dimensional (3-D) depth image with respect to the object and the second input image is a two-dimensional (2-D) color image with respect to the object.

4. The image processing system of claim 1, wherein the output image is a three-dimensional (3-D) depth image with respect to the object.

5. The image processing system of claim 1, wherein the up-sampling unit is configured to perform the joint bilateral up-sampling by

6. The image processing system of claim 5, wherein the confidence map C(q↓

) expressed by

C(q↓

)=w_AC_A+w_BC_B+w_REC_RE,

wherein

7. The image processing system of claim 5, wherein each of f, g and h denotes a Gaussian function.

8. The image processing system of claim 5, wherein the blending function α

(Δ

_Ω

) is expressed by

9. The image processing system of claim 1, wherein the up-sampling unit is configured to perform the joint bilateral up-sampling based on reflectances of pixels of an image sensor configured to generate the first input image.

10. An image processing system comprising:

a first image sensor including a light source module, the first image sensor configured to generate a first input image with a first resolution based on a first received light reflected from an object due to an emitted light from the light source module, the first image sensor further configured to provide a confidence map with respect to a first input image based on the first input image;

a second image sensor configured to provide a second input image based on a second received light reflected from the object, the second image having a second resolution higher than the first resolution; and

an up-sampling unit configured to perform a joint bilateral up-sampling on depth information of the first input image based on the confidence map and the second input image, the up-sampling unit configured to provide an output image with the second resolution based on the up-sampling.

11. The image processing system of claim 10, wherein the first image sensor comprises:

a calculation unit configured to calculate a phase difference of the first received light, calculate an amplitude indicating an intensity of the first received light and calculate an offset of the first received light;

a reconstruction unit configured to reconstruct the first received light based on the phase difference and configured to provide a reconstruction error corresponding to intensity differences between sampling points of the first received light and corresponding sampling points of the reconstructed first received light; and

a confidence map estimation unit configured to determine a confidence map based on the amplitude, the offset and the reconstruction error.

12. The image processing system of claim 11, wherein the calculation unit is configured to calculate the amplitude and the offset by sampling the first received light at sampling points of the emitted light during one period of the emitted light.

13. The image processing system of claim 10, wherein the first input image is a three-dimensional (3-D) depth image with respect to the object and the second input image is a two-dimensional (2-D) color image with respect to the object.

14. The image processing system of claim 10, wherein the output image is a three-dimensional (3-D) depth image with respect to the object.

15. An image processing system comprising:

a first image sensor including a pixel array having a plurality of pixels, the pixels configured to produce pixel values, the first image sensor configured to generate a first input image based on the pixel values;

a second image sensor configured to generate a second input image; and

an up-sampling unit configured to up-sample the first input image based on the second input image and a variance of the pixel values associated with at least one pixel of the plurality of pixels.

16. The image processing system of claim 15, wherein

the first image sensor is configured to generate the first input image having a first resolution,

the second image sensor is configured to generate the second input image having a second resolution, the first and second resolutions are different.

17. The image processing system of claim 16, wherein the up-sampling unit is configured to up-sample the first input image to have the second resolution.

18. The image processing system of claim 15, wherein the first image sensor is a time-of-flight (TOF) image sensor and the second image sensor is a two-dimensional (2-D) color image sensor.