3D image acquisition apparatus and method of generating depth image in the 3D image acquisition apparatus

• US 9,749,613 B2
• Filed: 04/08/2014
• Issued: 08/29/2017
• Est. Priority Date: 04/08/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Park, Yong-hwa, Kim, Sun-kwon, Yoon, Hee-sun
• Primary Examiner(s): McInnish, Kevin
• Application Number:
  US14/247,907
• Publication Number:
  US 20140300701A1
• Time in Patent Office: 1,239 Days
• Field of Search: None
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4914 of detector arrays, e.g. charge-transfer gates
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

Provided are a three-dimensional (3D) image acquisition apparatus, and a method of generating a depth image in the 3D image acquisition apparatus. The method may include sequentially projecting a light transmission signal, which is generated from a light source, to a subject, modulating reflected light, which is reflected by the subject, using a light modulation signal, calculating a phase delay using a combination of a first plurality of images of two groups, from among a second plurality of images of all groups obtained by capturing the modulated reflected light, and generating a depth image based on the phase delay.

Claims

1. A method of generating a depth image, the method comprising:

sequentially projecting a light transmission signal, which is generated from a light source, to a subject;

modulating reflected light, which is reflected by the subject, using a light modulation signal;

obtaining a first plurality of images from the modulated reflected light, the first plurality of images comprising a first group of a second plurality of images and a second group of the second plurality of images, images in the first group and images in the second group have different phases from one another;

determining that the first plurality of images comprises a motion blur image in response to a difference between a first sum of intensities of the images in the first group and a second sum of intensities of the images in the second group being greater than or equal to a predetermined value;

calculating a phase delay using a combination of the first group and the second group of the second plurality of images; and

generating a depth image based on the phase delay.

2. The method of claim 1, wherein the light transmission signal has different phases, and is sequentially transmitted in an order of 0, 180, 90, and 270 degrees, and

wherein the phase delay is calculated using the first group which comprises a first image corresponding to 0 degrees and a second image corresponding to 180 degrees, and the second group which comprises a third image corresponding to 90 degrees and a fourth image corresponding to 270 degrees.

3. The method of claim 1, wherein the light modulation signal has different phases, and is sequentially transmitted in an order of 0, 180, 90, and 270 degrees, and

wherein the phase delay is calculated using the first group which comprises a first image corresponding to 0 degrees and a second image corresponding to 180 degrees, and the second group which comprises a third image corresponding to 90 degrees and a fourth image corresponding to 270 degrees.

4. The method of claim 1, further comprising:

wherein the modulated reflected light is captured in an order of phases of 0, 180, 90, and 270 degrees, and the first plurality of images comprise a first image corresponding to 0 degrees, a second image corresponding to 180 degrees, a third image corresponding to 90 degrees, and a fourth image corresponding to 270 degrees, and

wherein the images in the first group correspond to the first image and the second image, and the images in the second group correspond to the third image and the fourth image is equal to or greater than a first critical value.

5. The method of claim 4, further comprising:

determining a group where the motion blur image is generated, from among the first group and the second group if the motion blur image is generated as a result of the determining,

wherein the determining the group where the motion blur image is generated comprises determining that the motion blur image is generated in the group that has a larger range of variation in luminance of images is than the other group from among the first group and the second group.

6. The method of claim 5, further comprising:

restoring the motion blur image,

wherein the restoring the motion blur image comprises restoring images of the determined group including the motion blur image, using images of the other group.

7. The method of claim 6, wherein the restoring the motion blur image comprises restoring the motion blur image using the images of the other group and a demodulation constant which is a ratio between an alternating current (AC) variation and a direct current (DC) offset value of an image due to the phase delay.

8. The method of claim 1, wherein the first plurality of images further comprise a first group of a third plurality of images and a second group of the third plurality of images,

the third plurality of images are obtained subsequent to the second plurality of images, and

a first depth image is generated by calculating the phase delay with respect to the first group and the second group of the second plurality of images, and a second depth image is generated by calculating the phase delay with respect to the second group of the second plurality of images and the first group of the third plurality of images.

9. A method of generating a depth image, the method comprising:

sequentially projecting a light transmission signal, which is generated in a light source, to a subject;

modulating reflected light, which is reflected by the subject, using a light modulation signal;

obtaining a first image corresponding to 0 degrees, a second image corresponding to 180 degrees, a third image corresponding to 90 degrees, and a fourth image corresponding to 270 degrees, wherein 0 degrees, 180 degrees, 90 degrees, and 270 degrees are phases of the modulated reflected light;

determining whether a motion blur image generated by motion of the subject is present by determining whether a difference between a first sum of images in a first group which comprises the first image and the second image and a second sum of images in a second group which comprises the third image and the fourth image is equal to or greater than a first critical value;

determining, when there is a motion blur image, a group including the motion blur image by comparing a difference of images in the first group and a difference of images in the second group, and restoring the images of the group including the motion blur image using images of the other group;

calculating a phase delay using the images of the other group and the images of the restored group; and

generating a depth image based on the phase delay.

10. A three-dimensional (3D) image acquisition apparatus comprising:

a light source configured to sequentially project a light transmission signal to a subject;

a light modulator configured to modulate reflected light which is reflected by the subject, using a light modulation signal having a gain waveform;

a capturing device configured to capture the modulated reflected light, which is modulated by the light modulator, to generate a first plurality of images comprising a first group of a second plurality of images and a second group of the second plurality of images, images in the first group and images in the second group have different phases from one another; and

a depth image processor configured to determine that the first plurality of images comprises a motion blur image in response to a difference between a first sum of intensities of the images in the first group and a second sum of intensities of the images in the second group being greater than or equal to a predetermined value, calculate a phase delay using a combination of the first group and the second group of the second plurality of image, and generate a depth image based on the phase delay.

11. The 3D image acquisition apparatus of claim 10, further comprising:

a light source driver configured to control a waveform of the light transmission signal by driving the light source;

a light modulator driver configured to control the gain waveform by driving the light modulator; and

a controller configured to control operations of the light source driver, the light modulator driver, and the capturing device.

12. The 3D image acquisition apparatus of claim 11, wherein the controller is configured to control the light source driver such that the light transmission signal has different phases, and is sequentially projected in an order of 0, 180, 90, and 270 degrees,

wherein the depth image processor is configured to calculate the phase delay using the first group which comprises a first image corresponding to 0 degrees and a second image corresponding to 180 degrees, and the second group which comprises a third image corresponding to 90 degrees and a fourth image corresponding to 270 degrees.

13. The 3D image acquisition apparatus of claim 11, wherein the controller is configured to control the light modulator driver such that the light modulation signal has different phases, and the reflected light is sequentially modulated in an order of 0, 180, 90, and 270 degrees, and

wherein the depth image processor is configured to calculate the phase delay using the first group which comprises a first image corresponding to 0 degrees and a second image corresponding to 180 degrees, and the second group which comprises a third image corresponding to 90 degrees and a fourth image corresponding to 270 degrees.

14. The 3D image acquisition apparatus of claim 12, wherein the depth image processor is configured to restore images of the determined group including the motion blur image, using images of the other group and a demodulation constant which is a ratio between an alternating current (AC) variation and a direct current (DC) offset value of an image due to the phase delay.

15. The 3D image acquisition apparatus of claim 12, wherein the first plurality of images further comprises a first group of a third plurality of images and a second group of the third plurality of images,

the third plurality of images are obtained subsequent to the second plurality of images, and

the depth image processor is configured to generate a first depth image by calculating distances of the first group of the second plurality of images to the subject and the second group of the second plurality of images to the subject, and a second depth image by calculating the phase delay with respect to the second group of the second plurality of images and the first group of the third plurality of images.

16. The 3D image acquisition apparatus of claim 12, further comprising:

a first lens configured to focus the reflected light on a light incident surface of the light modulator within an area of the light modulator;

a filter configured to transmit light having a predetermined wavelength between the first lens and the light modulator, and configured to remove light of wavelengths other than the predetermined wavelength; and

a second lens configured to focus the modulated light between the light modulator and the capturing device within an area of the capturing device.

17. The 3D image acquisition apparatus of claim 12, wherein the capturing device comprises at least one of a charge-coupled device (CCD) image sensor which comprises a one-dimensional array or a two-dimensional array, a complementary metal oxide semiconductor (CMOS) image sensor, a photodiode array, and one photodiode, and

wherein the capturing device is configured to measure a distance to a single point.

18. The 3D image acquisition apparatus of claim 12, wherein the light source is one of a solid-state light-emitting device which comprises a laser diode (LD) and a light-emitting diode (LED) in an infrared ray band.

19. A method of restoring an image which includes motion blur, the method comprising:

determining whether a motion blur image occurs in a reflected light signal by determining whether a difference between a first sum of images in a first group and a second sum of images in a second group is equal to or greater than a first critical value;

determining a group which comprises the motion blur image by comparing a difference of images in the first group and a difference of images in the second group in response to the motion blur image occurring in the reflected light signal; and

restoring images of the group which comprises the motion blur image using images of the other group which is not the group comprising the motion blur image,

wherein the first group comprises a first image and a second image of the reflected light signal, and the second group comprises a third image and a fourth image of the reflected light signal, and

wherein the first image, the second image, the third image, and the fourth image each have different phases.

20. The method of claim 19, wherein the first image corresponds to 0 degrees, the second image corresponds to 180 degrees, the third image corresponds to 90 degrees, and the fourth image corresponds to 270 degrees.

21. The method of claim 19, wherein the reflected light has been modulated by a light modulation signal.

22. The method of claim 19, wherein the reflected light has been reflected by a subject.