METHOD AND APPARATUS FOR CONVERTING DEPTH IMAGE IN HIGH RESOLUTION

• US 20150036926A1
• Filed: 11/29/2012
• Published: 02/05/2015
• Est. Priority Date: 11/29/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Do Kyoon, Choi, Ouk, Lee, Kee Chang
• Granted Patent: US 9,613,293 B2
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

Provided are a method and an apparatus for image processing. A scene includes a low resolution depth image and a high resolution color image. A depth image up-scaler up-scales the depth image using an interpolation so as to improve the resolution of the depth image. A color-depth similarity based filter removes noise from the up-scaled depth image using the color image and the up-scaled depth image. A boundary sharpening filter converts a boundary between objects in the up-scaled depth image into a clearer boundary. The resolution of the depth image may increase to the level of the resolution of the color image by means of the depth image up-scaler, the color-depth similarity based filter and the boundary sharpening filter, and the depth of each pixel in the depth image may be adjusted.

Claims

1-17. -17. (canceled)

18. A method of processing a depth image and a color image associated with a scene, the method comprising:

increasing, by use of a depth image up-scaler, a depth image resolution of the depth image by up-scaling the depth image through use of interpolation to produce an up-scaled depth image;

removing, by use of a color-depth similarity-based filter, noise in the up-scaled depth image through use of the color image and the up-scaled depth image; and

converting, by use of a boundary sharpening filter, a boundary between objects in the up-scaled depth image,

wherein the depth image resolution of the up-scaled depth image matches a color image resolution of the color image.

19. The method of claim 18, wherein the color-depth similarity-based filter updates a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of neighboring pixels,

wherein:

the neighboring pixels correspond to pixels included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image; and

a similarity of a neighboring pixel of the neighboring pixels with respect to the target pixel is determined based on at least one of;

a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,

a second difference between a depth of the neighboring pixel and a depth of the target pixel, and

a third difference between a location of the neighboring pixel and a location of the target pixel.

20. The method of claim 19, wherein the color-depth similarity-based filter updates the depth of the target pixel to a value obtained by dividing a first result value by a second result value,

wherein:

the first result value is a sum of products between depths of the neighboring pixels and similarities of the neighboring pixels with respect to the target pixel; and

the second result value is a sum of similarities of the neighboring pixels with respect to the target pixel.

21. The method of claim 19, wherein:

the pixels in the up-scaled depth image are sequentially selected as the target pixel and updated; and

the target pixel is repeatedly updated, at least once, based on previously updated depths of neighboring pixels.

22. The method of claim 19, wherein the target pixel is updated within a range between a minimum value and a maximum value, the minimum value being a minimum depth from among depths of neighboring pixels before updating, and the maximum value being a maximum depth from among depths of neighboring pixels before updating.

23. The method of claim 19, further comprising:

generating the color image by down-sampling an original color image associated with the scene to match the depth image resolution of the depth image,

wherein the increasing of the depth image resolution, generating of the color image, and removing of noise are repeatedly performed until the depth image resolution of the up-scaled depth image matches an original resolution of the original color image.

24. The method of claim 18, wherein the converting further comprises:

identifying a maximum depth and a minimum depth from among depths of neighboring pixels of a target pixel in the up-scaled depth image;

determining whether the target pixel exists in a boundary area between objects in the depth image; and

replacing a depth of the target pixel with a replacement depth that is closer to the depth of the target pixel between the maximum depth and the minimum depth when the target pixel exists in the boundary area,

wherein the neighboring pixels correspond to pixels included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image.

26. A non-transitory computer-readable medium comprising a program for instructing a computer to perform the method of claim 18.

25. The method of claim 25, wherein the target pixel is determined to exist in the boundary area when a difference between the maximum depth and the minimum depth is greater than a predetermined threshold value.

27. An apparatus for processing a depth image and a color image associated with a scene, the apparatus comprising:

a depth image up-scaler to perform up-scaling of the depth image through use of interpolation to increase a depth image resolution of the depth image and produce an up-scaled depth image;

a color-depth similarity-based filter to remove noise in the up-scaled depth image using the color image and the up-scaled depth image; and

a boundary sharpening filter to convert a boundary between objects in the up-scaled depth image,

wherein the depth image resolution of the up-scaled depth image matches a color image resolution of the color image.

28. The apparatus of claim 27, wherein the color-depth similarity-based filter updates a depth of a target pixel in the up-scaled depth image based on similarities of neighboring pixels with respect to the target pixel and depths of neighboring pixels,

wherein:

the neighboring pixels correspond to pixels, included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image; and

a similarity of a neighboring pixel with respect to the target pixel is determined based on at least one of

a first difference between a color of a pixel of the color image corresponding to the neighboring pixel and a color of a pixel of the color image corresponding to the target pixel,

a second difference between a depth of the neighboring pixel and a depth of the target pixel, and

a third difference between a location of the neighboring pixel and a location of the target pixel.

29. The apparatus of claim 28, wherein the color-depth similarity-based filter updates the depth of the target pixel to a value obtained by dividing a first result value by a second result value,

wherein:

the first result value is a sum of products between the depths of neighboring pixels and similarities of neighboring pixels with respect to the target pixel; and

the second result value is a sum of the similarities of neighboring pixels with respect to the target pixel.

30. The apparatus of claim 28, wherein:

the pixels in the up-scaled depth image are sequentially selected as the target pixel and updated; and

the target pixel is repeatedly updated, at least once, based on previously updated depths of neighboring pixels.

31. The apparatus of claim 28, wherein the target pixel is updated within a range between a minimum value and a maximum value, the minimum value being a minimum depth from among depths of neighboring pixels before updating, and the maximum value being a maximum depth from among depths of neighboring pixels before updating.

32. The apparatus of claim 28, further comprising:

an original color image down-sampler to generate the color image by down-sampling an original color image associated with the scene to match the depth image resolution of the depth image,

wherein the depth image up-scaler, the original color image down-sampler, and the color-depth similarity-based filter repeatedly operate until the depth image resolution of the up-scaled depth image matches an original resolution of the original color image.

33. The apparatus of claim 27, wherein, to convert the boundary between the objects in the up-scaled depth image, the boundary sharpening filter performs:

identifying of a maximum depth and a minimum depth from among depths of neighboring pixels of a target pixel in the up-scaled depth image;

determining of whether the target pixel exists in a boundary area between objects in the depth image; and

replacing of a depth of the target pixel with a replacement depth that is closer to the depth of the target pixel from among the maximum depth and the minimum depth when the target pixel exists in the boundary area,

wherein the neighboring pixels correspond to pixels, included in a window of which a center is the target pixel, from among pixels in the up-scaled depth image.

34. The apparatus of claim 33, wherein the boundary sharpening filter determines that the target pixel exists in the boundary area when a difference between the maximum depth and the minimum depth is greater than a predetermined threshold value.

35. A method, comprising:

increasing a depth image resolution of a depth image to match a color image resolution of a two dimensional color image of a same scene;

removing noise in the depth image;

sharpening a boundary of an object in the depth image; and

generating a three dimensional color model of the scene using the depth image and the two dimensional color image.

36. A method as recited in claim 35, wherein the sharpening adjusts a pixel depth of a boundary pixel to a closest one of a maximum and a minimum pixel depth of neighboring pixels in the boundary.