Depth value restoration method and system

• US 9,483,835 B2
• Filed: 04/28/2015
• Issued: 11/01/2016
• Est. Priority Date: 05/09/2014
• Status: Expired due to Fees

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Liang, Lingyan, Zhao, Ying, Liu, Liyan
• Primary Examiner(s): Ahmed, Samir
• Application Number:
  US14/697,804
• Publication Number:
  US 20150326845A1
• Time in Patent Office: 553 Days
• Field of Search: 382/154, 348/42, 345/419, 345/427, 356/12
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity

Abstract

Disclosed is a method comprising steps of conducting image preprocessing so as to respectively obtain a candidate object region including a foreground image in a depth map and a color image; determining whether it is necessary to conduct a region growing process with respect to the candidate object region of the depth map; in the former case, conducting the region growing process with respect to the candidate object region of the depth map; and conducting, after the region growing process is conducted with respect to the candidate object region of the depth map, a depth value restoration process with respect to a candidate region.

Claims

1. A depth value restoration method comprising:

conducting image noise reduction with regard to a depth map and a corresponding color image to establish a candidate object region in each of the depth map and the corresponding color image, respectively, the candidate object region of the depth map including a foreground image from the depth map and the candidate object region of the corresponding color image including a foreground image from the corresponding color image;

identifying an area of the candidate object region of the depth map associated with invalid depth values based on a determination that the area is associated with depth values that exceed a threshold range of valid depth values, the area being an invalid region of the depth map;

determining whether an area ratio of the invalid region to the candidate object region of the depth map is greater than a particular threshold value;

conducting region growing of the candidate object region of the depth map to generate a grown candidate object region of the depth map, based on

determining whether at least one candidate growing point grows, wherein determining whether the at least one candidate growing point grows includes

determining whether a depth value of the at least one candidate growing point is within a threshold range of valid depth values such that the depth value is valid; and

if the depth value of the at least one candidate growing point is valid,

calculating a depth value difference between the at least one candidate growing point and a seed point, and

if the depth value difference is less than or equal to a predetermined depth similarity threshold value, adding the at least one candidate growing point into a growing region, or

if the depth value of the at least one candidate growing point is invalid,

calculating a color difference between the at least one candidate growing point and the seed point, and

if the color difference is less than or equal to a predetermined color similarity threshold value,

giving the depth value of the seed point to the at least one candidate growing point, and

adding the at least one candidate growing point into the growing region, and

determining that the area ratio is greater than the particular threshold value; and

restoring depth values of one or more pixel points of the grown candidate object region of the depth map according to depth values of each pixel point located in a neighboring region of the depth map to generate a processed depth map.

2. The depth value restoration method according to claim 1, further comprising:

restoring depth values of one or more pixel points of the candidate object region of the depth map according to depth values of each pixel point located in a neighboring region of the depth map to generate a processed depth map, based on determining that the area ratio is less than or equal to the particular threshold value.

3. The depth value restoration method according to claim 1, wherein conducting region growing of the candidate object region of the depth map includes

selecting a pixel point in the foreground image of the depth map as the seed point, the selected pixel point having a valid depth value;

selecting each pixel point from a neighboring region of the seed point in the candidate object region of the depth map as a separate candidate growing point in the candidate object region of the depth map; and

adding the at least one candidate growing point into the growing region based on determining that the at least one candidate growing point grows.

4. The depth value restoration method according to claim 1, wherein restoring depth values of one or more pixel points of the grown candidate object region of the depth map includes

selecting a pixel point from the grown candidate object region, and generating a neighboring region, the generated neighboring region including a central point that is the selected pixel point;

calculating a weighting coefficient of a depth value of each pixel point in the generated neighboring region to a depth value of the central point, based on a relationship between each pixel point in the generated neighboring region and the central point; and

calculating a second depth value of the central point, based on both the calculated weighting coefficients and the depth value of each pixel point in the generated neighboring region; and

giving the calculated second depth value to the central point to serve as a new depth value of the central point.

5. The depth value restoration method according to claim 4, wherein

the relationship between each pixel point in the generated neighboring region and the central point includes

a spatial relationship,

a color relationship,

a depth relationship, and

a gradient relationship.

6. The depth value restoration method according to claim 5, further comprising:

obtaining the weighting coefficient of the depth value of each pixel point in the generated neighboring region to the depth value of the central point based on calculating a product of a spatial weighting coefficient, a color weighting coefficient, a gradient weighting coefficient, and/or a depth weighting coefficient between the corresponding pixel point and the central point.

7. The depth value restoration method according to claim 6, wherein

the spatial weighting coefficient, the color weighting coefficient, the gradient weighting coefficient, and the depth weighting coefficient are calculated based on utilizing a weighting kernel function, respectively.

8. The depth value restoration method according to claim 4, wherein

an area of the generated neighboring region is inversely proportional to a depth value of the selected pixel point.

9. A depth value restoration system comprising:

a memory storing computer-readable instructions; and

a processor configured to execute the computer-readable instructions to

conduct image noise reduction with regard to a depth map and a corresponding color image to establish a candidate object region in each of the depth map and the corresponding color image, respectively, the candidate object region of the depth map including a foreground image from the depth map and the candidate object region of the corresponding color image including a foreground image from the corresponding color image;

identify an area of the candidate object region of the depth map associated with invalid depth values based on a determination that the area is associated with depth values that exceed a threshold range of valid depth values, the area being an invalid region of the depth map;

determine whether an area ratio of the invalid region to the candidate object region of the depth map is greater than a particular threshold value;

conduct region growing of the candidate object region of the depth map to generate a grown candidate object region of the depth map, based on

determine whether at least one of the candidate growing points grows, wherein determining whether the at least one candidate growing point grows includes

determining whether a depth value of the at least one candidate growing point is within the threshold range of valid depth values such that the depth value is valid; and

if the depth value of the at least one candidate growing point is valid,

calculating a depth value difference between the at least one candidate growing point and a seed point, and

if the depth value difference is less than or equal to a predetermined depth similarity threshold value, adding the at least one candidate growing point into a growing region, or

if the depth value of the at least one candidate growing point is invalid,

calculating a color difference between the at least one candidate growing point and the seed point, and

if the color difference is less than or equal to a predetermined color similarity threshold value,

giving the depth value of the seed point to the at least one candidate growing point, and

adding the at least one candidate growing point into the growing region, and

determining that the area ratio is greater than the particular threshold value; and

restore depth values of one or more pixel points of the grown candidate object region of the depth map according to depth values of each pixel point located in a neighboring region of the depth map to generate a processed depth map.

10. The depth value restoration system according to claim 9, wherein the processor is configured to execute the computer-readable instructions to restore depth values of one or more pixel points of the candidate object region of the depth map according to depth values of each pixel point located in a neighboring region of the depth map to generate a processed depth map, based on determining that the area ratio is less than or equal to the particular threshold value.

11. The depth value restoration system according to claim 9, wherein the processor is configured to execute the computer-readable instructions interoperable with the memory to

select a pixel point in the foreground image of the depth map as the seed point, the selected pixel point having a valid depth value;

and

select each pixel point from a neighboring region of the seed point in the candidate object region of the depth map as a separate candidate growing point in the candidate object region of the depth map; and

add the at least one candidate growing point into the growing region based on determining that the at least one candidate growing point grows.

12. The depth value restoration system according to claim 9, wherein the processor is configured to execute the computer-readable instructions to

select a pixel point from the grown candidate object region, and generating a neighboring region, the generated neighboring region including a central point that is the selected pixel point;

calculate a weighting coefficient of a depth value of each pixel point in the generated neighboring region to a depth value of the central point, based on a relationship between each pixel point in the generated neighboring region and the central point; and

calculate a second depth value of the central point, based on both the calculated weighting coefficients and the depth value of each pixel point in the generated neighboring region; and

give the calculated second depth value to the central point to serve as a new depth value of the central point.

13. The depth value restoration system according to claim 12, wherein

the relationship between each pixel point in the generated neighboring region and the central point includes

a spatial relationship,

a color relationship,

a depth relationship, and

a gradient relationship.

14. The depth value restoration system according to claim 13, wherein the processor is configured to execute the computer-readable instructions to

obtain the weighting coefficient of the depth value of each pixel point in the generated neighboring region to the depth value of the central point based on calculating a product of a spatial weighting coefficient, a color weighting coefficient, a gradient weighting coefficient, and/or a depth weighting coefficient between the corresponding pixel point and the central point.

15. The depth value restoration system according to claim 14, wherein

the spatial weighting coefficient, the color weighting coefficient, the gradient weighting coefficient, and the depth weighting coefficient are calculated based on utilizing a weighting kernel function, respectively.

16. The depth value restoration system according to claim 12, wherein

an area of the generated neighboring region is inversely proportional to a depth value of the selected pixel point.

17. A non-transitory computer readable storage medium storing a program of instructions, the program of instructions being executable by a computer system to cause the computer system to:

generate a processed depth map based on a depth map and a corresponding color image, each of the depth map and the corresponding color image including a common candidate object region, respectively, the generating including,

conducting region growing of the candidate object region of the depth map to generate a grown candidate object region of the depth map, based on

determining whether the at least one candidate growing point grows, wherein determining whether the at least one candidate growing point grows includes

determining whether a depth value of the at least one candidate growing point is within a threshold range of valid depth values such that the depth value is valid; and

if the depth value of the at least one candidate growing point is valid,

calculating a depth value difference between the at least one candidate growing point and a seed point, and

if the depth value difference is less than or equal to a predetermined depth similarity threshold value, adding the at least one candidate growing point into a growing region, or

if the depth value of the at least one candidate growing point is invalid,

calculating a color difference between the at least one candidate growing point and the seed point, and

if the color difference is less than or equal to a predetermined color similarity threshold value,

giving the depth value of the seed point to the at least one candidate growing point, and

adding the at least one candidate growing point into the growing region, and

determining that a ratio of an area of the depth map having pixel points associated with invalid depth values to an area of the candidate object region of the depth map is greater than a particular threshold value, the invalid depth values being depth values external to a threshold depth value range; and

restoring depth values of one or more pixel points of the grown candidate object region of the depth map according to depth values of each pixel point located in a neighboring region of the depth map.

18. The non-transitory computer readable storage medium of claim 17, wherein restoring depth values of one or more pixel points of the grown candidate object region includes,

selecting a pixel point from the grown candidate object region and generating a neighboring region, the generated neighboring region including a central point that is the selected pixel point;

calculating a weighting coefficient of a depth value of each pixel point in the generated neighboring region to a depth value of the central point, based on a relationship between each pixel point in the generated neighboring region and the central point;

calculating a second depth value of the central point, based on both the calculated weighting coefficients and the depth value of each pixel point in the generated neighboring region; and

giving the calculated second depth value to the central point to serve as a new depth value of the central point.