IMAGE PROCESSING METHOD AND APPARATUS FOR ELIMINATION OF DEPTH ARTIFACTS

• US 20140240467A1
• Filed: 05/17/2013
• Published: 08/28/2014
• Est. Priority Date: 10/24/2012
• Status: Abandoned Application

• Current Assignee: Avago Technologies General IP PTE Limited (Broadcom, Inc.)
• Original Assignee: LSI Corporation (Broadcom, Inc.)
• Inventors: Mazurenko, Ivan L., Parfenov, Denis V., Babin, Dmitry N., Petyushko, Alexander A., Kholodenko, Alexander B.
• Application Number:
  US14/232,143
• Publication Number:
  US 20140240467A1
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An image processing system comprises an image processor configured to identify one or more potentially defective pixels associated with at least one depth artifact in a first image, and to apply a super resolution technique utilizing a second image to reconstruct depth information of the one or more potentially defective pixels. Application of the super resolution technique produces a third image having the reconstructed depth information. The first image may comprise a depth image and the third image may comprise a depth image corresponding generally to the first image but with the depth artifact substantially eliminated. An additional super resolution technique may be applied utilizing a fourth image. Application of the additional super resolution technique produces a fifth image having increased spatial resolution relative to the third image.

Claims

1. A method comprising:

identifying one or more potentially defective pixels associated with at least one depth artifact in a first image; and

applying a super resolution technique utilizing a second image to reconstruct depth information of said one or more potentially defective pixels;

wherein application of the super resolution technique produces a third image having the reconstructed depth information;

wherein the identifying and applying steps are implemented in at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the first image comprises a depth image and the third image comprises a depth image corresponding generally to the first image but with said at least one depth artifact substantially eliminated.

3. The method of claim 1 further comprising:

applying an additional super resolution technique utilizing a fourth image;

wherein application of the additional super resolution technique produces a fifth image having increased spatial resolution relative to the third image.

4. The method of claim 3 wherein the first image comprises a depth image and the fifth image comprises a depth image generally corresponding to the first image but with said at least one depth artifact substantially eliminated and the resolution increased.

5. The method of claim 1 wherein identifying one or more potentially defective pixels comprises:

marking at least a subset of the potentially defective pixels; and

removing the marked potentially defective pixels from the first image prior to applying the super resolution technique.

6. The method of claim 1 wherein the first image comprises a depth image of a first resolution from a first image source and the second image comprises a two-dimensional image of substantially the same scene and having a resolution substantially the same as the first resolution from another image source different than the first image source.

7. The method of claim 3 wherein the first image comprises a depth image of a first resolution from a first image source and the fourth image comprises a two-dimensional image of substantially the same scene and having a resolution substantially greater than the first resolution from another image source different than the first image source,

8. The method of claim 1 wherein identifying one or more potentially defective pixels comprises detecting pixels of the first image having depth values set to respective predetermined error values by an associated depth imager.

9. The method of claim 1 wherein identifying one or more potentially defective pixels comprises detecting an area of contiguous pixels having respective unexpected depth values that differ substantially from depth values of pixels outside of the area.

10. The method of claim 9 wherein the area of contiguous pixels having respective unexpected depth values is defined so as to satisfy the following inequality with reference to a peripheral border of the area:

|statistic{d_i;

pixel i is in the area}−

statistic{d_j;

pixel j is in the border}|>

d_T

where d_T is a threshold value, and statistic denotes one of mean, median and distance metric.

11. The method of claim 1 wherein identifying one or more potentially defective pixels comprises:

identifying a particular one of the pixels;

identifying a neighborhood of pixels for the particular pixel; and

identifying the particular pixel as a potentially defective pixel based on a depth value of the particular pixel and at least one of a mean and a standard deviation of depth values of the respective pixels in the neighborhood of pixels.

12. The method of claim 11 wherein identifying a neighborhood of pixels for the particular pixel comprises identifying a set S_p of n neighbors of particular pixel p:

S_p{p₁, . . . , p_n},

where the n neighbors each satisfy the inequality;

∥

p−

p_i∥

<

d,

where d is a neighborhood radius and ∥

·

∥

denotes a distance metric between pixels p and p_i in an x-y plane.

13. The method of claim 11 wherein identifying the particular pixel as a potentially defective pixel comprises identifying the particular pixel as a potentially defective pixel if the following inequality is satisfied:

|z_p−

m|>

kσ

,

where z_p is the depth value of the particular pixel, in and r are the mean and standard deviation, respectively, of the depth values of the respective pixels in the neighborhood of pixels, and k is a multiplying factor specifying a degree of confidence.

14. The method of claim 1 wherein applying the super resolution technique comprises applying a super resolution technique that is based at least in part on a Markov random field model.

15. The method of claim 3 wherein applying the additional super resolution technique comprises applying a super resolution technique that is based at least in part on bilateral filters.

16. A computer-readable storage medium having computer program code embodied therein, wherein the computer program code when executed in the processing device causes the processing device to perform the method of claim 1.

17. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

wherein said at least one processing device comprises;

a pixel identification module configured to identify one or more potentially defective pixels associated with at least one depth artifact in a first image; and

a super resolution module configured to utilize a second image to reconstruct depth information of said one or more potentially defective pixels;

wherein the super resolution module produces a third image having the reconstructed depth information.

18. The apparatus of claim 17 wherein the super resolution module is further configured to process the third image utilizing a fourth image in order to produce a fifth image having increased spatial resolution relative to the third image.

19. The apparatus of claim 17 wherein the first image comprises a depth image of a first resolution from a first image source and the second image comprises a two-dimensional image of substantially the same scene and having a resolution substantially the same as the first resolution from another image source different than the first image source

20. The apparatus of claim 19 wherein the first image source comprises a three-dimensional image source including one of a structured light camera and a time of flight camera.

21. The apparatus of claim 19 wherein the second image source comprises a two-dimensional image source configured to generate the second image as one of an infrared image, a gray scale image and a color image.

22. The apparatus of claim 18 wherein the first image comprises a depth image of a first resolution from a first image source and the fourth image comprises a two-dimensional image of substantially the same scene and having a resolution substantially greater than the first resolution from another image source different than the first image source.

23. An image processing system comprising the apparatus of claim 17.

24. A gesture detection system comprising the image processing system of claim 23.