Depth map aligning method and system

• US 9,462,251 B2
• Filed: 02/25/2014
• Issued: 10/04/2016
• Est. Priority Date: 01/02/2014
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Tu, Jih-Sheng, Kao, Jung-Yang
• Primary Examiner(s): JONES, HEATHER RAE
• Application Number:
  US14/189,258
• Publication Number:
  US 20150189253A1
• Time in Patent Office: 952 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity

Abstract

A depth map aligning method and a system are provided. The depth map aligning method includes the following steps. A view warping is performed to a left-view image and a right-view image of a real-view image according to a corresponding depth map to obtain a right-virtual view image, a left hole information, and a right hole information thereof. A residual difference of pixel, which is not a hole, is obtained by pixel to pixel. If the residual difference of the pixel is larger than a first threshold value, a reverse warping is performed to obtain a coordinate of the pixel. The depth value of the pixel is aligned according to the depth value of one or more pixels in a vicinity of the obtained coordinate.

Claims

1. A depth map aligning method, comprising:

performing a view warping to a left-view image and a right-view image of a real-view image according to a corresponding depth map to obtain a left-virtual-view image, a right-virtual-view image, a left hole information, and a right hole information;

obtaining by pixel to pixel a residual difference of at least one pixel, which is not a hole, by subtracting pixel value of the left-virtual-view image and the right-virtual-view image;

performing a reverse warping to obtain a coordinate of the at least one pixel in the real-view image, if the residual difference of the at least one pixel is larger than a first threshold value; and

performing a depth map aligning by correcting a depth value of the at least one pixel according to depth values of one or more pixels in a vicinity of the obtained coordinate.

2. The depth map aligning method according to claim 1, wherein the left hole information includes a number of left hole, and the step of performing the reverse warping further comprises:

performing the reverse warping to obtain a warping coordinate in the left-view image, if the number of left hole of the at least one pixel is larger than a second threshold value.

3. The depth map aligning method according to claim 2, wherein after the reverse warping, the depth map aligning method further comprises:

obtaining the one or more pixels of a left vicinity for a current pixel location of the left-view image; and

performing the depth map aligning, if depth value of the current pixel is smaller than the depth value of each of the one or more pixels.

4. The depth map aligning method according to claim 2, further comprising:

synthesizing a depth map of the left-virtual-view image by using the depth map of the left-view image to perform the reverse warping.

5. The depth map aligning method according to claim 2, further comprising:

producing a left lookup table to perform the reverse warping, for each pixel of the left-virtual-view image, the left lookup table records a warping distance projected from the real-view image.

6. The depth map aligning method according to claim 2, wherein the second threshold value is a setting value.

7. The depth map aligning method according to claim 2, wherein the second threshold value is a positive integer greater than or equal to 1, and

the second threshold value is

8. The depth map aligning method according to claim 1, wherein the right hole information includes a plurality of right hole, and the step of performing the reverse warping further comprises:

performing the reverse warping to obtain a warping coordinate in the right-view image, if the number of right hole of the at least one pixel is larger than a second threshold value.

9. The depth map aligning method according to claim 8, further comprises:

synthesizing a depth map of the right-virtual-view image by using the depth map of the right-view image to perform the reverse warping.

10. The depth map aligning method according to claim 8, further comprising:

producing a right lookup table to perform the reverse warping, for each pixel of the right-virtual-view image, the right lookup table records a warping distance projected from the real-view image.

11. The depth map aligning method according to claim 8, wherein after the reverse warping, the depth map aligning method further comprises:

obtaining the one or more pixels of a right vicinity for a current pixel location of the right-view image; and

performing the depth map aligning, if depth value of the current pixel is smaller than the depth value of each of the one or more pixels.

12. The depth map aligning method according to claim 8, wherein the second threshold value is a setting value.

13. The depth map aligning method according to claim 8, wherein the second threshold value is a positive integer greater than or equal to 1, and

the second threshold value is

14. The depth map aligning method according to claim 1, wherein the vicinity of the coordinate includes a setting distance of a setting direction that surrounding the at least one pixel.

15. The depth map aligning method according to claim 14, wherein the depth value of the at least one pixel is corrected by obtaining a arithmetic mean or a median of the depth values of one or more pixels in the vicinity of the coordinate.

16. The depth map aligning method according to claim 1, wherein the first threshold is a setting value.

17. The depth map aligning method according to claim 1, further comprising:

dividing all of the residual difference of the at least one pixel into M×

N blocks, wherein M and N are positive integers; and

obtaining an arithmetic mean or a median of each of the blocks to form the first threshold for each block.

18. The depth map aligning method according to claim 1, wherein after the reverse warping, the depth map aligning method further comprises:

obtaining the one or more pixels of a left vicinity for a current pixel location of the left-view image; and

performing the depth map aligning, if depth value of the current pixel is smaller than the depth value of each of the one or more pixels.

19. The depth map aligning method according to claim 1, wherein after the reverse warping, the depth map aligning method further comprises:

obtaining the one or more pixels of a right vicinity for a current pixel location of the right-view image; and

performing the depth map aligning, if depth value of the current pixel is smaller than the depth value of each of the one or more pixels.

20. The depth map aligning method according to claim 1, wherein the depth value of the at least one pixel is corrected by obtaining a arithmetic mean or a median of the depth values of one or more pixels in the vicinity of the coordinate.

21. A depth map aligning system, comprising:

a view warping unit for obtaining a left-virtual-view image, a right-virtual-view image, a left hole information, and a right hole information according to a corresponding depth map, and performing a view warping to a left-view image and a right-view image of a real-view image;

a residual difference unit for subtracting pixel value of the left-virtual-view image and the right-virtual-view image by pixel to pixel, to obtain a residual difference of at least one pixel, which is not a hole;

a first decision unit for determining whether the residual difference of the at least one pixel is larger than a first threshold value, wherein if the residual difference of the at least one pixel is larger than the first threshold value, then a reverse warping is performed;

a reverse warping unit for obtaining a coordinate of the at least one pixel in the real-view image, and performing the reverse warping; and

a depth map aligning unit for performing a depth map aligning by correcting a depth value of the at least one pixel according to depth values of one or more pixels in a vicinity of the coordinate obtained.

22. The depth map aligning system according to claim 21, wherein the left hole information includes a plurality of left hole, the right hole information includes a plurality of right hole, and system further performs the reverse warping to obtain a warping coordinate if the number of left hole or the number of right hole of the at least one pixel is larger than a second threshold value.

23. The depth map aligning system according to claim 22, further comprising:

a third decision unit for obtaining the one or more pixels of a right vicinity or a left vicinity for a current pixel location of the right-view image or left-view image respectively after performing the reverse warping and judging if depth value of the current pixel is smaller than the depth value of each of the one or more.

24. The depth map aligning system according to claim 21, further comprising:

a third decision unit, obtaining the one or more pixels of a right vicinity or a left vicinity for a current pixel location of the right-view image or left-view image respectively after performing the reverse warping and judging if depth value of the current pixel is smaller than the depth value of each of the one or more pixels.

25. The depth map aligning system according to claim 21, wherein the vicinity of the coordinate includes a setting distance of a setting direction that surrounds the at least one pixel.

26. The depth map aligning system according to claim 21, wherein the depth value of the at least one pixel is corrected by obtaining a arithmetic mean or a median of the depth value of one or more pixels in the vicinity of the coordinate.