MULTI-VIEW RENDERING APPARATUS AND METHOD USING BACKGROUND PIXEL EXPANSION AND BACKGROUND-FIRST PATCH MATCHING

• US 20120212480A1
• Filed: 01/12/2012
• Published: 08/23/2012
• Est. Priority Date: 01/13/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: CHO, Yang Ho, Park, Du Sik, Lee, Ho Young, Hwang, Kyu Young, Jeong, Young Ju
• Granted Patent: US 9,582,928 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

An apparatus and method for restoring a hole generated in multi-view rendering are provided. A hole in an output view may be restored using temporally neighboring images.

Claims

1. An image processing apparatus, comprising:

a processor to control one or more processor-executable units;

an image warping unit to generate an output view image by image warping using a reference view image, and binocular disparity information of the reference view image; and

a neighboring image-based hole restoration unit to restore a hole generated by the image warping using one or more temporally neighboring images.

2. The image processing apparatus of claim 1, wherein the one or more temporally neighboring images are temporally adjacent to the reference view image.

3. The image processing apparatus of claim 1, wherein the one or more temporally neighboring images are temporally adjacent to the output view image.

4. The image processing apparatus of claim 1, wherein the neighboring image-based hole restoration unit restores a pixel in the hole, using color values of pixels included in the temporally neighboring images and correspond to the pixel in the hole.

5. The image processing apparatus of claim 4, wherein, when all of the reference view image and the temporally neighboring images are moved over time, the neighboring image-based hole restoration unit selects the pixels corresponding to the pixel in the hole from the temporally neighboring images, based on a movement of the reference view image and the temporally neighboring images.

6. The image processing apparatus of claim 4, wherein the neighboring image-based hole restoration unit restores the hole, by excluding one or more hole pixels in the temporally neighboring images corresponding to the pixel in the hole.

7. The image processing apparatus of claim 4, wherein the neighboring image-based hole restoration unit restores the hole, using one or more background pixels among pixels corresponding to the pixel in the hole.

8. The image processing apparatus of claim 1, further comprising:

a buffer zone setting unit to expand the hole.

9. The image processing apparatus of claim 8, wherein the buffer zone setting unit expands the hole by regarding a buffer region adjacent to the hole as the hole, and

wherein, when the pixel in the hole is included in the buffer region, the neighboring image-based hole restoration unit restores the pixel in the hole based on a color value of the pixel in the hole.

10. The image processing apparatus of claim 1, further comprising:

a binocular disparity crack detection unit to set, as the hole, a crack in the output view image.

11. The image processing apparatus of claim 10, wherein, when a total sum of differences in a binocular disparity value between a pixel and neighboring pixels is greater than a predetermined value, the binocular disparity crack detection unit detects the pixel as a crack.

12. An image processing apparatus, comprising:

a processor to control one or more processor-executable units;

an image warping unit to generate an output view image by image warping using a reference view image and binocular disparity information of the reference view image; and

a neighboring pixel scaling-based hole restoration unit to restore a hole by scaling at least one pixel, the hole being generated by the image warping, and the at least one pixel being adjacent to the hole.

13. The image processing apparatus of claim 12, wherein the neighboring pixel scaling-based hole restoration unit performs scaling on one or more background pixels among the at least one pixel.

14. The image processing apparatus of claim 12, wherein the hole and the at least one pixel are on a same horizontal line.

15. The image processing apparatus of claim 12, wherein the neighboring pixel scaling-based hole restoration unit performs scaling on the at least one pixel in a direction perpendicular to a gradient of a background pixel among the at least one pixel adjacent to the hole.

16. An image processing apparatus, comprising:

a processor to control one or more processor-executable units;

an image warping unit to generate an output view image by image warping using a reference view image and binocular disparity information of the reference view image; and

an optimal patch search-based hole restoration unit to locate a patch from a background, and to restore a hole using the located patch, the hole being generated by the image warping, and the patch being most similar to a region including the hole.

17. The image processing apparatus of claim 16, wherein the region including the hole comprises a hole region and a background region, and

wherein the optimal patch search-based hole restoration unit restores the hole using a portion of the located patch that corresponds to the hole region.

18. The image processing apparatus of claim 16, wherein the optimal patch search-based hole restoration unit searches for a first patch for a first region of the output view image, and a second patch for a second region of the output view image, and restores an overlapping hole region between the first region and the second region using an average value of the first patch and the second patch.

19. An image processing method, comprising:

generating, by way of a processor, an output view image by image warping a reference view image, and based on binocular disparity information of the reference view image; and

restoring a hole generated in the output view image using one or more temporally neighboring images.

20. The image processing method of claim 19, wherein the one or more temporally neighboring images are temporally adjacent to the reference view image.

21. The image processing method of claim 19, wherein the one or more temporally neighboring images are temporally adjacent to the output view image.

22. The image processing method of claim 19, further comprising:

expanding the hole; and

setting, as the hole, a crack in the output view image.

23. The image processing method of claim 19, further comprising:

restoring the hole by scaling at least one pixel, the at least one pixel being adjacent to hole; and

restoring the hole by searching for a patch from a background, and using the found patch, the patch being most similar to a region including the hole.

24. A non-transitory computer readable recording medium storing a program to cause a computer to implement the method of claim 19.

25. An apparatus for generating multi-views, the apparatus comprising:

a processor to control one or more processor-executable units;

an image generating unit to generate an output view image based on a reference view image and binocular disparity information of the reference view image; and

a neighboring image-based hole restoration unit to restore a hole in the generated output view image using background information of one or more temporally neighboring images, the hole being generated as a result of the generating of the output view image.

26. The apparatus of claim 25, wherein the one or more temporally neighboring images are temporally adjacent to the reference view image.

27. The apparatus of claim 25, wherein the one or more temporally neighboring images are temporally adjacent to the output view image.

28. The apparatus of claim 25, wherein the image generating unit generates the output view image by interpolating or extrapolating data from the reference view.

29. An apparatus for generating multi-views, the apparatus comprising:

a processor to control one or more processor-executable units;

an image generating unit to generate an output view image based on at least one reference view image;

a binocular disparity crack detection unit to detect a crack in an object of the generated output view image, the object having different binocular disparity values allocated to different portions of the object, wherein the crack occurs in the object due to the generating of the output view image based on the at least one reference view image; and

a multi-view generating unit to reassign the crack as a hole and to restore the hole existing in a current frame of the generated output view image using background information of one or more temporally neighboring frames.

30. The apparatus of claim 29, wherein the one or more temporally neighboring frames are temporally adjacent to the reference view image.

31. The apparatus of claim 29, wherein the one or more temporally neighboring frames are temporally adjacent to the output view image.

32. The apparatus of claim 29, wherein when a total sum of differences in a binocular disparity value between a pixel and neighboring pixels is greater than a predetermined value, the binocular disparity crack detection unit may detect the pixel as a crack.

33. The apparatus of claim 29, wherein the temporally neighboring frames include at least one of a frame occurring prior in time to the current frame and a frame occurring after the current frame.

34. A method of generating multi-views, the method comprising:

generating, by way of a processor, an output view image based on at least one reference view image;

detecting a crack in an object of the generated output view image, the object having different binocular disparity values allocated to different portions of the object, wherein the crack occurs in the object due to the generating of the output view image based on the at least one reference view image; and

reassigning the crack as a hole and restoring the hole existing in a current frame of the generated output view image using background information of one or more temporally neighboring frames.

35. The method of claim 34, wherein the one or more temporally neighboring frames are temporally adjacent to the reference view image.

36. The method of claim 34, wherein the one or more temporally neighboring frames are temporally adjacent to the output view image.

37. The image processing method of claim 34, wherein when a total sum of differences in a binocular disparity value between a pixel and neighboring pixels is greater than a predetermined value, the binocular disparity crack detection unit may detect the pixel as a crack.

38. The image processing method of claim 34, wherein the temporally neighboring frames include at least one of a frame occurring prior in time to the current frame and a frame occurring after the current frame.

39. A display device including an image processing apparatus, the display device comprising:

an image generating unit to generate an output view image based on a reference view image and binocular disparity information of the reference view image;

a neighboring image-based hole restoration unit to restore a hole using one or more temporally neighboring images, the hole being generated by the generating of the output view image; and

a controller to generate a signal to be displayed by the display device based on the generated output view image having the hole restored by the neighboring image-based hole restoration unit.

40. The display device of claim 39, wherein the one or more temporally neighboring images are temporally adjacent to the reference view image.

41. The display device of claim 39, wherein the one or more temporally neighboring images are temporally adjacent to the output view image.