Intermediate View Synthesis and Multi-View Data Signal Extraction

• US 20110261050A1
• Filed: 04/01/2011
• Published: 10/27/2011
• Est. Priority Date: 10/02/2008
• Status: Active Grant

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Inventors: Dix, Kristina, Smolic, Aljosa, Mueller, Karsten
• Granted Patent: US 9,214,040 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 2213/005 Aspects relating to the "3D+depth" image format

Abstract

An intermediate view synthesis apparatus for synthesizing an intermediate view image from a first image corresponding to a first view and a second image corresponding to a second view different from the first view, the first and second images including depth information, wherein the second image being is divided-up into a non-boundary portion, and a foreground/background boundary region, wherein the intermediate view synthesis apparatus is configured to project and merge the first image and the second image into the intermediate view to obtain an intermediate view image, with treating the foreground/background boundary region subordinated relative to the non-boundary portion. A multi-view data signal extraction apparatus for extracting a multiview data signal from a multi-view representation including a first image corresponding to a first view and a second image corresponding to a second view being different from the first view is also described, the first and second images including depth information.

Claims

1. An intermediate view synthesis apparatus for synthesizing an intermediate view image from a first image corresponding to a first view and a second image corresponding to a second view different from the first view, the first and second images comprising depth information, wherein the second image is divided-up into a non-boundary portion and a foreground/background boundary region, wherein the foreground/background boundary region of the second image comprises a foreground boundary portion and a background boundary portion being disjoint to the foreground boundary portion, and wherein the intermediate view synthesis apparatus is configured to project and merge the first image and the second image into the intermediate view to achieve an intermediate view image, with using the background boundary portion of the foreground/background boundary region merely to fill holes otherwise occurring in the intermediate view image, and using the foreground boundary portion of the foreground/background boundary region, to project and merge into the intermediate view, wherein the intermediate view synthesis apparatus comprises:

a projector/merger for projecting and merging the non-boundary portion and the foreground boundary portion of the second image along with at least a part of the first image into the intermediate view to achieve a preliminary intermediate view image; and

a projector/filler for projecting the background boundary portion of the second image into, and filling the holes in the preliminary intermediate view image by the projected background boundary portion of the second image.

2. The intermediate view synthesis apparatus according to claim 1, further comprising

a detector for detecting the foreground/background boundary region in the second image;

a divider for dividing-up the foreground/background boundary region of the second image into the foreground boundary portion and the background boundary portion so that the second image is divided-up into the non-boundary portion, the foreground boundary portion and the background boundary portion.

3. The intermediate view synthesis apparatus according to claim 1, wherein the intermediate view synthesis apparatus is configured to acquire layer information discriminating the non-boundary portion, the foreground boundary portion and the background boundary portion of the second image from a multi-view data signal.

4. The intermediate view synthesis apparatus according to claim 2, wherein the detector is configured to detect also a foreground/background boundary region in the first image, the divider is configured to also divide-up the foreground/background boundary region of the first image into a foreground boundary portion and a background boundary portion so that the first image is also divided-up into a non-boundary portion, a foreground boundary portion and a background boundary portion, and the projector/merger is configured to project and merge the non-boundary portion and the foreground boundary portion of the first and second images into the intermediate view to achieve the preliminary intermediate view image, and the projector/filler is configured to project the background boundary portion of the first and second images into and fill the holes in the preliminary intermediate view image by the projected background boundary portion of the first and second images.

5. The intermediate view synthesis apparatus according to claim 1, wherein the intermediate view synthesis apparatus is configured to acquire layer information discriminating, at least, a non-boundary portion and a background boundary portion of the first image from a multi-view data signal, wherein the projector/merger is configured to project and merge the non-boundary portion and the foreground boundary portion of the second image and the non-boundary portion of the first image into the intermediate view to achieve the preliminary intermediate view image, and the projector/filler is configured to project the background boundary portion of the first and second images into and fill the holes in the preliminary intermediate view image by the projected background boundary portion of the first and second images.

6. The intermediate view synthesis apparatus according to claim 1, wherein the intermediate view synthesis apparatus is configured to acquire layer information discriminating a non-boundary portion, a foreground boundary portion and a background boundary portion of the first image, wherein the projector/merger is configured to project and merge the non-boundary portion and the foreground boundary portion of the first and second images into the intermediate view to achieve the preliminary intermediate view image, and the projector/filler is configured to project the background boundary portion of the first and second images into and fill the holes in the preliminary intermediate view image by the projected background boundary portion of the first and second images.

7. The intermediate view synthesis apparatus according to claim 2, wherein the detector for detecting a foreground/background boundary region comprises an edge detector for detecting an edge in the depth information.

8. The intermediate view synthesis apparatus according to claim 2, wherein the divider is configured to split an n-sample-wide area around the edge as the foreground/background boundary region into the foreground boundary portion and the background boundary portion.

9. The intermediate view synthesis apparatus according to claim 2, wherein the divider is configured to perform the splitting by comparing a depth of the depth information at a sample of the foreground/background boundary region with a threshold to decide as to whether the sample belongs to the foreground boundary portion or the background boundary portion.

10. The intermediate view synthesis apparatus according to claim 9, wherein the divider is configured to derive the threshold value by computing a central tendency of depth values of the depth information at samples of the foreground/background boundary region neighboring the sample.

11. The intermediate view synthesis apparatus according to claim 4, wherein the projector/merger is configured to project and merge the non-boundary portion of the first and second images into the intermediate view to achieve a non-boundary intermediate view image, with the merging of the non-boundary portion comprising averaging between the first and second images, and project and merge the foreground boundary portion of the first and second images into the non-boundary intermediate view image with the merging of the foreground boundary portion comprising a selection between samples of the non-boundary intermediate view image and co-located samples resulting from the projection of the foreground boundary portion of the first and second images rather than an averaging thereof.

12. The intermediate view synthesis apparatus according to claim 1, wherein the projector/merger and the projector/filler are configured to perform the respective projection into the intermediate view by projecting samples from 2D to 3D by use of the associated depth information to achieve 3D world points and projecting the 3D world points into the intermediate view.

13. The intermediate view synthesis apparatus according to claim 12, wherein the m projector/merger and the projector/filler are configured to compute a projecting matrix by interpolating between projecting matrices of the first view and the second view and perform the projection of the 3D world points by use of the projection matrix.

14. The intermediate view synthesis apparatus according to claim 1, wherein the projector/merger and the projector/filler are configured to perform the respective projection into the intermediate view by in-pixel-plane shifting the sample positions at an amount depending on a deviation from the intermediate view.

15. The intermediate view synthesis apparatus according to claim 1, wherein the intermediate view synthesis apparatus is configured to extract from a multi-view data signal the first image corresponding to the first view and the second image corresponding to the second view, comprising the depth information of the first and second images, wherein merely the part of the first image is comprised by the multi-view data signal while the disjoint portion thereof is discarded in the multi-view data signal, the part of the first image depending on a location of a disoccluded area within a projected version of the second image resulting from projecting the second image without the background boundary portion into the first view.

16. The intermediate view synthesis apparatus according to claim 1, further comprising:

an enhancer for enhancing the intermediate view image by filtering and/or extrapolation into remaining holes of the intermediate view image.

17. The intermediate view synthesis apparatus according to claim 1, further comprising:

an enhancer for enhancing the intermediate view image by detecting a foreground/background boundary region of the intermediate view image and low-pass filtering within the foreground/background boundary region of the intermediate view image.

18. The intermediate view synthesis apparatus according to claim 1, wherein the first and second images are part of an MVD or LDV video.

19. The intermediate view synthesis apparatus according to claim 1, wherein the first and second images comprise color information associated with the depth information.

20. The intermediate view synthesis apparatus according to claim 1, the intermediate view synthesis apparatus being part of a 3DTV display system.

21. A multi-view data signal extraction apparatus for extracting a multiview data signal from a multi-view representation comprising a first image corresponding to a first view and a second image corresponding to a second view being different from the first view, the first and second images comprising depth information, the multiview data signal extraction apparatus comprising:

a detector for detecting a foreground/background boundary in the second image;

a determiner for determining a background boundary portion of the second image along the foreground/background boundary;

a projector for projecting the second image without the background boundary portion into the first view to achieve a projected version of the second image comprising a disoccluded area; and

an inserter for inserting the second image along with a portion of the first image, depending on a location of the disoccluded area within the projected version of the second image, into the multiview data signal.

22. The multi-view data signal extraction apparatus according to claim 21, further comprising a determiner for determining the portion of the first image to be inserted into the multiview data signal as an area of the first image over which the disoccluded area within the projected version of the second image extends.

23. The multi-view data signal extraction apparatus according to claim 21, wherein the projector is configured to project the second image without the background boundary portion also into a third view to achieve a further projected version of the second image comprising a disoccluded area and wherein the inserter is configured to insert the second image, the portion of the first image and a portion of the third image, depending on a location of the disoccluded area within the further projected version of the second image, into the multiview data signal, wherein the second view lies between the first and third view.

24. The multi-view data signal extraction apparatus according to claim 23, wherein the inserter is configured to insert a layer information into the multi-view data signal, the layer information discriminating, at least, the non-boundary portion and the background boundary portion of the second image.

25. The multi-view data signal extraction apparatus according to claim 21, further comprising a filler for filling small holes resulting from the projection of the second image and/or filtering of the projected version of the second image.

26. The multi-view data signal extraction apparatus according to claim 21, wherein the inserter is configured to discard a remaining portion of the first image disjoint from the portion depending on the location of the disoccluded area within the projected version of the second image and to signal the discarding in the multi-view data signal.

27. The multi-view data signal extraction apparatus according to claim 21, wherein the detector for detecting a foreground/background boundary comprises an edge detector for detecting an edge in the depth information.

28. The multi-view data signal extraction apparatus according to claim 27, wherein the determiner is configured to split an n-sample-wide area around the edge into a foreground boundary portion and the background boundary portion.

29. The multi-view data signal extraction apparatus according to claim 28, wherein the determiner is configured to perform the splitting by comparing a depth of the depth information at a sample of the n-sample-wide area with a threshold to decide as to whether the sample belongs to the foreground boundary portion or the background boundary portion.

30. The multi-view data signal extraction apparatus according to claim 29, wherein the determiner up is configured to derive the threshold value by computing a central tendency of depth values of the depth information at samples of the n-sample-wide area neighboring the sample.

31. The multi-view data signal extraction apparatus according to claim 21, wherein the projector is configured to perform the respective projection into the first view by projecting samples from 2D to 3D by use of the associated depth information to achieve 3D world points and projecting the 3D world points into the intermediate view.

32. The multi-view data signal extraction apparatus according to claim 31, wherein the projector is configured to compute a projecting matrix by interpolating between projecting matrices of the first view and the second view and perform the projection of the 3D world points by use of the projection matrix.

33. The multi-view data signal extraction apparatus according to claim 21, wherein the projector is configured to perform the respective projection into the intermediate view by in-pixel-plane shifting the sample positions at an amount depending on a deviation from the intermediate view.

34. A multi-view data signal comprising data about a first image corresponding to a first view and a second image corresponding to a second view being different from the first view, the first and second images comprising depth information, wherein merely a portion of the first image is comprised by the multi-view data signal while the disjoint portion thereof is discarded in the multi-view data signal, the portion depending on a location of a disoccluded area within a projected version of the second image resulting from projecting the second image without a background boundary portion extending along a foreground/background boundary in the second image, into the first view.

35. An intermediate view synthesis method for synthesizing an intermediate view image from a first image corresponding to a first view and a second image corresponding to a second view different from the first view, the first and second images comprising depth information, wherein the second image is divided-up into a non-boundary portion and a foreground/background boundary region, wherein the foreground/background boundary region of the second image comprises a foreground boundary portion and a background boundary portion being disjoint to the foreground boundary portion, and wherein the intermediate view synthesis method comprises:

projecting and merging the non-boundary portion and the foreground boundary portion of the second image along with at least a part of the first image into the intermediate view to achieve a preliminary intermediate view image; and

projecting the background boundary portion of the second image into, and filling the holes in the preliminary intermediate view image by the projected background boundary portion of the second image.

36. A multi-view data signal extraction method for extracting a multiview data signal from a multi-view representation comprising a first image corresponding to a first view and a second image corresponding to a second view being different from the first view, the first and second images comprising depth information, the multiview data signal extraction method comprising:

detecting a foreground/background boundary in the second image;

determining a background boundary portion along the foreground/background boundary;

projecting the second image without the background boundary portion into the first view to achieve a projected version of the second image comprising a disoccluded area; and

inserting the second image along with a portion of the first image, depending on a location of the disoccluded area within the projected version of the second image, into the multiview data signal.

37. A computer program comprising instructions for performing, when running on a computer, a method for synthesizing an intermediate view image from a first image corresponding to a first view and a second image corresponding to a second view different from the first view, the first and second images comprising depth information, wherein the second image is divided-up into a non-boundary portion and a foreground/background boundary region, wherein the foreground/background boundary region of the second image comprises a foreground boundary portion and a background boundary portion being disjoint to the foreground boundary portion, and wherein the intermediate view synthesis method, wherein the intermediate view synthesis method comprises:

projecting and merging the non-boundary portion and the foreground boundary portion of the second image along with at least a part of the first image into the intermediate view to achieve a preliminary intermediate view image; and

projecting the background boundary portion of the second image into, and filling the holes in the preliminary intermediate view image by the projected background boundary portion of the second image.

38. A computer program comprising instructions for performing, when running on a computer, a method for extracting a multiview data signal from a multi-view representation comprising a first image corresponding to a first view and a second image corresponding to a second view being different from the first view, the first and second images comprising depth information, the multiview data signal extraction method comprising:

detecting a foreground/background boundary in the second image;

determining a background boundary portion along the foreground/background boundary;

projecting the second image without the background boundary portion into the first view to achieve a projected version of the second image comprising a disoccluded area; and

inserting the second image along with a portion of the first image, depending on a location of the disoccluded area within the projected version of the second image, into the multiview data signal.