COMBINING 3D IMAGE AND GRAPHICAL DATA

• US 20110304691A1
• Filed: 02/09/2010
• Published: 12/15/2011
• Est. Priority Date: 02/17/2009
• Status: Active Grant

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Newton, Philip Steven, De Haan, Wiebe, Talstra, Johan Cornelis, Bruls, Wilhelmus Hendrikus Alfonsus, Parlantzas, Georgios, Benien, Christian, Philomin, Vasanth, Varekamp, Christiaan, Van Der Heijden, Geradus Wilhelmus Theodorus, Helbing, Marc
• Granted Patent: US 9,438,879 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/183 On-screen display [OSD] information, e.g. subtitles or menus
• CPC Class Codes: H04N 13/293 Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
• CPC Class Codes: H04N 13/361 Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background

Abstract

Three dimensional [3D] image data and auxiliary graphical data are combined for rendering on a 3D display (30) by detecting depth values occurring in the 3D image data, and setting auxiliary depth values for the auxiliary graphical data (31) adaptively in dependence of the detected depth values. The 3D image data and the auxiliary graphical data at the auxiliary depth value are combined based on the depth values of the 3D image data. First an area of attention (32) in the 3D image data is detected. A depth pattern for the area of attention is determined, and the auxiliary depth values are set in dependence of the depth pattern.

Claims

1. Method of combining three dimensional [3D] image data and auxiliary graphical data, the method comprising:

obtaining scaling and/or shifting information for use with the three-dimensional [3D] image data,

scaling and/or shifting the three dimensional [3D] image data in accordance with the scaling and/or shifting information respectively,

combining the scaled and/or shifted three dimensional [3D] image data and the auxiliary graphical data such that at least part of the auxiliary graphical data is placed within a spatial area defined by the three dimensional [3D] image data which is not occupied by the scaled and/or shifted three dimensional [3D] image data.

2. Method of claim 1, wherein the auxiliary graphical data is at least one of:

two dimensional subtitle information,

two dimensional subpicture information,

three dimensional subtitle information and

three dimensional subpicture information.

3. Method of claim 1, wherein the scaling information and/or shifting information relate to scaling and or shifting within the direction of the display plane.

4. Method of claim 3, wherein the scaling and/or shifting information do not relate to scaling and/or shifting within the depth direction perpendicular to the display plane.

5. Method of claim 1, wherein the scaling information is a scale factor.

6. Method of claim 5, wherein the scale factor applies to scaling in both x and y direction of the display plane.

7. Method of claim 1, wherein the shift information is a offset in at least one of:

the horizontal direction of the display plane and

the vertical direction of the display plane.

8. Method of claim 1, wherein the auxiliary graphical information is located entirely within a spatial area defined by the three dimensional [3D] image data which is not occupied by the scaled and/or shifted three dimensional [3D] image data.

9. Method of claim 1 wherein the spatial area defined by the three dimensional [3D] image data which is not occupied by the scaled and/or shifted three dimensional [3D] image data is padded with a black background information.

23. Computer program product for combining three dimensional [3D] image data and auxiliary graphical data, which program is operative to cause a processor to perform, the respective steps of the method as claimed in claim 1.

10. 3D source device (10) for combining three dimensional [3D] image data and auxiliary graphical data, the device comprising:

means for obtaining scaling and/or shifting information for use with the three-dimensional [3D] image data,

means for scaling and/or shifting the three dimensional [3D] image data in accordance with the scaling and/or shifting information respectively,

means for combining the scaled and/or shifted three dimensional [3D] image data and the auxiliary graphical data such that at least part of the auxiliary graphical data is placed within a spatial area defined by the three dimensional [3D] image data which is not occupied by the scaled and/or shifted three dimensional [3D] image data.

11. 3D display device (13) for combining three dimensional [3D] image data and auxiliary graphical data, the device comprising:

means for obtaining scaling and/or shifting information for use with the three-dimensional [3D] image data,

means for scaling and/or shifting the three dimensional [3D] image data in accordance with the scaling and/or shifting information respectively,

means for combining the scaled and/or shifted three dimensional [3D] image data and the auxiliary graphical data such that at least part of the auxiliary graphical data is placed within a spatial area defined by the three dimensional [3D] image data which is not occupied by the scaled and/or shifted three dimensional [3D] image data.

12. Information carrier comprising three dimensional [3D] image data and auxiliary graphical data, the information carrier further comprising:

scaling and/or shifting information for use with the three-dimensional [3D] image data, so as to enable scaling and/or shifting the three dimensional [3D] image data in accordance with the scaling and/or shifting information respectively and subsequent combining of the scaled and/or shifted three dimensional [3D] image data and the auxiliary graphical data such that at least part of the auxiliary graphical data is placed within a spatial area defined by the three dimensional [3D] image data which is not occupied by the scaled and/or shifted three dimensional [3D] image data.

13. Method of combining three dimensional [3D] image data and auxiliary graphical data, the method comprising:

detecting depth values occurring in the 3D image data, and

setting auxiliary depth values for the auxiliary graphical data adaptively in dependence of the detected depth values for generating a 3D display signal for rendering the image content on a display area by combining the 3D image data and the auxiliary graphical data based on the auxiliary depth values,

wherein

detecting the depth values comprises detecting an area of attention in the 3D image data and determining a depth pattern for the area of attention, and

setting the auxiliary depth values comprises setting the auxiliary depth values in dependence of the depth pattern.

14. Method as claimed in claim 13, wherein detecting an area of attention comprises detecting an object of attention in the 3D image data, and the depth pattern is based on depth values of the object.

15. Method as claimed in claim 13, wherein detecting the object of attention in the 3D image data is based on at least one of

detecting image elements that are in focus, with respect to other image elements that are out of focus;

detecting, for image elements, an amount of additional 3D data for rendering the image element with respect to the background, such as occlusion data or transparency data;

detecting, for image elements, depth clues comprising at least one of differences in movement, depth, luminance and color with respect to the background;

detecting predetermined image elements such as human faces;

detecting, for image elements, position clues comprising at least one of being located near the center of the display area, and having at least a predetermined size relative to the display area.

16. Method as claimed in claim 13, wherein detecting the area of attention comprises selecting a target region for positioning the auxiliary graphical data, and

the depth pattern is based on depth values of the target region.

17. Method as claimed in claim 16, wherein selecting the target region comprises:

subdividing the display area in multiple regions, and

detecting the depth pattern is based on spatially filtering the depth values of the multiple regions according to a spatial filter function in dependence of the target region.

18. Method as claimed in claim 16, wherein selecting the target region comprises at least one of

selecting, as the target region, a region of the image data where no depth values occur larger than the auxiliary depth values;

selecting a period in time for displaying the auxiliary data such that, in the target region, no depth values occur larger than the auxiliary depth values;

selecting, as the target region, an area of the display where no image data is displayed, and accordingly reducing the size of the image data to fit in the remaining display area.

19. Method as claimed in claim 13, wherein determining the depth pattern comprises detecting depth values in multiple frames of video content, and temporally filtering the depth values according to a temporal filter function.

20. Method as claimed in claim 19, wherein determining the depth pattern comprises setting a time window for the temporal filter function based on detecting shot boundaries in the multiple frames of video content.

21. 3D source device (10) for combining three dimensional [3D] image data and auxiliary graphical data, the device comprising 3D image processing means (11,52) for

detecting depth values occurring in the 3D image data, and

setting auxiliary depth values for the auxiliary graphical data adaptively in dependence of the detected depth values for generating a 3D display signal for rendering the image content on a display area by combining the 3D image data and the auxiliary graphical data based on the auxiliary depth values,

wherein the 3D image processing means (11,52) are arranged for

detecting the depth values comprises detecting an area of attention in the 3D image data and determining a depth pattern for the area of attention, and

setting the auxiliary depth values comprises setting the auxiliary depth values in dependence of the depth pattern.

22. 3D display device (13) for combining three dimensional [3D] image data and auxiliary graphical data, the device comprising:

a 3D display (17) for displaying 3D image data, and

3D image processing means (18,19) for

detecting depth values occurring in the 3D image data, and

setting auxiliary depth values for the auxiliary graphical data adaptively in dependence of the detected depth values for generating a 3D display signal for rendering the image content on a display area by combining the 3D image data and the auxiliary graphical data based on the auxiliary depth values,

wherein the 3D image processing means (18,19) are arranged for

detecting the depth values comprises detecting an area of attention in the 3D image data and determining a depth pattern for the area of attention, and

setting the auxiliary depth values comprises setting the auxiliary depth values in dependence of the depth pattern.