IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD

• US 20130050412A1
• Filed: 08/08/2012
• Published: 02/28/2013
• Est. Priority Date: 08/24/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Shinohara, Takayuki, MORISADA, Hidehiko, KAWAI, Aritoki
• Granted Patent: US 9,118,894 B2
• Time in Patent Office: Days
• US Class Current: 348/42
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A viewpoint position sets, when a stereoscopic video image is provided that includes a left-eye parallax image and a right-eye parallax image having a predetermined parallax distribution and that is observed from a given viewpoint, a virtual viewpoint for observing a subject from another viewpoint. A parallax image generation unit generates a left-eye and a right-eye parallax image for providing a desired parallax distribution that are obtained when an observation is made from the virtual viewpoint, by shifting the image cut-out position of at least one of the left-eye or the right-eye parallax image. When the shift amount at the image cut-out position changes due to a position change of the virtual viewpoint, the parallax image generation unit generates a parallax image while changing the shift amount in stages from a shift amount obtained before the change to a shift amount obtained after the change.

Claims

1. An image processing apparatus comprising:

a viewpoint setting unit configured to set, when a stereoscopic video image is provided that includes a left-eye parallax image and a right-eye parallax image having a predetermined parallax distribution and that is observed from a given viewpoint, a virtual viewpoint for observing a subject in the stereoscopic video image from another viewpoint; and

a parallax image generation unit configured to generate a left-eye parallax image and a right-eye parallax image for providing a desired parallax distribution obtained when an observation is made from the viewpoint set by the viewpoint setting unit, by shifting an image cut-out position of at least one of the left-eye parallax image or the right-eye parallax image,

wherein, when a shift amount at the image cut-out position changes due to a change made in the position of the virtual viewpoint, the parallax image generation unit generates a parallax image while changing the shift amount in stages, from a shift amount obtained before the change to a shift amount obtained after the change.

2. The image processing apparatus according to claim 1 further comprising:

a reference coordinate storage unit configured to store the position coordinates of a shift-amount calculation reference point that is determined as a reference for calculating a shift amount at an image cut-out position in a virtual three-dimensional space of the stereoscopic video image,

wherein, the parallax image generation unit acquires, in reference to the reference coordinate storage unit, the respective position coordinates of shift-amount calculation reference points that exist near the position of the viewpoint set by the viewpoint setting unit and further includes a shift amount calculation unit configured to calculate, as the value of a shift amount, a value that is greater than or equal to a minimum shift amount value but less than or equal to a maximum shift amount value, all inclusive, calculated based on stereoscopic video images of the subject obtained when an observation is made at the respective position coordinates.

3. The image processing apparatus according to claim 2,

wherein the shift amount calculation unit includes:

a reference coordinate acquisition unit configured to acquire, in reference to the reference coordinate storage unit, the respective position coordinates of the shift-amount calculation reference points that exist near the position of the viewpoint set by the viewpoint setting unit;

a parallax distribution generation unit configured to generate the distribution of parallaxes between a left-eye parallax image and a right-eye parallax image of a stereoscopic video image obtained when an observation is made after providing viewpoint at each set of position coordinates acquired by the reference coordinate acquisition unit;

a local shift amount calculation unit configured to generate, based on the distribution of parallaxes generated by the parallax distribution generation unit, a parallax adjustment amount such that the parallaxes between the left-eye parallax image and the right-eye parallax image of the stereoscopic video image are within a predetermined parallax range and to obtain a shift amount that allows many subjects to be localized near a display device for displaying the stereoscopic video image; and

an adjustment unit configured to set a weighted average of shift amounts obtained at the respective position coordinates of the reference points calculated by the local shift amount calculation unit to be the value of a shift amount obtained when an observation is made at the viewpoint set by the viewpoint setting unit.

4. The image processing apparatus according to claim 3,

wherein the adjustment unit sets, as a weight used when obtaining the weighted average, the ratio of the volume of each of a plurality of sub-solids to the volume of a solid,

the solid having, as a vertex, each set of position coordinates acquired by the reference coordinate acquisition unit,

the plurality of sub-solids being obtained by dividing the solid by three planes that are perpendicular to one another, and

the three planes that are perpendicular to one another having a point of intersection at the position of the viewpoint set by the viewpoint setting unit.

5. The image processing apparatus according to claim 4 further comprising:

a synthesis unit configured to acquire a first stereoscopic video image and a second stereoscopic video image different from the first stereoscopic video image and then superimpose, on a part of the first stereoscopic video image, the second stereoscopic video image,

wherein the synthesis unit identifies, based on the distribution of parallaxes generated by the parallax distribution generation unit for the first stereoscopic video image, a subject localized closest to the viewpoint from the display device in the first stereoscopic video image and superimposes the second stereoscopic video image such that the second stereoscopic video image is localized even closer to the viewpoint compared to the subject.

6. The image processing apparatus according to claim 5, wherein the shift amount calculation unit generates a shift amount treating the synthesized first and second stereoscopic video images as a single stereoscopic video image.

7. An image processing method for allowing a processor to:

set, when a stereoscopic video image is provided that includes a left-eye parallax image and a right-eye parallax image having a predetermined parallax distribution and that is observed from a given viewpoint, a virtual viewpoint for observing a subject in the stereoscopic video image from another viewpoint;

generate a left-eye parallax image and a right-eye parallax image for providing a desired parallax distribution when an observation is made from the viewpoint, by shifting an image cut-out position of one of the left-eye parallax image or the right-eye parallax image; and

generate, when a shift amount at the cut-out position changes due to a change made in the position of the virtual viewpoint, a parallax image while changing the shift amount in stages, from a shift amount obtained before the change to a shift amount obtained after the change.

8. A computer program embedded on a non-transitory computer-readable recording medium, comprising:

a module configured to set, when a stereoscopic video image is provided that includes a left-eye parallax image and a right-eye parallax image having a predetermined parallax distribution and that is observed from a given viewpoint, a virtual viewpoint for observing a subject in the stereoscopic video image from another viewpoint; and

a module configured to generate a left-eye parallax image and a right-eye parallax image for providing a desired parallax distribution obtained when an observation is made from the viewpoint, by shifting an image cut-out position of one of the left-eye parallax image or the right-eye parallax image and configured to generate, when a shift amount at the cut-out position changes due to a change made in the position of the virtual viewpoint, a parallax image while changing the shift amount in stages, from a shift amount obtained before the change to a shift amount obtained after the change.