IMAGE PROCESSING DEVICE AND METHOD, RECORDING MEDIUM, AND PROGRAM

• US 20130135448A1
• Filed: 11/12/2012
• Published: 05/30/2013
• Est. Priority Date: 11/28/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nagumo, Takefumi, Murayama, Jun, Yamazaki, Toshio, Tamayama, Ken
• Granted Patent: US 9,392,257 B2
• Time in Patent Office: Days
• US Class Current: 348/49
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/225 using parallax barriers
• CPC Class Codes: H04N 13/229 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Abstract

An image processing device includes a viewpoint separating unit configured to separate multi-viewpoint image data, including images of multiple viewpoints and representing intensity distribution of light and the direction of travel of light according to positions and pixel values of pixels, into a plurality of single-viewpoint image data for each of the individual viewpoints; and a parallax control unit configured to control amount of parallax between the plurality of single-viewpoint image data obtained by separation into individual viewpoints by the viewpoint separating unit.

Claims

1. An image processing device comprising:

a viewpoint separating unit configured to separate multi-viewpoint image data, including images of multiple viewpoints and representing intensity distribution of light and the direction of travel of light according to positions and pixel values of pixels, into a plurality of single-viewpoint image data for each of the individual viewpoints; and

a parallax control unit configured to control amount of parallax between the plurality of single-viewpoint image data obtained by separation into individual viewpoints by the viewpoint separating unit.

2. The image processing device according to claim 1, wherein the parallax control unit controls the amount of parallax by adding or subtracting a derivative signal, obtained by performing derivation of the single-viewpoint image data, to or from each single-viewpoint image data.

3. The image processing device according to claim 2, wherein the parallax control unit

detects disparity, which indicates amount of parallax of an object to be controlled, and

based on the detected disparity, corrects the derivative signal and controls the amount of parallax by adding or subtracting the derivative signal following correction to or from each single-viewpoint image data.

4. The image processing device according to claim 1, further comprising:

a super-resolution processing unit configured to perform super-resolution processing where image resolution is raised to high resolution for each of the plurality of single-viewpoint image data obtained by separation into individual viewpoints by the viewpoint separating unit;

wherein the parallax control unit controls amount of parallax between the plurality of single-viewpoint image data of which the resolution has been raised to high resolution by the super-resolution processing unit.

5. The image processing device according to claim 4, wherein the super-resolution processing unit detects disparity which indicates amount of parallax, between each viewpoint, and performs super-resolution processing of the single-viewpoint image data, using the detected disparity.

6. The image processing device according to claim 5, wherein the parallax control unit controls the amount of parallax using the disparity detected by the super-resolution processing unit.

7. The image processing device according to claim 4, further comprising:

a noise reduction processing unit configured to perform noise reduction processing to reduce noise on each of the plurality of single-viewpoint image data, of which the resolution has been raised to high resolution by the super-resolution processing unit;

wherein the parallax control unit controls the amount of parallax between the plurality of single-viewpoint image data, of which noise has been reduced by the noise reduction processing unit.

8. The image processing device according to claim 7, wherein the noise reduction processing unit

performs motion detection with images before and after processing,

performs motion compensation on an image after processing using the detected motion vector, and

calculates the arithmetic mean of

the image following processing that has been subjected to motion compensation, and

the image before processing.

9. The image processing device according to claim 8, wherein the noise reduction processing unit uses the detected motion vector to perform the noise reduction processing on each of the plurality of single-viewpoint image data.

10. The image processing device according to claim 1, further comprising:

a super-resolution and noise reduction processing unit configured to perform, on each of the plurality of single-viewpoint image data obtained by separation into individual viewpoints by the viewpoint separating unit,

super-resolution processing to raise the resolution of images to high resolution, and

noise reduction processing to reduce noise;

wherein the parallax control unit controls the amount of parallax between the plurality of single-viewpoint image data, of which resolution has been raised to high resolution and noise has been reduced by the super-resolution and noise reduction processing unit.

11. The image processing device according to claim 10, wherein the super-resolution and noise reduction processing unit

performs the noise reduction processing on each of the super-resolution processing results of the multiple single-viewpoint image data, and

performs the super-resolution processing using the noise reduction processing results.

12. The image processing device according to claim 10, further comprising:

an initial image generating unit configured to generate an initial image using an image of a frame of interest which is to be processed, and an image of a past frame processed prior to the frame of interest;

wherein the super-resolution and noise reduction processing unit performs the super-resolution processing using the initial image generated by the initial image generating unit.

13. The image processing device according to claim 12, wherein the initial image generating unit

detects motion between the image of the frame of interest and the image of the past frame,

performs motion compensation of the image of the past frame using the detected motion vector, and

generates the initial image using the image of the past frame subjected to motion compensation and the image of the frame of interest.

14. The image processing device according to claim 12, wherein the initial image generating unit generates the initial image using an image of a viewpoint of interest to be processed in the frame of interest, an image of the viewpoint of interest in the past frame, and an image of another viewpoint which is not the viewpoint of interest in the past frame.

15. The image processing device according to claim 1, further comprising:

a super-resolution processing unit configured to perform super-resolution processing of the single-viewpoint image data to raise the resolution of an image to high resolution;

a noise reduction processing unit configured to perform noise reduction processing to reduce noise of the single-viewpoint image data of which the resolution has been raised to high resolution by the super-resolution processing unit; and

an image generating unit configured to generate the plurality of single-viewpoint image data, using the single-viewpoint image data of which the noise has been reduced by the noise reduction processing unit;

wherein the parallax control unit controls the amount of parallax between the plurality of single-viewpoint image data generated by the image generating unit.

16. The image processing device according to claim 15, further comprising:

an initial image generating unit configured to generate an initial image, using an image of a frame of interest to be processed, and an image of a past frame processed prior to the frame of interest;

wherein the super-resolution processing unit performs the super-resolution processing using the initial image generated by the initial image generating unit.

17. The image processing device according to claim 1, further comprising:

a first storage unit configured to store the plurality of single-viewpoint image data obtained by separation into individual viewpoints by the viewpoint separating unit;

a viewpoint sequential readout unit configured to read out the plurality of single-viewpoint image data stored in the first storage unit, one viewpoint at a time;

a super-resolution processing unit configured to perform super-resolution processing to raise the resolution of the single-viewpoint image data read out from the viewpoint sequential readout unit to high resolution;

a noise reduction processing unit configured to perform noise reduction processing to reduce the noise of the single-viewpoint image data of which the resolution has been raised to high resolution by the super-resolution processing unit; and

a second storage unit configured to store the single-viewpoint image data of which noise has been reduced by the noise reduction processing unit;

wherein the parallax control unit controls the amount of parallax between the plurality of single-viewpoint image data stored in the second storage unit.

18. An image processing method of an image processing device, the method comprising:

a viewpoint separating unit separating multi-viewpoint image data, including images of multiple viewpoints and representing intensity distribution of light and the direction of travel of light according to positions and pixel values of pixels, into a plurality of single-viewpoint image data for each of the individual viewpoints; and

a parallax control unit controlling amount of parallax between the plurality of single-viewpoint image data obtained by separation into individual viewpoints.

19. A computer-readable recording medium in which is recorded a program, to cause a computer to function as:

a viewpoint separating unit configured to separate multi-viewpoint image data, including images of multiple viewpoints and representing intensity distribution of light and the direction of travel of light according to positions and pixel values of pixels, into a plurality of single-viewpoint image data for each of the individual viewpoints; and

a parallax control unit configured to control amount of parallax between the plurality of single-viewpoint image data obtained by separation into individual viewpoints by the viewpoint separating unit.

20. A program to cause a computer to function as:

a viewpoint separating unit configured to separate multi-viewpoint image data, including images of multiple viewpoints and representing intensity distribution of light and the direction of travel of light according to positions and pixel values of pixels, into a plurality of single-viewpoint image data for each of the individual viewpoints; and

a parallax control unit configured to control amount of parallax between the plurality of single-viewpoint image data obtained by separation into individual viewpoints by the viewpoint separating unit.