Image signal processor and method for synthesizing super-resolution images from non-linear distorted images

• US 9,609,181 B2
• Filed: 12/20/2011
• Issued: 03/28/2017
• Est. Priority Date: 12/27/2010
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Wang, Qinhe, Tabei, Kenji
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Matt, Marnie
• Application Number:
  US13/992,807
• Publication Number:
  US 20130258048A1
• Time in Patent Office: 1,925 Days
• Field of Search: 348/38
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/407 Control or modification of tonal gradation or of extreme levels, e.g. background level
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

An image signal processing apparatus and image signal processing method obtains a high super-resolution effect for distorted images captured by a specialized optical system, such as a 360-degree camera. A distortion-correction conversion-formula calculation section calculates a distortion correction conversion formula for correcting the nonlinear distortion. A distortion correction and interpolation processing section performs distortion correction and interpolation processing on each cropped region. An alignment conversion formula calculation section calculates an alignment conversion formula for alignment between each cropped image after distortion correction and interpolation processing is completed. A pixel coordinate conversion section rearranges pixel coordinates for each frame image before interpolation processing by using the distortion correction conversion formula and the alignment conversion formula corresponding to the cropped regions. An image synthesis section synthesizes the cropped images.

Claims

1. A video signal processing apparatus, comprising:

a first calculator that receives a plurality of original input image frames, each of which having a different non-linear distortion, and calculates first coordinate conversion formulas, corresponding to each of the original input image frames, correcting non-linear distortion corresponding to a cropped region, capturing an object of each of the frame images;

a distortion correction and interpolation processor that generates a plurality of distortion corrected images by performing distortion correction on the cropped region of each of the original input frame images according to each of the first coordinate conversion formulas and generates a plurality of interpolation processed images by performing interpolation processing on each of the distortion corrected images;

a second calculator that calculates second coordinate conversion formulas correcting the interpolation processed images so that a position of the object captured in each of the interpolation processed images matches each other;

an image coordinate conversion formula calculator that calculates integrated coordinate conversion formulas by integrating the first coordinate conversion formulas and the second coordinate conversion formulas;

a pixel coordinate converter that generates pixel rearranged images by performing rearrangement of pixel coordinates on at least the cropped region of each of the original input frame images using the integrated coordinate conversion formula corresponding to each of the original input image frames; and

a synthesizer that generates a synthesized image by synthesizing the rearranged images.

2. The video signal processing apparatus according to claim 1, further comprising:

an interpolation processor that performs interpolation processing on the synthesized image obtained by the synthesizer.

3. A camera system, comprising:

a camera that photographs an image having non-linear distortion; and

the video signal processing apparatus according to claim 1 that performs processing on the photographed image.

5. The video signal processing apparatus according to claim 1, further comprising:

a region cropper that decides the cropped region corresponding to at least a portion of each of the original input image frames so that the cropped region captures the object.

6. The video signal processing apparatus according to claim 5,

wherein the region cropper performs tracking of the object captured in each of the original input image frames captured by a camera at different times.

7. The video signal processing apparatus according to claim 6,

wherein the region cropper decides a position of the cropped region based on the tracking.

8. The video signal processing apparatus according to claim 5,

wherein the region cropper decides a position of the cropped region in response to a user input.

9. The video signal processing apparatus according to claim 1, further comprising:

a memory that stores parameters for calculating a distortion corresponding to each of positions of the original input image frames.

10. The video signal processing apparatus according to claim 1,

wherein the first calculator calculates the first coordinate conversion formula based on position information of the cropped region and parameters for calculating a distortion corresponding to each of positions of the original input image frame.

11. The video signal processing apparatus according to claim 1, further comprising:

an output that outputs the synthesized image or an image further processed based on the synthesized image.

12. The video signal processing apparatus according to claim 1,

wherein the pixel coordinate convertor performs (1) distortion correction according to the first coordinate conversion formulas, and (2) rearrangement of pixel coordinates according to the second coordinate conversion formulas, on the cropped region corresponding to each of the original input frame images.

4. A video signal processing method, comprising:

receiving a plurality of original input image frames, each having a different non-linear distortion;

calculating first coordinate formulas, corresponding to each of the original input image frames, correcting non-linear distortion corresponding to a cropped region, capturing an object, of each of the frame images;

generating a plurality of distortion corrected images by performing distortion correction on the cropped region of each of the original input frame images according to each of the first coordinate conversion formulas, and generating a plurality of interpolation processed images by performing interpolation processing on each of the distortion corrected images;

calculating second coordinate conversion formulas correcting the interpolation processed images so that pixel positions in the object captured correspond with pixel positions in each of the interpolation processed images;

calculating integrated coordinate conversion formulas by integrating the first coordinate conversion formulas and the second coordinate conversion formulas;

generating pixel rearranged images by performing rearrangement of pixel coordinates on at least the cropped region of each of the original input frame images using the integrated coordinate conversion formula corresponding to each of the original input image frames; and

generating a synthesized image by synthesizing the rearranged images.

13. A video signal processing apparatus, comprising:

a first calculator that receives a plurality of original input image frames, each of which having a different non-linear distortion, and calculates first coordinate conversion formulas, corresponding to each of the original input image frames, correcting non-linear distortion corresponding to a cropped region, capturing an object of each of the frame images;

a distortion correction and interpolation processor that generates a plurality of distortion corrected images by performing distortion correction on the cropped region of each of the original input frame images according to each of the first coordinate conversion formulas and generates a plurality of interpolation processed images by performing interpolation processing on each of the distortion corrected images;

a second calculator that calculates second coordinate conversion formulas correcting the interpolation processed images so that a position of the object captured in each of the interpolation processed images matches each other;

a pixel coordinate converter that generates pixel rearranged images by performing rearrangement of pixel coordinates on at least the cropped region of each of the original input frame images using the first coordinate conversion formula for correcting each non-linear distortion of the original input image frames and the second coordinate conversion formula for correcting the original input image frames so that a position of the object captured in each of the original input image frames matches each other; and

a synthesizer that generates a synthesized image by synthesizing the rearranged images.