Image processing apparatus and method

• US 8,717,419 B2
• Filed: 04/13/2011
• Issued: 05/06/2014
• Est. Priority Date: 04/16/2010
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Yahata, Kazuhiro, Kotani, Takuya
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): ELAHI, SHAN E
• Application Number:
  US13/086,261
• Publication Number:
  US 20110254927A1
• Time in Patent Office: 1,119 Days
• Field of Search: 348/46
• US Class Current: 348/46
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

An image processing apparatus executes a distortion correction on coordinates of a target pixel in a virtual viewpoint image based on distortion characteristics of a virtual camera and calculates coordinates in the virtual viewpoint image after the distortion correction. The image process apparatus calculates ideal coordinates in a captured image from the coordinates in the virtual viewpoint image after the distortion correction and calculates real coordinates in the captured image from the ideal coordinates in the captured image based on distortion characteristics of an imaging unit. The image process apparatus calculates a pixel value corresponding to the real coordinates from image data of the virtual viewpoint image and corrects the pixel value corresponding to the real coordinates based on ambient light amount decrease characteristics of the imaging unit and ambient light amount decrease characteristics of the virtual camera.

Claims

1. An image processing apparatus generating a virtual viewpoint image from a captured image acquired by an imaging unit having distortion characteristics and vignetting characteristics, based on distortion characteristics and vignetting characteristics of a virtual camera, the image processing apparatus comprising:

a first calculation unit configured to execute a first distortion correction on coordinates of a target pixel in the virtual viewpoint image based on the distortion characteristics of the virtual camera and to calculate coordinates in the virtual viewpoint image after the first distortion correction;

a second calculation unit configured to calculate ideal coordinates that are not distorted and correspond to a space position captured at the target pixel in the captured image from the coordinates of the target pixel in the virtual viewpoint image after the distortion correction;

a third calculation unit configured to execute a second distortion correction on coordinates of the captured image and to calculate real coordinates in the captured image from the ideal coordinates based on the distortion characteristics of the imaging unit;

a fourth calculation unit configured to calculate a pixel value of the target pixel based on the real coordinates and the captured image data; and

a correction unit configured to correct the pixel value based on the vignetting characteristics of the imaging unit and the vignetting characteristics of the virtual camera.

2. The image processing apparatus according to claim 1,

wherein the second calculation unit comprises:

a fifth calculation unit configured to calculate a space position of a captured point on a base level from the coordinates in the virtual viewpoint image after the first distortion correction; and

a sixth calculation unit configured to use position information about the imaging unit and to calculate the ideal coordinates in the captured image from the space position on the base level.

3. The image processing apparatus according to claim 1,

wherein, if the real coordinates represent an area outside the captured image, a previously stored texture image is synthesized in an area of the virtual viewpoint image.

4. The image processing apparatus according to claim 1,

wherein the third calculation unit calculates the real coordinates in the captured image from ideal coordinates of the captured image, based on the distortion characteristics of the imaging unit and an aberration of the imaging unit.

5. A non-transitory computer-readable storage medium storing a program for causing a computer to implement the image processing apparatus according to claim 1.

6. An image processing method for generating a virtual viewpoint image from a captured image acquired by an imaging unit having distortion characteristics and vignetting characteristics, based on distortion characteristics and vignetting characteristics of a virtual camera, the image processing method comprising:

executing a first distortion correction on coordinates of a target pixel in the virtual viewpoint image based on the distortion characteristics of the virtual camera and calculating coordinates in the virtual viewpoint image after the distortion correction;

calculating ideal coordinates that are not distorted and correspond to a space position captured at the target pixel in the captured image from the coordinates of the target pixel in the virtual viewpoint image after the distortion correction;

executing a second distortion correction on coordinates of the captured image and calculating real coordinates in the captured image from the ideal coordinates based on the distortion characteristics of the imaging unit;

calculating a pixel value of the target pixel based on the real coordinates and the captured image data; and

correcting the pixel value based on the vignetting characteristics of the imaging unit and the vignetting characteristics of the virtual camera.