Image processing apparatus and image processing method, storage medium and computer program

• US 7,426,317 B2
• Filed: 11/21/2002
• Issued: 09/16/2008
• Est. Priority Date: 12/07/2001
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nielsen, Frank
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): CHU, RANDOLPH I
• Application Number:
  US10/467,143
• Publication Number:
  US 20040076340A1
• Time in Patent Office: 2,126 Days
• Field of Search: 382/284, 382/285, 345629-641, 348584-601
• US Class Current: 382/284
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Corresponding points are searched based on an error between light beam vectors projected on a projection plane without performing comparison between pixel values at the corresponding points. The necessity for use of cameras having a same camera lens or a same distortion parameter is eliminated, and picked up images of different camera models can be connected to each other. Since original picked up images are pasted directly to an output frame based on errors between light beam vectors without transforming any picked up image once into a pinhole image, deterioration of pixels can be suppressed. Accordingly, picked up images of various cameras which are different in terms of the lens distortion or the camera model from each other can be suitably pasted together.

Claims

1. An image processing apparatus for pasting together a plurality of image frames picked up by different cameras, the apparatus comprising:

mesh camera generating means for generating a mesh camera based on a camera model, the mesh camera establishing in advance a relation between vertex coordinates of polygons on a projection plane and coordinates of light beam vectors corresponding to the vertex coordinates;

light beam vector calculation means for calculating a light beam vector projected at a point of a projection plane by said cameras used for image pickup based on the mesh camera; and

optimization means for minimizing an error of information obtained from light beam vectors of characteristic points at neighboring image frames so as to paste the image frames, and optimizing an evaluation expression based on an output format by varying the light beam vector of the vertex coordinates of polygons of the pasted image frames mapped to a surface of the output format.

2. An image processing apparatus according to claim 1, wherein the mesh camera generating means divides the projection plane of each of said cameras into numerous polygons, and said optimization means determines a transform function for transforming the coordinate values of individual points in the polygons on the projection plane based on the light beam vectors at the vertices of the polygons and uses the transform function to calculate the light beam vectors at the characteristic points.

3. An image processing apparatus according to claim 2, wherein said optimization means calculates the transform function from a characteristic point into a light beam vector in a unit of a polygon and performs an optimization process in a unit of an image frame so that an error of information based on the light beam vectors at the characteristic points may be minimized while fixed transform functions are used as the transform functions.

4. An image processing apparatus according to claim 2, wherein said optimization means calculates the transform function from a characteristic point into a light beam vector in a unit of a polygon and performs an optimization process in a unit of a polygon so that an error of information based on the light beam vectors at the characteristic points may be minimized while variable transform functions are used as the transform functions.

5. An image processing apparatus according to claim 2, wherein said optimization means takes into consideration a light intensity of each pixel, a reliability of each pixel, and a solid angle of each pixel on the surface of the output format to minimize the evaluation expression.

6. An image processing method for pasting together a plurality of image frames picked up by different cameras, the method comprising:

generating a mesh camera based on a camera model, the mesh camera establishing in advance a relation between vertex coordinates of polygons on a projection plane and coordinates of light beam vectors corresponding to the vertex coordinates;

calculating a light beam vector projected at a point of a projection plane by said cameras used for image pickup based on the mesh camera; and

minimizing an error of information obtained from light beam vectors of corresponding characteristic points at neighboring image frames so as to paste the image frames, and optimizing an evaluation expression based on an output format by varying the light beam vector of the vertex coordinates of polygons of the pasted image frames mapped to a surface of the output format.

7. An image processing method according to claim 6, wherein generating the mesh camera further comprises dividing the projection plane of each of said cameras into numerous polygons, and minimizing the error further comprises determining a transform function for transforming the coordinate values of individual points in the polygons on the projection plane based on the light beam vectors at the vertices of the polygons and using the transform function to calculate the light beam vectors at the characteristic points.

8. An image processing method according to claim 7, wherein minimizing the error further comprises calculating the transform function from a characteristic point into a light beam vector in a unit of a polygon and performing an optimization process in a unit of an image frame so that an error of information based on the light beam vectors at the characteristic points may be minimized while fixed transform functions are used as the transform functions.

9. An image processing method according to claim 7, wherein minimizing the error further comprises calculating the transform function from a characteristic point into a light beam vector in a unit of a polygon and performing an optimization process in a unit of a polygon so that an error of information based on the light beam vectors at the characteristic points may be minimized while variable transform functions are used as the transform functions.

10. An image processing method according to claim 7, wherein minimizing the error further comprises taking into consideration a light intensity of each pixel, a reliability of each pixel, and a solid angle of each pixel on the surface of the output format to minimize the evaluation expression.

11. A computer readable medium in or on which computer software which describes a method for pasting together a plurality of image frames picked up by different cameras so that the method may be executed on a computer system is physically stored in a computer-readable form, the method comprising:

generating a mesh camera based on a camera model, the mesh camera establishing in advance a relation between vertex coordinates of polygons on a projection plane and coordinates of light beam vectors corresponding to the vertex coordinates;

calculating a light beam vector projected at a point of a projection plane by said cameras used for image pickup based on the mesh camera; and

minimizing an error of information obtained from light beam vectors of corresponding characteristic points at neighboring image frames so as to paste the image frames, and optimizing an evaluation expression based on an output format by varying the light beam vector of the vertex coordinates of polygons of the pasted image frames mapped to a surface of the output format.