Image processing apparatus and image processing method, storage medium and computer program

• US 20040076340A1
• Filed: 08/04/2003
• Published: 04/22/2004
• Est. Priority Date: 12/07/2001
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nielsen, Frank
• Granted Patent: US 7,426,317 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Corresponding points are searched based on an error between light beam vectors projected on a projection plane without performing comparison between pixel values at the corresponding points. The necessity for use of cameras having a same camera lens or a same distortion parameter is eliminated, and picked up images of different camera models can be connected to each other. Since original picked up images are pasted directly to an output frame based on errors between light beam vectors without transforming any picked up image once into a pinhole image, deterioration of pixels can be suppressed. Accordingly, picked up images of various cameras which are different in terms of the lens distortion or the camera model from each other can be suitably pasted together.

Claims

1. An image processing apparatus for pasting together a plurality of image frames picked up by different cameras comprising:

light beam vector calculation means for calculating a light beam vector projected at each point on a projection plane by said cameras used for image pickup based on a camera model; and

optimization means for optimizing pasting of images so that an error of information based on light beam vectors at each corresponding characteristic points between neighboring image frames may be minimized.

2. An image processing apparatus according to claim 1, wherein

said light beam vector calculation means divides the projection plane of each of said cameras into numerous polygons and produces a mesh camera which describes a corresponding relationship between coordinates of vertices of the polygons on the projection planes and light beam vectors corresponding to the coordinates in advance, and

said optimization means determines a transform function for transforming the coordinate values of individual points in the polygons on the projection plane based on the light beam vectors at the vertices of the polygons and uses the transform function to calculate the light beam vectors at the characteristic points.

3. An image processing apparatus according to claim 2, wherein said optimization means calculates the transform function from a characteristic point into a light beam vector in a unit of a polygon and performs an optimization process in a unit of an image frame so that an error of information based on the light beam vectors at the characteristic points may be minimized while fixed transform functions are used as the transform functions.

4. An image processing apparatus according to claim 2, wherein said optimization means calculates the transform function from a characteristic point into a light beam vector in a unit of a polygon and performs an optimization process in a unit of a polygon so that an error of information based on the light beam vectors at the characteristic points may be minimized while variable transform functions are used as the transform functions.

5. An image processing apparatus according to claim 2, wherein said optimization means takes a light intensity of each pixel, a reliability of each pixel and a solid angle of each pixel on an output format face into consideration to perform an optimization process in a unit of a pixel so that an error of information based on the light beam vectors at the pixels may be minimized.

6. An image processing method for pasting together a plurality of image frames picked up by different cameras comprising:

a light beam vector calculation step of calculating a light beam vector projected at each point on a projection plane by said cameras used for image pickup based on a camera model; and

an optimization step of optimizing pasting of images so that an error of information based on light beam vectors at each corresponding characteristic points between neighboring image frames may be minimized.

7. An image processing method according to claim 6, wherein

the light beam vector calculation step divides the projection plane of each of said cameras into numerous polygons and produces a mesh camera which describes a corresponding relationship between coordinates of vertices of the polygons on the projection planes and light beam vectors corresponding to the coordinates in advance, and

the optimization step determines a transform function for transforming the coordinate values of individual points in the polygons on the projection plane based on the light beam vectors at the vertices of the polygons and uses the transform function to calculate the light beam vectors at the characteristic points.

8. An image processing method according to claim 7, wherein the optimization step calculates the transform function from a characteristic point into a light beam vector in a unit of a polygon and performs an optimization process in a unit of an image frame so that an error of information based on the light beam vectors at the characteristic points may be minimized while fixed transform functions are used as the transform functions.

9. An image processing method according to claim 7, wherein the optimization step calculates the transform function from a characteristic point into a light beam vector in a unit of a polygon and performs an optimization process in a unit of a polygon so that an error of information based on the light beam vectors at the characteristic points may be minimized while variable transform functions are used as the transform functions.

10. An image processing method according to claim 7, wherein the optimization step takes a light intensity of each pixel, a reliability of each pixel and a solid angle of each pixel on an output format face into consideration to perform an optimization process in a unit of a pixel so that an error of information based on the light beam vectors at the pixels may be minimized.

11. A storage medium in or on which computer software which describes a process for pasting together a plurality of image frames picked up by different cameras so that the process may be executed on a computer system is physically stored in a computer-readable form, said computer software comprising:

a light beam vector calculation step of calculating a light beam vector projected at each point on a projection plane by said cameras used for image pickup based on a camera model; and

an optimization step of optimizing pasting of images so that an error of information based on light beam vectors at each corresponding characteristic points between neighboring image frames may be minimized.

12. A computer program which describes a process for pasting together a plurality of image frames picked up by different cameras in a computer-readable form so that the process may be executed on a computer system, said computer program comprising:

a light beam vector calculation step of calculating a light beam vector projected at each point on a projection plane by said cameras used for image pickup based on a camera model; and

an optimization step of optimizing pasting of images so that an error of information based on light beam vectors at each corresponding characteristic points between neighboring image frames may be minimized.

13. An image processing apparatus for interpolating a pixel value at a point surrounded by a plurality of vertices having known pixel values in an image frame picked up by a camera comprising:

light beam vector calculation means for calculating light beam vectors projected at an interpolation object point and vertices surrounding the interpolation object point on a projection plane by a camera used for image pickup; and

pixel value interpolation means for interpolating a pixel value of the interpolation object point using the light beam vectors at the vertices.

14. An image processing method for interpolating a pixel value at a point surrounded by a plurality of vertices having known pixel values in an image frame picked up by a camera comprising:

a light beam vector calculation step of calculating light beam vectors projected at an interpolation object point and vertices surrounding the interpolation object point on a projection plane by a camera used for image pickup; and

a pixel value interpolation step of interpolating a pixel value of the interpolation object point using the light beam vectors at the vertices.

15. A storage medium in or on which computer software which describes a process for interpolating a pixel value at a point surrounded by a plurality of vertices having known pixel values in an image frame picked up by a camera is physically stored in a computer-readable form, said computer software comprising:

a light beam vector calculation step of calculating light beam vectors projected at an interpolation object point and vertices surrounding the interpolation object point on a projection plane by a camera used for image pickup; and

a pixel value interpolation step of interpolating a pixel value of the interpolation object point using the light beam vectors at the vertices.

16. A computer program which describes a process for interpolating a pixel value at a point surrounded by a plurality of vertices having known pixel values in an image frame picked up by a camera in a computer-readable form so that the process may be executed on a computer system, said computer program comprising:

a light beam vector calculation step of calculating light beam vectors projected at an interpolation object point and vertices surrounding the interpolation object point on a projection plane by a camera used for image pickup; and

a pixel value interpolation step of interpolating a pixel value of the interpolation object point using the light beam vectors at the vertices.