Image taking apparatus, image processing method, and image processing program for connecting into a single image a plurality of images taken by a plurality of imaging units disposed such that viewpoints coincide with each other

• US 7,764,309 B2
• Filed: 12/13/2006
• Issued: 07/27/2010
• Est. Priority Date: 12/14/2005
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Deguchi, Satoshi
• Primary Examiner(s): Villecco; John M
• Application Number:
  US11/610,225
• Publication Number:
  US 20070132863A1
• Time in Patent Office: 1,322 Days
• Field of Search: 348/36, 348/38, 348/47, 348/48, 348/218.1, 382/294, 382/295, 382/284, 382/285
• US Class Current: 348/218.1
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

An image taking apparatus including a plurality of imaging units each including a lens with a variable focal distance and an image sensing device is provided. The plurality of imaging units are disposed such that their viewpoints are substantially coincident with each other and image areas partially overlap each other so as to allow it obtain a wide angle image by connecting a plurality of images obtained by the imaging units. Each imaging unit includes a set of mapping tables each of which represents the correspondence, for a particular focal distance, between a particular coordinate system and a coordinate system in which an image taken by each image sensing device is represented. Each imaging unit further includes a controller adapted to select a mapping table corresponding to the focal distance, and a mapping unit adapted to map the image into the particular coordinate system according to the selected mapping table.

Claims

1. An image taking apparatus comprising a plurality of imaging means each including a lens with a variable focal distance and an image sensing device, the plurality of imaging means being disposed such that their viewpoints are substantially coincident with each other and image areas partially overlap each other so as to allow it obtain a wide angle image by connecting a plurality of images obtained by the respective imaging means,

each imaging means including:

a set of a plurality of mapping tables prepared for various respective values of the focal distance, each mapping table representing the correspondence for one of the various values of the focal distance between a particular coordinate system and a coordinate system in which an image taken by each image sensing device is represented;

a controller adapted to select a mapping table corresponding to a focal distance used to take images by the image sensing devices; and

a mapping unit adapted to map each image into the particular coordinate system in accordance with the correspondence described in the mapping table selected by the controller.

2. The image taking apparatus according to claim 1, wherein the mapping tables in the set of mapping tables are prepared for respective different values off the focal distance, each mapping table representing, for one of various different focal distances, the correspondence in mapping from an image taken by each image sensing device to a non-planar image in a three-dimensional coordinate system or further to a two-dimensional plane image.

3. The image taking apparatus according to claim 1, wherein the image taken by each image sensing device includes an image with a standard image size as measured on a display that is a unit of a wide-angle image composed of a plurality of units and further includes an image in an overlapping area in which the image overlaps an image with the standard image size as measured on the display taken by adjacent imaging means, the size as measured on the display of the image in the overlapping area being maintained fixed regardless of the focal distance.

4. The image taking apparatus according to claim 1, wherein in first and second imaging means adjacent to each other,

an image taken by the first imaging means extends, beyond a connection boundary with an image with the standard image size as measured on a display taken by the second imaging means, into a first overlapping area in which the image taken by the first imaging means overlaps the image taken by the second imaging means, and

the image taken by the second imaging means extends, beyond the connection boundary with the image with the standard image size as measured on the display taken by the first imaging means, into a second overlapping area in which the image taken by the second imaging means overlaps the image taken by the first imaging means, the first overlapping area having the same size as the size of the second overlapping area.

5. The image taking apparatus according to claim 3, wherein each imaging means acquires, from adjacent imaging means, a result of an evaluation of an overlapping part between the image with the standard image size as measured on the display taken by the current imaging means and the image taken by the adjacent imaging means, and corrects a difference in color of the overlapping part with respect to the color of the image taken by the adjacent imaging means on the basis of the acquired evaluation result.

6. The image taking apparatus according to claim 5, wherein each imaging means further includes

an aperture, and

an overlapping area evaluation unit adapted to evaluate an overlapping part between an image with the standard image size as measured on the display taken by the present imaging means and an image taken by adjacent imaging means,

and wherein the controller of each imaging means adjusts the aperture value of the imaging means and/or the image quality of the image taken by the image sensing device so that the difference in color is minimized in accordance with the evaluation result supplied from the overlapping area evaluation unit of the adjacent imaging means.

7. The image taking apparatus according to claim 3, wherein each imaging means sends, to the other imaging means, pixel data of the image in the overlapping area in which the image with the standard image size as measured on the display taken by the imaging means overlaps the image taken by the adjacent imaging means, each imaging means receives pixel data sent from the other imaging means, and each imaging means performs a blending process on the image in the overlapping area in accordance with the received pixel data.

8. The image taking apparatus according to claim 7, wherein

each imaging means further includes an overlapping area processing unit adapted to perform a blending process on the overlapping part between the image with the standard image size as measured on the display taken by the imaging means and the image taken by the adjacent imaging means, and

the overlapping area processing unit acquires the pixel data from the adjacent imaging means, performs the blending process on the overlapping part, and outputs the image with the standard image size as measured on the display including the overlapping part subjected to the blending process.

9. An image processing method of obtaining a wide-angle image by connecting images taken by a plurality of imaging means each including a lens with a variable focal distance and an image sensing device, the plurality of imaging means being disposed such that their viewpoints are substantially coincident with each other and image areas partially overlap each other, comprising:

selecting a mapping table from a set of a plurality of mapping tables prepared for various respective values of the focal distance used in an image taking operation, each mapping table representing the correspondence for one of the various values of the focal distance between a particular coordinate system and a coordinate system in which an image taken by each image sensing device is represented; and

mapping each image taken by each image sensing device into the particular coordinate system in accordance with the correspondence described in the selected mapping table, so that the image mapped into the particular coordinate system in accordance with the selected mapping table and the image mapped into the particular coordinate system by an adjacent imaging means overlap each other in an overlapping area which is maintained fixed regardless of the focal distance.

10. A computer readable medium storing a computer program for obtaining a wide-angle image by connecting images taken by a plurality of imaging means each including a lens with a variable focal distance and an image sensing device, the plurality of imaging means being disposed such that their viewpoints are substantially coincident with each other and image areas partially overlap each other, the computer program structured to cause a computer to:

select a mapping table from a set of a plurality of mapping tables prepared for various respective values of the focal distance used in an image taking operation, each mapping table representing the correspondence for one of the various values of the focal distance between a particular coordinate system and a coordinate system in which an image taken by each image sensing device is represented; and

map each image taken by each image sensing device into the particular coordinate system in accordance with the correspondence described in the selected mapping table so that the image mapped into the particular coordinate system in accordance with the selected mapping table overlaps, in an overlapping area, with an image taken by adjacent imaging means and mapped into the particular coordinate system, the overlapping area being fixed regardless of the focal distance.

11. An image taking apparatus comprising a plurality of imaging units each including a lens with a variable focal distance and an image sensing device, the plurality of imaging units being disposed such that their viewpoints are substantially coincident with each other and image areas partially overlap each other so as to allow it obtain a wide angle image by connecting a plurality of images obtained by the respective imaging means,

each imaging unit including:

a set of a plurality of mapping tables prepared for various respective values of the focal distance, each mapping table representing the correspondence for one of the various values of the focal distance between a particular coordinate system and a coordinate system in which an image taken by each image sensing device is represented;

a controller adapted to select a mapping table corresponding to a focal distance used to take images by the image sensing devices; and

a mapping unit adapted to map each image into the particular coordinate system in accordance with the correspondence described in the mapping table selected by the controller.