Image processing method and apparatus, digital camera, image processing system and computer readable medium

• US 7,268,803 B1
• Filed: 08/25/2000
• Issued: 09/11/2007
• Est. Priority Date: 08/26/1999
• Status: Expired due to Fees

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Aoki, Shin, Kitaguchi, Takashi, Murata, Norihiko
• Primary Examiner(s): Ho; Tuan
• Assistant Examiner(s): Henn; Timothy J.
• Application Number:
  US09/645,511
• Time in Patent Office: 2,573 Days
• Field of Search: 348/218.1, 348/239, 348/36, 348/37, 348/42, 348/43, 348/44, 348/46, 348 47- 50, 352/275, 352/284, 352/287, 352/293, 352/294
• US Class Current: 348/218.1
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

In image processing method and apparatus of the present invention, an image distortion caused by an oblique image pickup is corrected. A plurality of images of an object on an object plane are input, the plurality of images including at least a pair of partially overlapping first and second images that are taken at two viewpoints, the first and second images sharing a common location on the object plane. A feature point of the first image corresponding to the common location and a matched point of the second image corresponding to the feature point of the first image are determined. A direction of the object plane is calculated based on the feature point and the matched point. A distortion-corrected image on a projection plane, which is parallel to the object plane, is generated by projecting one of the first and second images onto the projection plane based on the direction of the object plane through a perspective projection.

Claims

1. An image processing method for correcting an image distortion caused by an oblique image pickup, comprising:

moving a single image pickup device to at least two different positions so as to obtain a plurality of images of an object on an object plane, the plurality of images including at least two partially overlapping images, the at least two partially overlapping images sharing a common location on the object plane;

determining a feature point of a first image of the at least two partially overlapping images corresponding to the common location and a matched point of a second image of the at least two partially overlapping images corresponding to the feature point of the first image;

calculating a set of parameters based on the feature point and the matched point, the set of parameters including a unit vector indicating a direction of the object plane, an attitude value indicative of the attitude of the image pickup device at a viewpoint when taking the first image relative to that at a viewpoint when taking the second image, and a unit translation vector of the image pickup device from the viewpoint when taking the first image to the viewpoint when taking the second image; and

generating a distortion-corrected image on a projection surface based on the set of parameters calculated in the calculating step, the distortion-corrected image appearing as if the distortion-corrected image was taken from a viewpoint perpendicular to the object plane, by projecting at least one of the at least two partially overlapping images onto the projection surface based on the direction of the object plane through a perspective projection.

2. The image processing method according to claim 1, wherein said calculating the direction of the object plane includes calculating a perspective projection formula for the at least two partially overlapping images based on a relation of the feature point and the matched point.

3. The image processing method according to claim 1, further comprising combining respective images, which are projected onto an image surface of a final one of the plurality of images with respect to each of the plurality of images, so that a composite image is formed on the image surface.

4. The image processing method according to claim 1, further comprising detecting geometric data of an image surface of the first image and geometric data of an image surface of the second image, respectively, and detecting geometric relation information indicating a relation between the geometric data of the first image and the geometric data of the second image, wherein the geometric relation information is used in said determining to determine the feature point and the matched point, and used in said calculating to calculate the direction of the object plane.

5. The image processing method according to claim 1, further comprising correcting an inclination of the distortion-corrected image with respect to a display monitor by rotating the distortion-corrected image on an image surface of the monitor around an axis perpendicular to the image surface.

6. The image processing method according to claim 1, wherein said generating the distortion-corrected image includes adjusting a distance from an optical system center of an image pickup device to the projection surface.

7. The image processing method according to claim 1, wherein the projection surface is being set by considering angles of the overlapping images to the object plane.

8. An image processing system comprising:

an image pickup device and an image processing apparatus coupled to each other, the image pickup device configured to move to at least two different positions so as to obtain at least two images of an object on an object plane and to generate an image signal indicating the image of the object, the image processing apparatus configured to generate a processed image signal by processing the image signal supplied by the image pickup device, wherein the image processing apparatus comprises;

an image memory unit configured to store a plurality of image signals, supplied by the image pickup device, the plurality of image signals including at least two partially overlapping images taken from the image pickup at a plurality of different positions, the at least two partially overlapping images sharing a common location on the object plane;

a correspondence detection unit configured to determine a feature point of a first image of the at least two partially overlapping images corresponding to the common location and a matched point of a second image of the at least two partially overlapping images corresponding to the feature point of the first image; and

a distortion correction unit configured to calculate a set of parameters based on the feature point and the matched point, the set of parameters including a unit vector indicating a direction of the object plane, an attitude value indicative of the attitude of the image pickup device at a viewpoint when taking the first image relative to that at a viewpoint when taking the second image, and a unit translation vector of the image pickup device from the viewpoint when taking the first image to the viewpoint when taking the second image, and to generate a distortion-corrected image on a projection surface, the distortion-corrected image appearing as if the distortion-corrected image was taken from a viewpoint perpendicular to the object plane, by projecting at least one of the at least two partially overlapping images onto the projection surface based on the direction of the object plane through a perspective projection.

9. The image processing system according to claim 8, wherein the image processing apparatus further comprises an image composition means for combining respective images, which are projected onto an image surface of a final one of the plurality of images with respect to each of the plurality of images, so that a composite image is formed on the image surface.

10. The image processing system according to claim 8, wherein the image processing apparatus further comprises a geometric relation detection means for detecting geometric data of an image surface of the first image and geometric data of an image surface of the second image, respectively, and for detecting geometric relation information indicating a relation between the geometric data of the first image and the geometric data of the second image, wherein said correspondence detection means uses the geometric relation information to determine the feature point and the matched point, and said distortion correction means uses the geometric relation information to calculate the direction of the object plane.

11. The image processing system according to claim 8, wherein the image processing apparatus further comprises an inclination correction means for correcting an inclination of the distortion-corrected image with respect to a display monitor by rotating the distortion-corrected image on an image surface of the monitor around an axis perpendicular to the image surface.

12. The image processing system according to claim 8, wherein said distortion correction means includes an image size setting means for adjusting a distance from an optical system center of the image pickup device to the projection surface.

13. The image processing system according to claim 8, wherein the distortion correction unit is configured to further consider angles of the overlapping images to the object plane, to set the projection surface.

14. A digital camera comprising:

an image pickup device and an image processing apparatus coupled to each other, the image pickup device configured to move to at least two different positions so as to obtain at least two images of an object on an object plane and to generate an image signal indicating the image of the object, the image processing apparatus configured to generate a processed image signal by processing the image signal supplied by the image pickup device, wherein the image processing apparatus comprises;

an image memory unit configured to store a plurality of image signals, supplied by the image pickup device, the plurality of image signals including at least two partially overlapping images taken from the image pickup at a plurality of different positions, the at least two partially overlapping images sharing a common location on the object plane;

a correspondence detection unit configured to determine a feature point of a first image of the at least two partially overlapping images corresponding to the common location and a matched point of a second image of the at least two partially overlapping images corresponding to the feature point of the first image; and

a distortion correction unit configured to calculate a set of parameters based on the feature point and the matched point, the set of parameters including a unit vector indicating a direction of the object plane, an attitude value indicative of the attitude of the image pickup device at a viewpoint when taking the first image relative to that at a viewpoint when taking the second image, and a unit translation vector of the image pickup device from the viewpoint when taking the first image to the viewpoint when taking the second image, and to generate a distortion-corrected image on a projection surface, the distortion-corrected image appearing as if the distortion-corrected image was taken from a viewpoint perpendicular to the object plane, by projecting at least one of the at least two partially overlapping images onto the projection surface based on the direction of the object plane through a perspective projection.

15. The digital camera according to claim 14, wherein the distortion correction unit is configured to further consider angles of the overlapping images to the object plane, to set the projection surface.

16. An image processing apparatus for correcting an image distortion caused by an oblique image pickup, comprising:

a single image pickup device configured to move to at least two different positions so as to obtain a plurality of images of an object on an object plane, the plurality of images including at least two partially overlapping images, the at least two partially overlapping images sharing a common location on the object plane;

a correspondence detection unit configured to determine a feature point of a first image of the at least two partially overlapping images corresponding to the common location and a matched point of a second image of the at least two partially overlapping images corresponding to the feature point of the first image; and

a distortion correction unit configured to calculate a set of parameters based on the feature point and the matched point, the set of parameters including a unit vector indicating a direction of the object plane, an attitude value indicative of the attitude of the image pickup device at a viewpoint when taking the first image relative to that at a viewpoint when taking the second image, and a unit translation vector of the image pickup device from the viewpoint when taking the first image to the viewpoint when taking the second image, and to generate a distortion-corrected image on a projection surface, the distortion-corrected image appearing as if the distortion-corrected image was taken from a viewpoint perpendicular to the object plane, by projecting at least one of the at least two partially overlapping images onto the projection surface based on the direction of the object plane through a perspective projection.

17. The image processing apparatus according to claim 16, wherein the distortion correction unit is configured to further consider angles of the overlapping images to the object plane, to set the projection surface.

18. A computer readable medium storing program code instructions, which when executed by a computer cause the computer to perform an image distortion correction processing method, the image processing method comprising:

moving a single image pickup device to at least two different positions so as to obtain a plurality of images of an object on an object plane, the plurality of images including at least two partially overlapping images, the at least two partially overlapping images sharing a common location on the object plane;

determining a feature point of a first image of the at least two partially overlapping images corresponding to the common location and a matched point of a second image of the at least two partially overlapping images corresponding to the feature point of the first image;

calculating a set of parameters based on the feature point and the matched point, the set of parameters including a unit vector indicating a direction of the object plane, an attitude value indicative of the attitude of the image pickup device at a viewpoint when taking the first image relative to that at a viewpoint when taking the second image, and a unit translation vector of the image pickup device from the viewpoint when taking the first image to the viewpoint when taking the second image; and

generating a distortion-corrected image on a projection surface, the distortion-corrected image appearing as if the distortion-corrected image was taken from a viewpoint perpendicular to the object plane, by projecting at least one of the at least two partially overlapping images onto the projection surface based on the direction of the object plane through a perspective projection.

19. The computer readable medium according to claim 18, wherein the method further comprises considering angles of the overlapping images to the object plane, to set the projection surface.