Stereoscopic image processing device, method, recording medium and stereoscopic imaging apparatus

• US 8,111,910 B2
• Filed: 11/25/2009
• Issued: 02/07/2012
• Est. Priority Date: 11/27/2008
• Status: Expired due to Fees

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Tanaka, Koichi
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US13/131,496
• Publication Number:
  US 20110235899A1
• Time in Patent Office: 804 Days
• Field of Search: 382/154, 382/209, 382/276, 345419-420, 348/47, 348/48, 348/50, 348/51
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras

Abstract

An apparatus (10) includes a device for acquiring a plurality of images of an identical subject taken from a plurality of viewpoints; a device for selecting a prescribed image as a reference image, selecting an image other than the reference image as a target image from among the images, and detecting feature points from the reference image and corresponding points from the target image to generate pairs of the feature point and corresponding point, wherein feature of the feature point and the corresponding point in the same pair are substantially identical; a device for estimating geometrical transformation parameters for geometrically-transforming the target image such that y-coordinate values of the feature point and the corresponding point included in the same pair are substantially identical, wherein y-direction is orthogonal to a parallax direction of the viewpoints; and a device for geometrically-transforming the target image based on the parameters.

Claims

1. A stereoscopic image processing apparatus, comprising:

an image acquisition device for acquiring a plurality of images of an identical subject taken from a plurality of viewpoints;

a corresponding point detection device for selecting a prescribed image as a reference image from among the acquired plurality of images, selecting an image other than the reference image as a target image from among the acquired plurality of images, and detecting a plurality of feature points from the reference image and a plurality of corresponding points from the target image to generate a plurality of pairs of the feature point and corresponding point, wherein feature of the feature point and the corresponding point included in the same pair are substantially identical to each other;

a parameter estimation device for estimating geometrical transformation parameters for geometrically transforming the target image such that y coordinate values of the feature point and the corresponding point included in the same pair are substantially identical to each other, wherein y direction is orthogonal to a parallax direction of the plurality of viewpoints; and

an image transformation device for geometrically transforming the target image on the basis of the estimated geometrical transformation parameters.

2. The stereoscopic image processing apparatus according to claim 1, wherein the parameter estimation device estimates at least some parameters in the geometrical transformation parameters based on at least the x and y coordinate values of the corresponding points and the y coordinate values of the feature points.

3. The stereoscopic image processing apparatus according to claim 2, wherein the parameter estimation device calculates parameters other than said some parameters estimated by the parameter estimation device based on said some parameters.

4. The stereoscopic image processing apparatus according to claim 2, wherein

the geometrical transformation parameters are a projective transformation parameters, and

the parameter estimation device selects five or more pairs from among the plurality of pairs of the feature point and the corresponding point, and estimates parameters for determining y coordinate values of projective-transformed corresponding points on the basis of the coordinate values the feature point and the corresponding point of each of the selected pairs.

5. The stereoscopic image processing apparatus according to claim 4, wherein the parameter estimation device calculates the other parameters required to determine the x coordinate values of the projective-transformed corresponding points on the basis of the parameters for determining y coordinate values of projective-transformed corresponding points.

6. The stereoscopic image processing apparatus according to claim 2, wherein

the geometrical transformation parameters are Helmert transformation parameters, and

the parameter estimation device selects three or more pairs from among the detected plurality of pairs of the feature point and the corresponding point, and estimates the Helmert transformation parameters on the basis of the coordinate values of the feature points and the corresponding points of each of the selected pairs.

7. A stereoscopic imaging apparatus, comprising:

a stereoscopic image processing device according to claim 1; and

a plurality of imaging units which are disposed at a plurality of viewpoints along the parallax direction respectively, and take images of the identical subject from their viewpoints, wherein

the image acquisition device acquires the plurality of images taken by the plurality of imaging units, respectively.

8. A stereoscopic image processing method, comprising:

an image acquisition step of acquiring a plurality of images of an identical subject taken from a plurality of viewpoints;

a corresponding point detection step of selecting a prescribed image as a reference image from among the acquired plurality of images, selecting an image other than the reference image as a target image from among the acquired plurality of images, and detecting a plurality of feature points from the reference image and a plurality of corresponding points from the target image to generate a plurality of pairs of the feature point and corresponding point, wherein feature of the feature point and the corresponding point included in the same pair are substantially identical to each other;

a parameter estimation step of estimating geometrical transformation parameters for geometrically transforming the target image such that y coordinate values of the feature points and the corresponding points included in the same pair are substantially identical to each other, wherein y direction is orthogonal to a parallax direction of the plurality of viewpoints; and

an image transformation step of geometrically transforming the target image on the basis of the estimated geometrical transformation parameters.

9. The stereoscopic image processing method according to claim 8, wherein, in the parameter estimation step, at least some parameters in the geometrical transformation parameters are estimated based on at least the x and y coordinate values of the corresponding points and the y coordinate values of the feature points.

10. The stereoscopic image processing method according to claim 9, wherein the parameter estimation step includes:

a first step of randomly selecting a certain number of pairs required to estimate the geometrical transformation parameters for determining y coordinate values of the transformed corresponding points from among N pairs, provided that a total number of plurality of pairs is N;

a second step of calculating the y coordinate values of the transformed corresponding points with respect to each of the N pairs based on the parameter estimated on the basis of the coordinate values of each corresponding point randomly selected by the first step;

a third step of calculating a difference between the y coordinate value of the transformed corresponding point calculated by the second step and the y coordinate value of the feature point for each of the N pairs;

a fourth step of counting a number of pairs of the feature point and the corresponding point whose difference calculated by the third step is less than a predetermined first threshold;

a fifth step of determining a confidence level of the estimated parameter on the basis of a ratio between the counted number of pairs and the N; and

a step of iterating the first to fifth steps until the determined confidence level reaches a predetermined confidence level, or the number of iterations reaches a predetermined number of iterations.

11. The stereoscopic image processing method according to claim 10, wherein the parameter estimation step includes:

a sixth step of calculating the y coordinate value of the transformed corresponding point for each of the N pairs based on the estimated parameter when the determined confidence level reaches the predetermined confidence level or the estimated parameter when the confidence level is highest among the levels at the respective iterations;

a seventh step of calculating a difference between the y coordinate value of the transformed corresponding point calculated by the sixth step and the y coordinate value of the feature point for each of the N pairs;

an eighth step of selecting only the pairs of the feature point and the corresponding point whose difference calculated by the seventh step is less than a predetermined second threshold from the N pairs; and

a ninth step of calculating the plurality of parameters using only the pairs of the feature point and the corresponding point selected in the eighth step.

12. The stereoscopic image processing method according to claim 11, wherein, in the ninth step, the plurality of parameters which minimize a square sum of differences between the y coordinate values of the transformed corresponding points in the plurality of pairs selected by the eighth step and the y coordinate values of the feature points.

13. A non-transitory computer-readable medium comprising a computer program causing a computer to execute a process for a stereoscopic image processing, the process comprising:

acquiring a plurality of images of an identical subject taken from a plurality of viewpoints;

selecting a prescribed image as a reference image from among the acquired plurality of images, selecting an image other than the reference image as a target image from among the acquired plurality of images, and detecting a plurality of feature points from the reference image and a plurality of corresponding points from the target image to generate a plurality of pairs of the feature points and corresponding points, wherein features of the feature points and the corresponding points included in the same pair are substantially identical to the feature points respectively;

estimating geometrical transformation parameters for geometrically transforming the target image such that y coordinate values of the feature points and the corresponding points included in the same pair are substantially identical to each other, wherein y direction is orthogonal to a parallax direction of the plurality of viewpoints; and

geometrically transforming the target image on the basis of the estimated geometrical transformation parameters.