Context and epsilon stereo constrained correspondence matching

• US 8,773,513 B2
• Filed: 07/01/2011
• Issued: 07/08/2014
• Est. Priority Date: 07/01/2011
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Ding, Yuanyuan, Xiao, Jing
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): HUANG, FRANK F
• Application Number:
  US13/175,114
• Publication Number:
  US 20130002828A1
• Time in Patent Office: 1,103 Days
• Field of Search: 348/49, 348/335
• US Class Current: 348/49
• CPC Class Codes: G02B 17/08 Catadioptric systems used in non-imaging applications G02B19/00
• CPC Class Codes: G02B 17/0836 using more than three curved mirrors G02B17/0864, G02B17/0896 take precedence
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G03B 37/06 involving anamorphosis G03B37/02, G03B37/04 take precedence
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A catadioptric camera having a perspective camera and multiple curved mirrors, images the multiple curved mirrors and uses the epsilon constraint to establish a vertical parallax between points in one mirror and their corresponding reflection in another. An ASIFT transform is applied to all the mirror images to establish a collection of corresponding feature points, and edge detection is applied on mirror images to identify edge pixels. A first edge pixel in a first imaged mirror is selected, its nearest feature points are identified, and a rigid transform is applied to them. The rigid transform is fitted to corresponding feature points in a second imaged mirror. The closest edge pixel to the expected location as determined by the fitted rigid transform is identified, and its distance to the vertical parallax is determined. If the distance is not greater than predefined maximum, then it is deemed correlate to the edge pixel in the first imaged mirror.

Claims

1. A method of matching first pixels in a first image of a 3D scene to corresponding second pixels in a second image of the same 3D scene, said method comprising:

(a) obtaining said first image and said second image, wherein said first image is a first multi-perspective image and said second image is a second multi-perspective image;

(b) defining an index of matched pixel pairs;

(c) identifying a plurality of target pixels in said first image to be matched to pixels in said second image;

(d) for each identified target pixel;

(i) determining its potential corresponding pixel in said second image;

(ii) determining a vertical parallax in the second image for the identified target pixel, said vertical parallax being distinct from any horizontal parallax;

(iii) determining the minimum distance from said potential corresponding pixel to said vertical parallax, and

(iv) IF said minimum distance is not greater than a predefined maximum distance, THEN deeming said potential corresponding pixel to be a true match for said identified target pixel and adding the pixel pair comprised of said potential corresponding pixel and said identified target pixel to said index of matched pixel pairs, ELSE deeming said potential corresponding pixel to not be a match for said identified target pixel and omitting said target pixel and said potential corresponding pixel from the index of matched pairs,

wherein in step (c), said plurality of target pixels are edge pixels identified by application of an edge detection algorithm, and

step (c) further includes applying a feature based correspondence matching algorithm to said first and second images to render a collection of feature point pairs, each feature point pair including a first feature point in said first image and a corresponding second feature point in said second image; and

step (i) includes;

(I) identifying N first feature points nearest to a current target pixel, wherein N is a fixed, predefined number;

(II) defining a rigid transform T(.) for the current target pixel using the identified N first feature points;

(III) fitting the rigid transform to the corresponding N second feature points in the second image, identifying an edge pixel in said second image that is nearest to an expected position relative to the N second feature points as determined from the fitted rigid transform, the identified nearest edge pixel T(p) being said potential corresponding pixel.

2. The method of claim 1, wherein:

step (c) includes applying an affine scale-invariant feature transform (ASIFT) to said first and second images to render a collection of feature point pairs, each feature point pair including a first feature point in said first image and a corresponding second feature point in said second image, and the first feature points being said plurality of target pixels;

in step (i), the potential corresponding pixel of each identified target pixel is the second feature point that makes up the feature point pair of the identified target pixel;

in step (iv), said index of matched pairs overrides the correspondence index matches produced by said ASIFT.

3. The method of claim 1, wherein said vertical parallax is a line segment having known position and length.

4. The method of claim 1, wherein said predefined maximum distance is a distance defined by 5 adjacent pixel positions.

5. The method of claim 1, wherein said edge detection algorithm is applied to said first and second images, and in step (III) said identified an edge pixel is selected among the edge detected pixels identified by said edge detection algorithm.

6. The method of claim 1, wherein N is not more than 25.

7. The method of claim 1, wherein said first and second multi-perspective images are catadioptric images.

8. A method of matching edge pixels in a first image of a 3D scene to corresponding edge pixels in a second image of the same 3D scene, said method comprising:

(a) obtaining said first image and said second image, wherein said first image is a first catadioptric image and said second image is a second catadioptric image;

(b) applying an affine scale-invariant feature transform (ASIFT) to said first and second catadioptric images to render a collection of feature point pairs, each feature point pair including a first feature point in said first image and a corresponding second feature point in said second image;

(c) applying edge pixel detection to said first and second catadioptric images;

(d) identifying as a target pixel, in turn, each detected edge pixel in said first image, and for each identified target pixel;

(I) determining its vertical parallax in the second image, said vertical parallax being distinct from any horizontal parallax;

(II) identifying N first feature points nearest to the target pixel, wherein N is a fixed, predefined number;

(III) defining a rigid transform T(.) for the current target pixel using the identified N first feature points;

(IV) fitting the rigid transform to N corresponding second feature points in the second image that correspond to the N first feature points of the first image, identifying a detected edge pixel in said second image that is nearest to the expected position of a corresponding target pixel relative to the N second feature points in the second image as determined from the fitted rigid transform, the identified closest edge pixel T(p) being a potential corresponding pixel;

(V) determining the minimum distance from said potential corresponding pixel to said vertical parallax; and

(VI) IF said minimum distance is not greater than a predefined maximum distance, THEN deeming said potential corresponding pixel to be a true match for said identified target pixel and retaining the edge pixel pair comprised of said potential corresponding pixel and said identified target pixel, ELSE deeming said potential corresponding pixel to not be a match for said identified target pixel, and discarding the edge pixel pair comprised of said potential corresponding pixel and said identified target pixel.

9. The method of claim 8, wherein prior to step (d), step (b) includes:

for each feature point pair,

(ii) determining a vertical parallax in the second image for the first feature point, said vertical parallax being distinct from any horizontal parallax;

(iii) determining the minimum distance from the second feature point to said vertical parallax, and

(iv) IF said minimum distance is not greater than a predefined maximum distance, THEN deeming said first feature point to be a true match for said second feature point and retaining said feature point pair, ELSE deeming said first feature point to not be a match for said second feature point and removing said feature point pair from the collection of feature point pairs.

10. The method of claim 8, wherein said vertical parallax is defined by an epsilon stereo constraint.

11. The method of claim 8, wherein said predefined maximum distance is a distance defined by 5 adjacent pixel positions.

12. The method of claim 8, wherein N is 25.

13. The method of claim 8, wherein in step (a), obtaining first and second catadioptric images includes calibrating a first catadioptric camera and a second catadioptric camera, each catadioptric camera being defined by a curved mirror and a perspective camera;

said first catadioptric image being imaged by said first catadioptric camera and said second catadioptric image being imaged by said second catadioptric camera.

14. The method of claim 13, wherein:

said perspective camera is a pinhole camera;

and said first catadioptric cameras is defined by a first curved mirror, said second catadioptric cameras is defined by second curved mirror, and said first and second catadioptric cameras share the same pinhole camera; and

the image reflected from said first mirror provides a first field of vision, and the image reflected from said second mirror provides a second field of vision that at least partially overlaps said first field of vision.

15. A catadioptric camera system, comprising:

a first plurality of curved mirrors;

a first perspective camera with a field of vision encompassing said first plurality of curved mirrors, said first camera being configured to image said first plurality of curved mirrors simultaneously, each curved mirror providing a respective catadioptric image;

a data processing device for applying the following steps to a given pair of first and second catadioptric images;

(b) applying an affine scale-invariant feature transform (ASIFT) to said first and second catadioptric images to render a collection of feature point pairs, each feature point pair including a first feature point in said first image and a corresponding second feature point in said second image;

(c) applying edge pixel detection to said first and second catadioptric images;

(d) identifying as a target pixel, in turn, each detected edge pixel in said first image, and for each identified target pixel;

(I) determining its vertical parallax in the second image, said vertical parallax being distinct from any horizontal parallax;

(II) identifying N first feature points nearest to the target pixel, wherein N is a fixed, predefined number;

(III) defining a rigid transform T(.) for the current target pixel using the identified N first feature points;

(IV) fitting the rigid transform to the N corresponding second feature points that constitute the feature point pair of each of the N first feature points, identifying a detected edge pixel in said second image that is nearest to the expected position of a corresponding target pixel relative to the N second feature points as determined from the fitted rigid transform, the identified closest edge pixel T(p) being a potential corresponding pixel;

(V) determining the minimum distance from said potential corresponding pixel to said vertical parallax; and

(VI) IF said minimum distance is not greater than a predefined maximum distance,

THEN deeming said potential corresponding pixel to be a true match for said identified target pixel, and retaining the edge pixel pair comprised of said potential corresponding pixel and said identified target pixel,

ELSE deeming said potential corresponding pixel to not be a match for said identified target pixel, and discarding the edge pixel pair comprised of said potential corresponding pixel and said identified target pixel.

16. The catadioptric camera system of claim 15, wherein said predefined maximum distance is a distance defined by 5 adjacent pixel positions, and N is 25.

17. The catadioptric camera system of claim 15, wherein;

said plurality of curved mirrors forms an array of curved mirrors; and

said perspective camera and array of curved mirrors are calibrated to each other.

18. The catadioptric camera system of claim 15, further including;

a second plurality of curved mirrors;

a second perspective camera with a field of vision encompassing said second plurality of curved mirrors, said second camera being configured to image said second plurality of curved mirrors simultaneously, each curved mirror providing a respective catadioptric image;

wherein;

said the field of vision (FOV) of said second perspective camera at least partially overlaps the FOV of said first perspective camera, and the FOV of said first perspective camera at least overlaps the FOV of said second perspective camera; and

in said given pair of first and second catadioptric images, said first catadioptric image is imaged from a mirror in said first plurality of curved mirrors, and said second catadioptric image is imaged from a mirror in said second plurality of curved mirrors.