Structured light matching of a set of curves from two cameras

• US 10,271,039 B2
• Filed: 02/04/2015
• Issued: 04/23/2019
• Est. Priority Date: 02/05/2014
• Status: Active Grant

• Current Assignee: Creaform, Inc. (Ametek Incorporated)
• Original Assignee: Creaform, Inc. (Ametek Incorporated)
• Inventors: Tubic, Dragan, Soucy, Martin, Gignac, Olivier, Caron, Antoine Thomas
• Primary Examiner(s): Yeh, Euengnan
• Application Number:
  US15/114,563
• Publication Number:
  US 20160350929A1
• Time in Patent Office: 1,539 Days
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20068 Projection on vertical or horizontal image axis
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras

Abstract

A method for matching points between two images of a scene acquired by two cameras, the method including: extracting blobs from the reflection in the images and individually labelling the blobs with a unique identifier; selecting a selected epipolar plane and defining a conjugate epipolar line on each of the images; identifying plausible combinations including a light sheet label of the projected pattern and the unique identifier for a plausible blob selected from the blobs in each of the images; calculating a matching error for each of the plausible combinations; repeating the steps of selecting, identifying and calculating for at least some epipolar planes; determining a most probable combination by computing a figure of merit for the plausible combinations; identifying matching points in the frame from the most probable combination.

Claims

1. A computer-implemented method performed by at least one computer processor for matching points between two images of a scene, the method including:

retrieving two images acquired by a sensor, said sensor comprising at least a first camera, a second camera and a light projector unit, the light projector unit projecting a projected pattern onto said scene, said two images forming a frame captured at a single relative position between the scene and the sensor, said images each containing a reflection of the projected pattern projected by the light projector unit on said scene, wherein the projected pattern comprises at least one sheet of light, each sheet of light being labelled with a corresponding light sheet label;

extracting blobs from said reflection in said images and individually labelling said blobs with a unique identifier;

selecting a selected epipolar plane from a set of epipolar planes and defining a conjugate epipolar line on each of said images, wherein the selected epipolar plane of the set of epipolar planes is defined by a center of projection of each of the first and the second cameras and a corresponding point located on an extracted blob;

identifying plausible combinations, said plausible combinations including a light sheet label of said projected pattern and said unique identifier for a plausible blob selected from said blobs in each of said images, said plausible blob crossing the conjugate epipolar line;

calculating a matching error for each of the plausible combinations, wherein the matching error of a given plausible combination is indicative of a proximity between intersections of each pair of three rays, wherein a first ray originates from the center of projection of the first camera and reaches the plausible blob crossing the conjugate epipolar line, wherein a second ray originates from the center of projection of the second camera and reaches the plausible blob crossing the conjugate epipolar line and the third ray originates from a center of projection of the light projector unit and reaches an intersection between the selected epipolar plane and a corresponding sheet of light of the at least one sheet of light;

repeating said steps of selecting, identifying and calculating for each epipolar plane of said set of epipolar planes;

determining a most probable combination by computing a figure of merit for said plausible combinations using said matching error for each epipolar plane of said set of epipolar planes;

identifying matching points between the two images in said frame using said determined most probable combination and the set of epipolar planes;

generating an output identifying said matching points in said frame.

2. The method as claimed in claim 1, wherein the first camera, the second camera and the light projector unit are calibrated in a common coordinate system.

3. The method as claimed in claim 1, further comprising validating said matching points to discard said matching points if said figure of merit fails to meet a quality of match threshold.

4. The method as claimed in claim 1, further comprising repeating said computer-implemented method for a plurality of frames.

5. The method as claimed in claim 1, further comprising calculating sets of 3D points using said matching points in said frame.

6. The method as claimed in claim 5, further comprising estimating a position of the sensor at each frame to integrate said 3D points in a common coordinate system, wherein said estimating said position of the sensor includes using at least one of targets affixed to said scene, a geometry of said scene and a texture of said scene.

7. The method as claimed in claim 5, further comprising exploiting the images for said calculating said sets of 3D points to obtain segments of 3D curves.

8. A system for matching points between two images of a scene, including:

a sensor including

a light projector unit projecting at least one sheet of light onto said scene thereby providing a projected pattern on said scene, each sheet of light being labeled with a corresponding light sheet label; and

at least a first camera and a second camera, each camera for acquiring a respective image of said two images of said scene, said two images forming a frame captured at a single relative position between the scene and the sensor, said images each containing a reflection of said projected pattern on said scene;

wherein the first camera, the second camera and the light projector unit are calibrated in a common coordinate system; and

at least one computer processor in electronic communications with said sensor for retrieving said two images acquired by said cameras;

extracting blobs from said reflection in said images and individually labelling said blobs with a unique identifier;

selecting a selected epipolar plane from a set of epipolar planes and defining a conjugate epipolar line on each of said images, wherein the selected epipolar plane of the set of epipolar planes is defined by a center of projection of each of the first and the second cameras and a corresponding point located on an extracted blob;

identifying plausible combinations, said plausible combinations including a light sheet label of said projected pattern and said unique identifier for a plausible blob selected from said blobs in each of said images, said plausible blob crossing the conjugate epipolar line;

calculating a matching error for each of the plausible combinations, wherein the matching error of a given plausible combination is indicative of a proximity between intersections of each pair of three rays, wherein a first ray originates from the center of projection of the first camera and reaches the plausible blob crossing the conjugate epipolar line, wherein a second ray originates from the center of projection of the second camera and reaches the plausible blob crossing the conjugate epipolar line and the third ray originates from a center of projection of the light projector unit and reaches an intersection between the selected epipolar plane and a corresponding sheet of light of the at least one sheet of light;

repeating said steps of selecting, identifying and calculating for each epipolar plane of said set of epipolar planes;

determining a most probable combination by computing a figure of merit for said plausible combinations using said matching error for each epipolar plane of said set of epipolar planes; and

identifying matching points between the two images in said frame using the determined most probable combination and the set of epipolar planes;

at least one computer output in electronic communications with said processor for generating an output identifying said matching points in said frame.

9. The system as claimed in claim 8, wherein said light projector unit includes two light projectors each projecting at least one sheet of light onto said scene, said projected pattern being provided by a single one of said two light projectors at each frame and said projected pattern differing for each of said two light projectors.

10. The system as claimed in claim 8, wherein said at least one sheet of light is a plane of light.

11. A computer readable memory having recorded thereon statements and instructions for execution by at least one computer processor, said statements and instructions comprising:

code means for retrieving two images acquired by a sensor, said sensor comprising a first camera, a second camera and a light projector unit, the light projector unit projecting a projected pattern onto said scene, said two images forming a frame captured at a single relative position between the scene and the sensor, said images each containing a reflection of a projected pattern on said scene, wherein the projected pattern comprises at least one sheet of light, each sheet of light being labeled with a corresponding light sheet label;

code means for extracting blobs from said reflection in said images and individually labelling said blobs with a unique identifier;

code means for selecting a selected epipolar plane from a set of epipolar planes and defining a conjugate epipolar line on each of said images;

code means for identifying plausible combinations, said plausible combinations including a light sheet label of said projected pattern and said unique identifier for a plausible blob selected from said blobs in each of said images, said plausible blob crossing the conjugate epipolar line;

code means for calculating a matching error for each of the plausible combinations, wherein the matching error of a given plausible combination is indicative of a proximity between intersections of each pair of three rays, wherein a first ray originates from the center of projection of the first camera and reaches the plausible blob crossing the conjugate epipolar line, wherein a second ray originates from the center of projection of the second camera and reaches the plausible blob crossing the conjugate epipolar line and the third ray originates from a center of projection of the light projector unit and reaches an intersection between the selected epipolar plane and a corresponding sheet of light of the at least one sheet of light;

code means for repeating said steps of selecting, identifying and calculating for each epipolar plane of said set of epipolar planes;

code means for determining a most probable combination by computing a figure of merit for said plausible combinations using said matching error for each epipolar plane of said set of epipolar planes;

code means for identifying matching points between the two images in said frame using the determined most probable combination and the set of epipolar planes;

code means for generating an output identifying said matching points in said frame.

12. A non-transitory computer readable storage medium for storing computer-executable instructions which, when executed, cause a computer to a method for performing matching points between two images of a scene, the method including:

retrieving two images acquired by a sensor, said sensor comprising at least a first camera, a second camera and a light projector unit, the light projector unit projecting a projected pattern onto said scene, said two images forming a frame captured at a single relative position between the scene and the sensor, said images each containing a reflection of the projected pattern projected by the light projector unit on said scene;

wherein the projected pattern comprises at least one sheet of light, each sheet of light being labelled with a corresponding light sheet label;

extracting blobs from said reflection in said images and individually labelling said blobs with a unique identifier;

selecting a selected epipolar plane from a set of epipolar planes and defining a conjugate epipolar line on each of said images;

wherein the selected epipolar plane of the set of epipolar planes is defined by a center of projection of each of the first and the second cameras and a corresponding point located on an extracted blob;

identifying plausible combinations, said plausible combinations including a light sheet label of said projected pattern and said unique identifier for a plausible blob selected from said blobs in each of said images, said plausible blob crossing the conjugate epipolar line;

calculating a matching error for each of the plausible combinations;

wherein the matching error of a given plausible combination is indicative of a proximity between intersections of each pair of three rays, wherein a first ray originates from the center of projection of the first camera and reaches the plausible blob crossing the conjugate epipolar line, wherein a second ray originates from the center of projection of the second camera and reaches the plausible blob crossing the conjugate epipolar line and the third ray originates from a center of projection of the light projector unit and reaches an intersection between the selected epipolar plane and a corresponding sheet of light of the at least one sheet of light;

repeating said steps of selecting, identifying and calculating for each epipolar plane of said set of epipolar planes;

determining a most probable combination by computing a figure of merit for said plausible combinations using said matching error for each epipolar plane of said set of epipolar planes;

identifying matching points between the two images in said frame using said determined most probable combination and the set of epipolar planes;

generating an output identifying said matching points in said frame.