Systems, methods, and computer program products for runtime adjustment of image warping parameters in a multi-camera system

• US 9,338,439 B2
• Filed: 03/29/2013
• Issued: 05/10/2016
• Est. Priority Date: 04/02/2012
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Grossmann, Etienne G., Gordon, Gaile, Woodfill, John I.
• Primary Examiner(s): Patel, Jayanti K
• Assistant Examiner(s): Aghevli, Reza
• Application Number:
  US13/995,861
• Publication Number:
  US 20140125771A1
• Time in Patent Office: 1,138 Days
• Field of Search: 382/103, 382/154, 348/46, 348/42
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/246 Calibration of cameras

Abstract

Methods, systems, and computer program products for determining, without interrupting a video processing pipeline, whether currently used warping parameters are appropriate for a multi-camera system. New warping parameters may be determined, where these parameters cause the images rectified with these new parameters to have the desired geometric properties. This determination may be made without interrupting the video processing pipeline. Warping parameters may be determined, for example, such that the epipolar lines of the two cameras are horizontal and properly aligned, as defined below, and the metric 3D position of a point observed by both cameras can be determined and is accurate.

Claims

1. A system for image processing, comprising:

an image processing pipeline that comprises

two or more input imaging units rigidly positioned with respect to each other, each with a lens and an imager, wherein for each imaging unit the lens and imager are rigidly positioned with respect to each other and the lens has a focal length that is fixed;

a warping module connected to each input imaging unit and configured to receive raw images from the connected input imaging units and to apply warping parameters to output a rectified image; and

a diagnostic module configured to;

determine a number of pixels in a first rectified image associated with a first of the input imaging units and identified by a dense stereo correspondence algorithm as matching pixels in a second rectified image associated with a second of the input imaging units;

compare the number of pixels identified as matching with a predetermined threshold number of pixels for which correspondence is estimated to be found in response to the warping parameters being correct; and

detect epipolar misalignment in response to the number of pixels identified as matching failing to satisfy the predetermined threshold; and

an adjustment module configured to adjust the warping parameters in response to detection of the epipolar misalignment.

2. The system of claim 1, wherein the adjustment module is configured to adjust the warping parameters so that when adjusted warping parameters are used by the warping module, the image processing pipeline outputs images with the predetermined geometric or optical properties.

3. The system of claim 2, wherein said diagnostic module further comprises:

circuit logic configured to;

compare corresponding features in the first and second rectified images; and

detect the epipolar misalignment by compounding an output of the corresponding feature comparison and the comparison of the number of pixels identified as matching to the predetermined threshold.

4. The system of claim 3, wherein said adjustment module comprises:

circuit logic configured to execute a feature-based matching process on the first and second rectified images; and

circuit logic configured to perform an optimization parameters to minimize a measure of vertical misalignment, the optimization performed over a search space restricted to include projective transformations AR where R is a rotation matrix and A is a projective transformation matrix A associated with a prior warping parameter value.

5. The system of claim 2, wherein said diagnostic module further comprises:

circuit logic configured to identify the locations of projections of 3D points with known geometric properties in a rectified image;

circuit logic configured to use the projections to compute estimated positions of the 3D points; and

circuit logic configured to determine whether the estimated positions agree with the geometric or optical properties.

6. The system of claim 5, wherein said adjustment module comprises:

circuit logic configured to identify, in the first and second rectified images, a set of respective pairs of estimated locations of projections of the 3D points; and

circuit logic configured to determine warping parameters that, when used in a warping process, minimize discrepancy between the estimated locations and actual locations of the 3D points.

7. A method of image processing, comprising:

creating two or more raw images using two or more input imaging units rigidly positioned with respect to each other, each with a lens and an imager, wherein for each imaging unit the lens and imager are rigidly positioned with respect to each other and the lens has a focal length that is fixed;

warping the raw images with warping parameters and outputting respective rectified images; and

determining a number of pixels in a first rectified image associated with a first of the input imaging units and identified by a dense stereo correspondence algorithm as matching pixels in a second rectified image associated with a second of the input imaging units;

comparing the number of pixels identified as matching with a predetermined threshold number of pixels for which correspondence is estimated to be found in response to the warping parameters being correct; and

detecting epipolar misalignment in response to the number of pixels identified as matching failing to satisfy the predetermined threshold; and

adjusting the warping parameters in response to the detecting of the epipolar misalignment.

8. The method of claim 7, further comprising:

adjusting the warping parameters so that when adjusted warping parameters are used by said warping, the rectified images have a predetermined geometric or optical properties.

9. The method of claim 8, wherein said determining further comprises:

comparing corresponding features in the first and second rectified images; and

wherein the detecting further comprises compounding an output of the corresponding feature comparison and the comparison of the number of pixels identified as matching to the predetermined threshold.

10. The method of claim 9, wherein said adjusting of the warping parameters comprises:

executing a feature-based matching process on the first and second rectified images; and

performing an optimization of warping parameters to minimize a measure of vertical misalignment, the optimization performed over a search space restricted to include projective transformations AR where R is a rotation matrix and A is a projective transformation matrix A associated with a prior warping parameter value.

11. The method of claim 8, wherein said determining further comprises:

identifying locations of projections of 3D points with known geometric properties in a rectified image;

using the projections to compute estimated positions of the 3D points; and

determining whether the estimated positions agree with the geometric or optical properties.

12. The method of claim 11, wherein said adjusting of the warping parameters comprises:

identifying, in the first and second rectified images, a set of respective pairs of estimated locations of projections of the 3D points; and

determining warping parameters that, when used in a warping process, minimize discrepancy between the estimated locations and actual locations of the 3D points.

13. A non-transitory computer-readable storage medium having computer program instructions stored therein, that when executed on a processor cause the processor to perform a method comprising:

creating two or more raw images using two or more input imaging units rigidly positioned with respect to each other, each with a lens and an imager, wherein for each imaging unit the lens and imager are rigidly positioned with respect to each other and the lens has a focal length that is fixed;

warping the raw images with warping parameters and outputting respective rectified images; and

determining a number of pixels in a first rectified image associated with a first of the input imaging units and identified by a dense stereo correspondence algorithm as matching pixels in a second rectified image associated with a second of the input imaging units;

comparing the number of pixels identified as matching with a predetermined threshold number of pixels for which correspondence is estimated to be found in response to the warping parameters being correct; and

detecting epipolar misalignment in response to the number of pixels identified as matching failing to satisfy the predetermined threshold; and

adjusting the warping parameters in response to the detecting of the epipolar misalignment.

14. The non-transitory computer-readable storage medium of claim 13, further comprising instructions to cause the processor to perform the method further comprising:

adjusting the warping parameters so that when adjusted warping parameters are used by said warping, the rectified images have a predetermined geometric or optical properties.

15. The non-transitory computer-readable storage medium of claim 14, wherein said instructions cause the processor to perform the method further comprising:

comparing corresponding features in the first and second rectified images; and

wherein the detecting further comprises compounding an output of the corresponding feature comparison and the comparison of the number of pixels identified as matching to the predetermined threshold.

16. The non-transitory computer-readable storage medium of claim 15, wherein said instructions cause the processor to perform the method further comprising:

executing a feature-based matching process on the first and second rectified images; and

performing an optimization of warping parameters to minimize a measure of vertical misalignment, the optimization performed over a search space restricted to include projective transformations AR where R is a rotation matrix and A is a projective transformation matrix A associated with a prior warping parameter value.

17. The non-transitory computer-readable storage medium of claim 14, wherein said instructions cause the processor to perform the method further comprising:

identifying locations of projections of 3D points with known geometric properties in a rectified image;

using the projections to compute estimated positions of the 3D points; and

determining whether the estimated positions agree with the geometric or optical properties.

18. The non-transitory computer-readable storage medium of claim 17, wherein said instructions cause the processor to perform the method further comprising:

identifying, in the first and second rectified images, a set of respective pairs of estimated locations of projections of the 3D points; and

determining warping parameters that, when used in a warping process, minimize discrepancy between the estimated locations and actual locations of the 3D points.