System and method for automatic stereo measurement of a point of interest in a scene

• US 8,107,722 B2
• Filed: 08/09/2010
• Issued: 01/31/2012
• Est. Priority Date: 04/02/2004
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Oldroyd, Lawrence A.
• Primary Examiner(s): Rahmjoo, Mike
• Application Number:
  US12/852,776
• Publication Number:
  US 20110007948A1
• Time in Patent Office: 540 Days
• Field of Search: 382/154, 382/106, 382/283, 382/285, 382/286, 382/305
• US Class Current: 382/154
• CPC Class Codes: G01C 11/00 Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/13 Satellite images

Abstract

A system for performing a three dimensional stereo measurement that uses a sensor for obtaining a sensor image of a scene, and a database for providing first and second reference images of the scene that are a stereo pair of images. At least one processing system is responsive to an output of the sensor and in communication with the database. The processing system registers the sensor image with the first reference image, and also selects a point of interest from one of the sensor image and the first reference image. The processing system performs a stereo point measurement from the selected point of interest and the first reference image to determine a point in the second reference image that represents a stereo mate of the selected point in the first reference image.

Claims

1. A system for performing three dimensional, stereo measurement, the system comprising:

a sensor for obtaining a sensor image of a scene;

a database for providing first and second reference images of the scene that are a stereo pair of images;

at least one processing system responsive to an output of the sensor and in communication with the database, and operable to;

register the sensor image with the first reference image;

select a point of interest from the sensor image;

derive a point of interest in the first reference image from the selected point of interest in the sensor image and obtain a registration between the sensor image and the first reference image;

perform a stereo point measurement of the selected point of interest in the first reference image to determine a point in the second reference image that represents a stereo mate of the selected point of interest in the first reference image; and

further analyze the selected point of interest and the point in the second reference image to select an adjacent three dimensional scene coordinate which more accurately corresponds with the selected point of interest in the sensor image;

wherein said adjacent three dimensional stereo measurement of the selected point of interest in the sensor image is in the three dimensional scene coordinate.

2. The system of claim 1, further comprising:

a display for displaying both of said first and second reference images with said points marked on each of said reference images to enable an individual to visually confirm an accuracy of said stereo point measurement operation.

3. The system of claim 1, wherein:

the first and second reference images are each comprised of a plurality of pixels; and

wherein the selected point of interest in the first reference image is defined by a first pixel, and the point in the second reference image comprises a second pixel that is the stereo mate in the second reference image of the first pixel.

4. The system of claim 3, wherein:

the selected point of interest in the first reference image is used by the processing system to obtain coordinate data that corresponds to the pixel of the selected point of interest in the first reference image; and

the point in the second reference image is used by the at least one processing system to obtain coordinate data therefor.

5. The system of claim 3, wherein the at least one processing system is configured to perform a stereo point measurement.

6. The system of claim 5, wherein the stereo point measurement is accomplished by:

obtaining a first sub-image of the first reference image in an area circumscribing the selected point of interest in the first reference image;

obtaining a second sub-image of the second reference image in an area circumscribing the point in the second reference image, and where the second sub-image is larger than the first sub-image;

overlaying at least a portion of the first sub-image over the second sub-image;

matching the two sub-images to determine a first offset of the first sub-image within the second image, and a second offset of the second sub-image relative to the second reference image;

using the first and second offsets to translate coordinates of the first pixel to stereo matching coordinates of the second pixel, and

wherein the stereo matching coordinates of the second pixel represent a specific point on the second reference image that corresponds to a location of the first pixel on the first reference image.

7. The system of claim 6, wherein the at least one processing system is configured to obtain the stereo matching coordinates of the second pixel by:

determining column and row offset values that represent the first offset of the first sub-image within the second sub-image;

determining column and row offset values that represent the second offset of the second sub-image relative to the second reference image;

adding column and row coordinates of the first pixel in the first sub-image to the column and row offset values of the first sub-image offset to produce a modified first sub-image offset having associated column and row offset values; and

adding the modified first sub-image column and row offset values to the column and row offset values of the second sub-image, to produce coordinates of the second pixel.

8. The system of claim 3, wherein the at least one processing system is configured to:

define a three dimensional grid of points with the three dimensional measurement of the selected point of interest at a volumetric center of the three dimensional grid of points;

process each one of said three dimensional grid of points using parametric data for said first and second reference images to identify a subset of the grid points that process to derive a location of the first pixel within the first reference image;

use the subset of said grid points to derive a location of a corresponding, stereo matching pixel in the second reference image;

determine a distance between the derived location of said corresponding, stereo matching pixel and the second pixel, within the second reference image; and

repeat the processing of each one of the three dimensional grid of points and using the subset of the grid points to derive a location of a specific three dimensional grid point that, when processed using the parametric data, yields a location for a specific matching pixel that has a minimum distance from the second pixel, and denoting a three dimensional grid point of the specific matching pixel as an optimum three dimensional stereo coordinate location for the selected point of interest in the sensor image.

9. The system of claim 1, wherein:

the first reference image comprises a plurality of pixels;

the second reference image comprises a plurality of pixels;

the first reference image is provided with a corresponding elevation model; and

three dimensional scene coordinates are associated with each said pixel in the first reference image.

10. A system for performing three dimensional, stereo measurement, the system comprising:

a sensor for obtaining a sensor image of a scene;

a database for providing first and second reference images of the scene that are a stereo pair of images;

at least one processing system responsive to an output of the sensor and in communication with the database, and operable to;

register the sensor image with the first reference image;

select a point of interest in the sensor image;

derive a point of interest in the first reference image from the selected point of interest in the sensor image and a registration between the sensor image and the first reference image;

perform a stereo point measurement from the selected point of interest in the first reference image to determine a point in the second reference image that represents a stereo mate of the selected point of interest in the first reference image;

using the stereo point measurement to select a three dimensional scene coordinate which more accurately corresponds with the selected point of interest in the sensor image; and

a display for displaying both of said first and second reference images with said points displayed on each of said reference images to enable an individual to visually confirm an accuracy of said stereo point measurement operation.

11. The system of claim 10, wherein:

the first and second reference images are each comprised of a plurality of pixels; and

wherein the selected point of interest in the first reference image is defined by a first pixel, and the point in the second reference image comprises a second pixel that is the stereo mate in the second reference image of the first pixel.

12. The system of claim 11, wherein:

the selected point of interest of the first reference image is used by the at least one processing system to obtain coordinate data that corresponds to a pixel of the selected point of interest in the first reference image; and

the point in the second reference image is used by the at least one processing system to obtain coordinate data therefor.

13. The system of claim 10, wherein the at least one processing system is configured to perform a stereo point measurement by:

obtaining a first sub-image of the first reference image in an area circumscribing the selected point of interest in the first reference image;

obtaining a second sub-image of the second reference image in an area circumscribing the point in the second reference image, and where the second sub-image is larger than the first sub-image;

overlaying at least a portion of the first sub-image over the second sub-image;

analyzing the two sub-images to determine a first offset of the first sub-image within the second image, and a second offset of the second sub-image relative to the second reference image;

using the first and second offsets to translate coordinates of the first pixel to stereo matching coordinates of the second pixel, and

wherein the stereo matching coordinates of the second pixel represent a specific point on the second reference image that corresponds to a location of the first pixel on the first reference image.

14. The system of claim 13, wherein the at least one processing system is configured to obtain the stereo matching coordinates of the second pixel by:

determining column and row offset values that represent the first offset of the first sub-image within the second sub-image;

determining column and row offset values that represent the second offset of the second sub-image relative to the second reference image;

adding column and row coordinates of the first pixel in the first sub-image to the column and row offset values of the first sub-image offset to produce a modified first sub-image offset having associated column and row offset values; and

adding the modified first sub-image column and row offset values to the column and row offset values of the second sub-image, to produce the coordinates of the second pixel.

15. The system of claim 11, wherein the at least one processing system is configured to:

define a three dimensional grid of points with the three dimensional measurement of the selected point of interest at a volumetric center of the three dimensional grid of points;

process each one of said three dimensional grid of points using parametric data for said first and second reference images to identify a subset of the three dimensional grid of points that process to derive the location of the first pixel within the first reference image;

use the subset of said three dimensional grid of points to derive a location of a corresponding, stereo matching pixel in the second reference image;

determine a distance between the derived location of said corresponding, stereo matching pixel and the second pixel, within the second reference image; and

repeat the processing of each one of the three dimensional grid of points and using the subset of the grid points to derive a location of a corresponding, stereo matching pair, to determine a specific three dimensional grid point that, when processed using the parametric data, yields a location for a specific matching pixel that has a minimum distance from the second pixel, and denoting a three dimensional grid point of the specific matching pixel as an optimum three dimensional stereo coordinate location for the selected point of interest in the sensor image.

16. The system of claim 10, wherein:

the first reference image comprises a plurality of pixels;

the second reference image comprises a plurality of pixels;

the first reference image is provided with a corresponding elevation model; and

coordinates are associated with each said pixel in the first reference image.

17. A system for performing three dimensional, stereo measurement, the system comprising:

a sensor for obtaining a sensor image of a scene;

a database for providing first and second reference images of the scene that are a stereo pair of images, the first and second reference images each being comprised of a plurality of pixels, and the first reference image further being provided with a corresponding elevation model, and wherein coordinates are associated with each said pixel in the first reference image;

at least one processing system responsive to an output of the sensor and in communication with the database, and operable to;

register the sensor image with the first reference image;

select a point of interest from the sensor image;

derive a point of interest in the first reference image from a selected point of interest in the sensor image and a registration between the sensor image and the first reference image, wherein the selected point of interest in the first reference image is defined by a first pixel;

perform a stereo point measurement from the selected point of interest in the first reference image to determine a point in the second reference image that defines a pixel that represents a stereo mate of the selected point of interest in the first reference image;

using the stereo point measurement to select a three dimensional scene coordinate which more accurately corresponds with the selected point of interest in the sensor image; and

a display for displaying both of said first and second reference images with said points displayed on each of said reference images to enable an individual to visually confirm an accuracy of said stereo point measurement.

18. The system of claim 17, wherein:

the selected point of interest of the first reference image is used by the at least one processing system to obtain coordinate data that corresponds to the pixel of the selected point of interest in the first reference image; and

the point in the second reference image is used by the at least one processing system to obtain coordinate data therefor.

19. The system of claim 18, wherein the at least one processing system is configured to perform a stereo point measurement, and wherein the stereo point measurement is accomplished by:

obtaining a first sub-image of the first reference image in an area circumscribing the selected point of interest in the first reference image;

obtaining a second sub-image of the second reference image in an area circumscribing the point in the second reference image, and where the second sub-image is larger than the first sub-image;

overlaying at least a portion of the first sub-image over the second sub-image;

analyzing the two sub-images to determine a first offset of the first sub-image within the second image, and a second offset of the second sub-image relative to the second reference image;

using the first and second offsets to translate coordinates of the first pixel to stereo matching coordinates of the second pixel, and

wherein the stereo matching coordinates of the second pixel represent a specific point on the second reference image that corresponds to the location of the first pixel on the first reference image.

20. The system of claim 19, wherein the at least one processing system is configured to obtain the stereo matching coordinates of the second pixel by:

determining column and row offset values that represent the first offset of the first sub-image within the second sub-image;

determining column and row offset values that represent the second offset of the second sub-image relative to the second reference image;

adding column and row coordinates of the first pixel in the first sub-image to the column and row offset values of the first sub-image offset to produce a modified first sub-image offset having associated column and row offset values; and

adding the modified first sub-image column and row offset values to the column and row offset values of the second sub-image, to produce coordinates of the second pixel.