Automated graph local constellation (GLC) method of correspondence search for registration of 2-D and 3-D data

• US 9,754,165 B2
• Filed: 07/30/2013
• Issued: 09/05/2017
• Est. Priority Date: 07/30/2013
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Mundhenk, Terrell Nathan, Kim, Kyungnam, Owechko, Yuri, Chen, Hai-Wen
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Sun, Jiangeng
• Application Number:
  US13/953,757
• Publication Number:
  US 20150036870A1
• Time in Patent Office: 1,498 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/176 Urban or other man-made structures

Abstract

According to an embodiment, a 2-dimensional (2-D) image of a geographical region is transformed via a regional maxima transform (RMT) or an edge segmenting and boundary filling (ESBF) transform to produce a filtered 2-D image. The filtered 2-D image is iteratively eroded and opened to produce a processed EO 2-D image, 2-D object shape morphology is extracted from the processed EO 2-D image, and 2-D shape properties are extracted from the 2-D object shape morphology. A height slice of a 3-dimensional (3-D) point cloud comprising 3-D coordinate and intensity measurements of the geographical region is generated, and slice object shape morphology is extracted from the height slice. Slice shape properties from the slice object shape morphology are extracted, and the 2-D image is constellation matched to the height slice based on the 2-D shape properties and the slice shape properties.

Claims

1. A method for registration of two and three-dimensional (3-D) images, the method comprising:

capturing a 2-dimensional (2-D) image comprising 2-D image points of a geographical region by a camera attached to a first vehicle;

transforming the 2-dimensional (2-D) image of the geographical region via a regional maxima transform or an edge segmenting and boundary filling (ESBF) transform to produce a filtered 2-D image;

iteratively eroding and opening the filtered 2-D image to produce a processed electro-optical (EO) 2-D image;

extracting 2-D object shape morphology from the processed EO 2-D image;

extracting 2-D shape properties from the 2-D object shape morphology;

scanning the geographical region by a Light Detection and Ranging (LIDAR) system attached to a second vehicle to produce a 3-dimensional point cloud comprising 3-D cloud image points thereof;

generating a height slice of the 3-dimensional point cloud comprising 3-D coordinate and intensity measurements of the geographical region;

extracting slice object shape morphology from the height slice resulting in labeled shapes found in the 2-D slice image;

extracting slice shape properties from the slice object shape morphology;

constellation matching the 2-D image to the height slice based on the 2-D shape properties and the slice shape properties;

determining matching points between the 2-D image points and the 3-D cloud image points;

estimating a pose position of the camera attached to the first vehicle based on the matching points to provide an estimated pose position;

determining a geo-location of the first vehicle based on the estimated pose position of the camera attached to the first vehicle; and

based on the matching points, generating a colorized 3-dimensional point cloud wherein at least some of the 3-D cloud image points of the 3-dimensional point cloud are given a color of an object taken from the 2-D image.

2. The method of claim 1, wherein the 3-D coordinate and intensity measurements comprise Light Detection and Ranging (LIDAR) measurements.

3. The method of claim 1, further comprising fusing the 2-D image points and the 3-D cloud image points based on the estimated pose position.

4. The method of claim 1, further comprising:

finding a plurality of shapes for a plurality of objects;

identifying the shapes for further extraction to produce identified shapes;

extracting properties of the identified shapes comprising shape corners and shape dimensions; and

matching a constellation of objects via matching LIDAR object to EO objects based on the properties of the identified shapes.

9. The method of claim 1, wherein the 3-dimensional point cloud is calibrated into world coordinates, and the method further comprises:

determining the geo-location of the first vehicle relative to the world coordinates of the matching points of the 3-D cloud image points of the 3-dimensional point cloud and the 2-D image points.

5. A system for registration of two and three-dimensional images, the system comprising:

a first vehicle system configured to image a 2-dimensional (2-D) image comprising 2-D image points of a geographical region by a camera attached to a first vehicle;

a second vehicle system configured to scan a 3-dimensional point cloud comprising 3-D cloud image points of the geographical region by a Light Detection and Ranging (LIDAR) system attached to a second vehicle;

a processor;

memory storing instructions executable by the processor to perform functions comprising;

transforming the 2-dimensional (2-D) image of the geographical region via a regional maxima transform or an edge segmenting and boundary filling (ESBF) transform to produce a filtered 2-D image;

iteratively eroding and opening the filtered 2-D image to produce a processed electro-optical (EO) 2-D image;

extracting 2-D object shape morphology from the processed EO 2-D image;

extracting 2-D shape properties from the 2-D object shape morphology;

generating a height slice of the 3-dimensional (3-D) point cloud comprising LIDAR measurements of the geographical region;

extracting slice object shape morphology from the height slice resulting in labeled shapes found in the 2-D slice image;

extracting slice shape properties from the slice object shape morphology;

constellation matching the 2-D image to the height slice based on the 2-D shape properties and the slice shape properties;

matching points of the 2-D image and the 3-dimensional point cloud based on the 2-D shape properties and the slice shape properties to produce a plurality of matched points;

estimating a pose position of the camera based on the matched points to provide an estimated pose position;

determining a geo-location of the first vehicle based on the estimated pose position of the camera attached to the first vehicle; and

based on the matching points, generating a colorized 3-dimensional point cloud wherein at least some of the 3-D cloud image points of the 3-dimensional point cloud are given a color of an object taken from the 2-D image.

10. The system of claim 5, wherein the 3-dimensional point cloud is calibrated into world coordinates, and the functions further comprise:

determining the geo-location of the first vehicle relative to the world coordinates of the matching points of the 3-D cloud image points of the 3-dimensional point cloud and the 2-D image points.

6. A non-transitory computer readable storage medium comprising computer-executable instructions for registration of two and three-dimensional (3-D) images, the computer-executable instructions comprising:

capturing a 2-dimensional (2-D) image comprising 2-D image points of a geographical region by a camera attached to a first vehicle;

transforming the 2-dimensional (2-D) image of the geographical region via a regional maxima transform or an edge segmenting and boundary filling (ESBF) transform to produce a filtered 2-D image;

iteratively eroding and opening the filtered 2-D image to produce a processed electro-optical (EO) 2-D image;

extracting 2-D object shape morphology from the processed EO 2-D image;

extracting 2-D shape properties from the 2-D object shape morphology;

scanning the geographical region by a Light Detection and Ranging (LIDAR) system attached to a second vehicle to produce a 3-dimensional point cloud comprising 3-D cloud image points thereof;

generating a height slice of the 3-dimensional point cloud comprising LIDAR measurements of the geographical region;

extracting slice object shape morphology from the height slice resulting in labeled shapes found in the 2-D slice image;

extracting slice shape properties from the slice object shape morphology;

constellation matching the 2-dimensional (2-D) image to the height slice based on the 2-D shape properties and the slice shape properties;

determining matching points between 2-D image points and 3-D cloud image points to produce a plurality of matched points;

estimating a pose position of a camera based on the matched points to provide an estimated pose position;

determining a geo-location of the first vehicle based on the estimated pose position of the camera attached to the first vehicle; and

based on the matching points, generating a colorized 3-dimensional point cloud wherein at least some of the 3-D cloud image points of the 3-dimensional point cloud are given a color of an object taken from the 2-D image.

7. The non-transitory computer readable storage medium of claim 6, further comprising computer-executable instructions for:

fusing the 2-D image points and the 3-D cloud image points based on the estimated pose position.

8. The non-transitory computer readable storage medium of claim 6, further comprising computer-executable instructions for:

finding a plurality of shapes for a plurality of objects;

identifying the shapes for further extraction to produce identified shapes;

extracting properties of the identified shapes comprising shape corners and shape dimensions; and

matching a constellation of objects via matching LIDAR object to EO objects based on the properties of the identified shapes.

11. The non-transitory computer readable storage medium of claim 6, wherein the 3-dimensional point cloud is calibrated into world coordinates, and further comprising computer-executable instructions for:

determining the geo-location of the first vehicle relative to the world coordinates of the matching points of the 3-D cloud image points of the 3-dimensional point cloud and the 2-D image points.

12. A method for registration of two and three-dimensional (3-D) images, the method comprising:

capturing a 2-dimensional (2-D) image comprising 2-D image points of a geographical region by a camera attached to a first vehicle;

scanning the geographical region by a Light Detection and Ranging (LIDAR) system attached to a second vehicle to produce a 3-dimensional point cloud comprising 3-D cloud image points thereof;

determining matching points between the 2-D image points and the 3-D cloud image points by;

finding a plurality of shapes for a plurality of objects;

identifying the shapes for further extraction to produce identified shapes;

extracting properties of the identified shapes comprising shape corners and shape dimensions; and

matching a constellation of objects via matching LIDAR object to 2D objects based on the properties of the identified shapes;

estimating a pose position of the camera attached to the first vehicle based on the matching points to provide an estimated pose position;

determining a geo-location of the first vehicle based on the estimated pose position of the camera attached to the first vehicle; and

based on the matching points, generating a colorized 3-dimensional point cloud wherein at least some of the 3-D cloud image points of the 3-dimensional point cloud are given a color of an object taken from the 2-D image.

13. The method of claim 12, wherein the 3-dimensional point cloud is calibrated into world coordinates, and the method further comprises:

determining the geo-location of the first vehicle relative to the world coordinates of the matching points of the 3-D cloud image points of the 3-dimensional point cloud and the 2-D image points.

14. The method of claim 12, further comprising fusing the 2-D image points and the 3-D cloud image points based on the estimated pose position.

15. The method of claim 12, wherein determining matching points between the 2-D image points and the 3-D cloud image points comprises:

transforming the 2-dimensional (2-D) image of the geographical region via a regional maxima transform or an edge segmenting and boundary filling (ESBF) transform to produce a filtered 2-D image.

16. The method of claim 15, wherein determining matching points between the 2-D image points and the 3-D cloud image points further comprises:

iteratively eroding and opening the filtered 2-D image to produce a processed electro-optical (EO) 2-D image;

extracting 2-D object shape morphology from the processed EO 2-D image; and

extracting 2-D shape properties from the 2-D object shape morphology.

17. The method of claim 12, further comprising:

generating a height slice of the 3-dimensional point cloud comprising 3-D coordinate and intensity measurements of the geographical region;

extracting slice object shape morphology from the height slice resulting in labeled shapes found in the 2-D slice image;

extracting slice shape properties from the slice object shape morphology; and

constellation matching the 2-D image to the height slice.

18. The method of claim 17, wherein the 3-D coordinate and intensity measurements comprise Light Detection and Ranging (LIDAR) measurements.

19. The method of claim 12, wherein the 3-D cloud image points have an intensity corresponding to a reflectivity of an object represented by the 3-D cloud image points, and wherein generating the colorized 3-dimensional point cloud comprises:

generating an enhanced point cloud with true colors.