Registration of 3D point cloud data to 2D electro-optical image data

• US 8,290,305 B2
• Filed: 02/13/2009
• Issued: 10/16/2012
• Est. Priority Date: 02/13/2009
• Status: Expired due to Fees

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Inventors: Minear, Kathleen, Smith, Anthony O'Neil
• Primary Examiner(s): Bhatnagar, Anand
• Application Number:
  US12/371,436
• Publication Number:
  US 20100209013A1
• Time in Patent Office: 1,341 Days
• Field of Search: 382/100, 382/103, 382/106, 382/154, 382/293, 382/294, 382/295, 345619-689, 348/580, 348/584, 700/57
• US Class Current: 382/294
• CPC Class Codes: G06T 7/32 using correlation-based methods

Abstract

Method and system for registration of a two dimensional image data set and a three-dimensional image comprising point cloud data. The method begins by cropping a three-dimensional volume of point cloud data comprising a three-dimensional image data to remove a portion of the point cloud data comprising a ground surface within a scene, and dividing the three-dimensional volume into a plurality of m sub-volumes. Thereafter, the method continues by edge-enhancing a two-dimensional image data. Then, for each qualifying sub-volume, creating a filtered density image, calculating a two-dimensional correlation surface based on the filtered density image and the two-dimensional image data that has been edge enhanced, finding a peak of the two-dimensional correlation surface, determining a corresponding location of the peak within the two-dimensional image, defining a correspondence point set; and storing the correspondence point set in a point set list. Finally, a transformation is determined that minimizes the error between a plurality of the correspondence point sets contained in the point set list.

Claims

1. A method for registration of a plurality of images, comprising:

acquiring for a common scene a two dimensional image data and a three-dimensional image data;

cropping a three-dimensional volume of point cloud data comprising said three-dimensional image data to remove a portion of said point cloud data comprising a ground surface within said scene;

dividing said three-dimensional volume into a plurality of m sub-volumes, where m is greater than or equal to one;

edge-enhancing said two-dimensional image data;

for each qualifying sub-volume, creating a filtered density image, calculating a two-dimensional correlation surface based on said filtered density image and said two-dimensional image data that has been edge enhanced, finding a peak of the two-dimensional correlation surface, determining a corresponding location of said peak within the two-dimensional image, define a correspondence point set; and

storing said correspondence point set in a point set list;

finding a transformation that minimizes the error between a plurality of said correspondence point sets contained in said point set list; and

applying the transformation to said points in a target data selected from group consisting of the three-dimensional image data and said two-dimensional image data.

2. The method according to claim 1, wherein said qualifying sub-volume is identified based on one or more selected characteristics associated with a particular sub-volume n selected from the group consisting of a predetermined number of data points therein, and the presence of a blob-like structure therein.

3. The method according to claim 1, wherein said step of creating a filtered density image comprises projecting said point cloud data within a sub-volume n to a registration plane defined by a two-dimensional image data sensor attitude to form a density image.

4. The method according to claim 3, wherein said step of creating said filtered density image further comprises median filtering said density image.

5. The method according to claim 4, wherein said step of creating said filtered density image further comprises edge filtering said density image to enhance at least one edge in said density image.

6. The method according to claim 5, wherein said edge filtering method is selected to be a Sobel edge filter.

7. The method according to claim 1, wherein said step of calculating a two-dimensional correlation surface includes using either said two-dimensional image as a reference image, and said filtered density image as a target image, or said filtered density image as a reference image and said two-dimensional image as a target image.

8. The method according to claim 1, further comprising acquiring said point cloud data using a LIDAR sensor.

9. The method according to claim 1, further comprising acquiring said two-dimensional image data using an electro-optical sensor.

10. The method according to claim 1, wherein said step of calculating said two-dimensional correlation surface further comprises performing a normalized cross-correlation.

11. A system for registration of a plurality of images, comprising processing means programmed with a set of instructions for performing a series of steps including:

cropping a three-dimensional volume of point cloud data comprising a three-dimensional image data to remove a portion of said point cloud data comprising a ground surface within a scene;

dividing said three-dimensional volume into a plurality of m sub-volumes, where m is greater than or equal to one;

edge-enhancing a two-dimensional image data;

for each qualifying sub-volume, creating a filtered density image, calculating a two-dimensional correlation surface based on said filtered density image and said two-dimensional image data that has been edge enhanced, finding a peak of the two-dimensional correlation surface, determining a corresponding location of said peak within the two-dimensional image, defining a correspondence point set; and

storing said correspondence point set in a point set list;

finding a transformation that minimizes the error between a plurality of said correspondence point sets contained in said point set list; and

applying the transformation to said points in a target data selected from group consisting of the three-dimensional image data and said two-dimensional image data.

12. The system according to claim 11, wherein said qualifying sub-volume is identified based on one or more selected characteristics associated with a particular sub-volume n selected from the group consisting of a predetermined number of data points therein, and the presence of a blob-like structure therein.

13. The system according to claim 11, wherein said step of creating a filtered density image comprises projecting said point cloud data within a sub-volume n to a registration plane defined by a two-dimensional image data sensor attitude to form a density image.

14. The system according to claim 13, wherein said step of creating said filtered density image further comprises median filtering said density image.

15. The system according to claim 14, wherein said step of creating said filtered density image further comprises edge filtering said density image to enhance at least one edge in said density image.

16. The system according to claim 15, wherein said edge filtering system is selected to be a Sobel edge filter.

17. The system according to claim 11, wherein said step of calculating a two-dimensional correlation surface includes using either said two-dimensional image as a reference image, and said filtered density image as a target image, or said filtered density image as a reference image and said two-dimensional image as a target image.

18. The system according to claim 11, further comprising acquiring said point cloud data using a LIDAR sensor.

19. The system according to claim 11, further comprising acquiring said two-dimensional image data using an electro-optical sensor.

20. The system according to claim 11, wherein said step of calculating said two-dimensional correlation surface further comprises performing a normalized cross-correlation.

21. A computer program embodied on a non-transitory computer-readable medium for performing a series of steps comprising:

cropping a three-dimensional volume of point cloud data comprising a three-dimensional image data to remove a portion of said point cloud data comprising a ground surface within a scene;

dividing said three-dimensional volume into a plurality of m sub-volumes, where m is greater than or equal to one;

edge-enhancing said two-dimensional image data;

for each qualifying sub-volume, creating a filtered density image, calculating a two-dimensional correlation surface based on said filtered density image and said two-dimensional image data that has been edge enhanced, finding a peak of the two-dimensional correlation surface, determining a corresponding location of said peak within the two-dimensional image, define a correspondence point set; and

storing said correspondence point set in a point set list;

finding a transformation that minimizes the error between a plurality of said correspondence point sets contained in said point set list; and

applying the transformation to said points in a target data selected from group consisting of the three-dimensional image data and said two-dimensional image data.